slatedb 0.6.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +114 -44
- data/ext/slatedb/Cargo.toml +2 -3
- data/ext/slatedb/src/checkpoint.rs +47 -0
- data/ext/slatedb/src/database.rs +42 -17
- data/ext/slatedb/src/lib.rs +2 -0
- data/ext/slatedb/src/merge_ops.rs +2 -185
- data/ext/slatedb/src/reader.rs +21 -6
- data/ext/slatedb/src/runtime.rs +1 -52
- data/ext/slatedb/src/snapshot.rs +4 -0
- data/ext/slatedb/src/transaction.rs +4 -0
- data/lib/slatedb/database.rb +40 -26
- data/lib/slatedb/reader.rb +25 -19
- data/lib/slatedb/snapshot.rb +17 -9
- data/lib/slatedb/transaction.rb +17 -9
- data/lib/slatedb/version.rb +1 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d78fa73dbdaa8c8bd6bdacebc65b3f69b3df642eab3f7e8d7bdd08bde68b0ba0
|
|
4
|
+
data.tar.gz: 9dd7197dea492b9a146053f1aadad0bc233f258d5cceea2aadda64ad8c69a5c6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 783e32df3d81361f30ab91003b9fa10848e342b548219eb4b622be0d760c2ba3d6a5f1211b455ea7ccd7d05e6f13dc793700878c6ac370cd4204a92de4b8cc8d
|
|
7
|
+
data.tar.gz: 5239793e435456d610b5a2eea877e58873f4d48a7e80f47dcc9b191cc82857d0f5669dd6780e0d26997b616e4bc4e501ac917b7f4860dce9d832269969689177
|
data/README.md
CHANGED
|
@@ -30,6 +30,38 @@ gem install slatedb
|
|
|
30
30
|
|
|
31
31
|
Precompiled native gems are published for common Linux and Darwin platforms. Building from source still requires a Rust toolchain.
|
|
32
32
|
|
|
33
|
+
## Upgrading to SlateDB 0.18.1
|
|
34
|
+
|
|
35
|
+
This update moves the Rust engine from 0.17.0 to 0.18.1. Existing database files
|
|
36
|
+
remain readable. New Ruby options keep the existing defaults unless enabled.
|
|
37
|
+
|
|
38
|
+
- Two-phase checkpoints: `begin_checkpoint` returns a handle with `id` and `wait`.
|
|
39
|
+
Both checkpoint methods accept `scope: :all` to capture pending writes, or
|
|
40
|
+
`scope: :durable` (the existing default) for writes already durable.
|
|
41
|
+
- Point reads accept `max_concurrent_sources:` on databases, readers, snapshots,
|
|
42
|
+
and transactions; database metadata reads accept it too.
|
|
43
|
+
- `Database.open` accepts `experimental_incremental_compaction:` and
|
|
44
|
+
`sst_block_alignment:`. Both are disabled by default; see the warnings below.
|
|
45
|
+
- Snapshot and transaction range/prefix scans now accept `order:`, like database
|
|
46
|
+
and reader scans. Reader `merge_operator:` is now applied rather than ignored.
|
|
47
|
+
- **Breaking:** Ruby Proc/lambda merge operators are no longer accepted. They
|
|
48
|
+
silently used concatenation on other threads, which could change stored values.
|
|
49
|
+
Keep `:concat` / `:string_concat` for concatenation. For custom updates, use a
|
|
50
|
+
serializable transaction that reads the value and writes the computed result;
|
|
51
|
+
handle `SlateDb::TransactionError` conflicts in the application. Databases with
|
|
52
|
+
unresolved custom merge operands need an application-specific migration before
|
|
53
|
+
upgrading. Do not reopen them with `:concat` unless concatenation was their
|
|
54
|
+
intended merge rule.
|
|
55
|
+
- Reader disk-cache keywords remain supported, but upstream deprecated the
|
|
56
|
+
underlying `CachedObjectStore` in 0.18.1. The new mirror and Rust trait-based
|
|
57
|
+
extension APIs are not exposed by this gem yet.
|
|
58
|
+
|
|
59
|
+
Upstream also fixes clone-compaction retention/data-loss cases, writer ownership
|
|
60
|
+
validation after WAL replay, write stalls, upload retry/concurrency behavior, and
|
|
61
|
+
iterator fetch cancellation. It improves memtable seeks and point-read/scan work.
|
|
62
|
+
See the full [0.18.0 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.0)
|
|
63
|
+
and [0.18.1 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.1).
|
|
64
|
+
|
|
33
65
|
## Usage
|
|
34
66
|
|
|
35
67
|
### Basic Operations
|
|
@@ -64,6 +96,19 @@ SlateDb::Database.open("/tmp/mydb") do |db|
|
|
|
64
96
|
end # automatically closed
|
|
65
97
|
```
|
|
66
98
|
|
|
99
|
+
### Point-Read Concurrency
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
db.get("key", max_concurrent_sources: 4, cache_blocks: false)
|
|
103
|
+
db.get_entry("key", max_concurrent_sources: 4) # value plus sequence/timestamps
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`max_concurrent_sources` defaults to 1 (serial fetching); zero also means 1.
|
|
107
|
+
SlateDB checks the newest source alone, then fetches up to this many sources
|
|
108
|
+
concurrently after a miss. Higher values can reduce latency but increase object
|
|
109
|
+
store requests and bandwidth. The same keyword works with `get` on readers,
|
|
110
|
+
snapshots, and transactions. It does not change isolation or durability.
|
|
111
|
+
|
|
67
112
|
### Compaction Controls
|
|
68
113
|
|
|
69
114
|
SlateDB 0.17 adds controls for how long compaction inputs remain protected from
|
|
@@ -87,6 +132,32 @@ projected sorted-run count exceeds the threshold. It is not a hard limit.
|
|
|
87
132
|
Zero disables it (the default); use at least two to enable it. SlateDB warns and
|
|
88
133
|
disables the trigger for one. Low thresholds can increase storage reads and writes.
|
|
89
134
|
|
|
135
|
+
SlateDB 0.18 adds an opt-in experimental mode:
|
|
136
|
+
|
|
137
|
+
```ruby
|
|
138
|
+
SlateDb::Database.open("mydb", url: "s3://mybucket/path",
|
|
139
|
+
experimental_incremental_compaction: true) do |db|
|
|
140
|
+
db.put("key", "value")
|
|
141
|
+
end
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
This publishes output from sorted-run compactions at most once a minute before
|
|
145
|
+
the full compaction finishes. It can reduce read amplification and allow earlier
|
|
146
|
+
storage reclamation after the compactor checkpoint expires. Compactions with L0
|
|
147
|
+
inputs still publish only when complete. It is disabled by default, experimental,
|
|
148
|
+
and may change or disappear upstream.
|
|
149
|
+
|
|
150
|
+
### SST Block Alignment
|
|
151
|
+
|
|
152
|
+
`Database.open(..., sst_block_alignment: true)` pads SST data blocks to 4 KiB
|
|
153
|
+
boundaries, including oversized entries. It is disabled by default, uses more
|
|
154
|
+
storage, and cannot be combined with compression.
|
|
155
|
+
|
|
156
|
+
**Upgrade every reader before enabling alignment on a writer.** Older engines
|
|
157
|
+
that do not understand padded SSTs cannot read them. Once padded files exist,
|
|
158
|
+
rolling those readers back is not safe; disabling the option does not rewrite
|
|
159
|
+
existing files.
|
|
160
|
+
|
|
90
161
|
### Key Size
|
|
91
162
|
|
|
92
163
|
Keys can now exceed the previous 65,535-byte limit, up to SlateDB's new
|
|
@@ -341,53 +412,19 @@ db.batch do |b|
|
|
|
341
412
|
end
|
|
342
413
|
```
|
|
343
414
|
|
|
344
|
-
####
|
|
345
|
-
|
|
346
|
-
You can provide a Ruby Proc/lambda as a custom merge operator:
|
|
347
|
-
|
|
348
|
-
```ruby
|
|
349
|
-
# Counter merge operator (adds numbers)
|
|
350
|
-
counter_merge = ->(key, existing, new_value) {
|
|
351
|
-
existing_num = existing ? existing.to_i : 0
|
|
352
|
-
(existing_num + new_value.to_i).to_s
|
|
353
|
-
}
|
|
354
|
-
|
|
355
|
-
SlateDb::Database.open("/tmp/mydb", merge_operator: counter_merge) do |db|
|
|
356
|
-
db.merge("visits", "1")
|
|
357
|
-
db.merge("visits", "1")
|
|
358
|
-
db.merge("visits", "1")
|
|
359
|
-
|
|
360
|
-
db.get("visits") # => "3"
|
|
361
|
-
end
|
|
415
|
+
#### Available Merge Operators
|
|
362
416
|
|
|
363
|
-
|
|
364
|
-
max_merge = ->(key, existing, new_value) {
|
|
365
|
-
existing_num = existing ? existing.to_i : 0
|
|
366
|
-
new_num = new_value.to_i
|
|
367
|
-
[existing_num, new_num].max.to_s
|
|
368
|
-
}
|
|
417
|
+
- `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
|
|
369
418
|
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
db.merge("high_score", "250")
|
|
373
|
-
db.merge("high_score", "150")
|
|
419
|
+
Ruby Proc/lambda operators are rejected. Use a serializable read/modify/write
|
|
420
|
+
transaction for custom logic, with application-level conflict handling:
|
|
374
421
|
|
|
375
|
-
|
|
422
|
+
```ruby
|
|
423
|
+
db.transaction(isolation: :serializable) do |txn|
|
|
424
|
+
txn.put("visits", (txn.get("visits").to_i + 1).to_s)
|
|
376
425
|
end
|
|
377
426
|
```
|
|
378
427
|
|
|
379
|
-
The proc receives three arguments:
|
|
380
|
-
- `key` - The key being merged
|
|
381
|
-
- `existing` - The existing value (nil if no value exists)
|
|
382
|
-
- `new_value` - The new merge operand
|
|
383
|
-
|
|
384
|
-
**Note:** Custom Proc merge operators work best with direct `db.merge()` calls. When used with transactions or batches, some merge operations may be processed on background threads and fall back to string concatenation.
|
|
385
|
-
|
|
386
|
-
#### Available Merge Operators
|
|
387
|
-
|
|
388
|
-
- `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
|
|
389
|
-
- Any `Proc` or `lambda` - Custom merge logic
|
|
390
|
-
|
|
391
428
|
### Write Batches
|
|
392
429
|
|
|
393
430
|
Perform multiple writes atomically:
|
|
@@ -495,6 +532,29 @@ SlateDb::Database.open("/tmp/mydb", url: "file:///tmp/mydb") do |db|
|
|
|
495
532
|
end
|
|
496
533
|
```
|
|
497
534
|
|
|
535
|
+
`create_checkpoint` waits until the checkpoint is stored. Its default
|
|
536
|
+
`scope: :durable` preserves the existing behavior. To include writes made with
|
|
537
|
+
`await_durable: false`, use `scope: :all`; no separate `flush` is needed.
|
|
538
|
+
|
|
539
|
+
To separate capturing the write boundary from waiting for storage:
|
|
540
|
+
|
|
541
|
+
```ruby
|
|
542
|
+
db.put("key", "before", await_durable: false)
|
|
543
|
+
handle = db.begin_checkpoint(scope: :all, name: "backup", lifetime: 3_600_000)
|
|
544
|
+
id = handle.id # UUID is available immediately
|
|
545
|
+
db.put("key", "after") # not part of this checkpoint
|
|
546
|
+
checkpoint = handle.wait # => { id: id, manifest_id: ... }
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
With `scope: :all`, writes completed before `begin_checkpoint` starts are included;
|
|
550
|
+
writes started after it returns are excluded. Concurrent writes during the call
|
|
551
|
+
may be included. Starting the checkpoint can still wait for internal flush work;
|
|
552
|
+
it is not a fully nonblocking Ruby call. `wait` releases the Ruby GVL while
|
|
553
|
+
waiting, consumes the handle, and raises `SlateDb::ClosedError` if called again.
|
|
554
|
+
`id` remains readable. Dropping the handle does not cancel or delete the checkpoint.
|
|
555
|
+
If waiting fails, use the UUID with `Admin#list_checkpoints` to check its outcome
|
|
556
|
+
before starting another checkpoint.
|
|
557
|
+
|
|
498
558
|
### Snapshots
|
|
499
559
|
|
|
500
560
|
Point-in-time consistent reads:
|
|
@@ -517,6 +577,11 @@ value = snap.get("key")
|
|
|
517
577
|
snap.close
|
|
518
578
|
```
|
|
519
579
|
|
|
580
|
+
Range and prefix scans on snapshots and transactions accept the same `order:`
|
|
581
|
+
keyword as databases and readers, including `:asc`/`:ascending` and
|
|
582
|
+
`:desc`/`:descending`. Bounds remain lower-inclusive and upper-exclusive in
|
|
583
|
+
either direction.
|
|
584
|
+
|
|
520
585
|
### Reader (Read-Only Access)
|
|
521
586
|
|
|
522
587
|
Open a database in read-only mode, useful for replicas:
|
|
@@ -568,6 +633,14 @@ SlateDb::Reader.open("/tmp/mydb",
|
|
|
568
633
|
end
|
|
569
634
|
```
|
|
570
635
|
|
|
636
|
+
When the writer uses `merge_operator: :concat`, pass the same operator to
|
|
637
|
+
`Reader.open` so WAL replay and reads apply the correct merge rule. Unknown
|
|
638
|
+
operator names now raise `SlateDb::InvalidArgumentError` instead of being ignored.
|
|
639
|
+
|
|
640
|
+
The `cache_root` and `max_open_file_handles` keywords still work, but SlateDB
|
|
641
|
+
0.18.1 deprecates this legacy disk cache in favor of its separate mirror crate or
|
|
642
|
+
tiered block cache. Those replacement APIs are not yet exposed here.
|
|
643
|
+
|
|
571
644
|
### Admin Operations
|
|
572
645
|
|
|
573
646
|
Administrative operations for database management:
|
|
@@ -594,9 +667,6 @@ admin.delete_checkpoint("uuid")
|
|
|
594
667
|
# Garbage Collection
|
|
595
668
|
admin.run_gc # Run with default settings
|
|
596
669
|
admin.run_gc(min_age: 3600_000) # Set min age for all directories (1 hour)
|
|
597
|
-
admin.run_gc(manifest_min_age: 86400_000) # Custom age for manifest (1 day)
|
|
598
|
-
admin.run_gc(wal_min_age: 60_000) # Custom age for WAL (1 minute)
|
|
599
|
-
admin.run_gc(compacted_min_age: 60_000) # Custom age for compacted (1 minute)
|
|
600
670
|
|
|
601
671
|
# Delete a database (SlateDB >= 0.16.0)
|
|
602
672
|
would_delete = admin.delete_db # Dry run: returns the object paths
|
data/ext/slatedb/Cargo.toml
CHANGED
|
@@ -11,7 +11,7 @@ name = "slatedb"
|
|
|
11
11
|
crate-type = ["cdylib"]
|
|
12
12
|
|
|
13
13
|
[dependencies]
|
|
14
|
-
slatedb = "0.
|
|
14
|
+
slatedb = "0.18.1"
|
|
15
15
|
magnus = { version = "0.9.3", features = ["rb-sys"] }
|
|
16
16
|
rb-sys = { version = "0.9.130", features = ["stable-api-compiled-fallback"] }
|
|
17
17
|
tokio = { version = "1.54.1", features = ["rt-multi-thread", "sync"] }
|
|
@@ -19,5 +19,4 @@ bytes = "1.12.1"
|
|
|
19
19
|
serde_json = "1.0.154"
|
|
20
20
|
url = "2.5.8"
|
|
21
21
|
once_cell = "1.21.4"
|
|
22
|
-
|
|
23
|
-
uuid = "1.29.0"
|
|
22
|
+
uuid = "1.29.2"
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
use magnus::prelude::*;
|
|
2
|
+
use magnus::{method, Error, RHash, Ruby};
|
|
3
|
+
use std::cell::RefCell;
|
|
4
|
+
|
|
5
|
+
use crate::errors::closed_error;
|
|
6
|
+
use crate::runtime::block_on_result;
|
|
7
|
+
|
|
8
|
+
#[magnus::wrap(class = "SlateDb::CheckpointHandle", free_immediately, size)]
|
|
9
|
+
pub struct CheckpointHandle {
|
|
10
|
+
id: String,
|
|
11
|
+
inner: RefCell<Option<slatedb::CheckpointHandle>>,
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
impl CheckpointHandle {
|
|
15
|
+
pub fn new(handle: slatedb::CheckpointHandle) -> Self {
|
|
16
|
+
Self {
|
|
17
|
+
id: handle.id().to_string(),
|
|
18
|
+
inner: RefCell::new(Some(handle)),
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
pub fn id(&self) -> String {
|
|
23
|
+
self.id.clone()
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
pub fn wait(&self) -> Result<RHash, Error> {
|
|
27
|
+
let handle = self
|
|
28
|
+
.inner
|
|
29
|
+
.borrow_mut()
|
|
30
|
+
.take()
|
|
31
|
+
.ok_or_else(|| closed_error("checkpoint handle has already been waited on"))?;
|
|
32
|
+
let result = block_on_result(handle.wait())?;
|
|
33
|
+
|
|
34
|
+
let ruby = Ruby::get().expect("Ruby runtime not available");
|
|
35
|
+
let hash = ruby.hash_new();
|
|
36
|
+
hash.aset(ruby.to_symbol("id"), result.id.to_string())?;
|
|
37
|
+
hash.aset(ruby.to_symbol("manifest_id"), result.manifest_id)?;
|
|
38
|
+
Ok(hash)
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
pub fn define_checkpoint_class(ruby: &Ruby, module: &magnus::RModule) -> Result<(), Error> {
|
|
43
|
+
let class = module.define_class("CheckpointHandle", ruby.class_object())?;
|
|
44
|
+
class.define_method("id", method!(CheckpointHandle::id, 0))?;
|
|
45
|
+
class.define_method("wait", method!(CheckpointHandle::wait, 0))?;
|
|
46
|
+
Ok(())
|
|
47
|
+
}
|
data/ext/slatedb/src/database.rs
CHANGED
|
@@ -10,9 +10,10 @@ use slatedb::config::{
|
|
|
10
10
|
use slatedb::object_store::memory::InMemory;
|
|
11
11
|
use slatedb::{Db, IsolationLevel, IterationOrder, KeyValue};
|
|
12
12
|
|
|
13
|
+
use crate::checkpoint::CheckpointHandle;
|
|
13
14
|
use crate::errors::invalid_argument_error;
|
|
14
15
|
use crate::iterator::Iterator;
|
|
15
|
-
use crate::merge_ops::
|
|
16
|
+
use crate::merge_ops::parse_merge_operator;
|
|
16
17
|
use crate::metrics::Metrics;
|
|
17
18
|
use crate::runtime::block_on_result;
|
|
18
19
|
use crate::snapshot::Snapshot;
|
|
@@ -76,6 +77,10 @@ impl Database {
|
|
|
76
77
|
opts.cache_blocks = cb;
|
|
77
78
|
}
|
|
78
79
|
|
|
80
|
+
if let Some(limit) = get_optional::<usize>(kwargs, "max_concurrent_sources")? {
|
|
81
|
+
opts.max_concurrent_sources = limit;
|
|
82
|
+
}
|
|
83
|
+
|
|
79
84
|
Ok(opts)
|
|
80
85
|
}
|
|
81
86
|
|
|
@@ -113,20 +118,25 @@ impl Database {
|
|
|
113
118
|
/// # Arguments
|
|
114
119
|
/// * `path` - The path identifier for the database
|
|
115
120
|
/// * `url` - Optional object store URL (e.g., "s3://bucket/path")
|
|
116
|
-
/// * `kwargs` - Additional options (merge_operator
|
|
121
|
+
/// * `kwargs` - Additional options (merge_operator and compaction settings)
|
|
117
122
|
///
|
|
118
123
|
/// # Returns
|
|
119
124
|
/// A new Database instance
|
|
120
125
|
pub fn open(path: String, url: Option<String>, kwargs: RHash) -> Result<Self, Error> {
|
|
121
|
-
|
|
122
|
-
let merge_operator = parse_merge_operator(&kwargs)?.or(parse_merge_operator_proc(&kwargs)?);
|
|
126
|
+
let merge_operator = parse_merge_operator(&kwargs)?;
|
|
123
127
|
let checkpoint_lifetime = get_optional::<u64>(&kwargs, "compactor_checkpoint_lifetime")?
|
|
124
128
|
.map(std::time::Duration::from_millis);
|
|
125
129
|
let consolidation_threshold =
|
|
126
130
|
get_optional::<usize>(&kwargs, "sorted_run_consolidation_threshold")?;
|
|
131
|
+
let incremental_compaction =
|
|
132
|
+
get_optional::<bool>(&kwargs, "experimental_incremental_compaction")?;
|
|
133
|
+
let block_alignment = get_optional::<bool>(&kwargs, "sst_block_alignment")?;
|
|
127
134
|
|
|
128
135
|
let mut settings = Settings::default();
|
|
129
136
|
if let Some(compactor) = settings.compactor_options.as_mut() {
|
|
137
|
+
if let Some(enabled) = incremental_compaction {
|
|
138
|
+
compactor.enable_experimental_incremental_compaction = enabled;
|
|
139
|
+
}
|
|
130
140
|
if let Some(lifetime) = checkpoint_lifetime {
|
|
131
141
|
compactor.checkpoint_lifetime = lifetime;
|
|
132
142
|
}
|
|
@@ -147,6 +157,9 @@ impl Database {
|
|
|
147
157
|
};
|
|
148
158
|
|
|
149
159
|
let mut builder = Db::builder(path, object_store).with_settings(settings);
|
|
160
|
+
if let Some(aligned) = block_alignment {
|
|
161
|
+
builder = builder.with_sst_block_alignment(aligned);
|
|
162
|
+
}
|
|
150
163
|
if let Some(merge_operator) = merge_operator {
|
|
151
164
|
builder = builder.with_merge_operator(merge_operator);
|
|
152
165
|
}
|
|
@@ -713,7 +726,10 @@ impl Database {
|
|
|
713
726
|
}
|
|
714
727
|
};
|
|
715
728
|
|
|
716
|
-
let txn =
|
|
729
|
+
let txn = self
|
|
730
|
+
.inner
|
|
731
|
+
.begin(isolation_level)
|
|
732
|
+
.map_err(crate::errors::map_error)?;
|
|
717
733
|
Ok(Transaction::new(txn))
|
|
718
734
|
}
|
|
719
735
|
|
|
@@ -722,7 +738,7 @@ impl Database {
|
|
|
722
738
|
/// # Returns
|
|
723
739
|
/// A new Snapshot instance
|
|
724
740
|
pub fn snapshot(&self) -> Result<Snapshot, Error> {
|
|
725
|
-
let snap =
|
|
741
|
+
let snap = self.inner.snapshot().map_err(crate::errors::map_error)?;
|
|
726
742
|
Ok(Snapshot::new(snap))
|
|
727
743
|
}
|
|
728
744
|
|
|
@@ -734,8 +750,25 @@ impl Database {
|
|
|
734
750
|
/// # Returns
|
|
735
751
|
/// Hash with id (UUID string) and manifest_id (int)
|
|
736
752
|
pub fn create_checkpoint(&self, kwargs: RHash) -> Result<RHash, Error> {
|
|
753
|
+
self.begin_checkpoint(kwargs)?.wait()
|
|
754
|
+
}
|
|
755
|
+
|
|
756
|
+
pub fn begin_checkpoint(&self, kwargs: RHash) -> Result<CheckpointHandle, Error> {
|
|
737
757
|
use slatedb::config::{CheckpointOptions, CheckpointScope};
|
|
738
758
|
|
|
759
|
+
let scope = match get_optional::<String>(&kwargs, "scope")?
|
|
760
|
+
.as_deref()
|
|
761
|
+
.unwrap_or("durable")
|
|
762
|
+
{
|
|
763
|
+
"durable" => CheckpointScope::Durable,
|
|
764
|
+
"all" => CheckpointScope::All,
|
|
765
|
+
other => {
|
|
766
|
+
return Err(invalid_argument_error(&format!(
|
|
767
|
+
"invalid scope: {} (expected 'durable' or 'all')",
|
|
768
|
+
other
|
|
769
|
+
)))
|
|
770
|
+
}
|
|
771
|
+
};
|
|
739
772
|
let lifetime =
|
|
740
773
|
get_optional::<u64>(&kwargs, "lifetime")?.map(std::time::Duration::from_millis);
|
|
741
774
|
let name = get_optional::<String>(&kwargs, "name")?;
|
|
@@ -746,18 +779,9 @@ impl Database {
|
|
|
746
779
|
name,
|
|
747
780
|
};
|
|
748
781
|
|
|
749
|
-
let result = block_on_result(async {
|
|
750
|
-
self.inner
|
|
751
|
-
.create_checkpoint(CheckpointScope::Durable, &options)
|
|
752
|
-
.await
|
|
753
|
-
})?;
|
|
754
|
-
|
|
755
|
-
let ruby = Ruby::get().expect("Ruby runtime not available");
|
|
756
|
-
let hash = ruby.hash_new();
|
|
757
|
-
hash.aset(ruby.to_symbol("id"), result.id.to_string())?;
|
|
758
|
-
hash.aset(ruby.to_symbol("manifest_id"), result.manifest_id)?;
|
|
782
|
+
let result = block_on_result(async { self.inner.begin_checkpoint(scope, &options).await })?;
|
|
759
783
|
|
|
760
|
-
Ok(
|
|
784
|
+
Ok(CheckpointHandle::new(result))
|
|
761
785
|
}
|
|
762
786
|
|
|
763
787
|
/// Flush the database to ensure durability.
|
|
@@ -859,6 +883,7 @@ pub fn define_database_class(ruby: &Ruby, module: &magnus::RModule) -> Result<()
|
|
|
859
883
|
"_create_checkpoint",
|
|
860
884
|
method!(Database::create_checkpoint, 1),
|
|
861
885
|
)?;
|
|
886
|
+
class.define_method("_begin_checkpoint", method!(Database::begin_checkpoint, 1))?;
|
|
862
887
|
class.define_method("_flush", method!(Database::flush, 0))?;
|
|
863
888
|
class.define_method(
|
|
864
889
|
"_flush_with_options",
|
data/ext/slatedb/src/lib.rs
CHANGED
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
use magnus::{Error, Ruby};
|
|
18
18
|
|
|
19
19
|
mod admin;
|
|
20
|
+
mod checkpoint;
|
|
20
21
|
mod database;
|
|
21
22
|
mod errors;
|
|
22
23
|
mod iterator;
|
|
@@ -40,6 +41,7 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
|
|
|
40
41
|
errors::define_exceptions(ruby, &module)?;
|
|
41
42
|
|
|
42
43
|
// Define core classes
|
|
44
|
+
checkpoint::define_checkpoint_class(ruby, &module)?;
|
|
43
45
|
database::define_database_class(ruby, &module)?;
|
|
44
46
|
iterator::define_iterator_class(ruby, &module)?;
|
|
45
47
|
write_batch::define_write_batch_class(ruby, &module)?;
|
|
@@ -1,11 +1,7 @@
|
|
|
1
1
|
use std::sync::Arc;
|
|
2
|
-
use std::thread;
|
|
3
2
|
|
|
4
3
|
use bytes::Bytes;
|
|
5
|
-
use
|
|
6
|
-
use magnus::rb_sys::{AsRawValue, FromRawValue};
|
|
7
|
-
use magnus::value::ReprValue;
|
|
8
|
-
use magnus::{Error, RHash, Ruby, Value};
|
|
4
|
+
use magnus::{Error, RHash};
|
|
9
5
|
use slatedb::{MergeOperator, MergeOperatorError};
|
|
10
6
|
|
|
11
7
|
use crate::errors::invalid_argument_error;
|
|
@@ -39,166 +35,6 @@ impl MergeOperator for StringConcatMergeOperator {
|
|
|
39
35
|
}
|
|
40
36
|
}
|
|
41
37
|
|
|
42
|
-
/// A merge operator that calls a Ruby block/proc.
|
|
43
|
-
///
|
|
44
|
-
/// This stores the raw Ruby VALUE and calls it via `with_gvl` when merge
|
|
45
|
-
/// operations are needed. The proc is called with (key, existing_value, new_value)
|
|
46
|
-
/// and should return the merged value as a String.
|
|
47
|
-
///
|
|
48
|
-
/// # Thread Safety
|
|
49
|
-
///
|
|
50
|
-
/// The Ruby proc can only be called from the Ruby thread that created this operator.
|
|
51
|
-
/// If the merge is called from a different thread (e.g., a Tokio worker thread during
|
|
52
|
-
/// background compaction), the merge will use a fallback string concatenation behavior.
|
|
53
|
-
///
|
|
54
|
-
/// # Safety
|
|
55
|
-
///
|
|
56
|
-
/// The Ruby proc must be kept alive (not garbage collected) for the lifetime
|
|
57
|
-
/// of this operator. This is typically handled by storing a reference to the
|
|
58
|
-
/// proc in the Ruby Database object.
|
|
59
|
-
pub struct RubyProcMergeOperator {
|
|
60
|
-
/// The raw Ruby VALUE of the proc. We store this as a raw value because
|
|
61
|
-
/// magnus::Value is not Send+Sync, but we need to be thread-safe.
|
|
62
|
-
/// We re-acquire the GVL before using it, which makes this safe.
|
|
63
|
-
proc_value: usize,
|
|
64
|
-
/// The thread ID of the Ruby thread that created this operator.
|
|
65
|
-
/// We can only safely call Ruby from this thread.
|
|
66
|
-
ruby_thread_id: thread::ThreadId,
|
|
67
|
-
}
|
|
68
|
-
|
|
69
|
-
// SAFETY: We only access the proc_value when we hold the GVL via the Ruby thread,
|
|
70
|
-
// which ensures thread-safe access to Ruby objects.
|
|
71
|
-
unsafe impl Send for RubyProcMergeOperator {}
|
|
72
|
-
unsafe impl Sync for RubyProcMergeOperator {}
|
|
73
|
-
|
|
74
|
-
impl RubyProcMergeOperator {
|
|
75
|
-
/// Create a new RubyProcMergeOperator from a Ruby proc/block.
|
|
76
|
-
///
|
|
77
|
-
/// # Safety
|
|
78
|
-
///
|
|
79
|
-
/// The caller must ensure the proc remains alive (not GC'd) for the
|
|
80
|
-
/// lifetime of this operator.
|
|
81
|
-
pub fn new(proc: Value) -> Self {
|
|
82
|
-
Self {
|
|
83
|
-
proc_value: proc.as_raw() as usize,
|
|
84
|
-
ruby_thread_id: thread::current().id(),
|
|
85
|
-
}
|
|
86
|
-
}
|
|
87
|
-
|
|
88
|
-
/// Check if we're on the Ruby thread that created this operator.
|
|
89
|
-
fn is_ruby_thread(&self) -> bool {
|
|
90
|
-
thread::current().id() == self.ruby_thread_id
|
|
91
|
-
}
|
|
92
|
-
|
|
93
|
-
/// Call the Ruby proc with the given arguments.
|
|
94
|
-
/// This must only be called from the Ruby thread (after checking is_ruby_thread).
|
|
95
|
-
fn call_proc_on_ruby_thread(
|
|
96
|
-
&self,
|
|
97
|
-
key: &str,
|
|
98
|
-
existing_value: Option<&str>,
|
|
99
|
-
new_value: &str,
|
|
100
|
-
) -> Result<Bytes, MergeOperatorError> {
|
|
101
|
-
// We're on the Ruby thread, so we can use with_gvl
|
|
102
|
-
// Import here to avoid the circular dependency at module level
|
|
103
|
-
use crate::runtime::with_gvl;
|
|
104
|
-
|
|
105
|
-
let key_owned = key.to_string();
|
|
106
|
-
let existing_owned = existing_value.map(|s| s.to_string());
|
|
107
|
-
let new_owned = new_value.to_string();
|
|
108
|
-
|
|
109
|
-
with_gvl(|| {
|
|
110
|
-
let ruby = Ruby::get().expect("Ruby runtime not available");
|
|
111
|
-
|
|
112
|
-
// Reconstruct the proc Value from the raw pointer
|
|
113
|
-
let proc = unsafe { Value::from_raw(self.proc_value as _) };
|
|
114
|
-
|
|
115
|
-
// Build arguments: (key, existing_value, new_value)
|
|
116
|
-
let existing_arg: Value = match &existing_owned {
|
|
117
|
-
Some(s) => ruby.str_new(s).as_value(),
|
|
118
|
-
None => ruby.qnil().as_value(),
|
|
119
|
-
};
|
|
120
|
-
|
|
121
|
-
// Call the proc
|
|
122
|
-
let result: Result<String, magnus::Error> = proc.funcall(
|
|
123
|
-
"call",
|
|
124
|
-
(
|
|
125
|
-
ruby.str_new(&key_owned),
|
|
126
|
-
existing_arg,
|
|
127
|
-
ruby.str_new(&new_owned),
|
|
128
|
-
),
|
|
129
|
-
);
|
|
130
|
-
|
|
131
|
-
match result {
|
|
132
|
-
Ok(merged) => Ok(Bytes::from(merged)),
|
|
133
|
-
Err(e) => {
|
|
134
|
-
error!("Ruby merge operator error: {}", e);
|
|
135
|
-
Err(MergeOperatorError::EmptyBatch)
|
|
136
|
-
}
|
|
137
|
-
}
|
|
138
|
-
})
|
|
139
|
-
}
|
|
140
|
-
|
|
141
|
-
/// Fallback merge when we're not on the Ruby thread.
|
|
142
|
-
/// Uses simple concatenation as a safe default.
|
|
143
|
-
fn fallback_merge(
|
|
144
|
-
&self,
|
|
145
|
-
existing_value: Option<&Bytes>,
|
|
146
|
-
new_value: &Bytes,
|
|
147
|
-
) -> Result<Bytes, MergeOperatorError> {
|
|
148
|
-
error!(
|
|
149
|
-
"Ruby merge operator called from non-Ruby thread, using fallback concatenation. \
|
|
150
|
-
This can happen during background compaction."
|
|
151
|
-
);
|
|
152
|
-
let mut result = existing_value.map(|v| v.to_vec()).unwrap_or_default();
|
|
153
|
-
result.extend_from_slice(new_value);
|
|
154
|
-
Ok(Bytes::from(result))
|
|
155
|
-
}
|
|
156
|
-
|
|
157
|
-
/// Call the Ruby proc with the given arguments, handling thread safety.
|
|
158
|
-
fn call_proc(
|
|
159
|
-
&self,
|
|
160
|
-
key: &Bytes,
|
|
161
|
-
existing_value: Option<&Bytes>,
|
|
162
|
-
new_value: &Bytes,
|
|
163
|
-
) -> Result<Bytes, MergeOperatorError> {
|
|
164
|
-
let key_str = String::from_utf8_lossy(key);
|
|
165
|
-
let existing_str = existing_value.map(|v| String::from_utf8_lossy(v));
|
|
166
|
-
let new_str = String::from_utf8_lossy(new_value);
|
|
167
|
-
|
|
168
|
-
if self.is_ruby_thread() {
|
|
169
|
-
self.call_proc_on_ruby_thread(&key_str, existing_str.as_deref(), &new_str)
|
|
170
|
-
} else {
|
|
171
|
-
// We're on a worker thread, use fallback
|
|
172
|
-
self.fallback_merge(existing_value, new_value)
|
|
173
|
-
}
|
|
174
|
-
}
|
|
175
|
-
}
|
|
176
|
-
|
|
177
|
-
impl MergeOperator for RubyProcMergeOperator {
|
|
178
|
-
fn merge(
|
|
179
|
-
&self,
|
|
180
|
-
key: &Bytes,
|
|
181
|
-
existing_value: Option<Bytes>,
|
|
182
|
-
value: Bytes,
|
|
183
|
-
) -> Result<Bytes, MergeOperatorError> {
|
|
184
|
-
self.call_proc(key, existing_value.as_ref(), &value)
|
|
185
|
-
}
|
|
186
|
-
|
|
187
|
-
fn merge_batch(
|
|
188
|
-
&self,
|
|
189
|
-
key: &Bytes,
|
|
190
|
-
existing_value: Option<Bytes>,
|
|
191
|
-
operands: &[Bytes],
|
|
192
|
-
) -> Result<Bytes, MergeOperatorError> {
|
|
193
|
-
// Apply operands one at a time through the Ruby proc
|
|
194
|
-
let mut current = existing_value;
|
|
195
|
-
for operand in operands {
|
|
196
|
-
current = Some(self.call_proc(key, current.as_ref(), operand)?);
|
|
197
|
-
}
|
|
198
|
-
Ok(current.unwrap_or_default())
|
|
199
|
-
}
|
|
200
|
-
}
|
|
201
|
-
|
|
202
38
|
pub fn parse_merge_operator(
|
|
203
39
|
kwargs: &RHash,
|
|
204
40
|
) -> Result<Option<Arc<dyn MergeOperator + Send + Sync>>, Error> {
|
|
@@ -211,7 +47,7 @@ pub fn parse_merge_operator(
|
|
|
211
47
|
"string_concat" | "concat" => Arc::new(StringConcatMergeOperator),
|
|
212
48
|
_ => {
|
|
213
49
|
return Err(invalid_argument_error(&format!(
|
|
214
|
-
"invalid merge_operator: {} (expected 'string_concat'
|
|
50
|
+
"invalid merge_operator: {} (expected 'string_concat' or 'concat')",
|
|
215
51
|
merge_operator
|
|
216
52
|
)))
|
|
217
53
|
}
|
|
@@ -219,22 +55,3 @@ pub fn parse_merge_operator(
|
|
|
219
55
|
|
|
220
56
|
Ok(Some(operator))
|
|
221
57
|
}
|
|
222
|
-
|
|
223
|
-
/// Parse a Ruby proc as a merge operator.
|
|
224
|
-
pub fn parse_merge_operator_proc(
|
|
225
|
-
kwargs: &RHash,
|
|
226
|
-
) -> Result<Option<Arc<dyn MergeOperator + Send + Sync>>, Error> {
|
|
227
|
-
let proc_value = get_optional::<Value>(kwargs, "merge_operator_proc")?;
|
|
228
|
-
let Some(proc) = proc_value else {
|
|
229
|
-
return Ok(None);
|
|
230
|
-
};
|
|
231
|
-
|
|
232
|
-
// Verify it's callable
|
|
233
|
-
if !proc.respond_to("call", false).unwrap_or(false) {
|
|
234
|
-
return Err(invalid_argument_error(
|
|
235
|
-
"merge_operator_proc must respond to 'call'",
|
|
236
|
-
));
|
|
237
|
-
}
|
|
238
|
-
|
|
239
|
-
Ok(Some(Arc::new(RubyProcMergeOperator::new(proc))))
|
|
240
|
-
}
|
data/ext/slatedb/src/reader.rs
CHANGED
|
@@ -9,6 +9,7 @@ use slatedb::IterationOrder;
|
|
|
9
9
|
|
|
10
10
|
use crate::errors::invalid_argument_error;
|
|
11
11
|
use crate::iterator::Iterator;
|
|
12
|
+
use crate::merge_ops::parse_merge_operator;
|
|
12
13
|
use crate::runtime::block_on_result;
|
|
13
14
|
use crate::utils::{get_optional, prefix_subrange_from_kwargs, resolve_object_store};
|
|
14
15
|
|
|
@@ -57,6 +58,7 @@ impl Reader {
|
|
|
57
58
|
let max_open_file_handles = get_optional::<usize>(&kwargs, "max_open_file_handles")?;
|
|
58
59
|
let cache_root = get_optional::<String>(&kwargs, "cache_root")?;
|
|
59
60
|
let follow_latest = get_optional::<bool>(&kwargs, "follow_latest")?.unwrap_or(false);
|
|
61
|
+
let merge_operator = parse_merge_operator(&kwargs)?;
|
|
60
62
|
|
|
61
63
|
// Resolve the reader mode from checkpoint_id / follow_latest.
|
|
62
64
|
let mode = match checkpoint_id {
|
|
@@ -96,14 +98,23 @@ impl Reader {
|
|
|
96
98
|
if let Some(skip_replay) = skip_wal_replay {
|
|
97
99
|
options.skip_wal_replay = skip_replay;
|
|
98
100
|
}
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
101
|
+
#[allow(deprecated)]
|
|
102
|
+
{
|
|
103
|
+
if let Some(ref root) = cache_root {
|
|
104
|
+
options.object_store_cache_options.root_folder =
|
|
105
|
+
Some(std::path::PathBuf::from(root));
|
|
106
|
+
}
|
|
107
|
+
if let Some(max_handles) = max_open_file_handles {
|
|
108
|
+
options.object_store_cache_options.max_open_file_handles = max_handles;
|
|
109
|
+
}
|
|
102
110
|
}
|
|
103
|
-
|
|
104
|
-
|
|
111
|
+
let mut builder = DbReader::builder(path, object_store)
|
|
112
|
+
.with_reader_mode(mode)
|
|
113
|
+
.with_options(options);
|
|
114
|
+
if let Some(operator) = merge_operator {
|
|
115
|
+
builder = builder.with_merge_operator(operator);
|
|
105
116
|
}
|
|
106
|
-
|
|
117
|
+
builder.build().await
|
|
107
118
|
})?;
|
|
108
119
|
|
|
109
120
|
Ok(Self {
|
|
@@ -150,6 +161,10 @@ impl Reader {
|
|
|
150
161
|
opts.cache_blocks = cb;
|
|
151
162
|
}
|
|
152
163
|
|
|
164
|
+
if let Some(limit) = get_optional::<usize>(&kwargs, "max_concurrent_sources")? {
|
|
165
|
+
opts.max_concurrent_sources = limit;
|
|
166
|
+
}
|
|
167
|
+
|
|
153
168
|
let result =
|
|
154
169
|
block_on_result(async { self.inner.get_with_options(key.as_bytes(), &opts).await })?;
|
|
155
170
|
Ok(result.map(|b| String::from_utf8_lossy(&b).to_string()))
|
data/ext/slatedb/src/runtime.rs
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
use magnus::Error;
|
|
2
2
|
use once_cell::sync::OnceCell;
|
|
3
|
-
use rb_sys::
|
|
3
|
+
use rb_sys::rb_thread_call_without_gvl;
|
|
4
4
|
use slatedb::Error as SlateError;
|
|
5
5
|
use std::ffi::c_void;
|
|
6
6
|
use std::future::Future;
|
|
@@ -98,54 +98,3 @@ where
|
|
|
98
98
|
|
|
99
99
|
closure.result.expect("closure did not run")
|
|
100
100
|
}
|
|
101
|
-
|
|
102
|
-
/// Execute a closure while holding the Ruby GVL.
|
|
103
|
-
///
|
|
104
|
-
/// This is the inverse of `without_gvl`. Use this when you need to call back
|
|
105
|
-
/// into Ruby from code that has previously released the GVL (e.g., inside
|
|
106
|
-
/// a future being executed by `block_on`).
|
|
107
|
-
///
|
|
108
|
-
/// # Safety
|
|
109
|
-
///
|
|
110
|
-
/// This function can ONLY be called from a Ruby thread that has previously
|
|
111
|
-
/// released the GVL via `without_gvl` or `rb_thread_call_without_gvl`.
|
|
112
|
-
/// Calling it from a non-Ruby thread (like a spawned Tokio task) will cause
|
|
113
|
-
/// a Ruby fatal error.
|
|
114
|
-
///
|
|
115
|
-
/// # Panics
|
|
116
|
-
///
|
|
117
|
-
/// Panics if called from a non-Ruby thread.
|
|
118
|
-
pub fn with_gvl<F, T>(f: F) -> T
|
|
119
|
-
where
|
|
120
|
-
F: FnOnce() -> T,
|
|
121
|
-
{
|
|
122
|
-
struct Closure<F, T> {
|
|
123
|
-
f: Option<F>,
|
|
124
|
-
result: Option<T>,
|
|
125
|
-
}
|
|
126
|
-
|
|
127
|
-
extern "C" fn call_closure<F, T>(data: *mut c_void) -> *mut c_void
|
|
128
|
-
where
|
|
129
|
-
F: FnOnce() -> T,
|
|
130
|
-
{
|
|
131
|
-
let closure = unsafe { &mut *(data as *mut Closure<F, T>) };
|
|
132
|
-
if let Some(f) = closure.f.take() {
|
|
133
|
-
closure.result = Some(f());
|
|
134
|
-
}
|
|
135
|
-
std::ptr::null_mut()
|
|
136
|
-
}
|
|
137
|
-
|
|
138
|
-
let mut closure = Closure {
|
|
139
|
-
f: Some(f),
|
|
140
|
-
result: None,
|
|
141
|
-
};
|
|
142
|
-
|
|
143
|
-
unsafe {
|
|
144
|
-
rb_thread_call_with_gvl(
|
|
145
|
-
Some(call_closure::<F, T>),
|
|
146
|
-
&mut closure as *mut _ as *mut c_void,
|
|
147
|
-
);
|
|
148
|
-
}
|
|
149
|
-
|
|
150
|
-
closure.result.expect("closure did not run")
|
|
151
|
-
}
|
data/ext/slatedb/src/snapshot.rs
CHANGED
|
@@ -73,6 +73,10 @@ impl Snapshot {
|
|
|
73
73
|
opts.cache_blocks = cb;
|
|
74
74
|
}
|
|
75
75
|
|
|
76
|
+
if let Some(limit) = get_optional::<usize>(&kwargs, "max_concurrent_sources")? {
|
|
77
|
+
opts.max_concurrent_sources = limit;
|
|
78
|
+
}
|
|
79
|
+
|
|
76
80
|
let guard = self.inner.borrow();
|
|
77
81
|
let snapshot = guard
|
|
78
82
|
.as_ref()
|
data/lib/slatedb/database.rb
CHANGED
|
@@ -9,15 +9,18 @@ module SlateDb
|
|
|
9
9
|
#
|
|
10
10
|
# @param path [String] The path identifier for the database
|
|
11
11
|
# @param url [String, nil] Optional object store URL (e.g., "s3://bucket/path")
|
|
12
|
-
# @param merge_operator [Symbol, String,
|
|
13
|
-
#
|
|
14
|
-
# that takes (key, existing_value, new_value) and returns the merged value.
|
|
12
|
+
# @param merge_operator [Symbol, String, nil] Optional built-in merge operator
|
|
13
|
+
# (:string_concat or :concat). Ruby callbacks are not supported.
|
|
15
14
|
# @param compactor_checkpoint_lifetime [Integer, nil] How long compaction inputs
|
|
16
15
|
# are protected from garbage collection, in milliseconds (default: 15 minutes).
|
|
17
16
|
# Reads using those inputs must finish within this time.
|
|
18
17
|
# @param sorted_run_consolidation_threshold [Integer, nil] Sorted-run count above
|
|
19
18
|
# which fallback consolidation starts. Zero disables it (default); use at least
|
|
20
19
|
# two to enable it. One is disabled by SlateDB with a warning. This is not a hard limit.
|
|
20
|
+
# @param experimental_incremental_compaction [Boolean, nil] Publish sorted-run
|
|
21
|
+
# compaction output before completion (experimental, disabled by default).
|
|
22
|
+
# @param sst_block_alignment [Boolean, nil] Align SST data blocks to 4 KiB
|
|
23
|
+
# boundaries (disabled by default; incompatible with compression).
|
|
21
24
|
# @yield [db] If a block is given, yields the database and ensures it's closed
|
|
22
25
|
# @return [Database] The opened database (or block result if block given)
|
|
23
26
|
#
|
|
@@ -34,32 +37,22 @@ module SlateDb
|
|
|
34
37
|
# @example Open with S3 backend
|
|
35
38
|
# db = SlateDb::Database.open("/tmp/mydb", url: "s3://mybucket/path")
|
|
36
39
|
#
|
|
37
|
-
# @example Open with a custom merge operator (Proc)
|
|
38
|
-
# # Custom merge that adds numbers
|
|
39
|
-
# db = SlateDb::Database.open("/tmp/mydb", merge_operator: ->(key, existing, new_val) {
|
|
40
|
-
# existing_num = existing ? existing.to_i : 0
|
|
41
|
-
# (existing_num + new_val.to_i).to_s
|
|
42
|
-
# })
|
|
43
|
-
# db.merge("counter", "5")
|
|
44
|
-
# db.merge("counter", "3")
|
|
45
|
-
# db.get("counter") # => "8"
|
|
46
|
-
#
|
|
47
40
|
def open(path, url: nil, merge_operator: nil,
|
|
48
|
-
compactor_checkpoint_lifetime: nil, sorted_run_consolidation_threshold: nil
|
|
41
|
+
compactor_checkpoint_lifetime: nil, sorted_run_consolidation_threshold: nil,
|
|
42
|
+
experimental_incremental_compaction: nil, sst_block_alignment: nil)
|
|
49
43
|
opts = {
|
|
50
44
|
compactor_checkpoint_lifetime: compactor_checkpoint_lifetime,
|
|
51
|
-
sorted_run_consolidation_threshold: sorted_run_consolidation_threshold
|
|
45
|
+
sorted_run_consolidation_threshold: sorted_run_consolidation_threshold,
|
|
46
|
+
experimental_incremental_compaction: experimental_incremental_compaction,
|
|
47
|
+
sst_block_alignment: sst_block_alignment
|
|
52
48
|
}.compact
|
|
53
49
|
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
opts[:merge_operator] = merge_operator.to_s
|
|
57
|
-
when Proc
|
|
58
|
-
# Store the proc to prevent GC and pass to Rust
|
|
59
|
-
@_merge_operator_proc = merge_operator
|
|
60
|
-
opts[:merge_operator_proc] = merge_operator
|
|
50
|
+
unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
|
|
51
|
+
raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
|
|
61
52
|
end
|
|
62
53
|
|
|
54
|
+
opts[:merge_operator] = merge_operator.to_s if merge_operator
|
|
55
|
+
|
|
63
56
|
db = _open(path, url, opts)
|
|
64
57
|
|
|
65
58
|
if block_given?
|
|
@@ -84,6 +77,8 @@ module SlateDb
|
|
|
84
77
|
# @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
|
|
85
78
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
86
79
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
80
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources to fetch after
|
|
81
|
+
# the first source misses (default: 1; zero also means serial fetching).
|
|
87
82
|
# @return [String, nil] The value, or nil if not found
|
|
88
83
|
#
|
|
89
84
|
# @example Basic get
|
|
@@ -92,11 +87,12 @@ module SlateDb
|
|
|
92
87
|
# @example Get with options
|
|
93
88
|
# value = db.get("mykey", durability_filter: "memory", dirty: true)
|
|
94
89
|
#
|
|
95
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
90
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
96
91
|
opts = {}
|
|
97
92
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
98
93
|
opts[:dirty] = dirty unless dirty.nil?
|
|
99
94
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
95
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
100
96
|
|
|
101
97
|
if opts.empty?
|
|
102
98
|
_get(key)
|
|
@@ -112,17 +108,20 @@ module SlateDb
|
|
|
112
108
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
113
109
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
114
110
|
# @return [Hash, nil] A hash with :key, :value, :seq, :create_ts, and :expire_ts, or nil if not found
|
|
111
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
112
|
+
# miss. See {#get}.
|
|
115
113
|
#
|
|
116
114
|
# @example Inspect metadata
|
|
117
115
|
# entry = db.get_key_value("mykey")
|
|
118
116
|
# entry[:value] # => "myvalue"
|
|
119
117
|
# entry[:seq] # => sequence number
|
|
120
118
|
#
|
|
121
|
-
def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
119
|
+
def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
122
120
|
opts = {}
|
|
123
121
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
124
122
|
opts[:dirty] = dirty unless dirty.nil?
|
|
125
123
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
124
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
126
125
|
|
|
127
126
|
if opts.empty?
|
|
128
127
|
_get_key_value(key)
|
|
@@ -494,6 +493,8 @@ module SlateDb
|
|
|
494
493
|
#
|
|
495
494
|
# @param lifetime [Integer, nil] Checkpoint lifetime in milliseconds
|
|
496
495
|
# @param name [String, nil] Optional name for the checkpoint
|
|
496
|
+
# @param scope [Symbol, String] :durable (default) captures durable writes;
|
|
497
|
+
# :all also flushes pending writes before storing the checkpoint.
|
|
497
498
|
# @return [Hash] Hash with :id (UUID string) and :manifest_id (integer)
|
|
498
499
|
#
|
|
499
500
|
# @example Create a named checkpoint
|
|
@@ -503,13 +504,26 @@ module SlateDb
|
|
|
503
504
|
# @example Create a checkpoint with lifetime
|
|
504
505
|
# checkpoint = db.create_checkpoint(lifetime: 3600_000) # 1 hour
|
|
505
506
|
#
|
|
506
|
-
def create_checkpoint(lifetime: nil, name: nil)
|
|
507
|
-
opts = {}
|
|
507
|
+
def create_checkpoint(lifetime: nil, name: nil, scope: :durable)
|
|
508
|
+
opts = { scope: scope.to_s }
|
|
508
509
|
opts[:lifetime] = lifetime if lifetime
|
|
509
510
|
opts[:name] = name if name
|
|
510
511
|
_create_checkpoint(opts)
|
|
511
512
|
end
|
|
512
513
|
|
|
514
|
+
# Start a checkpoint without waiting for it to be stored.
|
|
515
|
+
#
|
|
516
|
+
# With scope: :all, captures writes completed before this call and excludes
|
|
517
|
+
# writes started after it returns. Writes during the call may be included.
|
|
518
|
+
# Options match {#create_checkpoint}. Dropping the handle does not cancel it.
|
|
519
|
+
# @return [CheckpointHandle] #id is available immediately; #wait returns the
|
|
520
|
+
# :id/:manifest_id hash once stored and consumes the handle. A second wait
|
|
521
|
+
# raises ClosedError. #id remains available after waiting.
|
|
522
|
+
def begin_checkpoint(lifetime: nil, name: nil, scope: :durable)
|
|
523
|
+
opts = { scope: scope.to_s, lifetime: lifetime, name: name }.compact
|
|
524
|
+
_begin_checkpoint(opts)
|
|
525
|
+
end
|
|
526
|
+
|
|
513
527
|
# Get database metrics registry.
|
|
514
528
|
#
|
|
515
529
|
# @return [Metrics] Metrics registry
|
data/lib/slatedb/reader.rb
CHANGED
|
@@ -21,6 +21,7 @@ module SlateDb
|
|
|
21
21
|
# @param cache_root [String, nil] Root folder for the reader's local on-disk
|
|
22
22
|
# object-store cache. Setting this enables the cached object store; when it is
|
|
23
23
|
# not set the cache (and `max_open_file_handles`) has no effect.
|
|
24
|
+
# This legacy cache is deprecated upstream in SlateDB 0.18.1.
|
|
24
25
|
# @param max_open_file_handles [Integer, nil] Maximum number of file handles to keep
|
|
25
26
|
# open in the reader's file-handle cache. When the limit is reached, the least
|
|
26
27
|
# recently used handle is closed (default: 1000). Only takes effect when
|
|
@@ -54,29 +55,31 @@ module SlateDb
|
|
|
54
55
|
manifest_poll_interval: nil, checkpoint_lifetime: nil,
|
|
55
56
|
max_memtable_bytes: nil, cache_root: nil, max_open_file_handles: nil,
|
|
56
57
|
merge_operator: nil)
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
opts
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
58
|
+
unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
|
|
59
|
+
raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
opts = {
|
|
63
|
+
follow_latest: follow_latest,
|
|
64
|
+
manifest_poll_interval: manifest_poll_interval,
|
|
65
|
+
checkpoint_lifetime: checkpoint_lifetime,
|
|
66
|
+
max_memtable_bytes: max_memtable_bytes,
|
|
67
|
+
cache_root: cache_root,
|
|
68
|
+
max_open_file_handles: max_open_file_handles,
|
|
69
|
+
merge_operator: merge_operator&.to_s
|
|
70
|
+
}.compact
|
|
65
71
|
|
|
66
72
|
reader = _open(path, url, checkpoint_id, opts)
|
|
73
|
+
return reader unless block_given?
|
|
67
74
|
|
|
68
|
-
|
|
75
|
+
begin
|
|
76
|
+
yield reader
|
|
77
|
+
ensure
|
|
69
78
|
begin
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
reader.close
|
|
74
|
-
rescue StandardError
|
|
75
|
-
nil
|
|
76
|
-
end
|
|
79
|
+
reader.close
|
|
80
|
+
rescue StandardError
|
|
81
|
+
nil
|
|
77
82
|
end
|
|
78
|
-
else
|
|
79
|
-
reader
|
|
80
83
|
end
|
|
81
84
|
end
|
|
82
85
|
end
|
|
@@ -87,13 +90,16 @@ module SlateDb
|
|
|
87
90
|
# @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
|
|
88
91
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
89
92
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
93
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
94
|
+
# miss (default: 1; zero also means serial fetching).
|
|
90
95
|
# @return [String, nil] The value, or nil if not found
|
|
91
96
|
#
|
|
92
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
97
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
93
98
|
opts = {}
|
|
94
99
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
95
100
|
opts[:dirty] = dirty unless dirty.nil?
|
|
96
101
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
102
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
97
103
|
|
|
98
104
|
if opts.empty?
|
|
99
105
|
_get(key)
|
data/lib/slatedb/snapshot.rb
CHANGED
|
@@ -8,13 +8,16 @@ module SlateDb
|
|
|
8
8
|
# @param durability_filter [String, nil] Filter by durability level
|
|
9
9
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
10
10
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
11
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
12
|
+
# miss (default: 1; zero also means serial fetching).
|
|
11
13
|
# @return [String, nil] The value, or nil if not found
|
|
12
14
|
#
|
|
13
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
15
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
14
16
|
opts = {}
|
|
15
17
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
16
18
|
opts[:dirty] = dirty unless dirty.nil?
|
|
17
19
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
20
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
18
21
|
|
|
19
22
|
if opts.empty?
|
|
20
23
|
_get(key)
|
|
@@ -27,16 +30,19 @@ module SlateDb
|
|
|
27
30
|
#
|
|
28
31
|
# @param start_key [String] The start key (inclusive)
|
|
29
32
|
# @param end_key [String, nil] The end key (exclusive)
|
|
33
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
30
34
|
# @return [Iterator] An iterator over key-value pairs
|
|
31
35
|
#
|
|
32
36
|
def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
|
|
33
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
|
|
34
|
-
opts = {
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
37
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
|
|
38
|
+
opts = {
|
|
39
|
+
durability_filter: durability_filter&.to_s,
|
|
40
|
+
dirty: dirty,
|
|
41
|
+
read_ahead_bytes: read_ahead_bytes,
|
|
42
|
+
cache_blocks: cache_blocks,
|
|
43
|
+
max_fetch_tasks: max_fetch_tasks,
|
|
44
|
+
order: order&.to_s
|
|
45
|
+
}.compact
|
|
40
46
|
|
|
41
47
|
iter = if opts.empty?
|
|
42
48
|
_scan(start_key, end_key)
|
|
@@ -59,6 +65,7 @@ module SlateDb
|
|
|
59
65
|
# @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
|
|
60
66
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
61
67
|
# @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
|
|
68
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
62
69
|
# @param from [String, nil] Inclusive lower bound suffix, appended to the
|
|
63
70
|
# prefix, to start scanning from. Defaults to the start of the prefix.
|
|
64
71
|
# @param to [String, nil] Exclusive upper bound suffix, appended to the
|
|
@@ -66,7 +73,7 @@ module SlateDb
|
|
|
66
73
|
# @return [Iterator] An iterator over key-value pairs
|
|
67
74
|
#
|
|
68
75
|
def scan_prefix(prefix, durability_filter: nil, dirty: nil,
|
|
69
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
|
|
76
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
|
|
70
77
|
from: nil, to: nil, &)
|
|
71
78
|
opts = {
|
|
72
79
|
durability_filter: durability_filter&.to_s,
|
|
@@ -74,6 +81,7 @@ module SlateDb
|
|
|
74
81
|
read_ahead_bytes: read_ahead_bytes,
|
|
75
82
|
cache_blocks: cache_blocks,
|
|
76
83
|
max_fetch_tasks: max_fetch_tasks,
|
|
84
|
+
order: order&.to_s,
|
|
77
85
|
subrange_from: from,
|
|
78
86
|
subrange_to: to
|
|
79
87
|
}.compact
|
data/lib/slatedb/transaction.rb
CHANGED
|
@@ -8,13 +8,16 @@ module SlateDb
|
|
|
8
8
|
# @param durability_filter [String, nil] Filter by durability level
|
|
9
9
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
10
10
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
11
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
12
|
+
# miss (default: 1; zero also means serial fetching).
|
|
11
13
|
# @return [String, nil] The value, or nil if not found
|
|
12
14
|
#
|
|
13
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
15
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
14
16
|
opts = {}
|
|
15
17
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
16
18
|
opts[:dirty] = dirty unless dirty.nil?
|
|
17
19
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
20
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
18
21
|
|
|
19
22
|
if opts.empty?
|
|
20
23
|
_get(key)
|
|
@@ -66,16 +69,19 @@ module SlateDb
|
|
|
66
69
|
#
|
|
67
70
|
# @param start_key [String] The start key (inclusive)
|
|
68
71
|
# @param end_key [String, nil] The end key (exclusive)
|
|
72
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
69
73
|
# @return [Iterator] An iterator over key-value pairs
|
|
70
74
|
#
|
|
71
75
|
def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
|
|
72
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
|
|
73
|
-
opts = {
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
76
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
|
|
77
|
+
opts = {
|
|
78
|
+
durability_filter: durability_filter&.to_s,
|
|
79
|
+
dirty: dirty,
|
|
80
|
+
read_ahead_bytes: read_ahead_bytes,
|
|
81
|
+
cache_blocks: cache_blocks,
|
|
82
|
+
max_fetch_tasks: max_fetch_tasks,
|
|
83
|
+
order: order&.to_s
|
|
84
|
+
}.compact
|
|
79
85
|
|
|
80
86
|
iter = if opts.empty?
|
|
81
87
|
_scan(start_key, end_key)
|
|
@@ -98,6 +104,7 @@ module SlateDb
|
|
|
98
104
|
# @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
|
|
99
105
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
100
106
|
# @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
|
|
107
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
101
108
|
# @param from [String, nil] Inclusive lower bound suffix, appended to the
|
|
102
109
|
# prefix, to start scanning from. Defaults to the start of the prefix.
|
|
103
110
|
# @param to [String, nil] Exclusive upper bound suffix, appended to the
|
|
@@ -105,7 +112,7 @@ module SlateDb
|
|
|
105
112
|
# @return [Iterator] An iterator over key-value pairs
|
|
106
113
|
#
|
|
107
114
|
def scan_prefix(prefix, durability_filter: nil, dirty: nil,
|
|
108
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
|
|
115
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
|
|
109
116
|
from: nil, to: nil, &)
|
|
110
117
|
opts = {
|
|
111
118
|
durability_filter: durability_filter&.to_s,
|
|
@@ -113,6 +120,7 @@ module SlateDb
|
|
|
113
120
|
read_ahead_bytes: read_ahead_bytes,
|
|
114
121
|
cache_blocks: cache_blocks,
|
|
115
122
|
max_fetch_tasks: max_fetch_tasks,
|
|
123
|
+
order: order&.to_s,
|
|
116
124
|
subrange_from: from,
|
|
117
125
|
subrange_to: to
|
|
118
126
|
}.compact
|
data/lib/slatedb/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: slatedb
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- SlateDB Contributors
|
|
@@ -93,6 +93,7 @@ files:
|
|
|
93
93
|
- ext/slatedb/Cargo.toml
|
|
94
94
|
- ext/slatedb/extconf.rb
|
|
95
95
|
- ext/slatedb/src/admin.rs
|
|
96
|
+
- ext/slatedb/src/checkpoint.rs
|
|
96
97
|
- ext/slatedb/src/database.rs
|
|
97
98
|
- ext/slatedb/src/errors.rs
|
|
98
99
|
- ext/slatedb/src/iterator.rs
|