slatedb 0.6.1 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c46e1c81aa2f9f98834847d89ebe70a3caa18dd7307ec864d2d18fb448b8c366
4
- data.tar.gz: 1e24f5bb9dc914c3f9f5fe7025d481b0eb93160ffd5834f78d039c333d66e434
3
+ metadata.gz: d78fa73dbdaa8c8bd6bdacebc65b3f69b3df642eab3f7e8d7bdd08bde68b0ba0
4
+ data.tar.gz: 9dd7197dea492b9a146053f1aadad0bc233f258d5cceea2aadda64ad8c69a5c6
5
5
  SHA512:
6
- metadata.gz: 3b5fce35dda024e1011917c41685c88eadf9537558bcc643c06e4db988fc99dd3819f4d0b5f5be1ff5f660a33053b3c4b6fc579e1723523edc940cef67682d39
7
- data.tar.gz: e0025127be66530aa82e64bcc98141b20cf5e5fd0a66290f32bcd88b16b4b7143a144238f85b0f209c7f7fc56477d85cfa3556f5b1808940f31d55086bf7e99d
6
+ metadata.gz: 783e32df3d81361f30ab91003b9fa10848e342b548219eb4b622be0d760c2ba3d6a5f1211b455ea7ccd7d05e6f13dc793700878c6ac370cd4204a92de4b8cc8d
7
+ data.tar.gz: 5239793e435456d610b5a2eea877e58873f4d48a7e80f47dcc9b191cc82857d0f5669dd6780e0d26997b616e4bc4e501ac917b7f4860dce9d832269969689177
data/README.md CHANGED
@@ -30,6 +30,38 @@ gem install slatedb
30
30
 
31
31
  Precompiled native gems are published for common Linux and Darwin platforms. Building from source still requires a Rust toolchain.
32
32
 
33
+ ## Upgrading to SlateDB 0.18.1
34
+
35
+ This update moves the Rust engine from 0.17.0 to 0.18.1. Existing database files
36
+ remain readable. New Ruby options keep the existing defaults unless enabled.
37
+
38
+ - Two-phase checkpoints: `begin_checkpoint` returns a handle with `id` and `wait`.
39
+ Both checkpoint methods accept `scope: :all` to capture pending writes, or
40
+ `scope: :durable` (the existing default) for writes already durable.
41
+ - Point reads accept `max_concurrent_sources:` on databases, readers, snapshots,
42
+ and transactions; database metadata reads accept it too.
43
+ - `Database.open` accepts `experimental_incremental_compaction:` and
44
+ `sst_block_alignment:`. Both are disabled by default; see the warnings below.
45
+ - Snapshot and transaction range/prefix scans now accept `order:`, like database
46
+ and reader scans. Reader `merge_operator:` is now applied rather than ignored.
47
+ - **Breaking:** Ruby Proc/lambda merge operators are no longer accepted. They
48
+ silently used concatenation on other threads, which could change stored values.
49
+ Keep `:concat` / `:string_concat` for concatenation. For custom updates, use a
50
+ serializable transaction that reads the value and writes the computed result;
51
+ handle `SlateDb::TransactionError` conflicts in the application. Databases with
52
+ unresolved custom merge operands need an application-specific migration before
53
+ upgrading. Do not reopen them with `:concat` unless concatenation was their
54
+ intended merge rule.
55
+ - Reader disk-cache keywords remain supported, but upstream deprecated the
56
+ underlying `CachedObjectStore` in 0.18.1. The new mirror and Rust trait-based
57
+ extension APIs are not exposed by this gem yet.
58
+
59
+ Upstream also fixes clone-compaction retention/data-loss cases, writer ownership
60
+ validation after WAL replay, write stalls, upload retry/concurrency behavior, and
61
+ iterator fetch cancellation. It improves memtable seeks and point-read/scan work.
62
+ See the full [0.18.0 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.0)
63
+ and [0.18.1 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.1).
64
+
33
65
  ## Usage
34
66
 
35
67
  ### Basic Operations
@@ -64,6 +96,19 @@ SlateDb::Database.open("/tmp/mydb") do |db|
64
96
  end # automatically closed
65
97
  ```
66
98
 
99
+ ### Point-Read Concurrency
100
+
101
+ ```ruby
102
+ db.get("key", max_concurrent_sources: 4, cache_blocks: false)
103
+ db.get_entry("key", max_concurrent_sources: 4) # value plus sequence/timestamps
104
+ ```
105
+
106
+ `max_concurrent_sources` defaults to 1 (serial fetching); zero also means 1.
107
+ SlateDB checks the newest source alone, then fetches up to this many sources
108
+ concurrently after a miss. Higher values can reduce latency but increase object
109
+ store requests and bandwidth. The same keyword works with `get` on readers,
110
+ snapshots, and transactions. It does not change isolation or durability.
111
+
67
112
  ### Compaction Controls
68
113
 
69
114
  SlateDB 0.17 adds controls for how long compaction inputs remain protected from
@@ -87,6 +132,32 @@ projected sorted-run count exceeds the threshold. It is not a hard limit.
87
132
  Zero disables it (the default); use at least two to enable it. SlateDB warns and
88
133
  disables the trigger for one. Low thresholds can increase storage reads and writes.
89
134
 
135
+ SlateDB 0.18 adds an opt-in experimental mode:
136
+
137
+ ```ruby
138
+ SlateDb::Database.open("mydb", url: "s3://mybucket/path",
139
+ experimental_incremental_compaction: true) do |db|
140
+ db.put("key", "value")
141
+ end
142
+ ```
143
+
144
+ This publishes output from sorted-run compactions at most once a minute before
145
+ the full compaction finishes. It can reduce read amplification and allow earlier
146
+ storage reclamation after the compactor checkpoint expires. Compactions with L0
147
+ inputs still publish only when complete. It is disabled by default, experimental,
148
+ and may change or disappear upstream.
149
+
150
+ ### SST Block Alignment
151
+
152
+ `Database.open(..., sst_block_alignment: true)` pads SST data blocks to 4 KiB
153
+ boundaries, including oversized entries. It is disabled by default, uses more
154
+ storage, and cannot be combined with compression.
155
+
156
+ **Upgrade every reader before enabling alignment on a writer.** Older engines
157
+ that do not understand padded SSTs cannot read them. Once padded files exist,
158
+ rolling those readers back is not safe; disabling the option does not rewrite
159
+ existing files.
160
+
90
161
  ### Key Size
91
162
 
92
163
  Keys can now exceed the previous 65,535-byte limit, up to SlateDB's new
@@ -341,53 +412,19 @@ db.batch do |b|
341
412
  end
342
413
  ```
343
414
 
344
- #### Custom Merge Operators
345
-
346
- You can provide a Ruby Proc/lambda as a custom merge operator:
347
-
348
- ```ruby
349
- # Counter merge operator (adds numbers)
350
- counter_merge = ->(key, existing, new_value) {
351
- existing_num = existing ? existing.to_i : 0
352
- (existing_num + new_value.to_i).to_s
353
- }
354
-
355
- SlateDb::Database.open("/tmp/mydb", merge_operator: counter_merge) do |db|
356
- db.merge("visits", "1")
357
- db.merge("visits", "1")
358
- db.merge("visits", "1")
359
-
360
- db.get("visits") # => "3"
361
- end
415
+ #### Available Merge Operators
362
416
 
363
- # Max value merge operator
364
- max_merge = ->(key, existing, new_value) {
365
- existing_num = existing ? existing.to_i : 0
366
- new_num = new_value.to_i
367
- [existing_num, new_num].max.to_s
368
- }
417
+ - `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
369
418
 
370
- SlateDb::Database.open("/tmp/mydb", merge_operator: max_merge) do |db|
371
- db.merge("high_score", "100")
372
- db.merge("high_score", "250")
373
- db.merge("high_score", "150")
419
+ Ruby Proc/lambda operators are rejected. Use a serializable read/modify/write
420
+ transaction for custom logic, with application-level conflict handling:
374
421
 
375
- db.get("high_score") # => "250"
422
+ ```ruby
423
+ db.transaction(isolation: :serializable) do |txn|
424
+ txn.put("visits", (txn.get("visits").to_i + 1).to_s)
376
425
  end
377
426
  ```
378
427
 
379
- The proc receives three arguments:
380
- - `key` - The key being merged
381
- - `existing` - The existing value (nil if no value exists)
382
- - `new_value` - The new merge operand
383
-
384
- **Note:** Custom Proc merge operators work best with direct `db.merge()` calls. When used with transactions or batches, some merge operations may be processed on background threads and fall back to string concatenation.
385
-
386
- #### Available Merge Operators
387
-
388
- - `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
389
- - Any `Proc` or `lambda` - Custom merge logic
390
-
391
428
  ### Write Batches
392
429
 
393
430
  Perform multiple writes atomically:
@@ -495,6 +532,29 @@ SlateDb::Database.open("/tmp/mydb", url: "file:///tmp/mydb") do |db|
495
532
  end
496
533
  ```
497
534
 
535
+ `create_checkpoint` waits until the checkpoint is stored. Its default
536
+ `scope: :durable` preserves the existing behavior. To include writes made with
537
+ `await_durable: false`, use `scope: :all`; no separate `flush` is needed.
538
+
539
+ To separate capturing the write boundary from waiting for storage:
540
+
541
+ ```ruby
542
+ db.put("key", "before", await_durable: false)
543
+ handle = db.begin_checkpoint(scope: :all, name: "backup", lifetime: 3_600_000)
544
+ id = handle.id # UUID is available immediately
545
+ db.put("key", "after") # not part of this checkpoint
546
+ checkpoint = handle.wait # => { id: id, manifest_id: ... }
547
+ ```
548
+
549
+ With `scope: :all`, writes completed before `begin_checkpoint` starts are included;
550
+ writes started after it returns are excluded. Concurrent writes during the call
551
+ may be included. Starting the checkpoint can still wait for internal flush work;
552
+ it is not a fully nonblocking Ruby call. `wait` releases the Ruby GVL while
553
+ waiting, consumes the handle, and raises `SlateDb::ClosedError` if called again.
554
+ `id` remains readable. Dropping the handle does not cancel or delete the checkpoint.
555
+ If waiting fails, use the UUID with `Admin#list_checkpoints` to check its outcome
556
+ before starting another checkpoint.
557
+
498
558
  ### Snapshots
499
559
 
500
560
  Point-in-time consistent reads:
@@ -517,6 +577,11 @@ value = snap.get("key")
517
577
  snap.close
518
578
  ```
519
579
 
580
+ Range and prefix scans on snapshots and transactions accept the same `order:`
581
+ keyword as databases and readers, including `:asc`/`:ascending` and
582
+ `:desc`/`:descending`. Bounds remain lower-inclusive and upper-exclusive in
583
+ either direction.
584
+
520
585
  ### Reader (Read-Only Access)
521
586
 
522
587
  Open a database in read-only mode, useful for replicas:
@@ -568,6 +633,14 @@ SlateDb::Reader.open("/tmp/mydb",
568
633
  end
569
634
  ```
570
635
 
636
+ When the writer uses `merge_operator: :concat`, pass the same operator to
637
+ `Reader.open` so WAL replay and reads apply the correct merge rule. Unknown
638
+ operator names now raise `SlateDb::InvalidArgumentError` instead of being ignored.
639
+
640
+ The `cache_root` and `max_open_file_handles` keywords still work, but SlateDB
641
+ 0.18.1 deprecates this legacy disk cache in favor of its separate mirror crate or
642
+ tiered block cache. Those replacement APIs are not yet exposed here.
643
+
571
644
  ### Admin Operations
572
645
 
573
646
  Administrative operations for database management:
@@ -594,9 +667,6 @@ admin.delete_checkpoint("uuid")
594
667
  # Garbage Collection
595
668
  admin.run_gc # Run with default settings
596
669
  admin.run_gc(min_age: 3600_000) # Set min age for all directories (1 hour)
597
- admin.run_gc(manifest_min_age: 86400_000) # Custom age for manifest (1 day)
598
- admin.run_gc(wal_min_age: 60_000) # Custom age for WAL (1 minute)
599
- admin.run_gc(compacted_min_age: 60_000) # Custom age for compacted (1 minute)
600
670
 
601
671
  # Delete a database (SlateDB >= 0.16.0)
602
672
  would_delete = admin.delete_db # Dry run: returns the object paths
@@ -11,7 +11,7 @@ name = "slatedb"
11
11
  crate-type = ["cdylib"]
12
12
 
13
13
  [dependencies]
14
- slatedb = "0.17.0"
14
+ slatedb = "0.18.1"
15
15
  magnus = { version = "0.9.3", features = ["rb-sys"] }
16
16
  rb-sys = { version = "0.9.130", features = ["stable-api-compiled-fallback"] }
17
17
  tokio = { version = "1.54.1", features = ["rt-multi-thread", "sync"] }
@@ -19,5 +19,4 @@ bytes = "1.12.1"
19
19
  serde_json = "1.0.154"
20
20
  url = "2.5.8"
21
21
  once_cell = "1.21.4"
22
- log = "0.4.34"
23
- uuid = "1.29.0"
22
+ uuid = "1.29.2"
@@ -0,0 +1,47 @@
1
+ use magnus::prelude::*;
2
+ use magnus::{method, Error, RHash, Ruby};
3
+ use std::cell::RefCell;
4
+
5
+ use crate::errors::closed_error;
6
+ use crate::runtime::block_on_result;
7
+
8
+ #[magnus::wrap(class = "SlateDb::CheckpointHandle", free_immediately, size)]
9
+ pub struct CheckpointHandle {
10
+ id: String,
11
+ inner: RefCell<Option<slatedb::CheckpointHandle>>,
12
+ }
13
+
14
+ impl CheckpointHandle {
15
+ pub fn new(handle: slatedb::CheckpointHandle) -> Self {
16
+ Self {
17
+ id: handle.id().to_string(),
18
+ inner: RefCell::new(Some(handle)),
19
+ }
20
+ }
21
+
22
+ pub fn id(&self) -> String {
23
+ self.id.clone()
24
+ }
25
+
26
+ pub fn wait(&self) -> Result<RHash, Error> {
27
+ let handle = self
28
+ .inner
29
+ .borrow_mut()
30
+ .take()
31
+ .ok_or_else(|| closed_error("checkpoint handle has already been waited on"))?;
32
+ let result = block_on_result(handle.wait())?;
33
+
34
+ let ruby = Ruby::get().expect("Ruby runtime not available");
35
+ let hash = ruby.hash_new();
36
+ hash.aset(ruby.to_symbol("id"), result.id.to_string())?;
37
+ hash.aset(ruby.to_symbol("manifest_id"), result.manifest_id)?;
38
+ Ok(hash)
39
+ }
40
+ }
41
+
42
+ pub fn define_checkpoint_class(ruby: &Ruby, module: &magnus::RModule) -> Result<(), Error> {
43
+ let class = module.define_class("CheckpointHandle", ruby.class_object())?;
44
+ class.define_method("id", method!(CheckpointHandle::id, 0))?;
45
+ class.define_method("wait", method!(CheckpointHandle::wait, 0))?;
46
+ Ok(())
47
+ }
@@ -10,9 +10,10 @@ use slatedb::config::{
10
10
  use slatedb::object_store::memory::InMemory;
11
11
  use slatedb::{Db, IsolationLevel, IterationOrder, KeyValue};
12
12
 
13
+ use crate::checkpoint::CheckpointHandle;
13
14
  use crate::errors::invalid_argument_error;
14
15
  use crate::iterator::Iterator;
15
- use crate::merge_ops::{parse_merge_operator, parse_merge_operator_proc};
16
+ use crate::merge_ops::parse_merge_operator;
16
17
  use crate::metrics::Metrics;
17
18
  use crate::runtime::block_on_result;
18
19
  use crate::snapshot::Snapshot;
@@ -76,6 +77,10 @@ impl Database {
76
77
  opts.cache_blocks = cb;
77
78
  }
78
79
 
80
+ if let Some(limit) = get_optional::<usize>(kwargs, "max_concurrent_sources")? {
81
+ opts.max_concurrent_sources = limit;
82
+ }
83
+
79
84
  Ok(opts)
80
85
  }
81
86
 
@@ -113,20 +118,25 @@ impl Database {
113
118
  /// # Arguments
114
119
  /// * `path` - The path identifier for the database
115
120
  /// * `url` - Optional object store URL (e.g., "s3://bucket/path")
116
- /// * `kwargs` - Additional options (merge_operator, merge_operator_proc)
121
+ /// * `kwargs` - Additional options (merge_operator and compaction settings)
117
122
  ///
118
123
  /// # Returns
119
124
  /// A new Database instance
120
125
  pub fn open(path: String, url: Option<String>, kwargs: RHash) -> Result<Self, Error> {
121
- // Try string-based merge operator first, then proc-based
122
- let merge_operator = parse_merge_operator(&kwargs)?.or(parse_merge_operator_proc(&kwargs)?);
126
+ let merge_operator = parse_merge_operator(&kwargs)?;
123
127
  let checkpoint_lifetime = get_optional::<u64>(&kwargs, "compactor_checkpoint_lifetime")?
124
128
  .map(std::time::Duration::from_millis);
125
129
  let consolidation_threshold =
126
130
  get_optional::<usize>(&kwargs, "sorted_run_consolidation_threshold")?;
131
+ let incremental_compaction =
132
+ get_optional::<bool>(&kwargs, "experimental_incremental_compaction")?;
133
+ let block_alignment = get_optional::<bool>(&kwargs, "sst_block_alignment")?;
127
134
 
128
135
  let mut settings = Settings::default();
129
136
  if let Some(compactor) = settings.compactor_options.as_mut() {
137
+ if let Some(enabled) = incremental_compaction {
138
+ compactor.enable_experimental_incremental_compaction = enabled;
139
+ }
130
140
  if let Some(lifetime) = checkpoint_lifetime {
131
141
  compactor.checkpoint_lifetime = lifetime;
132
142
  }
@@ -147,6 +157,9 @@ impl Database {
147
157
  };
148
158
 
149
159
  let mut builder = Db::builder(path, object_store).with_settings(settings);
160
+ if let Some(aligned) = block_alignment {
161
+ builder = builder.with_sst_block_alignment(aligned);
162
+ }
150
163
  if let Some(merge_operator) = merge_operator {
151
164
  builder = builder.with_merge_operator(merge_operator);
152
165
  }
@@ -713,7 +726,10 @@ impl Database {
713
726
  }
714
727
  };
715
728
 
716
- let txn = block_on_result(async { self.inner.begin(isolation_level).await })?;
729
+ let txn = self
730
+ .inner
731
+ .begin(isolation_level)
732
+ .map_err(crate::errors::map_error)?;
717
733
  Ok(Transaction::new(txn))
718
734
  }
719
735
 
@@ -722,7 +738,7 @@ impl Database {
722
738
  /// # Returns
723
739
  /// A new Snapshot instance
724
740
  pub fn snapshot(&self) -> Result<Snapshot, Error> {
725
- let snap = block_on_result(async { self.inner.snapshot().await })?;
741
+ let snap = self.inner.snapshot().map_err(crate::errors::map_error)?;
726
742
  Ok(Snapshot::new(snap))
727
743
  }
728
744
 
@@ -734,8 +750,25 @@ impl Database {
734
750
  /// # Returns
735
751
  /// Hash with id (UUID string) and manifest_id (int)
736
752
  pub fn create_checkpoint(&self, kwargs: RHash) -> Result<RHash, Error> {
753
+ self.begin_checkpoint(kwargs)?.wait()
754
+ }
755
+
756
+ pub fn begin_checkpoint(&self, kwargs: RHash) -> Result<CheckpointHandle, Error> {
737
757
  use slatedb::config::{CheckpointOptions, CheckpointScope};
738
758
 
759
+ let scope = match get_optional::<String>(&kwargs, "scope")?
760
+ .as_deref()
761
+ .unwrap_or("durable")
762
+ {
763
+ "durable" => CheckpointScope::Durable,
764
+ "all" => CheckpointScope::All,
765
+ other => {
766
+ return Err(invalid_argument_error(&format!(
767
+ "invalid scope: {} (expected 'durable' or 'all')",
768
+ other
769
+ )))
770
+ }
771
+ };
739
772
  let lifetime =
740
773
  get_optional::<u64>(&kwargs, "lifetime")?.map(std::time::Duration::from_millis);
741
774
  let name = get_optional::<String>(&kwargs, "name")?;
@@ -746,18 +779,9 @@ impl Database {
746
779
  name,
747
780
  };
748
781
 
749
- let result = block_on_result(async {
750
- self.inner
751
- .create_checkpoint(CheckpointScope::Durable, &options)
752
- .await
753
- })?;
754
-
755
- let ruby = Ruby::get().expect("Ruby runtime not available");
756
- let hash = ruby.hash_new();
757
- hash.aset(ruby.to_symbol("id"), result.id.to_string())?;
758
- hash.aset(ruby.to_symbol("manifest_id"), result.manifest_id)?;
782
+ let result = block_on_result(async { self.inner.begin_checkpoint(scope, &options).await })?;
759
783
 
760
- Ok(hash)
784
+ Ok(CheckpointHandle::new(result))
761
785
  }
762
786
 
763
787
  /// Flush the database to ensure durability.
@@ -859,6 +883,7 @@ pub fn define_database_class(ruby: &Ruby, module: &magnus::RModule) -> Result<()
859
883
  "_create_checkpoint",
860
884
  method!(Database::create_checkpoint, 1),
861
885
  )?;
886
+ class.define_method("_begin_checkpoint", method!(Database::begin_checkpoint, 1))?;
862
887
  class.define_method("_flush", method!(Database::flush, 0))?;
863
888
  class.define_method(
864
889
  "_flush_with_options",
@@ -17,6 +17,7 @@
17
17
  use magnus::{Error, Ruby};
18
18
 
19
19
  mod admin;
20
+ mod checkpoint;
20
21
  mod database;
21
22
  mod errors;
22
23
  mod iterator;
@@ -40,6 +41,7 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
40
41
  errors::define_exceptions(ruby, &module)?;
41
42
 
42
43
  // Define core classes
44
+ checkpoint::define_checkpoint_class(ruby, &module)?;
43
45
  database::define_database_class(ruby, &module)?;
44
46
  iterator::define_iterator_class(ruby, &module)?;
45
47
  write_batch::define_write_batch_class(ruby, &module)?;
@@ -1,11 +1,7 @@
1
1
  use std::sync::Arc;
2
- use std::thread;
3
2
 
4
3
  use bytes::Bytes;
5
- use log::error;
6
- use magnus::rb_sys::{AsRawValue, FromRawValue};
7
- use magnus::value::ReprValue;
8
- use magnus::{Error, RHash, Ruby, Value};
4
+ use magnus::{Error, RHash};
9
5
  use slatedb::{MergeOperator, MergeOperatorError};
10
6
 
11
7
  use crate::errors::invalid_argument_error;
@@ -39,166 +35,6 @@ impl MergeOperator for StringConcatMergeOperator {
39
35
  }
40
36
  }
41
37
 
42
- /// A merge operator that calls a Ruby block/proc.
43
- ///
44
- /// This stores the raw Ruby VALUE and calls it via `with_gvl` when merge
45
- /// operations are needed. The proc is called with (key, existing_value, new_value)
46
- /// and should return the merged value as a String.
47
- ///
48
- /// # Thread Safety
49
- ///
50
- /// The Ruby proc can only be called from the Ruby thread that created this operator.
51
- /// If the merge is called from a different thread (e.g., a Tokio worker thread during
52
- /// background compaction), the merge will use a fallback string concatenation behavior.
53
- ///
54
- /// # Safety
55
- ///
56
- /// The Ruby proc must be kept alive (not garbage collected) for the lifetime
57
- /// of this operator. This is typically handled by storing a reference to the
58
- /// proc in the Ruby Database object.
59
- pub struct RubyProcMergeOperator {
60
- /// The raw Ruby VALUE of the proc. We store this as a raw value because
61
- /// magnus::Value is not Send+Sync, but we need to be thread-safe.
62
- /// We re-acquire the GVL before using it, which makes this safe.
63
- proc_value: usize,
64
- /// The thread ID of the Ruby thread that created this operator.
65
- /// We can only safely call Ruby from this thread.
66
- ruby_thread_id: thread::ThreadId,
67
- }
68
-
69
- // SAFETY: We only access the proc_value when we hold the GVL via the Ruby thread,
70
- // which ensures thread-safe access to Ruby objects.
71
- unsafe impl Send for RubyProcMergeOperator {}
72
- unsafe impl Sync for RubyProcMergeOperator {}
73
-
74
- impl RubyProcMergeOperator {
75
- /// Create a new RubyProcMergeOperator from a Ruby proc/block.
76
- ///
77
- /// # Safety
78
- ///
79
- /// The caller must ensure the proc remains alive (not GC'd) for the
80
- /// lifetime of this operator.
81
- pub fn new(proc: Value) -> Self {
82
- Self {
83
- proc_value: proc.as_raw() as usize,
84
- ruby_thread_id: thread::current().id(),
85
- }
86
- }
87
-
88
- /// Check if we're on the Ruby thread that created this operator.
89
- fn is_ruby_thread(&self) -> bool {
90
- thread::current().id() == self.ruby_thread_id
91
- }
92
-
93
- /// Call the Ruby proc with the given arguments.
94
- /// This must only be called from the Ruby thread (after checking is_ruby_thread).
95
- fn call_proc_on_ruby_thread(
96
- &self,
97
- key: &str,
98
- existing_value: Option<&str>,
99
- new_value: &str,
100
- ) -> Result<Bytes, MergeOperatorError> {
101
- // We're on the Ruby thread, so we can use with_gvl
102
- // Import here to avoid the circular dependency at module level
103
- use crate::runtime::with_gvl;
104
-
105
- let key_owned = key.to_string();
106
- let existing_owned = existing_value.map(|s| s.to_string());
107
- let new_owned = new_value.to_string();
108
-
109
- with_gvl(|| {
110
- let ruby = Ruby::get().expect("Ruby runtime not available");
111
-
112
- // Reconstruct the proc Value from the raw pointer
113
- let proc = unsafe { Value::from_raw(self.proc_value as _) };
114
-
115
- // Build arguments: (key, existing_value, new_value)
116
- let existing_arg: Value = match &existing_owned {
117
- Some(s) => ruby.str_new(s).as_value(),
118
- None => ruby.qnil().as_value(),
119
- };
120
-
121
- // Call the proc
122
- let result: Result<String, magnus::Error> = proc.funcall(
123
- "call",
124
- (
125
- ruby.str_new(&key_owned),
126
- existing_arg,
127
- ruby.str_new(&new_owned),
128
- ),
129
- );
130
-
131
- match result {
132
- Ok(merged) => Ok(Bytes::from(merged)),
133
- Err(e) => {
134
- error!("Ruby merge operator error: {}", e);
135
- Err(MergeOperatorError::EmptyBatch)
136
- }
137
- }
138
- })
139
- }
140
-
141
- /// Fallback merge when we're not on the Ruby thread.
142
- /// Uses simple concatenation as a safe default.
143
- fn fallback_merge(
144
- &self,
145
- existing_value: Option<&Bytes>,
146
- new_value: &Bytes,
147
- ) -> Result<Bytes, MergeOperatorError> {
148
- error!(
149
- "Ruby merge operator called from non-Ruby thread, using fallback concatenation. \
150
- This can happen during background compaction."
151
- );
152
- let mut result = existing_value.map(|v| v.to_vec()).unwrap_or_default();
153
- result.extend_from_slice(new_value);
154
- Ok(Bytes::from(result))
155
- }
156
-
157
- /// Call the Ruby proc with the given arguments, handling thread safety.
158
- fn call_proc(
159
- &self,
160
- key: &Bytes,
161
- existing_value: Option<&Bytes>,
162
- new_value: &Bytes,
163
- ) -> Result<Bytes, MergeOperatorError> {
164
- let key_str = String::from_utf8_lossy(key);
165
- let existing_str = existing_value.map(|v| String::from_utf8_lossy(v));
166
- let new_str = String::from_utf8_lossy(new_value);
167
-
168
- if self.is_ruby_thread() {
169
- self.call_proc_on_ruby_thread(&key_str, existing_str.as_deref(), &new_str)
170
- } else {
171
- // We're on a worker thread, use fallback
172
- self.fallback_merge(existing_value, new_value)
173
- }
174
- }
175
- }
176
-
177
- impl MergeOperator for RubyProcMergeOperator {
178
- fn merge(
179
- &self,
180
- key: &Bytes,
181
- existing_value: Option<Bytes>,
182
- value: Bytes,
183
- ) -> Result<Bytes, MergeOperatorError> {
184
- self.call_proc(key, existing_value.as_ref(), &value)
185
- }
186
-
187
- fn merge_batch(
188
- &self,
189
- key: &Bytes,
190
- existing_value: Option<Bytes>,
191
- operands: &[Bytes],
192
- ) -> Result<Bytes, MergeOperatorError> {
193
- // Apply operands one at a time through the Ruby proc
194
- let mut current = existing_value;
195
- for operand in operands {
196
- current = Some(self.call_proc(key, current.as_ref(), operand)?);
197
- }
198
- Ok(current.unwrap_or_default())
199
- }
200
- }
201
-
202
38
  pub fn parse_merge_operator(
203
39
  kwargs: &RHash,
204
40
  ) -> Result<Option<Arc<dyn MergeOperator + Send + Sync>>, Error> {
@@ -211,7 +47,7 @@ pub fn parse_merge_operator(
211
47
  "string_concat" | "concat" => Arc::new(StringConcatMergeOperator),
212
48
  _ => {
213
49
  return Err(invalid_argument_error(&format!(
214
- "invalid merge_operator: {} (expected 'string_concat', 'concat', or use merge_operator_proc for a custom block)",
50
+ "invalid merge_operator: {} (expected 'string_concat' or 'concat')",
215
51
  merge_operator
216
52
  )))
217
53
  }
@@ -219,22 +55,3 @@ pub fn parse_merge_operator(
219
55
 
220
56
  Ok(Some(operator))
221
57
  }
222
-
223
- /// Parse a Ruby proc as a merge operator.
224
- pub fn parse_merge_operator_proc(
225
- kwargs: &RHash,
226
- ) -> Result<Option<Arc<dyn MergeOperator + Send + Sync>>, Error> {
227
- let proc_value = get_optional::<Value>(kwargs, "merge_operator_proc")?;
228
- let Some(proc) = proc_value else {
229
- return Ok(None);
230
- };
231
-
232
- // Verify it's callable
233
- if !proc.respond_to("call", false).unwrap_or(false) {
234
- return Err(invalid_argument_error(
235
- "merge_operator_proc must respond to 'call'",
236
- ));
237
- }
238
-
239
- Ok(Some(Arc::new(RubyProcMergeOperator::new(proc))))
240
- }
@@ -9,6 +9,7 @@ use slatedb::IterationOrder;
9
9
 
10
10
  use crate::errors::invalid_argument_error;
11
11
  use crate::iterator::Iterator;
12
+ use crate::merge_ops::parse_merge_operator;
12
13
  use crate::runtime::block_on_result;
13
14
  use crate::utils::{get_optional, prefix_subrange_from_kwargs, resolve_object_store};
14
15
 
@@ -57,6 +58,7 @@ impl Reader {
57
58
  let max_open_file_handles = get_optional::<usize>(&kwargs, "max_open_file_handles")?;
58
59
  let cache_root = get_optional::<String>(&kwargs, "cache_root")?;
59
60
  let follow_latest = get_optional::<bool>(&kwargs, "follow_latest")?.unwrap_or(false);
61
+ let merge_operator = parse_merge_operator(&kwargs)?;
60
62
 
61
63
  // Resolve the reader mode from checkpoint_id / follow_latest.
62
64
  let mode = match checkpoint_id {
@@ -96,14 +98,23 @@ impl Reader {
96
98
  if let Some(skip_replay) = skip_wal_replay {
97
99
  options.skip_wal_replay = skip_replay;
98
100
  }
99
- if let Some(ref root) = cache_root {
100
- options.object_store_cache_options.root_folder =
101
- Some(std::path::PathBuf::from(root));
101
+ #[allow(deprecated)]
102
+ {
103
+ if let Some(ref root) = cache_root {
104
+ options.object_store_cache_options.root_folder =
105
+ Some(std::path::PathBuf::from(root));
106
+ }
107
+ if let Some(max_handles) = max_open_file_handles {
108
+ options.object_store_cache_options.max_open_file_handles = max_handles;
109
+ }
102
110
  }
103
- if let Some(max_handles) = max_open_file_handles {
104
- options.object_store_cache_options.max_open_file_handles = max_handles;
111
+ let mut builder = DbReader::builder(path, object_store)
112
+ .with_reader_mode(mode)
113
+ .with_options(options);
114
+ if let Some(operator) = merge_operator {
115
+ builder = builder.with_merge_operator(operator);
105
116
  }
106
- DbReader::open(path, object_store, mode, options).await
117
+ builder.build().await
107
118
  })?;
108
119
 
109
120
  Ok(Self {
@@ -150,6 +161,10 @@ impl Reader {
150
161
  opts.cache_blocks = cb;
151
162
  }
152
163
 
164
+ if let Some(limit) = get_optional::<usize>(&kwargs, "max_concurrent_sources")? {
165
+ opts.max_concurrent_sources = limit;
166
+ }
167
+
153
168
  let result =
154
169
  block_on_result(async { self.inner.get_with_options(key.as_bytes(), &opts).await })?;
155
170
  Ok(result.map(|b| String::from_utf8_lossy(&b).to_string()))
@@ -1,6 +1,6 @@
1
1
  use magnus::Error;
2
2
  use once_cell::sync::OnceCell;
3
- use rb_sys::{rb_thread_call_with_gvl, rb_thread_call_without_gvl};
3
+ use rb_sys::rb_thread_call_without_gvl;
4
4
  use slatedb::Error as SlateError;
5
5
  use std::ffi::c_void;
6
6
  use std::future::Future;
@@ -98,54 +98,3 @@ where
98
98
 
99
99
  closure.result.expect("closure did not run")
100
100
  }
101
-
102
- /// Execute a closure while holding the Ruby GVL.
103
- ///
104
- /// This is the inverse of `without_gvl`. Use this when you need to call back
105
- /// into Ruby from code that has previously released the GVL (e.g., inside
106
- /// a future being executed by `block_on`).
107
- ///
108
- /// # Safety
109
- ///
110
- /// This function can ONLY be called from a Ruby thread that has previously
111
- /// released the GVL via `without_gvl` or `rb_thread_call_without_gvl`.
112
- /// Calling it from a non-Ruby thread (like a spawned Tokio task) will cause
113
- /// a Ruby fatal error.
114
- ///
115
- /// # Panics
116
- ///
117
- /// Panics if called from a non-Ruby thread.
118
- pub fn with_gvl<F, T>(f: F) -> T
119
- where
120
- F: FnOnce() -> T,
121
- {
122
- struct Closure<F, T> {
123
- f: Option<F>,
124
- result: Option<T>,
125
- }
126
-
127
- extern "C" fn call_closure<F, T>(data: *mut c_void) -> *mut c_void
128
- where
129
- F: FnOnce() -> T,
130
- {
131
- let closure = unsafe { &mut *(data as *mut Closure<F, T>) };
132
- if let Some(f) = closure.f.take() {
133
- closure.result = Some(f());
134
- }
135
- std::ptr::null_mut()
136
- }
137
-
138
- let mut closure = Closure {
139
- f: Some(f),
140
- result: None,
141
- };
142
-
143
- unsafe {
144
- rb_thread_call_with_gvl(
145
- Some(call_closure::<F, T>),
146
- &mut closure as *mut _ as *mut c_void,
147
- );
148
- }
149
-
150
- closure.result.expect("closure did not run")
151
- }
@@ -73,6 +73,10 @@ impl Snapshot {
73
73
  opts.cache_blocks = cb;
74
74
  }
75
75
 
76
+ if let Some(limit) = get_optional::<usize>(&kwargs, "max_concurrent_sources")? {
77
+ opts.max_concurrent_sources = limit;
78
+ }
79
+
76
80
  let guard = self.inner.borrow();
77
81
  let snapshot = guard
78
82
  .as_ref()
@@ -74,6 +74,10 @@ impl Transaction {
74
74
  opts.cache_blocks = cb;
75
75
  }
76
76
 
77
+ if let Some(limit) = get_optional::<usize>(&kwargs, "max_concurrent_sources")? {
78
+ opts.max_concurrent_sources = limit;
79
+ }
80
+
77
81
  let guard = self.inner.borrow();
78
82
  let txn = guard
79
83
  .as_ref()
@@ -9,15 +9,18 @@ module SlateDb
9
9
  #
10
10
  # @param path [String] The path identifier for the database
11
11
  # @param url [String, nil] Optional object store URL (e.g., "s3://bucket/path")
12
- # @param merge_operator [Symbol, String, Proc, nil] Optional merge operator.
13
- # Can be a symbol/string ("string_concat" or "concat") or a Proc/lambda
14
- # that takes (key, existing_value, new_value) and returns the merged value.
12
+ # @param merge_operator [Symbol, String, nil] Optional built-in merge operator
13
+ # (:string_concat or :concat). Ruby callbacks are not supported.
15
14
  # @param compactor_checkpoint_lifetime [Integer, nil] How long compaction inputs
16
15
  # are protected from garbage collection, in milliseconds (default: 15 minutes).
17
16
  # Reads using those inputs must finish within this time.
18
17
  # @param sorted_run_consolidation_threshold [Integer, nil] Sorted-run count above
19
18
  # which fallback consolidation starts. Zero disables it (default); use at least
20
19
  # two to enable it. One is disabled by SlateDB with a warning. This is not a hard limit.
20
+ # @param experimental_incremental_compaction [Boolean, nil] Publish sorted-run
21
+ # compaction output before completion (experimental, disabled by default).
22
+ # @param sst_block_alignment [Boolean, nil] Align SST data blocks to 4 KiB
23
+ # boundaries (disabled by default; incompatible with compression).
21
24
  # @yield [db] If a block is given, yields the database and ensures it's closed
22
25
  # @return [Database] The opened database (or block result if block given)
23
26
  #
@@ -34,32 +37,22 @@ module SlateDb
34
37
  # @example Open with S3 backend
35
38
  # db = SlateDb::Database.open("/tmp/mydb", url: "s3://mybucket/path")
36
39
  #
37
- # @example Open with a custom merge operator (Proc)
38
- # # Custom merge that adds numbers
39
- # db = SlateDb::Database.open("/tmp/mydb", merge_operator: ->(key, existing, new_val) {
40
- # existing_num = existing ? existing.to_i : 0
41
- # (existing_num + new_val.to_i).to_s
42
- # })
43
- # db.merge("counter", "5")
44
- # db.merge("counter", "3")
45
- # db.get("counter") # => "8"
46
- #
47
40
  def open(path, url: nil, merge_operator: nil,
48
- compactor_checkpoint_lifetime: nil, sorted_run_consolidation_threshold: nil)
41
+ compactor_checkpoint_lifetime: nil, sorted_run_consolidation_threshold: nil,
42
+ experimental_incremental_compaction: nil, sst_block_alignment: nil)
49
43
  opts = {
50
44
  compactor_checkpoint_lifetime: compactor_checkpoint_lifetime,
51
- sorted_run_consolidation_threshold: sorted_run_consolidation_threshold
45
+ sorted_run_consolidation_threshold: sorted_run_consolidation_threshold,
46
+ experimental_incremental_compaction: experimental_incremental_compaction,
47
+ sst_block_alignment: sst_block_alignment
52
48
  }.compact
53
49
 
54
- case merge_operator
55
- when Symbol, String
56
- opts[:merge_operator] = merge_operator.to_s
57
- when Proc
58
- # Store the proc to prevent GC and pass to Rust
59
- @_merge_operator_proc = merge_operator
60
- opts[:merge_operator_proc] = merge_operator
50
+ unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
51
+ raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
61
52
  end
62
53
 
54
+ opts[:merge_operator] = merge_operator.to_s if merge_operator
55
+
63
56
  db = _open(path, url, opts)
64
57
 
65
58
  if block_given?
@@ -84,6 +77,8 @@ module SlateDb
84
77
  # @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
85
78
  # @param dirty [Boolean, nil] Whether to include uncommitted data
86
79
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
80
+ # @param max_concurrent_sources [Integer, nil] Concurrent sources to fetch after
81
+ # the first source misses (default: 1; zero also means serial fetching).
87
82
  # @return [String, nil] The value, or nil if not found
88
83
  #
89
84
  # @example Basic get
@@ -92,11 +87,12 @@ module SlateDb
92
87
  # @example Get with options
93
88
  # value = db.get("mykey", durability_filter: "memory", dirty: true)
94
89
  #
95
- def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
90
+ def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
96
91
  opts = {}
97
92
  opts[:durability_filter] = durability_filter.to_s if durability_filter
98
93
  opts[:dirty] = dirty unless dirty.nil?
99
94
  opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
95
+ opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
100
96
 
101
97
  if opts.empty?
102
98
  _get(key)
@@ -112,17 +108,20 @@ module SlateDb
112
108
  # @param dirty [Boolean, nil] Whether to include uncommitted data
113
109
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
114
110
  # @return [Hash, nil] A hash with :key, :value, :seq, :create_ts, and :expire_ts, or nil if not found
111
+ # @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
112
+ # miss. See {#get}.
115
113
  #
116
114
  # @example Inspect metadata
117
115
  # entry = db.get_key_value("mykey")
118
116
  # entry[:value] # => "myvalue"
119
117
  # entry[:seq] # => sequence number
120
118
  #
121
- def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
119
+ def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
122
120
  opts = {}
123
121
  opts[:durability_filter] = durability_filter.to_s if durability_filter
124
122
  opts[:dirty] = dirty unless dirty.nil?
125
123
  opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
124
+ opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
126
125
 
127
126
  if opts.empty?
128
127
  _get_key_value(key)
@@ -494,6 +493,8 @@ module SlateDb
494
493
  #
495
494
  # @param lifetime [Integer, nil] Checkpoint lifetime in milliseconds
496
495
  # @param name [String, nil] Optional name for the checkpoint
496
+ # @param scope [Symbol, String] :durable (default) captures durable writes;
497
+ # :all also flushes pending writes before storing the checkpoint.
497
498
  # @return [Hash] Hash with :id (UUID string) and :manifest_id (integer)
498
499
  #
499
500
  # @example Create a named checkpoint
@@ -503,13 +504,26 @@ module SlateDb
503
504
  # @example Create a checkpoint with lifetime
504
505
  # checkpoint = db.create_checkpoint(lifetime: 3600_000) # 1 hour
505
506
  #
506
- def create_checkpoint(lifetime: nil, name: nil)
507
- opts = {}
507
+ def create_checkpoint(lifetime: nil, name: nil, scope: :durable)
508
+ opts = { scope: scope.to_s }
508
509
  opts[:lifetime] = lifetime if lifetime
509
510
  opts[:name] = name if name
510
511
  _create_checkpoint(opts)
511
512
  end
512
513
 
514
+ # Start a checkpoint without waiting for it to be stored.
515
+ #
516
+ # With scope: :all, captures writes completed before this call and excludes
517
+ # writes started after it returns. Writes during the call may be included.
518
+ # Options match {#create_checkpoint}. Dropping the handle does not cancel it.
519
+ # @return [CheckpointHandle] #id is available immediately; #wait returns the
520
+ # :id/:manifest_id hash once stored and consumes the handle. A second wait
521
+ # raises ClosedError. #id remains available after waiting.
522
+ def begin_checkpoint(lifetime: nil, name: nil, scope: :durable)
523
+ opts = { scope: scope.to_s, lifetime: lifetime, name: name }.compact
524
+ _begin_checkpoint(opts)
525
+ end
526
+
513
527
  # Get database metrics registry.
514
528
  #
515
529
  # @return [Metrics] Metrics registry
@@ -21,6 +21,7 @@ module SlateDb
21
21
  # @param cache_root [String, nil] Root folder for the reader's local on-disk
22
22
  # object-store cache. Setting this enables the cached object store; when it is
23
23
  # not set the cache (and `max_open_file_handles`) has no effect.
24
+ # This legacy cache is deprecated upstream in SlateDB 0.18.1.
24
25
  # @param max_open_file_handles [Integer, nil] Maximum number of file handles to keep
25
26
  # open in the reader's file-handle cache. When the limit is reached, the least
26
27
  # recently used handle is closed (default: 1000). Only takes effect when
@@ -54,29 +55,31 @@ module SlateDb
54
55
  manifest_poll_interval: nil, checkpoint_lifetime: nil,
55
56
  max_memtable_bytes: nil, cache_root: nil, max_open_file_handles: nil,
56
57
  merge_operator: nil)
57
- opts = {}
58
- opts[:follow_latest] = follow_latest unless follow_latest.nil?
59
- opts[:manifest_poll_interval] = manifest_poll_interval if manifest_poll_interval
60
- opts[:checkpoint_lifetime] = checkpoint_lifetime if checkpoint_lifetime
61
- opts[:max_memtable_bytes] = max_memtable_bytes if max_memtable_bytes
62
- opts[:cache_root] = cache_root if cache_root
63
- opts[:max_open_file_handles] = max_open_file_handles if max_open_file_handles
64
- opts[:merge_operator] = merge_operator.to_s if merge_operator
58
+ unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
59
+ raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
60
+ end
61
+
62
+ opts = {
63
+ follow_latest: follow_latest,
64
+ manifest_poll_interval: manifest_poll_interval,
65
+ checkpoint_lifetime: checkpoint_lifetime,
66
+ max_memtable_bytes: max_memtable_bytes,
67
+ cache_root: cache_root,
68
+ max_open_file_handles: max_open_file_handles,
69
+ merge_operator: merge_operator&.to_s
70
+ }.compact
65
71
 
66
72
  reader = _open(path, url, checkpoint_id, opts)
73
+ return reader unless block_given?
67
74
 
68
- if block_given?
75
+ begin
76
+ yield reader
77
+ ensure
69
78
  begin
70
- yield reader
71
- ensure
72
- begin
73
- reader.close
74
- rescue StandardError
75
- nil
76
- end
79
+ reader.close
80
+ rescue StandardError
81
+ nil
77
82
  end
78
- else
79
- reader
80
83
  end
81
84
  end
82
85
  end
@@ -87,13 +90,16 @@ module SlateDb
87
90
  # @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
88
91
  # @param dirty [Boolean, nil] Whether to include uncommitted data
89
92
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
93
+ # @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
94
+ # miss (default: 1; zero also means serial fetching).
90
95
  # @return [String, nil] The value, or nil if not found
91
96
  #
92
- def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
97
+ def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
93
98
  opts = {}
94
99
  opts[:durability_filter] = durability_filter.to_s if durability_filter
95
100
  opts[:dirty] = dirty unless dirty.nil?
96
101
  opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
102
+ opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
97
103
 
98
104
  if opts.empty?
99
105
  _get(key)
@@ -8,13 +8,16 @@ module SlateDb
8
8
  # @param durability_filter [String, nil] Filter by durability level
9
9
  # @param dirty [Boolean, nil] Whether to include uncommitted data
10
10
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
11
+ # @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
12
+ # miss (default: 1; zero also means serial fetching).
11
13
  # @return [String, nil] The value, or nil if not found
12
14
  #
13
- def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
15
+ def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
14
16
  opts = {}
15
17
  opts[:durability_filter] = durability_filter.to_s if durability_filter
16
18
  opts[:dirty] = dirty unless dirty.nil?
17
19
  opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
20
+ opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
18
21
 
19
22
  if opts.empty?
20
23
  _get(key)
@@ -27,16 +30,19 @@ module SlateDb
27
30
  #
28
31
  # @param start_key [String] The start key (inclusive)
29
32
  # @param end_key [String, nil] The end key (exclusive)
33
+ # @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
30
34
  # @return [Iterator] An iterator over key-value pairs
31
35
  #
32
36
  def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
33
- read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
34
- opts = {}
35
- opts[:durability_filter] = durability_filter.to_s if durability_filter
36
- opts[:dirty] = dirty unless dirty.nil?
37
- opts[:read_ahead_bytes] = read_ahead_bytes if read_ahead_bytes
38
- opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
39
- opts[:max_fetch_tasks] = max_fetch_tasks if max_fetch_tasks
37
+ read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
38
+ opts = {
39
+ durability_filter: durability_filter&.to_s,
40
+ dirty: dirty,
41
+ read_ahead_bytes: read_ahead_bytes,
42
+ cache_blocks: cache_blocks,
43
+ max_fetch_tasks: max_fetch_tasks,
44
+ order: order&.to_s
45
+ }.compact
40
46
 
41
47
  iter = if opts.empty?
42
48
  _scan(start_key, end_key)
@@ -59,6 +65,7 @@ module SlateDb
59
65
  # @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
60
66
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
61
67
  # @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
68
+ # @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
62
69
  # @param from [String, nil] Inclusive lower bound suffix, appended to the
63
70
  # prefix, to start scanning from. Defaults to the start of the prefix.
64
71
  # @param to [String, nil] Exclusive upper bound suffix, appended to the
@@ -66,7 +73,7 @@ module SlateDb
66
73
  # @return [Iterator] An iterator over key-value pairs
67
74
  #
68
75
  def scan_prefix(prefix, durability_filter: nil, dirty: nil,
69
- read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
76
+ read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
70
77
  from: nil, to: nil, &)
71
78
  opts = {
72
79
  durability_filter: durability_filter&.to_s,
@@ -74,6 +81,7 @@ module SlateDb
74
81
  read_ahead_bytes: read_ahead_bytes,
75
82
  cache_blocks: cache_blocks,
76
83
  max_fetch_tasks: max_fetch_tasks,
84
+ order: order&.to_s,
77
85
  subrange_from: from,
78
86
  subrange_to: to
79
87
  }.compact
@@ -8,13 +8,16 @@ module SlateDb
8
8
  # @param durability_filter [String, nil] Filter by durability level
9
9
  # @param dirty [Boolean, nil] Whether to include uncommitted data
10
10
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
11
+ # @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
12
+ # miss (default: 1; zero also means serial fetching).
11
13
  # @return [String, nil] The value, or nil if not found
12
14
  #
13
- def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
15
+ def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
14
16
  opts = {}
15
17
  opts[:durability_filter] = durability_filter.to_s if durability_filter
16
18
  opts[:dirty] = dirty unless dirty.nil?
17
19
  opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
20
+ opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
18
21
 
19
22
  if opts.empty?
20
23
  _get(key)
@@ -66,16 +69,19 @@ module SlateDb
66
69
  #
67
70
  # @param start_key [String] The start key (inclusive)
68
71
  # @param end_key [String, nil] The end key (exclusive)
72
+ # @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
69
73
  # @return [Iterator] An iterator over key-value pairs
70
74
  #
71
75
  def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
72
- read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
73
- opts = {}
74
- opts[:durability_filter] = durability_filter.to_s if durability_filter
75
- opts[:dirty] = dirty unless dirty.nil?
76
- opts[:read_ahead_bytes] = read_ahead_bytes if read_ahead_bytes
77
- opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
78
- opts[:max_fetch_tasks] = max_fetch_tasks if max_fetch_tasks
76
+ read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
77
+ opts = {
78
+ durability_filter: durability_filter&.to_s,
79
+ dirty: dirty,
80
+ read_ahead_bytes: read_ahead_bytes,
81
+ cache_blocks: cache_blocks,
82
+ max_fetch_tasks: max_fetch_tasks,
83
+ order: order&.to_s
84
+ }.compact
79
85
 
80
86
  iter = if opts.empty?
81
87
  _scan(start_key, end_key)
@@ -98,6 +104,7 @@ module SlateDb
98
104
  # @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
99
105
  # @param cache_blocks [Boolean, nil] Whether to cache blocks
100
106
  # @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
107
+ # @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
101
108
  # @param from [String, nil] Inclusive lower bound suffix, appended to the
102
109
  # prefix, to start scanning from. Defaults to the start of the prefix.
103
110
  # @param to [String, nil] Exclusive upper bound suffix, appended to the
@@ -105,7 +112,7 @@ module SlateDb
105
112
  # @return [Iterator] An iterator over key-value pairs
106
113
  #
107
114
  def scan_prefix(prefix, durability_filter: nil, dirty: nil,
108
- read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
115
+ read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
109
116
  from: nil, to: nil, &)
110
117
  opts = {
111
118
  durability_filter: durability_filter&.to_s,
@@ -113,6 +120,7 @@ module SlateDb
113
120
  read_ahead_bytes: read_ahead_bytes,
114
121
  cache_blocks: cache_blocks,
115
122
  max_fetch_tasks: max_fetch_tasks,
123
+ order: order&.to_s,
116
124
  subrange_from: from,
117
125
  subrange_to: to
118
126
  }.compact
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SlateDb
4
- VERSION = "0.6.1"
4
+ VERSION = "0.7.0"
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: slatedb
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.1
4
+ version: 0.7.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - SlateDB Contributors
@@ -93,6 +93,7 @@ files:
93
93
  - ext/slatedb/Cargo.toml
94
94
  - ext/slatedb/extconf.rb
95
95
  - ext/slatedb/src/admin.rs
96
+ - ext/slatedb/src/checkpoint.rs
96
97
  - ext/slatedb/src/database.rs
97
98
  - ext/slatedb/src/errors.rs
98
99
  - ext/slatedb/src/iterator.rs