beagle-turso 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e153a3302d14dc6f72dcd76b510a0dadb1e16f4f50235258fd81a6ef67be4e91
-  data.tar.gz: b52eeb81d1fb68ce40efbe8691202f8c70e9cdceb6491fdce31ad58e4a6714d6
+  metadata.gz: f49be5566a3b64416bccffc8c792e167e770139827016890070836eac6ab2484
+  data.tar.gz: c8be2046fb4e78a337008dcdd488f5b8587d754c77692d73e15f90dc0e3d2f8b
 SHA512:
-  metadata.gz: bef6dc8666bdf7b163eb8b2c710b61b4db7cb33ee2b914b8c60904b2c13a333fb0a656f780ad6e909b186e273b1dc113558b5a0161b55c5cf8473760bcddc3b0
-  data.tar.gz: fd867e70cc1657e4c5922cf635b80eb4bade93805f41fcd343353975a582812bb18b7b405ecc3ec538fada798d665d76b2409e01ec09d83b4e588869631cd7ce
+  metadata.gz: 591b0e8a2e9e71d767008bac9a2742a2c93e4bb74424f586d2af035339aa1510c6fb883e770b20440a9ad561b5c88e9b113f63f8b4b506355e18f4e39442ee23
+  data.tar.gz: 53782d7a3a82cffb627bdbc107ab022619cae5fe49a7fdc85d233aa04910497da7932de863b384d14199d77f63d35473021bf5adc8d95d5d97c13b00355d3145

data/README.md

@@ -152,6 +152,72 @@ still a `RuntimeError`. A replica file already held by another process raises
   `closed?` is safe (it reports the handle as closed if it can't be inspected),
   so inherited handles need no cleanup in the child.
 
+## Background sync
+
+`Beagle::Turso::Syncer` keeps the remote copy current without a job system:
+
+```ruby
+db = Beagle::Turso::Database.open(local_path: "app.db", remote_url: url, auth_token: token)
+
+syncer = Beagle::Turso::Syncer.new(
+  db,
+  interval: 30,        # seconds between cycles
+  pull: true,          # pull after each push (false = push only)
+  max_backoff: 300,    # retry delay cap, seconds
+  on_error: ->(error, status) { ErrorTracker.notify(error) },
+  logger: Logger.new($stderr),
+).start
+
+syncer.status   # => #<data Status running=true, syncing=false, last_push_at=..., ...>
+syncer.sync_now # one cycle now, on this thread; :ok, or :busy if one is running
+                # (a failure is recorded, passed to on_error, then re-raised)
+
+# on shutdown, before closing the database:
+syncer.stop(timeout: 10)   # waits for an in-flight sync, then pushes once more
+# syncer.stop(timeout: 10, final_push: false)  # waits, but doesn't push again
+db.close
+```
+
+- **Opt-in.** Requiring the gem starts nothing. `start` runs one background
+  thread. `push` and `pull` release the GVL, so that thread doesn't hold up
+  requests on other threads or a Falcon reactor.
+- **Failures don't stop it.** Each failure is logged (at `warn`) and passed to
+  `on_error`. The next attempt waits `interval × 2ⁿ`, capped at
+  `max_backoff`, with ±20% jitter. A success resets the count. A failed push
+  skips that cycle's pull. A logger or `on_error` that raises a
+  `StandardError` is ignored (an `on_error` error is logged).
+- **`sync_now` re-raises.** A failed `sync_now` records the failure in
+  `status`, logs it, calls `on_error`, then raises the error to you. Since
+  `on_error` has already seen it, don't report it again from a `rescue`. If
+  the error message carried a URL or credential, both `on_error` and the
+  caller get a copy of the error (same class and backtrace) with the redacted
+  message.
+- **Status is for monitoring, not lag.** `last_push_at` says when a push last
+  succeeded, not how much is still unpushed. Alert when `consecutive_failures`
+  grows, or when `last_push_at` gets older than you can tolerate. The site keeps
+  serving from the local replica either way. `destination` is the remote host
+  only, and error messages are stripped of URLs and tokens.
+- **Shutdown is best-effort.** `stop(timeout:, final_push: true)` stops the
+  loop, waits for a cycle in flight on any thread (the loop or a concurrent
+  `sync_now`), then pushes once more unless `final_push: false`. It waits up
+  to `timeout` seconds in total. A push or pull already in flight can't be
+  interrupted, so if the deadline passes, `stop` returns with
+  `status.syncing == true` and the work finishes in the background; process
+  exit then waits for that native call to return (until your platform's kill
+  timeout). `stop(timeout: 0)` skips the final push. Called from inside
+  `on_error`, `stop` can't wait for the cycle it is part of, so it returns
+  without a final push. If the network is down, the deadline passes, or the
+  process is killed, writes stay local until the next process pushes them.
+  `stop` never closes the database; do that after it returns.
+- **Don't call `stop` from a signal handler.** Inside `Signal.trap` it raises
+  `ThreadError` (it takes locks). Call it from your server's shutdown hook or
+  `at_exit`.
+- **One syncer per process, created after fork.** A stopped syncer can't be
+  restarted, so create a new one. With Puma in single mode, create it in an
+  initializer or `on_booted`; in cluster mode, create it in `on_worker_boot`
+  (each worker gets its own) and call `stop` from `on_worker_shutdown` or
+  `at_exit`.
+
 ## Credential safety
 
 `auth_token` is never included in `inspect` output or in error messages —

data/lib/beagle/turso/redaction.rb

@@ -0,0 +1,41 @@
+require "uri"
+
+module Beagle
+  module Turso
+    # Strips credentials from text before it reaches a log, a status or a
+    # callback. Engine error messages can embed the remote URL; the auth token
+    # itself is sent as a header, but a URL can still carry userinfo or a
+    # token query parameter, so reduce every URL to scheme://host.
+    module Redaction
+      # scheme://[userinfo@]host[...]: keep scheme and host (a bracketed IPv6
+      # literal included), drop userinfo, port, path, query and fragment. The
+      # scheme length is bounded so a long run of "a.a.a." stays linear.
+      URL = %r{\b([a-z][a-z0-9+.-]{0,31})://(?:[^/\s?#"'@\[]+@)?(\[[0-9a-f:.]*\]|[^/\s?#"':\[]+)[^\s"']*}i
+      # Authorization schemes: keep the scheme word, blank the credential.
+      AUTH_SCHEME = /\b(Bearer|Basic)\s+[^\s"',;]+/i
+      # key=value / key: value / "key": "value" / key%3Dvalue, where the key is
+      # a whole word ending in a secret-ish name (turso_auth_token, x-api-key,
+      # access_token, ...). The lookbehind pins the key to a word start, which
+      # keeps the unbounded prefix linear.
+      SECRET_PARAM = /(?<![\w-])([\w-]*?(?:auth_?token|token|jwt|api[_-]?key|password|passwd|secret|authorization))(["']?\s*(?:[=:]|%3D)\s*["']?)(?!(?:Bearer|Basic)\s)[^\s"'&,;]+/i
+      # A bare JWT (header.payload.signature, base64url).
+      JWT = /\beyJ[\w-]+\.[\w-]+\.[\w-]+/
+
+      module_function
+
+      def message(text)
+        text.to_s
+          .gsub(URL) { "#{Regexp.last_match(1)}://#{Regexp.last_match(2)}" }
+          .gsub(JWT, "[redacted]")
+          .gsub(AUTH_SCHEME, '\1 [redacted]')
+          .gsub(SECRET_PARAM) { "#{Regexp.last_match(1)}#{Regexp.last_match(2)}[redacted]" }
+      end
+
+      def host(url)
+        URI.parse(url.to_s).host
+      rescue URI::InvalidURIError
+        nil
+      end
+    end
+  end
+end

data/lib/beagle/turso/syncer.rb

@@ -0,0 +1,293 @@
+module Beagle
+  module Turso
+    # Keeps a synced Database's remote copy current: runs push (then pull) on
+    # a background thread, backs off on failure, reports status, and pushes
+    # one last time on stop. Framework-neutral and opt-in: nothing runs until
+    # #start, and it never closes the database (its owner does, after #stop).
+    #
+    # push/pull release the GVL (see the README), so the syncer's thread does
+    # not stall other Ruby threads or a fiber scheduler's reactor.
+    class Syncer
+      Status = Data.define(
+        :running, :syncing,
+        :last_push_at, :last_pull_at,
+        :last_error_class, :last_error_message, :last_error_at,
+        :consecutive_failures, :destination
+      )
+
+      # Retry delays are spread by ±JITTER so restarted processes don't retry
+      # in lockstep.
+      JITTER = 0.2
+
+      # Backoff stops growing past interval * 2**MAX_EXPONENT (max_backoff
+      # caps it long before), so a long outage never builds a bignum.
+      MAX_EXPONENT = 32
+
+      # How often a final push polls for an in-flight cycle to finish.
+      LOCK_POLL = 0.005
+
+      def initialize(database, interval: 30, pull: true, max_backoff: 300, on_error: nil, logger: nil, random: Random.new)
+        unless database.synced?
+          raise NotSyncedError, "Syncer needs a synced database (opened with remote_url: and auth_token:)"
+        end
+        raise ArgumentError, "interval must be positive" unless interval.positive?
+        raise ArgumentError, "max_backoff must be at least interval" if max_backoff < interval
+
+        @database = database
+        @interval = interval
+        @pull = pull
+        @max_backoff = max_backoff
+        @on_error = on_error
+        @logger = logger
+        @random = random
+
+        @cycle_lock = Mutex.new # held for the whole of one push/pull cycle
+        @state_lock = Mutex.new # guards the fields below and the wake-up signal
+        @wake = ConditionVariable.new
+        @thread = nil
+        @stopping = false
+        @stopped = false
+        @syncing = false
+        @last_push_at = nil
+        @last_pull_at = nil
+        @last_error_class = nil
+        @last_error_message = nil
+        @last_error_at = nil
+        @consecutive_failures = 0
+      end
+
+      # Runs one cycle on the calling thread. Returns :ok, or :busy (without
+      # waiting) when another cycle is in progress. A failure is recorded in
+      # #status, reported to on_error/logger, then re-raised. If the message
+      # carried credentials, both on_error and the caller get a copy of the
+      # error (same class and backtrace) with the redacted message instead.
+      def sync_now
+        cycle(pull: @pull)
+      end
+
+      def status
+        @state_lock.synchronize do
+          Status.new(
+            running: loop_running?,
+            syncing: @syncing,
+            last_push_at: @last_push_at,
+            last_pull_at: @last_pull_at,
+            last_error_class: @last_error_class,
+            last_error_message: @last_error_message,
+            last_error_at: @last_error_at,
+            consecutive_failures: @consecutive_failures,
+            destination: @database.remote_host
+          )
+        end
+      end
+
+      # Starts the background thread. Idempotent while running. A stopped
+      # syncer cannot be restarted; create a new one.
+      def start
+        @state_lock.synchronize do
+          raise Error, "syncer has been stopped; create a new one" if @stopped
+          return self if @thread
+
+          @thread = Thread.new { run_loop }
+          @thread.name = "beagle-turso-syncer"
+        end
+        self
+      end
+
+      # True while the background thread is alive and not stopping. A loop
+      # thread that died (e.g. on_error raised a non-StandardError) is not
+      # running.
+      def running?
+        @state_lock.synchronize { loop_running? }
+      end
+
+      # Stops the loop, waits for a cycle in flight on any other thread
+      # (the loop or a concurrent sync_now), then, if final_push and the
+      # deadline allows, pushes once more. Waits at most `timeout` seconds in
+      # total: an in-flight push or pull cannot be interrupted, so if the
+      # deadline passes first this returns anyway with status.syncing true and
+      # the work finishes in the background. Called from inside a cycle (from
+      # on_error), it cannot wait for that cycle and skips the final push.
+      # Never closes the database. Idempotent.
+      def stop(timeout: 10, final_push: true)
+        unless timeout.is_a?(Numeric) && timeout >= 0
+          raise ArgumentError, "timeout must be a non-negative number of seconds"
+        end
+
+        first = @state_lock.synchronize do
+          next false if @stopped
+
+          @stopped = true
+          @stopping = true
+          @wake.broadcast
+          true
+        end
+        return status unless first
+
+        deadline = monotonic_now + timeout
+        # Joining our own thread (stop called from on_error) would raise; that
+        # thread exits by itself once the callback returns.
+        join_quietly(@thread, deadline) if @thread && @thread != Thread.current
+        # Inside a cycle (stop from on_error) this thread holds the cycle lock,
+        # so waiting for it would only burn the deadline.
+        unless @cycle_lock.owned? || loop_alive?
+          if final_push && remaining(deadline).positive?
+            pusher = Thread.new do
+              cycle(pull: false, wait_until: deadline)
+            rescue StandardError
+              nil # already recorded, logged and reported by cycle
+            end
+            join_quietly(pusher, deadline)
+          elsif lock_by(deadline)
+            @cycle_lock.unlock # only waiting for the in-flight cycle
+          end
+        end
+        status
+      end
+
+      private
+
+      def run_loop
+        loop do
+          return if @state_lock.synchronize { @stopping }
+
+          delay =
+            begin
+              cycle(pull: @pull)
+              @interval
+            rescue StandardError
+              retry_delay(@state_lock.synchronize { @consecutive_failures })
+            end
+
+          @state_lock.synchronize do
+            return if @stopping
+
+            @wake.wait(@state_lock, delay)
+          end
+        end
+      end
+
+      def loop_alive?
+        @thread&.alive? && @thread != Thread.current
+      end
+
+      # Caller holds @state_lock.
+      def loop_running?
+        !!@thread&.alive? && !@stopping
+      end
+
+      # Joins a helper thread until the deadline. A thread that died from an
+      # exception re-raises it here; stop must still finish, so log it.
+      def join_quietly(thread, deadline)
+        thread.join(remaining(deadline))
+      rescue Exception => e # rubocop:disable Lint/RescueException
+        raise if thread.alive? # not the thread's death: e.g. an interrupt aimed at us
+
+        safe_warn("beagle-turso syncer thread died (#{e.class}): #{Redaction.message(e.message)}")
+      end
+
+      # Takes the cycle lock, waiting for an in-flight cycle until the
+      # deadline. False if the deadline passed first.
+      def lock_by(deadline)
+        until @cycle_lock.try_lock
+          left = remaining(deadline)
+          return false unless left.positive?
+
+          sleep([LOCK_POLL, left].min)
+        end
+        true
+      end
+
+      def remaining(deadline)
+        [deadline - monotonic_now, 0].max
+      end
+
+      def monotonic_now
+        Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      end
+
+      # One push (then pull) under the cycle lock. Without wait_until it never
+      # waits (returns :busy); with it, it waits for the lock until then.
+      def cycle(pull:, wait_until: nil)
+        locked = wait_until ? lock_by(wait_until) : @cycle_lock.try_lock
+        return :busy unless locked
+
+        begin
+          set_syncing(true)
+          @database.push
+          @state_lock.synchronize { @last_push_at = Time.now }
+          if pull
+            @database.pull
+            @state_lock.synchronize { @last_pull_at = Time.now }
+          end
+          @state_lock.synchronize { @consecutive_failures = 0 }
+          :ok
+        rescue StandardError => e
+          reported = record_failure(e)
+          raise e if reported.equal?(e)
+
+          raise reported, cause: nil # the cause would carry the unredacted original
+        ensure
+          begin
+            set_syncing(false)
+          ensure
+            @cycle_lock.unlock
+          end
+        end
+      end
+
+      def set_syncing(value)
+        @state_lock.synchronize { @syncing = value }
+      end
+
+      # Records, logs and reports a failure. Returns the error to hand on:
+      # the original, or a redacted copy if its message carried credentials.
+      def record_failure(error)
+        message = Redaction.message(error.message).freeze
+        @state_lock.synchronize do
+          @consecutive_failures += 1
+          @last_error_class = error.class.to_s
+          @last_error_message = message
+          @last_error_at = Time.now
+        end
+        safe_warn("beagle-turso sync failed (#{error.class}): #{message}")
+        reported = redacted_error(error, message)
+        notify(reported)
+        reported
+      end
+
+      def redacted_error(error, message)
+        return error if message == error.message
+
+        copy = begin
+          error.exception(message)
+        rescue StandardError
+          nil
+        end
+        copy = Error.new(message) unless copy.is_a?(Exception) && copy.message == message
+        copy.set_backtrace(error.backtrace) if copy.backtrace.nil? && error.backtrace
+        copy
+      end
+
+      def notify(error)
+        @on_error&.call(error, status)
+      rescue StandardError => e
+        safe_warn("beagle-turso syncer on_error raised (#{e.class}): #{Redaction.message(e.message)}")
+      end
+
+      # A broken logger must not skip on_error or replace the sync error.
+      def safe_warn(message)
+        @logger&.warn(message)
+      rescue StandardError
+        nil
+      end
+
+      # Delay before the next attempt after `failures` consecutive failures:
+      # interval * 2**failures, capped at max_backoff, then spread by ±JITTER.
+      def retry_delay(failures)
+        base = [@interval * (2**[failures, MAX_EXPONENT].min), @max_backoff].min
+        base * (1 + @random.rand(-JITTER..JITTER))
+      end
+    end
+  end
+end

data/lib/beagle/turso/version.rb

@@ -1,5 +1,5 @@
 module Beagle
   module Turso
-    VERSION = "0.2.0"
+    VERSION = "0.3.0"
   end
 end

data/lib/beagle/turso.rb

@@ -1,5 +1,7 @@
 require_relative "turso/version"
 require_relative "turso/errors"
+require_relative "turso/redaction"
+require_relative "turso/syncer"
 
 # Load the native extension. Precompiled ("fat") gems ship the .so under a
 # per-Ruby-ABI subdir (lib/beagle_turso/<major.minor>/beagle_turso.so); a
@@ -17,12 +19,28 @@ module Beagle
       # Opens a local-only database, or one synced with a remote Turso
       # database when both `remote_url` and `auth_token` are given (see
       # `#push`/`#pull`). Delegates to the native `_open` primitive, which
-      # takes its arguments positionally.
+      # takes its arguments positionally. Only the remote host is kept on
+      # the Ruby object; the token is never stored.
       def self.open(local_path:, remote_url: nil, auth_token: nil, bootstrap_if_empty: true)
-        _open(local_path, remote_url, auth_token, bootstrap_if_empty)
+        db = _open(local_path, remote_url, auth_token, bootstrap_if_empty)
+        if remote_url && auth_token
+          db.instance_variable_set(:@synced, true)
+          db.instance_variable_set(:@remote_host, Redaction.host(remote_url))
+        end
+        db
       end
 
       private_class_method :_open
+
+      # True when opened with both remote_url: and auth_token:, i.e. when
+      # #push/#pull talk to a remote.
+      def synced?
+        @synced == true
+      end
+
+      # The remote database's host, without scheme, path, query or
+      # credentials; nil for a local-only database.
+      attr_reader :remote_host
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: beagle-turso
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - BeagleSoftwareUK
@@ -45,6 +45,8 @@ files:
 - ext/beagle_turso/src/nogvl.rs
 - lib/beagle/turso.rb
 - lib/beagle/turso/errors.rb
+- lib/beagle/turso/redaction.rb
+- lib/beagle/turso/syncer.rb
 - lib/beagle/turso/version.rb
 homepage: https://github.com/BeagleSoftwareUK/beagle-turso
 licenses: