esse-sequel 0.0.1.beta1

data/docs/usage.md

@@ -0,0 +1,211 @@
+# Usage Guide
+
+## Installation
+
+```ruby
+# Gemfile
+gem 'esse'
+gem 'esse-sequel'
+gem 'sequel'
+```
+
+Enable the plugin on any index:
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+end
+```
+
+## The `collection` DSL
+
+Pass a Sequel model (or dataset) to `collection`:
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User
+    document { |u, **| { _id: u.id, name: u.name } }
+  end
+end
+```
+
+Options:
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `batch_size` | `1000` | Records per batch |
+| `connect_with` | — | `{ server: :read_only }` (a server from the Sequel `:servers` option) |
+
+```ruby
+repository :user do
+  collection ::User, batch_size: 500, connect_with: { server: :read_only }
+  document { |u, **| { _id: u.id, name: u.name } }
+end
+```
+
+The same can be declared inside the collection block with `connected_to server: :read_only`. The collection dataset runs on that server, and when the `server_block` extension is loaded (`DB.extension :server_block`) the whole iteration, `batch_context` queries included, runs inside `DB.with_server`.
+
+## Scopes
+
+```ruby
+repository :user do
+  collection ::User do
+    scope :active, -> { where(active: true) }
+    scope :role,   ->(role) { where(role: role) }
+    scope :created_after, ->(date) { where(Sequel.lit('created_at > ?', date)) }
+  end
+
+  document { |u, **| { _id: u.id, name: u.name, role: u.role } }
+end
+
+UsersIndex.import(context: { active: true, role: 'admin' })
+```
+
+## Batch context
+
+```ruby
+repository :order do
+  collection ::Order do
+    batch_context :customers do |orders, **|
+      Customer.where(id: orders.map(&:customer_id)).all.index_by(&:id)
+    end
+  end
+
+  document do |order, customers: {}, **|
+    { _id: order.id, customer_name: customers[order.customer_id]&.name }
+  end
+end
+```
+
+## Eager loading
+
+```ruby
+repository :order do
+  collection ::Order.eager(:customer, :line_items)
+  document do |order, **|
+    { _id: order.id, customer: order.customer.name }
+  end
+end
+```
+
+## Automatic callbacks
+
+```ruby
+class User < Sequel::Model
+  include Esse::Sequel::Model
+  index_callback 'users_index:user'
+end
+```
+
+Options:
+
+```ruby
+index_callback 'users_index:user',
+  on:     %i[create update],
+  with:   :update,
+  if:     :active?,
+  unless: :deleted?
+```
+
+### Reference format
+
+Same as esse-active_record:
+
+- `'users'`
+- `'users_index'`
+- `'users_index:user'`
+- `'UsersIndex'`
+- `'UsersIndex::User'`
+- `'foo/v1/users_index:user'` (namespaced)
+
+### Routing changes
+
+When the document has a routing key, the update callback deletes the document from its previous routing when the routing value changes. The previous values come from the Sequel `dirty` plugin, so load it in those models:
+
+```ruby
+class City < Sequel::Model
+  plugin :dirty
+  include Esse::Sequel::Model
+
+  index_callback 'geos_index:city'
+end
+```
+
+Without the plugin the document is still indexed, but the stale copy under the old routing is not removed.
+
+### Index an associated record
+
+```ruby
+class City < Sequel::Model
+  many_to_one :state
+  include Esse::Sequel::Model
+
+  index_callback('geos_index:state') { state }
+end
+```
+
+### Update a lazy attribute
+
+```ruby
+class Comment < Sequel::Model
+  many_to_one :post
+  include Esse::Sequel::Model
+
+  update_lazy_attribute_callback('posts_index:post', 'comments_count') { post_id }
+end
+```
+
+## Disabling callbacks
+
+### Per-block
+
+```ruby
+Esse::Sequel::Hooks.without_indexing do
+  10_000.times { User.create(...) }
+end
+```
+
+### Per-repository
+
+```ruby
+Esse::Sequel::Hooks.without_indexing(UsersIndex, AccountsIndex) do
+  migrate_users!
+end
+```
+
+### Per-model
+
+```ruby
+User.without_indexing { User.create(...) }
+User.without_indexing(UsersIndex) { User.create(...) }
+```
+
+### Globally
+
+```ruby
+Esse::Sequel::Hooks.disable!
+# ... bulk work ...
+Esse::Sequel::Hooks.enable!
+```
+
+## Streaming by primary key range
+
+```ruby
+UsersIndex.import(context: { start: 1,     finish: 5000,  batch_size: 500 })
+UsersIndex.import(context: { start: 5001,  finish: 10000, batch_size: 500 })
+```
+
+## Differences from esse-active_record
+
+Behaviorally the plugins are identical. The differences:
+
+- Uses Sequel datasets (`where`, `eager`, `select`) in lieu of ActiveRecord scopes.
+- Collections accept a model class or a model dataset (`User.where(...)`), not `User.all` (which returns an Array in Sequel).
+- Callbacks run in the `after_create`/`after_update`/`after_destroy` model hooks and are deferred with `DB.after_commit`. Models overriding these hooks must call `super`.
+- `connect_with`/`connected_to` take a Sequel `server:` instead of ActiveRecord `role:`/`shard:`.
+- Routing changes on update require the `dirty` plugin (ActiveRecord tracks previous changes natively).
+- Async callbacks from esse-async_indexing are not available for Sequel models yet.
+- Hook store key differs (`:esse_sequel_hooks` vs `:esse_active_record_hooks`), so you can use both side-by-side in a single process.

data/lib/esse/plugins/sequel.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require "sequel"
+
+module Esse
+  module Plugins
+    module Sequel
+      module RepositoryClassMethods
+        # @param [Class, Sequel::Dataset] model_class The Sequel model class or model dataset
+        # @param [Hash] options The options
+        # @option options [Integer] :batch_size The batch size for the collection
+        # @option options [Hash] :connect_with The database server to run the collection queries on. Ex: `{ server: :read_only }`
+        def collection(*args, &block)
+          unless model_or_dataset?(args.first)
+            return super
+          end
+          kwargs = args.last.is_a?(Hash) ? args.pop.dup : {}
+          model_class = args.shift
+
+          repo = Class.new(Esse::Sequel::Collection)
+          repo.base_scope = -> { model_class }
+          repo.batch_size = kwargs.delete(:batch_size) if kwargs.key?(:batch_size)
+          repo.connect_with = kwargs.delete(:connect_with) if kwargs.key?(:connect_with)
+          repo.class_eval(&block) if block
+
+          super(repo, *args, **kwargs)
+        end
+        # Sequel models and datasets respond to #to_hash, so a **kwargs signature would make
+        # Ruby 2.7 implicitly convert the model argument into keyword arguments.
+        ruby2_keywords(:collection) if respond_to?(:ruby2_keywords, true)
+
+        def dataset(**params)
+          if @collection_proc.nil?
+            raise NotImplementedError, "Can't call `dataset' on a repository without a collection defined"
+          elsif @collection_proc.is_a?(Class) && @collection_proc < Esse::Sequel::Collection
+            @collection_proc.new(**params).dataset
+          elsif defined? super
+            super
+          else
+            raise NoMethodError, "undefined method `dataset' for #{self}"
+          end
+        end
+
+        private
+
+        def model_or_dataset?(klass)
+          return true if klass.is_a?(Class) && klass < ::Sequel::Model
+          return true if klass.is_a?(::Sequel::Dataset) && klass.respond_to?(:model)
+
+          false
+        end
+      end
+    end
+  end
+end

data/lib/esse/sequel/callbacks/indexing_on_create.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Esse::Sequel
+  module Callbacks
+    class IndexingOnCreate < Callback
+      def call(model)
+        record = block_result || model
+        document = repo.serialize(record)
+        repo.index.index(document, **options) if document && !document.ignore_on_index?
+        true
+      end
+    end
+
+    register_callback(:indexing, :create, IndexingOnCreate)
+  end
+end

data/lib/esse/sequel/callbacks/indexing_on_destroy.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Esse::Sequel
+  module Callbacks
+    class IndexingOnDestroy < Callback
+      def call(model)
+        record = block_result || model
+        document = repo.serialize(record)
+        repo.index.delete(document, **options) if document && !document.ignore_on_delete?
+        true
+      rescue Esse::Transport::NotFoundError
+        true
+      end
+    end
+
+    register_callback(:indexing, :destroy, IndexingOnDestroy)
+  end
+end

data/lib/esse/sequel/callbacks/indexing_on_update.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module Esse::Sequel
+  module Callbacks
+    class IndexingOnUpdate < Callback
+      def call(model)
+        record = block_result || model
+
+        document = repo.serialize(record)
+        return true unless document
+
+        update_document(document)
+        return true unless document.routing
+
+        prev_document = previous_document(model)
+
+        return true unless prev_document
+        return true if [prev_document.id, prev_document.routing].include?(nil)
+        return true if prev_document.routing == document.routing
+        return true if prev_document.id != document.id
+
+        begin
+          repo.index.delete(prev_document, **options)
+        rescue Esse::Transport::NotFoundError
+        end
+
+        true
+      end
+
+      protected
+
+      def update_document(document)
+        return if document.ignore_on_index?
+
+        if @with == :update || (@with.nil? && repo.lazy_document_attributes.any?)
+          begin
+            repo.index.update(document, **options)
+          rescue Esse::Transport::NotFoundError
+            repo.index.index(document, **options)
+          end
+        else
+          repo.index.index(document, **options)
+        end
+      end
+
+      # Previous column values are only known when the model uses the Sequel dirty plugin
+      def previous_document(model)
+        return unless model.respond_to?(:previous_changes)
+
+        changes = model.previous_changes
+        return if changes.nil? || changes.empty?
+
+        prev_values = model.values.merge(changes.transform_values(&:first))
+        repo.serialize(model.class.call(prev_values).freeze)
+      end
+    end
+
+    register_callback(:indexing, :update, IndexingOnUpdate)
+  end
+end

data/lib/esse/sequel/callbacks/update_lazy_attribute.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Esse::Sequel
+  module Callbacks
+    class UpdateLazyAttribute < Callback
+      attr_reader :attribute_name
+
+      def initialize(attribute_name:, **kwargs, &block)
+        @attribute_name = attribute_name
+        super(**kwargs, &block)
+      end
+
+      def call(model)
+        related_ids = Esse::ArrayUtils.wrap(block_result || model.id)
+        return true if related_ids.empty?
+
+        repo.update_documents_attribute(attribute_name, related_ids, options)
+
+        true
+      end
+    end
+
+    register_callback(:update_lazy_attribute, :create, UpdateLazyAttribute)
+    register_callback(:update_lazy_attribute, :update, UpdateLazyAttribute)
+    register_callback(:update_lazy_attribute, :destroy, UpdateLazyAttribute)
+  end
+end

data/lib/esse/sequel/callbacks.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module Esse
+  module Sequel
+    class Callback
+      attr_reader :repo, :options, :block_result
+
+      def initialize(repo:, block_result: nil, with: nil, **kwargs)
+        @repo = repo
+        @with = with
+        @options = kwargs
+        @block_result = block_result
+      end
+
+      def call(model)
+        raise NotImplementedError, "You must implement #call method"
+      end
+    end
+
+    module Callbacks
+      class << self
+        def to_h
+          @callbacks || {}.freeze
+        end
+
+        def register_callback(identifier, operation, callback_class)
+          unless callback_class < Esse::Sequel::Callback
+            raise ArgumentError, "callback_class must be a subclass of Esse::Sequel::Callback"
+          end
+
+          key = [operation, identifier].join("_").to_sym
+
+          @callbacks = @callbacks ? @callbacks.dup : {}
+          if @callbacks.key?(key)
+            raise ArgumentError, "callback #{identifier} for #{operation} operation already registered"
+          end
+
+          @callbacks[key] = callback_class
+        ensure
+          @callbacks&.freeze
+        end
+
+        def registered?(identifier, operation)
+          return false unless @callbacks
+
+          key = [operation, identifier].join("_").to_sym
+          @callbacks.key?(key)
+        end
+
+        def fetch!(identifier, operation)
+          key = [operation, identifier].join("_").to_sym
+          if registered?(identifier, operation)
+            [key, @callbacks[key]]
+          else
+            raise ArgumentError, "callback #{identifier} for #{operation} operation not registered"
+          end
+        end
+      end
+    end
+  end
+end
+
+require_relative "callbacks/indexing_on_create"
+require_relative "callbacks/indexing_on_update"
+require_relative "callbacks/indexing_on_destroy"
+require_relative "callbacks/update_lazy_attribute"

data/lib/esse/sequel/collection.rb

@@ -0,0 +1,191 @@
+# frozen_string_literal: true
+
+module Esse
+  module Sequel
+    class Collection
+      include Enumerable
+
+      class << self
+        # The model class or the dataset to be used as the scope
+        # @return [Proc]
+        attr_accessor :base_scope
+
+        # The number of records to be returned in each batch
+        # @return [Integer]
+        attr_accessor :batch_size
+
+        # Hash with the custom scopes defined on the collection
+        # @return [Hash]
+        attr_accessor :scopes
+
+        # The hash with the contexts as key and the transformer proc as value
+        # @return [Hash]
+        attr_accessor :batch_contexts
+
+        # The database server (shard) options for the collection query. Ex: `{ server: :read_only }`
+        # @return [Hash, nil]
+        attr_accessor :connect_with
+
+        def inspect
+          return super unless self < Esse::Sequel::Collection
+          return super unless base_scope
+
+          format("#<Esse::Sequel::Collection__%s>", model)
+        end
+
+        def model
+          raise(NotImplementedError, "No model defined for #{self}") unless base_scope
+
+          scope = base_scope.call
+          scope.is_a?(::Sequel::Dataset) ? scope.model : scope
+        end
+
+        def inherited(subclass)
+          super
+
+          subclass.base_scope = base_scope
+          subclass.batch_size = batch_size
+          subclass.scopes = scopes.dup
+          subclass.batch_contexts = batch_contexts.dup
+          subclass.connect_with = connect_with&.dup
+        end
+
+        def scope(name, proc = nil, override: false, &block)
+          proc = proc&.to_proc || block
+          raise ArgumentError, "proc or block required" unless proc
+          raise ArgumentError, "scope `#{name}' already defined" if !override && scopes.key?(name.to_sym)
+
+          scopes[name.to_sym] = proc
+        end
+
+        def batch_context(name, proc = nil, override: false, &block)
+          proc = proc&.to_proc || block
+          raise ArgumentError, "proc or block required" unless proc
+          raise ArgumentError, "batch_context `#{name}' already defined" if !override && batch_contexts.key?(name.to_sym)
+
+          batch_contexts[name.to_sym] = proc
+        end
+
+        # Runs the collection queries on the given database server (shard)
+        # @param [Symbol] server The server name as configured in the Sequel database :servers option
+        def connected_to(server:)
+          self.connect_with = {server: server}
+        end
+      end
+
+      self.scopes = {}
+      self.batch_contexts = {}
+
+      attr_reader :start, :finish, :batch_size, :params
+
+      # @param [Integer] start Specifies the primary key value to start from, inclusive of the value.
+      # @param [Integer] finish  Specifies the primary key value to end at, inclusive of the value.
+      # @param [Integer] batch_size The number of records to be returned in each batch. Defaults to 1000.
+      # @param [Hash] params The query criteria
+      # @return [Esse::Sequel::Collection]
+      def initialize(start: nil, finish: nil, batch_size: nil, **params)
+        @start = start
+        @finish = finish
+        @batch_size = batch_size || self.class.batch_size || 1000
+        @params = params
+      end
+
+      def each
+        with_connection do
+          find_in_batches(dataset) do |rows|
+            kwargs = params.dup
+            self.class.batch_contexts.each do |name, proc|
+              kwargs[name] = proc.call(rows, **params)
+            end
+            yield(rows, **kwargs)
+          end
+        end
+      end
+
+      def each_batch_ids
+        with_connection do
+          find_in_batches(dataset, ids_only: true) do |ids|
+            yield(ids)
+          end
+        end
+      end
+
+      def count
+        with_connection do
+          dataset.unordered.unlimited.ungrouped.count
+        end
+      end
+      alias_method :size, :count
+
+      def dataset(**kwargs)
+        query = self.class.base_scope&.call || raise(NotImplementedError, "No scope defined for #{self.class}")
+        query = query.dataset unless query.is_a?(::Sequel::Dataset)
+        query = query.unordered.unlimited
+        query = query.server(server) if server
+        params.merge(kwargs).each do |key, value|
+          if self.class.scopes.key?(key)
+            scope_proc = self.class.scopes[key]
+            query = if scope_proc.arity == 0
+              query.instance_exec(&scope_proc)
+            else
+              query.instance_exec(value, &scope_proc)
+            end
+          elsif query.model.columns.include?(key.to_sym)
+            query = query.where(::Sequel[query.model.table_name][key.to_sym] => value)
+          else
+            raise ArgumentError, "Unknown scope `#{key}'"
+          end
+        end
+
+        query
+      end
+
+      def inspect
+        return super unless self.class < Esse::Sequel::Collection
+        return super unless self.class.base_scope
+
+        vars = instance_variables.map do |n|
+          "#{n}=#{instance_variable_get(n).inspect}"
+        end
+        format("#<Esse::Sequel::Collection__%s:0x%x %s>", self.class.model, object_id, vars.join(", "))
+      end
+
+      protected
+
+      def server
+        self.class.connect_with&.fetch(:server, nil)
+      end
+
+      # Wraps the block with Database#with_server when the server_block extension is loaded,
+      # so queries made by batch contexts also run on the configured server.
+      def with_connection(&block)
+        db = server && self.class.model.db
+        if db.respond_to?(:with_server)
+          db.with_server(server, &block)
+        else
+          yield
+        end
+      end
+
+      # Keyset pagination over the primary key, inclusive of start and finish values.
+      def find_in_batches(query, ids_only: false)
+        pk = ::Sequel[query.model.table_name][query.model.primary_key]
+        query = query.where(pk >= start) if start
+        query = query.where(pk <= finish) if finish
+        query = query.order(pk).limit(batch_size)
+
+        last_id = nil
+        loop do
+          page = last_id.nil? ? query : query.where(pk > last_id)
+          rows = ids_only ? page.select_map(pk) : page.all
+          break if rows.empty?
+
+          yield rows
+          break if rows.size < batch_size
+
+          last_id = ids_only ? rows.last : rows.last.pk
+        end
+      end
+    end
+  end
+end

data/lib/esse/sequel/hooks.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Esse
+  module Sequel
+    module Hooks
+      STORE_STATE_KEY = :esse_sequel_hooks
+
+      include Esse::Hooks[store_key: STORE_STATE_KEY]
+    end
+  end
+end