esse-sequel 0.0.1.beta1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 556f57525adf41b371e1e117bf9a31d4c72967ec400a46bcd2de4d6b155b0536
+  data.tar.gz: 0ae837cc171d0065ec04fec57130f82613932de578c90a5c4fd98da982c98582
+SHA512:
+  metadata.gz: 46fd592b16c0f972389d6116fba050d5396ed26e2485964c18f09b1de23af37d739ae71e331daca1f1c2702e196191f90f0bd2b90e2517c56bde3b9c1944461b
+  data.tar.gz: 40a434be03f7339f8f39096777dd6e2d068b948b32c101cef1ec614af5ccc9eb7ccf063554a42de7cf4fe33502b20f3ef3e6b7863e0fcd8dca277b1dbfa203d0

data/.rubocop.yml

@@ -0,0 +1,37 @@
+inherit_mode:
+  merge:
+    - Exclude
+
+plugins:
+  - rubocop-performance
+  - rubocop-rspec
+
+require:
+  - standard/cop/block_single_line_braces
+
+inherit_gem:
+  standard: config/base.yml
+
+AllCops:
+  TargetRubyVersion: 2.6
+  SuggestExtensions: false
+  Exclude:
+    - "db/**/*"
+    - "tmp/**/*"
+    - "vendor/**/*"
+  NewCops: enable
+
+RSpec/MultipleExpectations:
+  Enabled: false
+
+RSpec/ExampleLength:
+  Enabled: false
+
+RSpec/MultipleMemoizedHelpers:
+  Enabled: false
+
+RSpec/MessageSpies:
+  Enabled: false
+
+RSpec/StubbedMock:
+  Enabled: false

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## 0.0.1.beta1
+The first release of the esse-sequel plugin
+* Added: Initial implementation of the plugin

data/Gemfile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
+
+gem "esse", "~> 0.3.4"
+gem "sqlite3"
+gem "esse-rspec"
+gem "logger"
+gem "elasticsearch", "~> 7.17", ">= 7.17.10"
+
+# Specify your gem's dependencies in esse-sequel.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2022 Marcos G. Zimmermann
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,265 @@
+# Esse Sequel Plugin
+
+This gem is a [esse](https://github.com/marcosgz/esse) plugin for the Sequel ORM. It provides a set of methods to simplify implementation of Sequel models as datasource of esse indexes.
+
+## Documentation
+
+Full guides, usage examples, and API reference are published at **[gems.marcosz.com.br/esse-sequel](https://gems.marcosz.com.br/esse-sequel/)** — part of the [marcosgz Ruby gem catalogue](https://gems.marcosz.com.br).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'esse-sequel'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install esse-sequel
+
+## Usage
+
+Add the `:sequel` plugin and configure the `collection` with the Sequel model you want to use.
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User
+    document # ...
+  end
+end
+```
+
+Using multiple repositories is also possible:
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :account do
+    collection ::Account
+    document # ...
+  end
+
+  repository :admin do
+    collection ::User.where(admin: true)
+    document # ...
+  end
+end
+```
+
+### Collection Scope
+It's also possible to specify custom scopes to the repository collection to be used to import data to the index:
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User do
+      scope :active, -> { where(active: true) }
+      scope :role, ->(role) { where(role: role) }
+    end
+    document # ...
+  end
+end
+
+# Import data using the scopes
+#   > UsersIndex.import(context: { active: true, role: 'admin' })
+#
+# Streaming data using the scopes
+#   > UsersIndex.documents(active: true, role: 'admin').first
+```
+
+## Collection Batch Context
+
+Assume that you have a collection of orders and you want to also include the customer data that lives in a external system. To avoid making a request for each order, you can use the `batch_context` to fetch the data in batches and make it available in the document context.
+
+```ruby
+class OrdersIndex < Esse::Index
+  plugin :sequel
+
+  repository :order do
+    collection ::Order do
+      batch_context :customers do |orders, **_existing_context|
+        # The return value will be available in the document context
+        # { customers: <value returned from this block> }
+        ExternalSystem::Customer.find_all_by_ids(orders.map(&:customer_id)).index_by(&:id) # => { 1 => <Customer>, 2 => <Customer> }
+      end
+    end
+    document do |order, customers: {}, **_|
+      customer = customers[order.customer_id]
+      {
+        id: order.id,
+        customer: {
+          id: customer&.id,
+          name: customer&.name
+        }
+      }
+    end
+  end
+end
+```
+
+For Sequel associations, you can define the repository collection by eager loading the associations as usual:
+
+```ruby
+
+class OrdersIndex < Esse::Index
+  plugin :sequel
+
+  repository :order do
+    collection ::Order.eager(:customer)
+    document do |order, **_|
+      {
+        id: order.id,
+        customer: {
+          id: order.customer&.id,
+          name: order.customer&.name
+        }
+      }
+    end
+  end
+end
+```
+
+### Data Streaming Options
+
+By default the collection supports 3 streaming options:
+* `batch_size`: the number of documents to be streamed in each batch. Default is 1000;
+* `start`: the primary key value to start from, inclusive of the value;
+* `finish`: the primary key value to end at, inclusive of the value;
+
+This is useful when you want to import simultaneous data. You can make one process import all records between 1 and 10,000, and another from 10,000 and beyond
+
+```ruby
+UsersIndex.import(context: { start: 1, finish: 10000, batch_size: 500 })
+```
+
+The default value of `batch_size` can be also defined in the `collection` configuration:
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User, batch_size: 500
+    document # ...
+  end
+end
+```
+
+### Database Server (Sharding)
+
+If the Sequel database is configured with multiple servers, the collection queries can be routed to one of them using the `connect_with` option or the `connected_to` method:
+
+```ruby
+DB = Sequel.connect(ENV["DATABASE_URL"], servers: { read_only: { host: "replica" } })
+
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User, connect_with: { server: :read_only }
+    # or
+    collection ::User do
+      connected_to server: :read_only
+    end
+    document # ...
+  end
+end
+```
+
+The collection dataset runs on the given server. When the [server_block](https://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/server_block_rb.html) extension is loaded (`DB.extension :server_block`), the whole batch iteration, including `batch_context` queries, runs inside `DB.with_server`.
+
+### Indexing Callbacks
+
+The `index_callback` callback can be used to automatically index or delete documents after commit on create/update/destroy events. Include the `Esse::Sequel::Model` module in the model to enable it.
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user, const: true do
+    collection ::User
+    document # ...
+  end
+end
+
+class User < Sequel::Model
+  include Esse::Sequel::Model
+  many_to_one :organization
+
+  # Using a index and repository as argument. Note that the index name is used instead of the
+  # of the constant name. it's so because index and model depends on each other should result in
+  # circular dependencies issues.
+  index_callback 'users_index:user'
+  # Using a block to direct a different object to be indexed
+  index_callback('organizations') { organization } # The `_index` suffix and repo name is optional on the index name
+end
+```
+
+Callbacks run in the `after_create`, `after_update` and `after_destroy` model hooks and are deferred with `DB.after_commit`, so documents are only indexed after the transaction commits, and are not indexed at all when it is rolled back. Models that override these hooks must call `super`.
+
+When a document uses a routing key, the update callback also deletes the document from its previous routing when the routing value changes. This requires the previous column values, so load the [dirty](https://sequel.jeremyevans.net/rdoc-plugins/classes/Sequel/Plugins/Dirty.html) plugin in those models:
+
+```ruby
+class City < Sequel::Model
+  plugin :dirty
+  include Esse::Sequel::Model
+
+  index_callback 'geos_index:city'
+end
+```
+
+Callbacks can also be disabled/enabled globally:
+
+```ruby
+Esse::Sequel::Hooks.disable!
+Esse::Sequel::Hooks.enable!
+Esse::Sequel::Hooks.without_indexing do
+  10.times { User.create! }
+end
+```
+
+or by some specific list of index or index's repository
+
+```ruby
+Esse::Sequel::Hooks.disable!(UsersIndex.repo)
+Esse::Sequel::Hooks.enable!(UsersIndex.repo)
+Esse::Sequel::Hooks.without_indexing(AccountsIndex, UsersIndex.repo) do
+  10.times { User.create! }
+end
+```
+
+or by the model that the callback is configured
+
+```ruby
+User.without_indexing do
+  10.times { User.create! }
+end
+User.without_indexing(AccountsIndex) do
+  10.times { User.create! }
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/marcosgz/esse-sequel.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/docs/README.md

@@ -0,0 +1,55 @@
+# esse-sequel
+
+Sequel ORM integration for [Esse](../../esse/docs/README.md). The API mirrors [esse-active_record](../../esse-active_record/docs/README.md) — if you know one, you know the other.
+
+## Contents
+
+- [Usage guide](usage.md)
+- [API reference](api.md)
+
+## What you get
+
+- `collection Model` DSL for repositories.
+- `scope`, `batch_context`, `connected_to` inside collections.
+- `index_callback` and `update_lazy_attribute_callback` on Sequel models.
+- `Esse::Sequel::Hooks` for enabling/disabling indexing globally or per-repo.
+
+## Quick start
+
+```ruby
+# Gemfile
+gem 'esse', '>= 0.3.0'
+gem 'esse-sequel'
+gem 'sequel', '>= 5.0'
+```
+
+```ruby
+class UsersIndex < Esse::Index
+  plugin :sequel
+
+  repository :user do
+    collection ::User do
+      scope :active, -> { where(active: true) }
+    end
+
+    document { |u, **| { _id: u.id, name: u.name } }
+  end
+end
+
+class User < Sequel::Model
+  include Esse::Sequel::Model
+  index_callback 'users_index:user'
+end
+```
+
+Every `User.create`, `update`, and `destroy` now syncs to Elasticsearch.
+
+## Version
+
+- Version: **0.0.1.beta1**
+- Ruby: `>= 2.4.0`
+- Depends on: `esse >= 0.3.0`, `esse-hooks`, `sequel >= 5.0`
+
+## License
+
+MIT.

data/docs/api.md

@@ -0,0 +1,119 @@
+# API Reference
+
+## `Esse::Plugins::Sequel`
+
+Registered with:
+
+```ruby
+plugin :sequel
+```
+
+### Repository class methods
+
+#### `collection(model, **opts, &block)`
+
+Accepts a `Sequel::Model` class or a `Sequel::Dataset`. Produces an `Esse::Sequel::Collection` subclass.
+
+| Argument | Description |
+|----------|-------------|
+| `model` | Sequel model class or dataset |
+| `batch_size:` | Records per batch (default `1000`) |
+| `connect_with:` | `{server: :name}` to run the queries on a Sequel server (shard) |
+| `&block` | Evaluated in the collection class scope (`scope`, `batch_context`, `connected_to`) |
+
+#### `dataset(**params)`
+
+Returns the filtered `Sequel::Dataset` after applying scopes.
+
+---
+
+## `Esse::Sequel::Collection`
+
+Base class. Same interface as `Esse::ActiveRecord::Collection`. Batches are fetched with keyset pagination over the model primary key.
+
+### Class-level DSL
+
+- `scope(name, proc = nil, override: false, &block)` — define a named scope.
+- `batch_context(name, proc = nil, override: false, &block)` — define a batch enrichment fetcher.
+- `connected_to(server:)` — run the queries on the given Sequel server. Uses `DB.with_server` for the whole iteration when the `server_block` extension is loaded.
+
+### Instance methods
+
+| Method | Description |
+|--------|-------------|
+| `each { |rows, **ctx| }` | Iterate batches with scopes + contexts applied |
+| `each_batch_ids { |ids| }` | Iterate ID-only batches |
+| `count` / `size` | Total row count |
+| `dataset(**kwargs)` | Filtered `Sequel::Dataset` |
+
+Constructor:
+
+```ruby
+CollectionClass.new(start: 1, finish: 10_000, batch_size: 500, **scope_args)
+```
+
+---
+
+## `Esse::Sequel::Model`
+
+Mixin for Sequel models. `include` it to enable callbacks.
+
+### `index_callback(reference, on: [...], with: nil, **opts, &block)`
+
+Registers a callback that runs after the transaction commits (`DB.after_commit` from the model `after_create`/`after_update`/`after_destroy` hooks).
+
+| Option | Description |
+|--------|-------------|
+| `reference` | `'index_name:repo_name'` (or class form) |
+| `on:` | Subset of `[:create, :update, :destroy]`, default all |
+| `with:` | `:update` for partial updates |
+| `if:` / `unless:` | Conditional predicates |
+| `&block` | Optional — return the object to index |
+
+Raises if the same repo already has a callback registered.
+
+### `update_lazy_attribute_callback(reference, attribute, on: [...], **opts, &block)`
+
+Registers a callback that calls `repo.update_documents_attribute(attribute, ids, opts)`.
+
+### `without_indexing(*repos, &block)`
+
+Temporarily disables callbacks for this model class.
+
+### `esse_callbacks`
+
+Returns a frozen hash of registered callbacks per reference.
+
+---
+
+## `Esse::Sequel::Hooks`
+
+Includes `Esse::Hooks[store_key: :esse_sequel_hooks]`. See [esse-hooks](../../esse-hooks/docs/README.md) for the full API. Methods include:
+
+- `disable!` / `enable!` (global or per-repo).
+- `with_indexing` / `without_indexing` (scoped).
+- `with_indexing_for_model` / `without_indexing_for_model`.
+- `enabled?` / `disabled?` / `enabled_for_model?`.
+- `resolve_index_repository(reference)`.
+
+---
+
+## Built-in callbacks
+
+Registered in `Esse::Sequel::Callbacks`. All inherit from `Esse::Sequel::Callback`.
+
+| Callback | Behavior |
+|----------|----------|
+| `IndexingOnCreate` | `repo.index(document)` on create |
+| `IndexingOnUpdate` | `repo.update(document)` or `repo.index(document)` (depending on `with:`). Handles routing changes by deleting the previous document at the old routing (requires the `dirty` plugin). |
+| `IndexingOnDestroy` | `repo.delete(document)` on destroy. Silently handles `NotFoundError`. |
+| `UpdateLazyAttribute` | `repo.update_documents_attribute(attribute, ids, opts)` |
+
+---
+
+## Deprecated methods
+
+| Deprecated | Use instead |
+|------------|-------------|
+| `index_callbacks` | `index_callback` |
+| `esse_index_repos` | `esse_callbacks` |