table_sync 5.1.0 → 6.0

data/docs/publishing.md

@@ -1,133 +1,63 @@
-# Publishing changes
+# Publishing
 
-Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. `:if` and `:unless` are supported for Sequel and ActiveRecord
+TableSync can be used to send data using RabbitMQ.
 
-Functioning `Rails.cache` is required
+You can do in two ways. Automatic and manual.
+Each one has its own pros and cons.
 
-Example:
-
-```ruby
-class SomeModel < Sequel::Model
-  TableSync.sync(self, { if: -> (*) { some_code } })
-end
-```
-
-#### #attributes_for_sync
-
-Models can implement `#attributes_for_sync` to override which attributes are published. If not present, all attributes are published
+Automatic is used to publish changes in realtime, as soon as the tracked entity changes.
+Usually syncs one entity at a time.
 
-#### #attrs_for_routing_key
+Manual allows to sync a lot of entities per message.
+But demands greater amount of work and data preparation.
 
-Models can implement `#attrs_for_routing_key` to override which attributes are given to `routing_key_callable`. If not present, published attributes are given
+## Automatic
 
-#### #attrs_for_metadata
+Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. 
 
-Models can implement `#attrs_for_metadata` to override which attributes are given to `metadata_callable`. If not present, published attributes are given
+Options:
 
-#### .table_sync_model_name
+- `if:` and `unless:` - Runs given proc in the scope of an instance. Skips sync on `false` for `if:` and on `true` for `unless:`.
+- `on:` - specify events (`create`, `update`, `destroy`) to trigger sync on. Triggered for all of them without this option.
+- `debounce_time` - min time period allowed between synchronizations.
 
-Models can implement `.table_sync_model_name` class method to override the model name used for publishing events. Default is model class name
+Functioning `Rails.cache` is required.
 
-#### .table_sync_destroy_attributes(original_attributes)
+How it works:
 
-Models can implement `.table_sync_destroy_attributes` class method to override the attributes used for publishing destroy events. Default is object's original attributes
+- `TableSync.sync(self)` - registers new callbacks (for `create`, `update`, `destroy`) for ActiveRecord model, and defines `after_create`, `after_update` and `after_destroy` callback methods for Sequel model.
 
-## Configuration
-
-- `TableSync.publishing_job_class_callable` is a callable which should resolve to a ActiveJob subclass that calls TableSync back to actually publish changes (required)
+- Callbacks call `TableSync::Publishing::Single#publish_later` with given options and object attributes. It enqueues a job which then publishes a message.
 
 Example:
 
 ```ruby
-class TableSync::Job < ActiveJob::Base
-  def perform(*args)
-    TableSync::Publishing::Publisher.new(*args).publish_now
-  end
+class SomeModel < Sequel::Model
+  TableSync.sync(self, { if: -> (*) { some_code }, unless: -> (*) { some_code }, on: [:create, :update] })
 end
-```
-
-- `TableSync.batch_publishing_job_class_callable` is a callable which should resolve to a ActiveJob subclass that calls TableSync batch publisher back to actually publish changes (required for batch publisher)
-
-- `TableSync.routing_key_callable` is a callable which resolves which routing key to use when publishing changes. It receives object class and published attributes (required)
-
-Example:
-
-```ruby
-TableSync.routing_key_callable = -> (klass, attributes) { klass.gsub('::', '_').tableize }
-```
-
-- `TableSync.routing_metadata_callable` is a callable that adds RabbitMQ headers which can be used in routing (optional). It receives object class and published attributes. One possible way of using it is defining a headers exchange and routing rules based on key-value pairs (which correspond to sent headers)
 
-Example:
-
-```ruby
-TableSync.routing_metadata_callable = -> (klass, attributes) { attributes.slice("project_id") }
+class SomeOtherModel < Sequel::Model
+  TableSync.sync(self)
+end
 ```
 
-- `TableSync.exchange_name` defines the exchange name used for publishing (optional, falls back to default Rabbit gem configuration).
-
-- `TableSync.notifier` is a module that provides publish and recieve notifications.
-
-# Manual publishing
-
-`TableSync::Publishing::Publisher.new(object_class, original_attributes, confirm: true, state: :updated, debounce_time: 45)`
-where state is one of `:created / :updated / :destroyed` and `confirm` is Rabbit's confirm delivery flag and optional param `debounce_time` determines debounce time in seconds, 1 minute by default.
-
-# Manual publishing with batches
-
-You can use `TableSync::Publishing::BatchPublisher` to publish changes in batches (array of hashes in `attributes`).
-
-When using `TableSync::Publishing::BatchPublisher`,` TableSync.routing_key_callable` is called as follows: `TableSync.routing_key_callable.call(klass, {})`, i.e. empty hash is passed instead of attributes. And `TableSync.routing_metadata_callable` is not called at all: metadata is set to empty hash.
+## Manual
 
-`TableSync::Publishing::BatchPublisher.new(object_class, original_attributes_array, **options)`, where `original_attributes_array` is an array with hash of attributes of published objects and `options` is a hash of options.
-
-`options` consists of:
-- `confirm`, which is a flag for RabbitMQ, `true` by default
-- `routing_key`, which is a custom key used (if given) to override one from `TableSync.routing_key_callable`, `nil` by default
-- `push_original_attributes` (default value is `false`), if this option is set to `true`,
-original_attributes_array will be pushed to Rabbit instead of fetching records from database and sending their mapped attributes.
-- `headers`, which is an option for custom headers (can be used for headers exchanges routes), `nil` by default
-- `event`, which is an option for event specification (`:destroy` or `:update`), `:update` by default
+Directly call one of the publishers. It's the best if you need to sync a lot of data.
+This way you don't even need for the changes to occur.
 
 Example:
 
 ```ruby
-TableSync::Publishing::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
+  TableSync::Publishing::Batch.new(
+    object_class: "User",
+    original_attributes: [{ id: 1 }, { id: 2 }],
+    event: :update,
+  ).publish_now
 ```
 
-# Manual publishing with batches (Russian)
-
-С помощью класса `TableSync::Publishing::BatchPublisher` вы можете опубликовать изменения батчами (массивом в `attributes`).
-
-При использовании `TableSync::Publishing::BatchPublisher`, `TableSync.routing_key_callable` вызывается следующим образом: `TableSync.routing_key_callable.call(klass, {})`, то есть вместо аттрибутов передается пустой хэш. А `TableSync.routing_metadata_callable` не вызывается вовсе: в метадате устанавливается пустой хэш.
-
-`TableSync::Publishing::BatchPublisher.new(object_class, original_attributes_array, **options)`, где `original_attributes_array` - массив с аттрибутами публикуемых объектов и `options`- это хэш с дополнительными опциями.
-
-`options` состоит из:
-- `confirm`, флаг для RabbitMQ, по умолчанию - `true`
-- `routing_key`, ключ, который (если указан) замещает ключ, получаемый из `TableSync.routing_key_callable`, по умолчанию - `nil`
-- `push_original_attributes` (значение по умолчанию `false`), если для этой опции задано значение true, в Rabbit будут отправлены original_attributes_array, вместо получения значений записей из базы непосредственно перед отправкой.
-- `headers`, опция для задания headers (можно использовать для задания маршрутов в headers exchange'ах), `nil` по умолчанию
-- `event`, опция для указания типа события (`:destroy` или `:update`), `:update` по умолчанию
+## Read More
 
-Example:
-
-```ruby
-TableSync::Publishing::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
-```
+- [Publishers](publishing/publishers.md)
+- [Configuration](publishing/configuration.md)
+- [Manual Sync (examples)](publishing/manual.md)

data/lib/table_sync/errors.rb

@@ -3,6 +3,35 @@
 module TableSync
   Error = Class.new(StandardError)
 
+  NoObjectsForSyncError = Class.new(Error)
+
+  class EventError < Error
+    def initialize(event)
+      super(<<~MSG.squish)
+        Event #{event.inspect} is invalid.#{' '}
+        Expected: #{TableSync::Event::VALID_RAW_EVENTS.inspect}.
+      MSG
+    end
+  end
+
+  class NoPrimaryKeyError < Error
+    def initialize(object_class, object_data, primary_key_columns)
+      super(<<~MSG.squish)
+        Can't find or init an object of #{object_class} with #{object_data.inspect}.
+        Incomplete primary key! object_data must contain: #{primary_key_columns.inspect}.
+      MSG
+    end
+  end
+
+  class NoCallableError < Error
+    def initialize(type)
+      super(<<~MSG.squish)
+        Can't find callable for #{type}!
+        Please initialize TableSync.#{type}_callable with the correct proc!
+      MSG
+    end
+  end
+
   class UpsertError < Error
     def initialize(data:, target_keys:, result:)
       super("data: #{data.inspect}, target_keys: #{target_keys.inspect}, result: #{result.inspect}")
@@ -25,28 +54,6 @@ module TableSync
     end
   end
 
-  # @api public
-  # @since 2.2.0
-  PluginError = Class.new(Error)
-
-  # @api public
-  # @since 2.2.0
-  class UnregisteredPluginError < PluginError
-    # @param plugin_name [Any]
-    def initialize(plugin_name)
-      super("#{plugin_name} plugin is not registered")
-    end
-  end
-
-  # @api public
-  # @since 2.2.0
-  class AlreadyRegisteredPluginError < PluginError
-    # @param plugin_name [Any]
-    def initialize(plugin_name)
-      super("#{plugin_name} plugin already exists")
-    end
-  end
-
   class InterfaceError < Error
     def initialize(object, method_name, parameters, description)
       parameters = parameters.map do |parameter|
@@ -54,7 +61,9 @@ module TableSync
 
         case type
         when :req
+          #:nocov:
           name.to_s
+          #:nocov:
         when :keyreq
           "#{name}:"
         when :block

data/lib/table_sync/event.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+class TableSync::Event
+  attr_reader :event
+
+  UPSERT_EVENTS          = %i[create update].freeze
+  VALID_RESOLVED_EVENTS  = %i[update destroy].freeze
+  VALID_RAW_EVENTS       = %i[create update destroy].freeze
+
+  def initialize(event)
+    @event = event
+
+    validate!
+  end
+
+  def validate!
+    raise TableSync::EventError.new(event) unless event.in?(VALID_RAW_EVENTS)
+  end
+
+  def resolve
+    destroy? ? :destroy : :update
+  end
+
+  def metadata
+    { created: event == :create }
+  end
+
+  def destroy?
+    event == :destroy
+  end
+
+  def upsert?
+    event.in?(UPSERT_EVENTS)
+  end
+end

data/lib/table_sync/orm_adapter/active_record.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module TableSync::ORMAdapter
+  class ActiveRecord < Base
+    def find
+      @object = object_class.find_by(needle)
+
+      super
+    end
+
+    def init
+      @object = object_class.new(object_data)
+
+      super
+    end
+
+    def attributes
+      object.attributes.symbolize_keys
+    end
+
+    def self.model_naming(object_class)
+      TableSync::NamingResolver::ActiveRecord.new(table_name: object_class.table_name)
+    end
+  end
+end

data/lib/table_sync/orm_adapter/base.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+module TableSync::ORMAdapter
+  class Base
+    attr_reader :object, :object_class, :object_data
+
+    def initialize(object_class, object_data)
+      @object_class = object_class
+      @object_data  = object_data.symbolize_keys
+
+      validate!
+    end
+
+    # VALIDATE
+
+    def validate!
+      if (primary_key_columns - object_data.keys).any?
+        raise TableSync::NoPrimaryKeyError.new(object_class, object_data, primary_key_columns)
+      end
+    end
+
+    # FIND OR INIT OBJECT
+
+    def init
+      self
+    end
+
+    def find
+      self
+    end
+
+    def needle
+      object_data.slice(*primary_key_columns)
+    end
+
+    # ATTRIBUTES
+
+    def attributes_for_update
+      if object.respond_to?(:attributes_for_sync)
+        object.attributes_for_sync
+      else
+        attributes
+      end
+    end
+
+    def attributes_for_destroy
+      if object.respond_to?(:attributes_for_destroy)
+        object.attributes_for_destroy
+      else
+        needle
+      end
+    end
+
+    def attributes_for_routing_key
+      if object.respond_to?(:attributes_for_routing_key)
+        object.attributes_for_routing_key
+      else
+        attributes
+      end
+    end
+
+    def attributes_for_headers
+      if object.respond_to?(:attributes_for_headers)
+        object.attributes_for_headers
+      else
+        attributes
+      end
+    end
+
+    def primary_key_columns
+      Array.wrap(object_class.primary_key).map(&:to_sym)
+    end
+
+    # MISC
+
+    def empty?
+      object.nil?
+    end
+
+    # NOT IMPLEMENTED
+
+    # :nocov:
+    def attributes
+      raise NotImplementedError
+    end
+
+    def self.model_naming
+      raise NotImplementedError
+    end
+    # :nocov:
+  end
+end

data/lib/table_sync/orm_adapter/sequel.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module TableSync::ORMAdapter
+  class Sequel < Base
+    def attributes
+      object.values
+    end
+
+    def init
+      @object = object_class.new(object_data.except(*primary_key_columns))
+
+      @object.set_fields(needle, needle.keys)
+
+      super
+    end
+
+    def find
+      @object = object_class.find(needle)
+
+      super
+    end
+
+    def self.model_naming(object_class)
+      TableSync::NamingResolver::Sequel.new(
+        table_name: object_class.table_name, db: object_class.db,
+      )
+    end
+  end
+end

data/lib/table_sync/publishing/batch.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+class TableSync::Publishing::Batch
+  include Tainbox
+
+  attribute :object_class
+  attribute :original_attributes
+
+  attribute :routing_key
+  attribute :headers
+
+  attribute :event, default: :update
+
+  def publish_later
+    job.perform_later(job_attributes)
+  end
+
+  def publish_now
+    message.publish
+  end
+
+  def message
+    TableSync::Publishing::Message::Batch.new(attributes)
+  end
+
+  alias_method :publish_async, :publish_later
+
+  private
+
+  # JOB
+
+  def job
+    if TableSync.batch_publishing_job_class_callable
+      TableSync.batch_publishing_job_class_callable.call
+    else
+      raise TableSync::NoCallableError.new("batch_publishing_job_class")
+    end
+  end
+
+  def job_attributes
+    attributes.merge(
+      original_attributes: serialized_original_attributes,
+    )
+  end
+
+  def serialized_original_attributes
+    original_attributes.map do |set_of_attributes|
+      TableSync::Publishing::Helpers::Attributes
+        .new(set_of_attributes)
+        .serialize
+    end
+  end
+end

data/lib/table_sync/publishing/data/objects.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module TableSync::Publishing::Data
+  class Objects
+    attr_reader :objects, :event
+
+    def initialize(objects:, event:)
+      @objects = objects
+      @event   = TableSync::Event.new(event)
+    end
+
+    def construct
+      {
+        model: model,
+        attributes: attributes_for_sync,
+        version: version,
+        event: event.resolve,
+        metadata: event.metadata,
+      }
+    end
+
+    private
+
+    def model
+      if object_class.respond_to?(:table_sync_model_name)
+        object_class.table_sync_model_name
+      else
+        object_class.name
+      end
+    end
+
+    def version
+      Time.current.to_f
+    end
+
+    def object_class
+      objects.first.object_class
+    end
+
+    def attributes_for_sync
+      objects.map do |object|
+        if event.destroy?
+          object.attributes_for_destroy
+        else
+          object.attributes_for_update
+        end
+      end
+    end
+  end
+end

data/lib/table_sync/publishing/data/raw.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module TableSync::Publishing::Data
+  class Raw
+    attr_reader :object_class, :attributes_for_sync, :event
+
+    def initialize(object_class:, attributes_for_sync:, event:)
+      @object_class = object_class
+      @attributes_for_sync = attributes_for_sync
+      @event = TableSync::Event.new(event)
+    end
+
+    def construct
+      {
+        model: object_class,
+        attributes: attributes_for_sync,
+        version: version,
+        event: event.resolve,
+        metadata: event.metadata,
+      }
+    end
+
+    def version
+      Time.current.to_f
+    end
+  end
+end

data/lib/table_sync/publishing/helpers/attributes.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module TableSync::Publishing::Helpers
+  class Attributes
+    BASE_SAFE_JSON_TYPES = [
+      NilClass,
+      String,
+      TrueClass,
+      FalseClass,
+      Numeric,
+      Symbol,
+    ].freeze
+
+    # add custom seializables?
+
+    NOT_MAPPED = Object.new
+
+    attr_reader :attributes
+
+    def initialize(attributes)
+      @attributes = attributes.deep_symbolize_keys
+    end
+
+    def serialize
+      filter_safe_for_serialization(attributes)
+    end
+
+    def filter_safe_for_serialization(object)
+      case object
+      when Array
+        object.each_with_object([]) do |value, memo|
+          value = filter_safe_for_serialization(value)
+          memo << value if object_mapped?(value)
+        end
+      when Hash
+        object.each_with_object({}) do |(key, value), memo|
+          key = filter_safe_for_serialization(key)
+          value = filter_safe_hash_values(value)
+          memo[key] = value if object_mapped?(key) && object_mapped?(value)
+        end
+      when Float::INFINITY
+        NOT_MAPPED
+      when *BASE_SAFE_JSON_TYPES
+        object
+      else # rubocop:disable Lint/DuplicateBranch
+        NOT_MAPPED
+      end
+    end
+
+    def filter_safe_hash_values(value)
+      case value
+      when Symbol
+        value.to_s # why?
+      else
+        filter_safe_for_serialization(value)
+      end
+    end
+
+    def object_mapped?(object)
+      object != NOT_MAPPED
+    end
+  end
+end