table_sync 2.3.0 → 4.0.0

data/docs/receiving.md

@@ -0,0 +1,341 @@
+# Receiving changes
+
+Naming convention for receiving handlers is `Rabbit::Handler::GROUP_ID::TableSync`,
+where `GROUP_ID` represents first part of source exchange name.
+Define handler class inherited from `TableSync::ReceivingHandler`
+and named according to described convention.
+You should use DSL inside the class.
+Suppose we will synchronize models {Project, News, User} project {MainProject}, then:
+
+```ruby
+class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
+  queue_as :custom_queue
+
+  receive "Project", to_table: :projects
+
+  receive "News", to_table: :news, events: :update do
+    after_commit_on_update do
+      NewsCache.reload
+    end
+  end
+
+  receive "User", to_table: :clients, events: %i[update destroy] do
+    mapping_overrides email: :project_user_email, id: :project_user_id
+
+    only :project_user_email, :project_user_id
+    target_keys :project_id, :project_user_id
+    rest_key :project_user_rest
+    version_key :project_user_version
+
+    additional_data do |project_id:|
+      { project_id: project_id }
+    end
+
+    default_values do
+      { created_at: Time.current }
+    end
+  end
+
+  receive "User", to_model: CustomModel.new(:users) do
+    rest_key nil
+  end
+end
+```
+
+### Handler class (`Rabbit::Handler::MainProject::TableSync`)
+
+In this case:
+- `TableSync` - RabbitMQ event type.
+- `MainProject` - event source.
+- `Rabbit::Handler` - module for our handlers of events from RabbitMQ (there might be others)
+
+Method `queue_as` allows you to set custom queue.
+
+### Recieving handler batch processing
+
+Receiving handler supports array of attributes in a single update or destroy event. Corresponding
+upsert-style logic in ActiveRecord and Sequel orm handlers are provided.
+
+### Config
+```ruby
+receive source, [to_table:, to_model:, events:, &block]
+```
+
+The method receives following arguments
+- `source` - string, name of source model (required)
+- `to_table` - destination table name (required if not set to_model)
+- `to_model` - destination model (required if not set to_table)
+- `events` - array of supported events (optional)
+- `block` - configuration block with options (optional)
+
+This method implements logic of mapping `source` to `to_table` (or to `to_model`) and allows customizing 
+the event handling logic with provided block.
+You can use one `source` for a lot of `to_table` or `to_moel`.
+
+### Options:
+
+Most of the options can be set as computed value or as a process.
+
+```ruby
+option(value)
+```
+
+```ruby
+option do |key params|
+  value
+end
+```
+
+Each of options can receive static value or code block which will be called for each event with the following arguments:
+- `event` - type of event (`:update` or `:destroy`)
+- `model` - source model (`Project`, `News`, `User` in example)
+- `version` - version of the data
+- `project_id` - id of project which is used in RabbitMQ
+- `raw_data` - raw data from event (before applying `mapping_overrides`, `only`, etc.)
+
+Blocks can receive any number of parameters from the list.
+
+All specific key params will be explained in examples for each option.
+
+#### only
+Whitelist for receiving attributes.
+
+```ruby
+only(instance of Array)
+```
+
+```ruby
+only do |row:|
+  return instance of Array
+end
+```
+
+default value is taken through the call `model.columns`
+
+#### target_keys
+Primary keys or unique keys.
+
+```ruby
+target_keys(instance of Array)
+```
+
+```ruby
+target_keys do |data:|
+  return instance of Array
+end
+```
+
+default value is taken through the call `model.primary_keys`
+
+#### rest_key
+Name of jsonb column for attributes which are not included in the whitelist.
+You can set the `rest_key(false)` if you won't need the rest data.
+
+```ruby
+rest_key(instance of Symbol)
+```
+
+```ruby
+rest_key do |row:, rest:|
+  return instance of Symbol
+end
+```
+default value is `:rest`
+
+#### version_key
+Name of version column.
+
+```ruby
+version_key(instance of Symbol)
+```
+
+```ruby
+version_key do |data:|
+  return instance of Symbol
+end
+```
+default value is `:version`
+
+#### except
+Blacklist for receiving attributes.
+
+```ruby
+except(instance of Array)
+```
+
+```ruby
+except do |row:|
+  return instance of Array
+end
+```
+
+default value is `[]`
+
+#### mapping_overrides
+Map for overriding receiving columns.
+
+```ruby
+mapping_overrides(instance of Hash)
+```
+
+```ruby
+mapping_overrides do |row:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### additional_data
+Additional data for insert or update (e.g. `project_id`).
+
+```ruby
+additional_data(instance of Hash)
+```
+
+```ruby
+additional_data do |row:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### default_values
+Values for insert if a row is not found.
+
+```ruby
+default_values(instance of Hash)
+```
+
+```ruby
+default_values do |data:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### skip
+Return truthy value to skip the row.
+
+```ruby
+skip(instance of TrueClass or FalseClass)
+```
+
+```ruby
+skip do |data:|
+  return instance of TrueClass or FalseClass
+end
+```
+
+default value is `false`
+
+#### wrap_receiving
+Proc that is used to wrap the receiving logic by custom block of code.
+
+```ruby
+wrap_receiving do |data:, target_keys:, version_key:, default_values: {}, &receiving_logic|
+  receiving_logic.call
+  return makes no sense
+end
+```
+
+default value is `proc { |&block| block.call }`
+
+#### before_update
+Perform code before updating data in the database.
+
+```ruby
+before_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+
+before_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### after_commit_on_update
+Perform code after updated data was committed.
+
+```ruby
+after_commit_on_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+
+after_commit_on_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### before_destroy
+Perform code before destroying data in database.
+
+```ruby
+before_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+
+before_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### after_commit_on_destroy
+Perform code after destroyed data was committed.
+
+```ruby
+after_commit_on_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+
+after_commit_on_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+### Custom model
+You can use custom model for receiving.
+```
+class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
+  receive "Project", to_model: CustomModel.new
+end
+```
+
+This model has to implement next interface:
+```
+def columns
+  return all columns from table
+end
+
+def primary_keys
+  return primary keys from table
+end
+
+def upsert(data: Array, target_keys: Array, version_key: Symbol, default_values: Hash)
+  return array with updated rows
+end
+
+def destroy(data: Array, target_keys: Array, version_key: Symbol)
+  return array with delited rows
+end
+
+def transaction(&block)
+  block.call
+  return makes no sense
+end
+
+def after_commit(&block)
+  block.call
+  return makes no sense
+end
+```

data/lib/table_sync.rb

@@ -1,66 +1,51 @@
 # frozen_string_literal: true
 
 require "memery"
+require "self_data"
 require "rabbit_messaging"
 require "rabbit/event_handler" # NOTE: from rabbit_messaging"
 require "active_support/core_ext/object/blank"
 require "active_support/core_ext/numeric/time"
 
 module TableSync
+  require_relative "table_sync/utils"
   require_relative "table_sync/version"
   require_relative "table_sync/errors"
-  require_relative "table_sync/plugins"
-  require_relative "table_sync/event_actions"
-  require_relative "table_sync/event_actions/data_wrapper"
-  require_relative "table_sync/config"
-  require_relative "table_sync/config/callback_registry"
-  require_relative "table_sync/config_decorator"
-  require_relative "table_sync/dsl"
-  require_relative "table_sync/receiving_handler"
-  require_relative "table_sync/base_publisher"
-  require_relative "table_sync/publisher"
-  require_relative "table_sync/batch_publisher"
-  require_relative "table_sync/orm_adapter/active_record"
-  require_relative "table_sync/orm_adapter/sequel"
-  require_relative "table_sync/model/active_record"
-  require_relative "table_sync/model/sequel"
   require_relative "table_sync/instrument"
   require_relative "table_sync/instrument_adapter/active_support"
   require_relative "table_sync/naming_resolver/active_record"
   require_relative "table_sync/naming_resolver/sequel"
-
-  # @api public
-  # @since 2.2.0
-  extend Plugins::AccessMixin
+  require_relative "table_sync/receiving"
+  require_relative "table_sync/publishing"
 
   class << self
-    include Memery
-
     attr_accessor :publishing_job_class_callable
     attr_accessor :batch_publishing_job_class_callable
     attr_accessor :routing_key_callable
     attr_accessor :exchange_name
     attr_accessor :routing_metadata_callable
     attr_accessor :notifier
+    attr_reader :orm
+    attr_reader :publishing_adapter
+    attr_reader :receiving_model
 
     def sync(klass, **opts)
-      orm.setup_sync(klass, opts)
+      publishing_adapter.setup_sync(klass, opts)
     end
 
     def orm=(val)
-      clear_memery_cache!
-      @orm = val
-    end
-
-    memoize def orm
-      case @orm
+      case val
       when :active_record
-        ORMAdapter::ActiveRecord
+        @publishing_adapter = Publishing::ORMAdapter::ActiveRecord
+        @receiving_model = Receiving::Model::ActiveRecord
       when :sequel
-        ORMAdapter::Sequel
+        @publishing_adapter = Publishing::ORMAdapter::Sequel
+        @receiving_model = Receiving::Model::Sequel
       else
-        raise "ORM not supported: #{@orm.inspect}"
+        raise ORMNotSupported.new(val.inspect)
       end
+
+      @orm = val
     end
   end
 end

data/lib/table_sync/errors.rb

@@ -4,37 +4,23 @@ module TableSync
   Error = Class.new(StandardError)
 
   class UpsertError < Error
-    def initialize(data:, target_keys:, version_key:, first_sync_time_key:, default_values:)
-      super <<~MSG
-        Upsert has changed more than 1 row;
-          data: #{data.inspect}
-          target_keys: #{target_keys.inspect}
-          version_key: #{version_key.inspect}
-          first_sync_time_key: #{first_sync_time_key.inspect}
-          default_values: #{default_values.inspect}
-      MSG
-    end
-  end
-
-  class UndefinedConfig < Error
-    def initialize(model)
-      super("Config not defined for model; model: #{model.inspect}")
+    def initialize(data:, target_keys:, result:)
+      super("data: #{data.inspect}, target_keys: #{target_keys.inspect}, result: #{result.inspect}")
     end
   end
 
   class DestroyError < Error
-    def initialize(data)
-      super("Destroy has changed more than 1 row; data: #{data.inspect}")
+    def initialize(data:, target_keys:, result:)
+      super("data: #{data.inspect}, target_keys: #{target_keys.inspect}, result: #{result.inspect}")
     end
   end
 
-  class UnprovidedEventTargetKeysError < Error
-    # @param target_keys [Array<Symbol,String>]
-    # @param target_attributes [Hash<Symbol|String,Any>]
-    def initialize(target_keys, target_attributes)
+  class DataError < Error
+    def initialize(data, target_keys, description)
       super(<<~MSG.squish)
-        Some target keys not found in received attributes!
-        (Expects: #{target_keys}, Actual: #{target_attributes.keys})
+        #{description}
+        target_keys: #{target_keys}
+        data: #{data}
       MSG
     end
   end
@@ -60,4 +46,34 @@ module TableSync
       super("#{plugin_name} plugin already exists")
     end
   end
+
+  class InterfaceError < Error
+    def initialize(object, method_name, parameters, description)
+      parameters = parameters.map do |parameter|
+        type, name = parameter
+
+        case type
+        when :req
+          name.to_s
+        when :keyreq
+          "#{name}:"
+        when :block
+          "&#{name}"
+        end
+      end
+
+      signature = "#{method_name}(#{parameters.join(", ")})"
+
+      super("#{object} has to implement method `#{signature}`\n#{description}")
+    end
+  end
+
+  UndefinedEvent = Class.new(Error)
+  ORMNotSupported = Class.new(Error)
+
+  class WrongOptionValue < Error
+    def initialize(model, option, value)
+      super("TableSync config for #{model.inspect} can't contain #{value.inspect} as #{option}")
+    end
+  end
 end