table_sync 2.3.0 → 4.0.0

data/docs/development.md

@@ -1,43 +0,0 @@
-# TableSync (development)
-
-## Table of Contents
-
-- [Creation and registration of the new plugin](#creation-and-registration-of-the-new-plugin)
-
-### Creation and registration of the new plugin
-
-* Create a class inherited from `TableSync::Plugins::Abstract` (recommendation: place it in the plugins directory (`lib/plugins/`));
-* Implement `.install!` method (`TableSync::Plugins::Abstract.install!`)
-* Register the newly created class in plugins ecosystem (`TableSync.register_plugin('plugin_name', PluginClass))`);
-* Usage: `TableSync.enable(:plugin_name)` / `TableSync.plugin(:plugin_name)` / `TableSync.load(:plugin_name)` (string name is supported too);
-
-Example:
-
-```ruby
-# 1) creation (lib/plugins/global_method.rb)
-class TableSync::Plugins::GlobalMethod < TableSync::Plugins::Abstract
-  class << self
-    # 2) plugin loader method implementation
-    def install!
-      ::TableSync.extend(Module.new do
-        def global_method
-          :works!
-        end
-      end)
-    end
-  end
-end
-
-# 3) plugin registration
-TableSync.register('global_method', TableSync::Plugins::GlobalMethod)
-
-# 4) enable registerd plugin
-TableSync.plugin(:global_method) # string is supported too
-# --- or ---
-TableSync.enable(:global_method) # string is supported too
-# --- or ---
-TableSync.load(:global_method) # string is supported too
-
-# Your new functionality
-TableSync.global_method # => :works!
-```

data/docs/synopsis.md

@@ -1,336 +0,0 @@
-# TableSync
-
-Table synchronization via RabbitMQ
-
-# Publishing changes
-
-Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. `:if` and `:unless` are
-supported for Sequel and ActiveRecord
-
-Functioning `Rails.cache` is required
-
-Example:
-
-```ruby
-class SomeModel < Sequel::Model
-    TableSync.sync(self, { if: -> (*) { some_code } })
-end
-```
-
-#### #attributes_for_sync
-
-Models can implement `#attributes_for_sync` to override which attributes are published. If not
-present, all attributes are published
-
-#### #attrs_for_routing_key
-
-Models can implement `#attrs_for_routing_key` to override which attributes are given to routing_key_callable. If not present, default attributes are given
-
-#### #attrs_for_metadata
-
-Models can implement `#attrs_for_metadata` to override which attributes are given to metadata_callable. If not present, default attributes are given
-
-#### .table_sync_model_name
-
-Models can implement `.table_sync_model_name` class method to override the model name used for
-publishing events. Default is model class name
-
-#### .table_sync_destroy_attributes(original_attributes)
-
-Models can implement `.table_sync_destroy_attributes` class method to override the attributes
-used for publishing destroy events. Default is object's primary key
-
-## Configuration
-
-- `TableSync.publishing_job_class_callable` is a callable which should resolve to a ActiveJob
-subclass that calls TableSync back to actually publish changes (required)
-
-Example:
-
-```ruby
-class TableSync::Job < ActiveJob::Base
-  def perform(*args)
-    TableSync::Publisher.new(*args).publish_now
-  end
-end
-```
-
-- `TableSync.batch_publishing_job_class_callable` is a callable which should resolve to a ActiveJob
-subclass that calls TableSync batch publisher back to actually publish changes (required for batch publisher)
-
-- `TableSync.routing_key_callable` is a callable which resolves which routing key to use when
-publishing changes. It receives object class and attributes (required)
-
-Example:
-
-```ruby
-TableSync.routing_key_callable = -> (klass, attributes) { klass.gsub('::', '_').tableize }
-```
-
-- `TableSync.routing_metadata_callable` is a callable that adds RabbitMQ headers which can be
-used in routing (optional). One possible way of using it is defining a headers exchange and
-routing rules based on key-value pairs (which correspond to sent headers)
-
-Example:
-
-```ruby
-TableSync.routing_metadata_callable = -> (klass, attributes) { attributes.slice("project_id") }
-```
-
-- `TableSync.exchange_name` defines the exchange name used for publishing (optional, falls back
-to default Rabbit gem configuration).
-
-- `TableSync.notifier` is a module that provides publish and recieve notifications.
-
-# Manual publishing
-
-`TableSync::Publisher.new(object_class, original_attributes, confirm: true, state: :updated, debounce_time: 45)`
-where state is one of `:created / :updated / :destroyed` and `confirm` is Rabbit's confirm delivery flag and optional param `debounce_time` determines debounce time in seconds, 1 minute by default.
-
-# Manual publishing with batches
-
-You can use `TableSync::BatchPublisher` to publish changes in batches (array of hashes in `attributes`).
-
-When using `TableSync::BatchPublisher`,` TableSync.routing_key_callable` is called as follows:
-`TableSync.routing_key_callable.call(klass, {})`, i.e. empty hash is passed instead of attributes.
-And `TableSync.routing_metadata_callable` is not called at all: metadata is set to empty hash.
-
-`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, where `original_attributes_array` is an array with hash of attributes of published objects and `options` is a hash of options.
-
-`options` consists of:
-- `confirm`, which is a flag for RabbitMQ, `true` by default
-- `routing_key`, which is a custom key used (if given) to override one from `TableSync.routing_key_callable`, `nil` by default
-- `push_original_attributes` (default value is `false`), if this option is set to `true`,
-original_attributes_array will be pushed to Rabbit instead of fetching records from database and sending their mapped attributes.
-- `headers`, which is an option for custom headers (can be used for headers exchanges routes), `nil` by default
-- `event`, which is an option for event specification (`:destroy` or `:update`), `:update` by default
-
-Example:
-
-```ruby
-TableSync::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
-```
-
-# Manual publishing with batches (Russian)
-
-С помощью класса `TableSync::BatchPublisher` вы можете опубликовать изменения батчами (массивом в `attributes`).
-
-При использовании `TableSync::BatchPublisher`, `TableSync.routing_key_callable` вызывается следующим образом:
-`TableSync.routing_key_callable.call(klass, {})`, то есть вместо аттрибутов передается пустой хэш.
-А `TableSync.routing_metadata_callable` не вызывается вовсе: в метадате устанавливается пустой хэш.
-
-`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, где `original_attributes_array` - массив с аттрибутами публикуемых объектов и `options`- это хэш с дополнительными опциями.
-
-`options` состоит из:
-- `confirm`, флаг для RabbitMQ, по умолчанию - `true`
-- `routing_key`, ключ, который (если указан) замещает ключ, получаемый из `TableSync.routing_key_callable`, по умолчанию - `nil`
-- `push_original_attributes` (значение по умолчанию `false`), если для этой опции задано значение true, в Rabbit будут отправлены original_attributes_array, вместо получения значений записей из базы непосредственно перед отправкой.
-- `headers`, опция для задания headers (можно использовать для задания маршрутов в headers exchange'ах), `nil` по умолчанию
-- `event`, опция для указания типа события (`:destroy` или `:update`), `:update` по умолчанию
-
-Example:
-
-```ruby
-TableSync::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
-```
-
-# Receiving changes
-
-Naming convention for receiving handlers is `Rabbit::Handler::GROUP_ID::TableSync`,
-where `GROUP_ID` represents first part of source exchange name.
-Define handler class inherited from `TableSync::ReceivingHandler`
-and named according to described convention.
-You should use DSL inside the class.
-Suppose we will synchronize models {Project, News, User} project {MainProject}, then:
-
-```ruby
-class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
-  queue_as :custom_queue
-
-  receive "Project", to_table: :projects
-
-  receive "News", to_table: :news, events: :update do
-    after_commit on: :update do
-      NewsCache.reload
-    end
-  end
-
-  receive "User", to_table: :clients, events: %i[update destroy] do
-    mapping_overrides email: :project_user_email, id: :project_user_id
-
-    only :project_user_email, :project_user_id
-    target_keys :project_id, :project_user_id
-    rest_key :project_user_rest
-    version_key :project_user_version
-
-    additional_data do |project_id:|
-      { project_id: project_id }
-    end
-
-    default_values do
-      { created_at: Time.current }
-    end
-  end
-
-  receive "User", to_table: :users do
-    rest_key nil
-  end
-end
-```
-
-### Handler class (`Rabbit::Handler::MainProject::TableSync`)
-
-In this case:
-- `TableSync` - RabbitMQ event type.
-- `MainProject` - event source.
-- `Rabbit::Handler` - module for our handlers of events from RabbitMQ (there might be others)
-
-Method `queue_as` allow you to set custom queue.
-
-### Recieving handler batch processing
-
-Receiving handler supports array of attributes in a single update event. Corresponding
-upsert-style logic in ActiveRecord and Sequel orm handlers is provided.
-
-### Config DSL
-```ruby
-receive source, to_table:, [events:, &block]
-```
-
-The method receives following arguments
-- `source` - string, name of source model (required)
-- `to_table` - destination_table hash (required)
-- `events` - array of supported events (optional)
-- `block` - configuration block (optional)
-
-This method implements logic of mapping `source` to `to_table` and allows customizing the event handling logic with provided block.
-You can use one `source` for a lot of `to_table`.
-
-The following options are available inside the block:
-- `on_destroy` - defines a custom logic and behavior for `destroy` event:
-  - definition:
-    ```ruby
-      on_destroy do |attributes:, target_keys:|
-        # your code here
-      end
-    ```
-  - `target_keys:` - primary keys or unique keys;
-  - `attributes:` - received model attributes;
-- `only` - whitelist for receiving attributes
-- `skip` - return truthy value to skip the row
-- `target_keys` - primary keys or unique keys
-- `rest_key` - name of jsonb column for attributes which are not included in the whitelist. You can set the `rest_key(false)` or `rest_key(nil)` if you won't need the rest data.
-- `version_key` - name of version column
-- `first_sync_time_key` - name of the column where the time of first record synchronization should be stored. Disabled by default.
-- `mapping_overrides` - map for overriding receiving columns
-- `additional_data` - additional data for insert or update (e.g. `project_id`)
-- `default_values` - values for insert if a row is not found
-- `partitions` - proc that is used to obtain partitioned data to support table partitioning. Must return a hash which
-  keys are names of partitions of partitioned table and values - arrays of attributes to be inserted into particular
-  partition `{ measurements_2018_01: [ { attrs }, ... ], measurements_2018_02: [ { attrs }, ... ], ...}`.
-  While the proc is called inside an upsert transaction it is suitable place for creating partitions for new data.
-  Note that transaction of proc is a TableSynk.orm transaction.
-  ```ruby
-  partitions do |data:|
-    data.group_by { |d| "measurements_#{d[:time].year}_#{d[:time].month}" }
-        .tap { |data| data.keys.each { |table| DB.run("CREATE TABLE IF NOT EXISTS #{table} PARTITION OF measurements") } }
-  end
-  ```
-- `wrap_reciving` - proc that is used to wrap the receiving logic by custom block of code. Receives `data` and `receiving` attributes
-  (received event data and receiving logic proc respectively). `receiving.call` runs receiving process (you should use it manually).
-    - example (concurrent receiving):
-    ```ruby
-      wrap_receiving do |data, receiving|
-        Locking.acquire("some-lock-key") { receiving.call }
-      end
-    ```
-  - `data` attribute:
-    - for `destroy` event - an instance of `TableSync::EventActions::DataWrapper::Destroy`;
-    - for `update` event - an instance of `TableSync::EventActions::DataWrapper::Update`;
-    - `#event_data` - raw recevied event data:
-      - for `destroy` event - simple `Hash`;
-      - for `update` event - `Hash` with `Hash<ModelKlass, Array<Hash<Symbol, Object>>>` signature;
-    - `#destroy?` / `#update?` - corresponding predicates;
-    - `#type` - indicates a type of data (`:destroy` and `:update` respectively);
-    - `#each` - iterates over `#event_data` elements (acts like an iteration over an array of elements);
-
-Each of options can receive static value or code block which will be called for each event with the following arguments:
-- `event` - type of event (`:update` or `:destroy`)
-- `model` - source model (`Project`, `News`, `User` in example)
-- `version` - version of the data
-- `project_id` - id of project which is used in RabbitMQ
-- `data` - raw data from event (before applying `mapping_overrides`, `only`, etc.)
-
-Also, the `additional_data`, `skip` has a `current_row` field, which gives you a hash of all parameters of the current row (useful when receiving changes in batches).
-
-Block can receive any number of parameters from the list.
-
-### Callbacks
-You can set callbacks like this:
-```ruby
-before_commit on: event, &block
-after_commit on: event, &block
-```
-TableSync performs this callbacks after transaction commit as to avoid side effects. Block receives array of record attributes.
-
-### Notifications
-
-#### ActiveSupport adapter
-
-You can use an already existing ActiveSupport adapter:
-```ruby
-  TableSync.notifier = TableSync::InstrumentAdapter::ActiveSupport
-```
-
-This instrumentation API is provided by Active Support. It allows to subscribe to notifications:
-
-```ruby
-ActiveSupport::Notifications.subscribe(/tablesync/) do |name, start, finish, id, payload|
-  # do something
-end
-```
-
-Types of events available:
-`"tablesync.receive.update"`, `"tablesync.receive.destroy"`, `"tablesync.publish.update"`
-and `"tablesync.publish.destroy"`.
-
-You have access to the payload, which contains  `event`, `direction`, `table`, `schema` and `count`.
-
-```
-{
-  :event => :update,       # one of update / destroy
-  :direction => :publish,  # one of publish / receive
-  :table => "users",
-  :schema => "public",
-  :count => 1
-}
-```
-
- See more at https://guides.rubyonrails.org/active_support_instrumentation.html
-
-
-#### Custom adapters
-
-You can also create a custom adapter. It is expected to respond to the following method:
-
-```ruby
-  def notify(table:, event:, direction:, count:)
-    # processes data about table_sync event
-  end
-```

data/lib/table_sync/config.rb

@@ -1,105 +0,0 @@
-# frozen_string_literal: true
-
-module TableSync
-  class Config
-    attr_reader :model, :events, :callback_registry
-
-    def initialize(model:, events: nil)
-      @model = model
-      @events = events.nil? ? nil : [events].flatten.map(&:to_sym)
-
-      @callback_registry = CallbackRegistry.new
-
-      # initialize default options
-      only(model.columns)
-      mapping_overrides({})
-      additional_data({})
-      default_values({})
-      @rest_key = :rest
-      @version_key = :version
-      @first_sync_time_key = nil
-      @on_destroy = nil
-      @wrap_receiving = nil
-      target_keys(model.primary_keys)
-    end
-
-    # add_option implements the following logic
-    # config.option - get value
-    # config.option(args) - set static value
-    # config.option { ... } - set proc as value
-    def self.add_option(name, &option_block)
-      ivar = "@#{name}".to_sym
-
-      # default option handler (empty handler)
-      # handles values when setting options
-      option_block ||= proc { |value| value }
-
-      define_method(name) do |*args, &block|
-        # logic for each option
-        if args.empty? && block.nil? # case for config.option, just return ivar
-          instance_variable_get(ivar)
-        elsif block # case for config.option { ... }
-          # get named parameters (parameters with :keyreq) from proc-value
-          params = block.parameters.map { |param| param[0] == :keyreq ? param[1] : nil }.compact
-
-          # wrapper for proc-value, this wrapper receives all params (model, version, project_id...)
-          # and filters them for proc-value
-          unified_block = proc { |hash = {}| block.call(**hash.slice(*params)) }
-
-          # set wrapped proc-value as ivar value
-          instance_variable_set(ivar, unified_block)
-        else # case for config.option(args)
-          # call option_block with args as params and set ivar to result
-          instance_variable_set(ivar, instance_exec(*args, &option_block))
-        end
-      end
-    end
-
-    def allow_event?(name)
-      return true if events.nil?
-      events.include?(name)
-    end
-
-    def before_commit(on:, &block)
-      callback_registry.register_callback(block, kind: :before_commit, event: on.to_sym)
-    end
-
-    def after_commit(on:, &block)
-      callback_registry.register_callback(block, kind: :after_commit, event: on.to_sym)
-    end
-
-    def on_destroy(&block)
-      block_given? ? @on_destroy = block : @on_destroy
-    end
-
-    def wrap_receiving(&block)
-      block_given? ? @wrap_receiving = block : @wrap_receiving
-    end
-
-    check_and_set_column_key = proc do |key|
-      key = key.to_sym
-      raise "#{model.inspect} doesn't have key: #{key}" unless model.columns.include?(key)
-      key
-    end
-
-    set_column_keys = proc do |*keys|
-      [keys].flatten.map { |k| instance_exec(k, &check_and_set_column_key) }
-    end
-
-    add_option(:only, &set_column_keys)
-    add_option(:target_keys, &set_column_keys)
-    add_option(:rest_key) do |value|
-      value ? instance_exec(value, &check_and_set_column_key) : nil
-    end
-    add_option(:version_key, &check_and_set_column_key)
-    add_option(:first_sync_time_key) do |value|
-      value ? instance_exec(value, &check_and_set_column_key) : nil
-    end
-
-    add_option(:mapping_overrides)
-    add_option(:additional_data)
-    add_option(:default_values)
-    add_option(:partitions)
-    add_option(:skip)
-  end
-end