table_sync 1.13.1 → 4.0.0

data/docs/notifications.md

@@ -0,0 +1,45 @@
+### Notifications
+
+#### ActiveSupport adapter
+
+You can use an already existing ActiveSupport adapter:
+```ruby
+  TableSync.notifier = TableSync::InstrumentAdapter::ActiveSupport
+```
+
+This instrumentation API is provided by Active Support. It allows to subscribe to notifications:
+
+```ruby
+ActiveSupport::Notifications.subscribe(/tablesync/) do |name, start, finish, id, payload|
+  # do something
+end
+```
+
+Types of events available:
+`"tablesync.receive.update"`, `"tablesync.receive.destroy"`, `"tablesync.publish.update"`
+and `"tablesync.publish.destroy"`.
+
+You have access to the payload, which contains  `event`, `direction`, `table`, `schema` and `count`.
+
+```
+{
+  :event => :update,       # one of update / destroy
+  :direction => :publish,  # one of publish / receive
+  :table => "users",
+  :schema => "public",
+  :count => 1
+}
+```
+
+ See more at https://guides.rubyonrails.org/active_support_instrumentation.html
+
+
+#### Custom adapters
+
+You can also create a custom adapter. It is expected to respond to the following method:
+
+```ruby
+  def notify(table:, event:, direction:, count:)
+    # processes data about table_sync event
+  end
+```

data/docs/publishing.md

@@ -0,0 +1,147 @@
+# Publishing changes
+
+Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. `:if` and `:unless` are
+supported for Sequel and ActiveRecord
+
+Functioning `Rails.cache` is required
+
+Example:
+
+```ruby
+class SomeModel < Sequel::Model
+    TableSync.sync(self, { if: -> (*) { some_code } })
+end
+```
+
+#### #attributes_for_sync
+
+Models can implement `#attributes_for_sync` to override which attributes are published. If not
+present, all attributes are published
+
+#### #attrs_for_routing_key
+
+Models can implement `#attrs_for_routing_key` to override which attributes are given to routing_key_callable. If not present, default attributes are given
+
+#### #attrs_for_metadata
+
+Models can implement `#attrs_for_metadata` to override which attributes are given to metadata_callable. If not present, default attributes are given
+
+#### .table_sync_model_name
+
+Models can implement `.table_sync_model_name` class method to override the model name used for
+publishing events. Default is model class name
+
+#### .table_sync_destroy_attributes(original_attributes)
+
+Models can implement `.table_sync_destroy_attributes` class method to override the attributes
+used for publishing destroy events. Default is object's primary key
+
+## Configuration
+
+- `TableSync.publishing_job_class_callable` is a callable which should resolve to a ActiveJob
+subclass that calls TableSync back to actually publish changes (required)
+
+Example:
+
+```ruby
+class TableSync::Job < ActiveJob::Base
+  def perform(*args)
+    TableSync::Publisher.new(*args).publish_now
+  end
+end
+```
+
+- `TableSync.batch_publishing_job_class_callable` is a callable which should resolve to a ActiveJob
+subclass that calls TableSync batch publisher back to actually publish changes (required for batch publisher)
+
+- `TableSync.routing_key_callable` is a callable which resolves which routing key to use when
+publishing changes. It receives object class and attributes (required)
+
+Example:
+
+```ruby
+TableSync.routing_key_callable = -> (klass, attributes) { klass.gsub('::', '_').tableize }
+```
+
+- `TableSync.routing_metadata_callable` is a callable that adds RabbitMQ headers which can be
+used in routing (optional). One possible way of using it is defining a headers exchange and
+routing rules based on key-value pairs (which correspond to sent headers)
+
+Example:
+
+```ruby
+TableSync.routing_metadata_callable = -> (klass, attributes) { attributes.slice("project_id") }
+```
+
+- `TableSync.exchange_name` defines the exchange name used for publishing (optional, falls back
+to default Rabbit gem configuration).
+
+- `TableSync.notifier` is a module that provides publish and recieve notifications.
+
+# Manual publishing
+
+`TableSync::Publisher.new(object_class, original_attributes, confirm: true, state: :updated, debounce_time: 45)`
+where state is one of `:created / :updated / :destroyed` and `confirm` is Rabbit's confirm delivery flag and optional param `debounce_time` determines debounce time in seconds, 1 minute by default.
+
+# Manual publishing with batches
+
+You can use `TableSync::BatchPublisher` to publish changes in batches (array of hashes in `attributes`).
+
+When using `TableSync::BatchPublisher`,` TableSync.routing_key_callable` is called as follows:
+`TableSync.routing_key_callable.call(klass, {})`, i.e. empty hash is passed instead of attributes.
+And `TableSync.routing_metadata_callable` is not called at all: metadata is set to empty hash.
+
+`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, where `original_attributes_array` is an array with hash of attributes of published objects and `options` is a hash of options.
+
+`options` consists of:
+- `confirm`, which is a flag for RabbitMQ, `true` by default
+- `routing_key`, which is a custom key used (if given) to override one from `TableSync.routing_key_callable`, `nil` by default
+- `push_original_attributes` (default value is `false`), if this option is set to `true`,
+original_attributes_array will be pushed to Rabbit instead of fetching records from database and sending their mapped attributes.
+- `headers`, which is an option for custom headers (can be used for headers exchanges routes), `nil` by default
+- `event`, which is an option for event specification (`:destroy` or `:update`), `:update` by default
+
+Example:
+
+```ruby
+TableSync::BatchPublisher.new(
+  "SomeClass",
+  [{ id: 1 }, { id: 2 }],
+  confirm: false,
+  routing_key: "custom_routing_key",
+  push_original_attributes: true,
+  headers: { key: :value },
+  event: :destroy,
+)
+```
+
+# Manual publishing with batches (Russian)
+
+С помощью класса `TableSync::BatchPublisher` вы можете опубликовать изменения батчами (массивом в `attributes`).
+
+При использовании `TableSync::BatchPublisher`, `TableSync.routing_key_callable` вызывается следующим образом:
+`TableSync.routing_key_callable.call(klass, {})`, то есть вместо аттрибутов передается пустой хэш.
+А `TableSync.routing_metadata_callable` не вызывается вовсе: в метадате устанавливается пустой хэш.
+
+`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, где `original_attributes_array` - массив с аттрибутами публикуемых объектов и `options`- это хэш с дополнительными опциями.
+
+`options` состоит из:
+- `confirm`, флаг для RabbitMQ, по умолчанию - `true`
+- `routing_key`, ключ, который (если указан) замещает ключ, получаемый из `TableSync.routing_key_callable`, по умолчанию - `nil`
+- `push_original_attributes` (значение по умолчанию `false`), если для этой опции задано значение true, в Rabbit будут отправлены original_attributes_array, вместо получения значений записей из базы непосредственно перед отправкой.
+- `headers`, опция для задания headers (можно использовать для задания маршрутов в headers exchange'ах), `nil` по умолчанию
+- `event`, опция для указания типа события (`:destroy` или `:update`), `:update` по умолчанию
+
+Example:
+
+```ruby
+TableSync::BatchPublisher.new(
+  "SomeClass",
+  [{ id: 1 }, { id: 2 }],
+  confirm: false,
+  routing_key: "custom_routing_key",
+  push_original_attributes: true,
+  headers: { key: :value },
+  event: :destroy,
+)
+```

data/docs/receiving.md

@@ -0,0 +1,341 @@
+# Receiving changes
+
+Naming convention for receiving handlers is `Rabbit::Handler::GROUP_ID::TableSync`,
+where `GROUP_ID` represents first part of source exchange name.
+Define handler class inherited from `TableSync::ReceivingHandler`
+and named according to described convention.
+You should use DSL inside the class.
+Suppose we will synchronize models {Project, News, User} project {MainProject}, then:
+
+```ruby
+class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
+  queue_as :custom_queue
+
+  receive "Project", to_table: :projects
+
+  receive "News", to_table: :news, events: :update do
+    after_commit_on_update do
+      NewsCache.reload
+    end
+  end
+
+  receive "User", to_table: :clients, events: %i[update destroy] do
+    mapping_overrides email: :project_user_email, id: :project_user_id
+
+    only :project_user_email, :project_user_id
+    target_keys :project_id, :project_user_id
+    rest_key :project_user_rest
+    version_key :project_user_version
+
+    additional_data do |project_id:|
+      { project_id: project_id }
+    end
+
+    default_values do
+      { created_at: Time.current }
+    end
+  end
+
+  receive "User", to_model: CustomModel.new(:users) do
+    rest_key nil
+  end
+end
+```
+
+### Handler class (`Rabbit::Handler::MainProject::TableSync`)
+
+In this case:
+- `TableSync` - RabbitMQ event type.
+- `MainProject` - event source.
+- `Rabbit::Handler` - module for our handlers of events from RabbitMQ (there might be others)
+
+Method `queue_as` allows you to set custom queue.
+
+### Recieving handler batch processing
+
+Receiving handler supports array of attributes in a single update or destroy event. Corresponding
+upsert-style logic in ActiveRecord and Sequel orm handlers are provided.
+
+### Config
+```ruby
+receive source, [to_table:, to_model:, events:, &block]
+```
+
+The method receives following arguments
+- `source` - string, name of source model (required)
+- `to_table` - destination table name (required if not set to_model)
+- `to_model` - destination model (required if not set to_table)
+- `events` - array of supported events (optional)
+- `block` - configuration block with options (optional)
+
+This method implements logic of mapping `source` to `to_table` (or to `to_model`) and allows customizing 
+the event handling logic with provided block.
+You can use one `source` for a lot of `to_table` or `to_moel`.
+
+### Options:
+
+Most of the options can be set as computed value or as a process.
+
+```ruby
+option(value)
+```
+
+```ruby
+option do |key params|
+  value
+end
+```
+
+Each of options can receive static value or code block which will be called for each event with the following arguments:
+- `event` - type of event (`:update` or `:destroy`)
+- `model` - source model (`Project`, `News`, `User` in example)
+- `version` - version of the data
+- `project_id` - id of project which is used in RabbitMQ
+- `raw_data` - raw data from event (before applying `mapping_overrides`, `only`, etc.)
+
+Blocks can receive any number of parameters from the list.
+
+All specific key params will be explained in examples for each option.
+
+#### only
+Whitelist for receiving attributes.
+
+```ruby
+only(instance of Array)
+```
+
+```ruby
+only do |row:|
+  return instance of Array
+end
+```
+
+default value is taken through the call `model.columns`
+
+#### target_keys
+Primary keys or unique keys.
+
+```ruby
+target_keys(instance of Array)
+```
+
+```ruby
+target_keys do |data:|
+  return instance of Array
+end
+```
+
+default value is taken through the call `model.primary_keys`
+
+#### rest_key
+Name of jsonb column for attributes which are not included in the whitelist.
+You can set the `rest_key(false)` if you won't need the rest data.
+
+```ruby
+rest_key(instance of Symbol)
+```
+
+```ruby
+rest_key do |row:, rest:|
+  return instance of Symbol
+end
+```
+default value is `:rest`
+
+#### version_key
+Name of version column.
+
+```ruby
+version_key(instance of Symbol)
+```
+
+```ruby
+version_key do |data:|
+  return instance of Symbol
+end
+```
+default value is `:version`
+
+#### except
+Blacklist for receiving attributes.
+
+```ruby
+except(instance of Array)
+```
+
+```ruby
+except do |row:|
+  return instance of Array
+end
+```
+
+default value is `[]`
+
+#### mapping_overrides
+Map for overriding receiving columns.
+
+```ruby
+mapping_overrides(instance of Hash)
+```
+
+```ruby
+mapping_overrides do |row:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### additional_data
+Additional data for insert or update (e.g. `project_id`).
+
+```ruby
+additional_data(instance of Hash)
+```
+
+```ruby
+additional_data do |row:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### default_values
+Values for insert if a row is not found.
+
+```ruby
+default_values(instance of Hash)
+```
+
+```ruby
+default_values do |data:|
+  return instance of Hash
+end
+```
+
+default value is `{}`
+
+#### skip
+Return truthy value to skip the row.
+
+```ruby
+skip(instance of TrueClass or FalseClass)
+```
+
+```ruby
+skip do |data:|
+  return instance of TrueClass or FalseClass
+end
+```
+
+default value is `false`
+
+#### wrap_receiving
+Proc that is used to wrap the receiving logic by custom block of code.
+
+```ruby
+wrap_receiving do |data:, target_keys:, version_key:, default_values: {}, &receiving_logic|
+  receiving_logic.call
+  return makes no sense
+end
+```
+
+default value is `proc { |&block| block.call }`
+
+#### before_update
+Perform code before updating data in the database.
+
+```ruby
+before_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+
+before_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### after_commit_on_update
+Perform code after updated data was committed.
+
+```ruby
+after_commit_on_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+
+after_commit_on_update do |data:, target_keys:, version_key:, default_values:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### before_destroy
+Perform code before destroying data in database.
+
+```ruby
+before_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+
+before_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+#### after_commit_on_destroy
+Perform code after destroyed data was committed.
+
+```ruby
+after_commit_on_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+
+after_commit_on_destroy do |data:, target_keys:, version_key:|
+  return makes no sense
+end
+```
+
+Сan be defined several times. Execution order guaranteed.
+
+### Custom model
+You can use custom model for receiving.
+```
+class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
+  receive "Project", to_model: CustomModel.new
+end
+```
+
+This model has to implement next interface:
+```
+def columns
+  return all columns from table
+end
+
+def primary_keys
+  return primary keys from table
+end
+
+def upsert(data: Array, target_keys: Array, version_key: Symbol, default_values: Hash)
+  return array with updated rows
+end
+
+def destroy(data: Array, target_keys: Array, version_key: Symbol)
+  return array with delited rows
+end
+
+def transaction(&block)
+  block.call
+  return makes no sense
+end
+
+def after_commit(&block)
+  block.call
+  return makes no sense
+end
+```