table_sync 6.0.4 → 6.1.0

data/docs/publishing/publishers.md

@@ -1,162 +0,0 @@
-# Publishers
-
-There are three publishers you can use to send data.
-
-- `TableSync::Publishing::Single` - sends one row with initialization.
-- `TableSync::Publishing::Batch` - sends a batch of rows with initialization.
-- `TableSync::Publishing::Raw`  - sends raw data without checks.
-
-## Single
-
-`TableSync::Publishing::Single` - sends one row with initialization.
-
-This is a publisher called by `TableSync.sync(self)`.
-
-### Expected parameters:
-
-- `object_class` - class (model) used to initialize published object
-- `original_attributes` - attributes used to initialize `object_class` with
-- `debounce_time` - minimum allowed time between delayed publishings
-- `event` - type of event that happened to the published object (`create`, `update`, `destroy`); `update` by default
-
-### What it does (when uses `#publish_now`):
-- takes in the `original_attributes`, `object_class`, `event`
-- constantizes `object_class`
-- extracts the primary key (`needle`) of the `object_class` from the `original_attributes`
-- queries the database for the object with the `needle` (for `update` and `create`) or initializes the `object_class` with `original_attributes` (for `destroy`)
-- constructs routing_key using `routing_key_callable` and `#attributes_for_routing_key` (if defined)
-- constructs headers using `headers_callable` and `#attributes_for_headers` (if defined)
-- publishes Rabbit message (uses attributes from queried/initialized object as data)
-- sends notification (if set up)
-
-### What it does (when uses `#publish_later`):
-- takes in the `original_attributes`, `object_class`, `debounce_time`, `event`
-- serializes the `original_attributes`, silently filters out unserializable keys/values
-- enqueues (or skips) the job with the `serialized_original_attributes` to be performed in time according to debounce
-- job (if enqueued) calls `TableSync::Publishing::Single#publish_now` with `serialized_original_attributes` and the same `object_class`, `debounce_time`, `event`
-
-### Serialization
-
-Currently allowed key/values are:
-  `NilClass`, `String`, `TrueClass`, `FalseClass`, `Numeric`, `Symbol`.
-
-### Job
-
-Job is defined in `TableSync.single_publishing_job_class_callable` as a proc. Read more in [Configuration](docs/publishing/configuration.md).
-
-### Example #1 (send right now)
-
-```ruby
-  TableSync::Publishing::Single.new(
-    object_class: "User",
-    original_attributes: { id: 1, name: "Mark" },
-    debounce_time: 60, # useless for #publish _now, can be skipped
-    event: :create,
-  ).publish_now
-```
-
-### Example #2 (enqueue job)
-
-```ruby
-  TableSync::Publishing::Single.new(
-    object_class: "User",
-    original_attributes: { id: 1, name: "Mark" }, # will be serialized!
-    debounce_time: 60,
-    event: :update,
-  ).publish_later
-```
-
-## Batch
-
-- `TableSync::Publishing::Batch` - sends a batch of rows with initialization.
-
-### Expected parameters:
-
-- `object_class` - class (model) used to initialize published objects
-- `original_attributes` - array of attributes used to initialize `object_class` with
-- `event` - type of event that happened to the published objects (`create`, `update`, `destroy`); `update` by default
-- `routing_key` - custom routing_key
-- `headers` - custom headers
-
-### What it does (when uses `#publish_now`):
-- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
-- constantizes `object_class`
-- extracts primary keys (`needles`) of the `object_class` from the array of `original_attributes`
-- queries the database for the objects with `needles` (for `update` and `create`) or initializes the `object_class` with `original_attributes` (for `destroy`)
-- constructs routing_key using `routing_key_callable` (ignores `#attributes_for_routing_key`) or uses `routing_key` if given
-- constructs headers using `headers_callable` (ignores `#attributes_for_headers`)  or uses `headers` if given
-- publishes Rabbit message (uses attributes from queried/initialized objects as data)
-- sends notification (if set up)
-
-### What it does (when uses `#publish_later`):
-- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
-- serializes the array of `original_attributes`, silently filters out unserializable keys/values
-- enqueues the job with the `serialized_original_attributes` 
-- job calls `TableSync::Publishing::Batch#publish_now` with `serialized_original_attributes` and the same `object_class`, `event`, `routing_key`, `headers`
-
-### Serialization
-
-Currently allowed key/values are:
-  `NilClass`, `String`, `TrueClass`, `FalseClass`, `Numeric`, `Symbol`.
-
-### Job
-
-Job is defined in `TableSync.batch_publishing_job_class_callable` as a proc. Read more in [Configuration](docs/publishing/configuration.md).
-
-### Example #1 (send right now)
-
-```ruby
-  TableSync::Publishing::Batch.new(
-    object_class: "User",
-    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
-    event: :create,
-    routing_key: :custom_key,   # optional
-    headers: { type: "admin" }, # optional
-  ).publish_now
-```
-
-### Example #2 (enqueue job)
-
-```ruby
-  TableSync::Publishing::Batch.new(
-    object_class: "User",
-    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
-    event: :create,
-    routing_key: :custom_key,   # optional
-    headers: { type: "admin" }, # optional
-  ).publish_later
-```
-
-## Raw
-- `TableSync::Publishing::Raw` - sends raw data without checks.
-
-Be carefull with this publisher. There are no checks for the data sent.
-You can send anything.
-
-### Expected parameters:
-
-- `object_class` - model
-- `original_attributes` - raw data that will be sent
-- `event` - type of event that happened to the published objects (`create`, `update`, `destroy`); `update` by default
-- `routing_key` - custom routing_key
-- `headers` - custom headers
-
-### What it does (when uses `#publish_now`):
-- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
-- constantizes `object_class`
-- constructs routing_key using `routing_key_callable` (ignores `#attributes_for_routing_key`) or uses `routing_key` if given
-- constructs headers using `headers_callable` (ignores `#attributes_for_headers`)  or uses `headers` if given
-- publishes Rabbit message (uses `original_attributes` as is)
-- sends notification (if set up)
-
-### Example
-
-```ruby
-  TableSync::Publishing::Raw.new(
-    object_class: "User",
-    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
-    event: :create,
-    routing_key: :custom_key,   # optional
-    headers: { type: "admin" }, # optional
-  ).publish_now
-```

data/docs/publishing.md

@@ -1,80 +0,0 @@
-# Publishing
-
-TableSync can be used to send data using RabbitMQ.
-
-You can do in two ways. Automatic and manual.
-Each one has its own pros and cons.
-
-Automatic is used to publish changes in realtime, as soon as the tracked entity changes.
-Usually syncs one entity at a time.
-
-Manual allows to sync a lot of entities per message.
-But demands greater amount of work and data preparation.
-
-## Automatic
-
-Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. 
-
-Options:
-
-- `if:` and `unless:` - Runs given proc in the scope of an instance. Skips sync on `false` for `if:` and on `true` for `unless:`.
-- `on:` - specify events (`create`, `update`, `destroy`) to trigger sync on. Triggered for all of them without this option.
-- `debounce_time` - min time period allowed between synchronizations.
-
-Functioning `Rails.cache` is required.
-
-How it works:
-
-- `TableSync.sync(self)` - registers new callbacks (for `create`, `update`, `destroy`) for ActiveRecord model, and defines `after_create`, `after_update` and `after_destroy` callback methods for Sequel model.
-
-- Callbacks call `TableSync::Publishing::Single#publish_later` with given options and object attributes. It enqueues a job which then publishes a message.
-
-Example:
-
-```ruby
-class SomeModel < Sequel::Model
-  TableSync.sync(self, { if: -> (*) { some_code }, unless: -> (*) { some_code }, on: [:create, :update] })
-end
-
-class SomeOtherModel < Sequel::Model
-  TableSync.sync(self)
-end
-```
-
-### Non persisted record destruction
-
-Sometimes destroy event can happen for non persisted record. In this case we can expect the following:
-
-For Sequel: 'Sequel::NoExistingObject' is raised. (This is default Sequel behaviour)
-For Active Record: Publishing is skipped.
-
-Example:
-
-```ruby
- # ActiveRecord
-  user = User.new.destroy! # Publishing is skipped.
-  
- # Sequel 
-  user = User.new.destroy! # raise Sequel::NoExistingObject
-```
-
-## Manual
-
-Directly call one of the publishers. It's the best if you need to sync a lot of data.
-This way you don't even need for the changes to occur.
-
-Example:
-
-```ruby
-  TableSync::Publishing::Batch.new(
-    object_class: "User",
-    original_attributes: [{ id: 1 }, { id: 2 }],
-    event: :update,
-  ).publish_now
-```
-
-## Read More
-
-- [Publishers](publishing/publishers.md)
-- [Configuration](publishing/configuration.md)
-- [Manual Sync (examples)](publishing/manual.md)

data/docs/receiving.md

@@ -1,346 +0,0 @@
-# Receiving changes
-
-Naming convention for receiving handlers is `Rabbit::Handler::GROUP_ID::TableSync`,
-where `GROUP_ID` represents first part of source exchange name.
-Define handler class inherited from `TableSync::ReceivingHandler`
-and named according to described convention.
-You should use DSL inside the class.
-Suppose we will synchronize models {Project, News, User} project {MainProject}, then:
-
-```ruby
-class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
-  queue_as :custom_queue
-
-  receive "Project", to_table: :projects
-
-  receive "News", to_table: :news, events: :update do
-    after_commit_on_update do
-      NewsCache.reload
-    end
-  end
-
-  receive "User", to_table: :clients, events: %i[update destroy] do
-    mapping_overrides email: :project_user_email, id: :project_user_id
-
-    only :project_user_email, :project_user_id, :project_id
-    target_keys :project_id, :project_user_id
-    rest_key :project_user_rest
-    version_key :project_user_version
-
-    additional_data do |project_id:|
-      { project_id: project_id }
-    end
-
-    default_values do
-      { created_at: Time.current }
-    end
-  end
-
-  receive "User", to_model: CustomModel.new(:users) do
-    rest_key false
-  end
-end
-```
-
-### Handler class (`Rabbit::Handler::MainProject::TableSync`)
-
-In this case:
-- `TableSync` - RabbitMQ event type.
-- `MainProject` - event source.
-- `Rabbit::Handler` - module for our handlers of events from RabbitMQ (there might be others)
-
-Method `queue_as` allows you to set custom queue.
-
-### Recieving handler batch processing
-
-Receiving handler supports array of attributes in a single update or destroy event. Corresponding
-upsert-style logic in ActiveRecord and Sequel orm handlers are provided.
-
-### Config
-```ruby
-receive source, [to_table:, to_model:, events:, &block]
-```
-
-The method receives following arguments
-- `source` - string, name of source model (required)
-- `to_table` - destination table name (required if not set to_model)
-- `to_model` - destination model (required if not set to_table)
-- `events` - array of supported events (optional)
-- `block` - configuration block with options (optional)
-
-This method implements logic of mapping `source` to `to_table` (or to `to_model`) and allows customizing
-the event handling logic with provided block.
-You can use one `source` for a lot of `to_table` or `to_moel`.
-
-### Options:
-
-Most of the options can be set as computed value or as a process.
-
-```ruby
-option(value)
-```
-
-```ruby
-option do |key params|
-  value
-end
-```
-
-Each of options can receive static value or code block which will be called for each event with the following arguments:
-- `event` - type of event (`:update` or `:destroy`)
-- `model` - source model (`Project`, `News`, `User` in example)
-- `version` - version of the data
-- `project_id` - id of project which is used in RabbitMQ
-- `raw_data` - raw data from event (before applying `mapping_overrides`, `only`, etc.)
-
-Blocks can receive any number of parameters from the list.
-
-All specific key params will be explained in examples for each option.
-
-#### only
-Whitelist for receiving attributes.
-
-```ruby
-only(instance of Array)
-```
-
-```ruby
-only do |row:|
-  return instance of Array
-end
-```
-
-default value is taken through the call `model.columns`
-
-#### target_keys
-Primary keys or unique keys.
-
-```ruby
-target_keys(instance of Array)
-```
-
-```ruby
-target_keys do |data:|
-  return instance of Array
-end
-```
-
-default value is taken through the call `model.primary_keys`
-
-#### rest_key
-Name of jsonb column for attributes which are not included in the whitelist.
-You can set the `rest_key(false)` if you won't need the rest data.
-
-```ruby
-rest_key(instance of Symbol)
-```
-
-```ruby
-rest_key do |row:, rest:|
-  return instance of Symbol
-end
-```
-default value is `:rest`
-
-#### version_key
-Name of version column.
-
-```ruby
-version_key(instance of Symbol)
-```
-
-```ruby
-version_key do |data:|
-  return instance of Symbol
-end
-```
-default value is `:version`
-
-#### except
-Blacklist for receiving attributes.
-
-```ruby
-except(instance of Array)
-```
-
-```ruby
-except do |row:|
-  return instance of Array
-end
-```
-
-default value is `[]`
-
-#### mapping_overrides
-Map for overriding receiving columns.
-
-```ruby
-mapping_overrides(instance of Hash)
-```
-
-```ruby
-mapping_overrides do |row:|
-  return instance of Hash
-end
-```
-
-default value is `{}`
-
-#### additional_data
-Additional data for insert or update (e.g. `project_id`).
-
-```ruby
-additional_data(instance of Hash)
-```
-
-```ruby
-additional_data do |row:|
-  return instance of Hash
-end
-```
-
-default value is `{}`
-
-#### default_values
-Values for insert if a row is not found.
-
-```ruby
-default_values(instance of Hash)
-```
-
-```ruby
-default_values do |data:|
-  return instance of Hash
-end
-```
-
-default value is `{}`
-
-#### skip
-Return truthy value to skip the row.
-
-```ruby
-skip(instance of TrueClass or FalseClass)
-```
-
-```ruby
-skip do |data:|
-  return instance of TrueClass or FalseClass
-end
-```
-
-default value is `false`
-
-#### wrap_receiving
-Proc that is used to wrap the receiving logic by custom block of code.
-
-```ruby
-wrap_receiving do |data:, target_keys:, version_key:, default_values: {}, event:, &receiving_logic|
-  receiving_logic.call
-  return makes no sense
-end
-```
-
-event option is current fired event
-default value is `proc { |&block| block.call }`
-
-#### before_update
-Perform code before updating data in the database.
-
-```ruby
-before_update do |data:, target_keys:, version_key:, default_values:|
-  return makes no sense
-end
-
-before_update do |data:, target_keys:, version_key:, default_values:|
-  return makes no sense
-end
-```
-
-Сan be defined several times. Execution order guaranteed.
-
-#### after_commit_on_update
-Perform code after updated data was committed.
-
-```ruby
-after_commit_on_update do |data:, target_keys:, version_key:, default_values:, results:|
-  return makes no sense
-end
-
-after_commit_on_update do |data:, target_keys:, version_key:, default_values:, results:|
-  return makes no sense
-end
-```
-
-- `results` - returned value from `model.upsert`
-
-Сan be defined several times. Execution order guaranteed.
-
-#### before_destroy
-Perform code before destroying data in database.
-
-```ruby
-before_destroy do |data:, target_keys:, version_key:|
-  return makes no sense
-end
-
-before_destroy do |data:, target_keys:, version_key:|
-  return makes no sense
-end
-```
-
-Сan be defined several times. Execution order guaranteed.
-
-#### after_commit_on_destroy
-Perform code after destroyed data was committed.
-
-```ruby
-after_commit_on_destroy do |data:, target_keys:, version_key:, results:|
-  return makes no sense
-end
-
-after_commit_on_destroy do |data:, target_keys:, version_key:, results:|
-  return makes no sense
-end
-```
-
-- `results` - returned value from `model.destroy`
-
-Сan be defined several times. Execution order guaranteed.
-
-### Custom model
-You can use custom model for receiving.
-```
-class Rabbit::Handler::MainProject::TableSync < TableSync::ReceivingHandler
-  receive "Project", to_model: CustomModel.new
-end
-```
-
-This model has to implement next interface:
-```
-def columns
-  return all columns from table
-end
-
-def primary_keys
-  return primary keys from table
-end
-
-def upsert(data: Array, target_keys: Array, version_key: Symbol, default_values: Hash)
-  return array with updated rows
-end
-
-def destroy(data: Array, target_keys: Array, version_key: Symbol)
-  return array with delited rows
-end
-
-def transaction(&block)
-  block.call
-  return makes no sense
-end
-
-def after_commit(&block)
-  block.call
-  return makes no sense
-end
-```