table_sync 4.2.1 → 6.0

data/docs/publishing/configuration.md

@@ -0,0 +1,143 @@
+# Configuration
+
+Customization, configuration and other options.
+
+## Model Customization
+
+There are methods you can define on a synched model to customize published messages for it.
+
+### `#attributes_for_sync`
+
+Models can implement `#attributes_for_sync` to override which attributes are published for `update` and `create` events. If not present, all attributes are published.
+
+### `#attributes_for_destroy`
+
+Models can implement `#attributes_for_destroy` to override which attributes are published for `destroy` events. If not present, `needle` (primary key) is published.
+
+### `#attributes_for_routing_key`
+
+Models can implement `#attributes_for_routing_key` to override which attributes are given to the `routing_key_callable`. If not present, published attributes are given.
+
+### `#attributes_for_headers`
+
+Models can implement `#attributes_for_headers` to override which attributes are given to the `headers_callable`. If not present, published attributes are given.
+
+### `.table_sync_model_name`
+
+Models can implement `.table_sync_model_name` class method to override the model name used for publishing events. Default is a model class name.
+
+## Callables
+
+Callables are defined once. TableSync will use them to dynamically resolve things like jobs, routing_key and headers.
+
+### Single publishing job (required for automatic and delayed publishing)
+
+- `TableSync.single_publishing_job_class_callable` is a callable which should resolve to a class that calls TableSync back to actually publish changes.
+
+It is expected to have `.perform_at(hash_with_options)` and it will be passed a hash with the following keys:
+
+- `original_attributes` - serialized `original_attributes`
+- `object_class` - model name
+- `debounce_time` - pause between publishing messages
+- `event` - type of event that happened to synched entity
+- `perform_at` - time to perform the job at (depends on debounce)
+
+Example:
+
+```ruby
+TableSync.single_publishing_job_class_callable = -> { TableSync::Job }
+
+class TableSync::Job < ActiveJob::Base
+  def perform(jsoned_attributes)
+    TableSync::Publishing::Single.new(
+      JSON.parse(jsoned_attributes),
+    ).publish_now
+  end
+
+  def self.perform_at(attributes)
+    set(wait_until: attributes.delete(:perform_at))
+      .perform_later(attributes.to_json)
+  end
+end
+
+# will enqueue the job described above
+
+TableSync::Publishing::Single.new(
+  object_class: "User",
+  original_attributes: { id: 1, name: "Mark" }, # will be serialized!
+  debounce_time: 60,
+  event: :update,
+).publish_later
+```
+
+### Batch publishing job (required only for `TableSync::Publishing::Batch#publish_later`)
+
+- `TableSync.batch_publishing_job_class_callable` is a callable which should resolve to a class that calls TableSync back to actually publish changes.
+
+It is expected to have `.perform_later(hash_with_options)` and it will be passed a hash with the following keys:
+
+- `original_attributes` - array of serialized `original_attributes`
+- `object_class` - model name
+- `event` - type of event that happened to synched entity
+- `routing_key` - custom routing_key (optional)
+- `headers` - custom headers (optional)
+
+More often than not this job is not very useful, since it makes more sense to use `#publish_now` from an already existing job that does a lot of things (not just publishing messages).
+
+### Example
+
+```ruby
+TableSync.batch_publishing_job_class_callable = -> { TableSync::BatchJob }
+
+class TableSync::BatchJob < ActiveJob::Base
+  def perform(jsoned_attributes)
+    TableSync::Publishing::Batch.new(
+      JSON.parse(jsoned_attributes),
+    ).publish_now
+  end
+
+  def self.perform_later(attributes)
+    super(attributes.to_json)
+  end
+end
+
+TableSync::Publishing::Batch.new(
+  object_class: "User",
+  original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }], # will be serialized!
+  event: :create,
+  routing_key: :custom_key,   # optional
+  headers: { type: "admin" }, # optional
+).publish_later
+```
+
+### Routing key callable (required)
+
+- `TableSync.routing_key_callable` is a callable that resolves which routing key to use when publishing changes. It receives object class and published attributes or `#attributes_for_routing_key` (if defined).
+
+Example:
+
+```ruby
+TableSync.routing_key_callable = -> (klass, attributes) { klass.gsub('::', '_').tableize }
+```
+
+### Headers callable (required)
+
+- `TableSync.headers_callable` is a callable that adds RabbitMQ headers which can be used in routing. It receives object class and published attributes or `#attributes_for_headers` (if defined).
+
+One possible way of using it is defining a headers exchange and routing rules based on key-value pairs (which correspond to sent headers).
+
+Example:
+
+```ruby
+TableSync.routing_metadata_callable = -> (klass, attributes) { attributes.slice("project_id") }
+```
+
+## Other
+
+- `TableSync.exchange_name` defines the exchange name used for publishing (optional, falls back to default Rabbit gem configuration).
+
+- `TableSync.notifier` is a module that provides publish and recieve notifications.
+
+- `TableSync.raise_on_empty_message` - raises an error on empty message if set to true.
+
+- `TableSync.orm` - set ORM (ActiveRecord or Sequel) used to process given entities. Required!

data/docs/publishing/manual.md

@@ -0,0 +1,155 @@
+# Manual Sync
+
+There are two ways you can manually publish large amounts of data.
+
+### `TableSync::Publishing:Batch`
+
+Easier to use, but does a lot of DB queries. May filter out invalid PK values.
+
+#### Pros:
+
+- requires less work
+- it will automatically use methods for data customization (`#attributes_for_sync`, `#attributes_for_destroy`)
+- you can be sure that data you publish is valid (more or less)
+- serializes values for (`#publish_later`)
+
+#### Cons:
+
+- it queries database for each entity in batch (for `create` and `update`)
+- it may also do a lot of queries if `#attributes_for_sync` contains additional data from other connected entities
+- serializes values for (`#publish_later`); if your PK contains invalid values (ex. Date) they will be filtered out
+
+### `TableSync::Publishing:Raw`
+
+More complex to use, but requires very few DB queries.
+You are responsible for the data you send!
+
+#### Pros:
+
+- very customizable; only requirement - `object_class` must exist
+- you can send whatever data you want
+
+#### Cons:
+
+- you have to manually prepare data for publishing
+- you have to be really sure you are sending valid data
+
+## Tips
+
+- **Don't make data batches too large!**
+
+It may result in failure to process them. Good rule is to send approx. 5000 rows in one batch.
+
+- **Make pauses between publishing data batches!**
+
+Publishing without pause may overwhelm the receiving side. Either their background job processor (ex. Sidekiq) may clog with jobs, or their consumers may not be able to get messages from Rabbit server fast enough.
+
+1 or 2 seconds is a good wait period.
+
+- **Do not use `TableSync::Publishing:Single` to send millions or even thousands of rows of data.**
+
+Or just calling update on rows with automatic sync.
+It WILL overwhelm the receiving side. Especially if they have some additional receiving logic.
+
+- **On the receiving side don't create job with custom logic (if it exists) for every row in a batch.**
+
+Better to process it whole. Otherwise three batches of 5000 will result in 15000 new jobs.
+
+- **Send one test batch before publishing the rest of the data.**
+
+Make sure it was received properly. This way you won't send a lot invalid messages.
+
+- **Check the other arguments.**
+
+    - Ensure the routing_key is correct if you are using a custom one. Remember, that batch publishing ignores `#attributes_for_routing_key` on a model.
+    - Ensure that `object_class` is correct. And it belongs to entities you want to send.
+    - Ensure that you chose the correct event.
+    - If you have some logic depending on headers, ensure they are also correct. Remember, that batch publishing ignores `#attributes_for_headers` on a model.
+
+- **You can check what you send before publishing with:**
+
+```ruby
+TableSync::Publishing:Batch.new(...).message.message_params
+
+TableSync::Publishing:Raw.new(...).message.message_params
+```
+
+## Examples
+
+### `TableSync::Publishing:Raw`
+
+```ruby
+  # For Sequel
+  # gem 'sequel-batches' or equivalent that will allow you to chunk data somehow
+  # or #each_page from Sequel
+
+  # this is a simple query
+  # they can be much more complex, with joins and other things
+  # just make sure that it results in a set of data you expect
+  data = User.in_batches(of: 5000).naked.select(:id, :name, :email)
+
+  data.each_with_index do |batch, i|
+    TableSync::Publishing::Raw.new(
+      object_class: "User",
+      original_attributes: batch,
+      event: :create,
+      routing_key: :custom_key,   # optional
+      headers: { type: "admin" }, # optional
+    ).publish_now
+
+    # make a pause between batches
+    sleep 1
+
+    # for when you are sending from terminal
+    # allows you to keep an eye on progress
+    # you can create more complex output
+    puts "Batch #{i} sent!"
+  end
+
+```
+
+#### Another way to gather data
+
+If you don't want to create a data query (maybe it's too complex) but there is a lot of quereing in `#attributes_for_sync` and you are willing to trade a little bit of perfomance, you can try the following.
+
+```ruby
+class User < Sequel
+  one_to_many :user_info
+
+  # For example our receiving side wants to know the ips user logged in under
+  # But doesn't want to sync the user_info
+  def attributes_for_sync
+    attributes.merge(
+      ips: user_info.ips
+    )
+  end
+end
+
+# to prevent the need to query for every piece of additional data we can user eager load
+# and construct published data by calling #attributes_for_sync
+# don't forget to chunk it into more managable sizes before trying to send
+data = User.eager(:statuses).map { |user| user.attributes_for_sync }
+```
+This way it will not make unnecessary queries.
+
+### `TableSync::Publishing::Batch`
+
+Remember, it will query or initialize each row.
+
+```ruby
+  # You can just send ids.
+  data = User.in_batches(of: 5000).naked.select(:id)
+
+  data.each_with_index do |data, i|
+    TableSync::Publishing::Batch.new(
+      object_class: "User",
+      original_attributes: data,
+      event: :create,
+      routing_key: :custom_key,   # optional
+      headers: { type: "admin" }, # optional
+    ).publish_now
+
+    sleep 1
+    puts "Batch #{i} sent!"
+  end
+```

data/docs/publishing/publishers.md

@@ -0,0 +1,162 @@
+# Publishers
+
+There are three publishers you can use to send data.
+
+- `TableSync::Publishing::Single` - sends one row with initialization.
+- `TableSync::Publishing::Batch` - sends a batch of rows with initialization.
+- `TableSync::Publishing::Raw`  - sends raw data without checks.
+
+## Single
+
+`TableSync::Publishing::Single` - sends one row with initialization.
+
+This is a publisher called by `TableSync.sync(self)`.
+
+### Expected parameters:
+
+- `object_class` - class (model) used to initialize published object
+- `original_attributes` - attributes used to initialize `object_class` with
+- `debounce_time` - minimum allowed time between delayed publishings
+- `event` - type of event that happened to the published object (`create`, `update`, `destroy`); `update` by default
+
+### What it does (when uses `#publish_now`):
+- takes in the `original_attributes`, `object_class`, `event`
+- constantizes `object_class`
+- extracts the primary key (`needle`) of the `object_class` from the `original_attributes`
+- queries the database for the object with the `needle` (for `update` and `create`) or initializes the `object_class` with `original_attributes` (for `destroy`)
+- constructs routing_key using `routing_key_callable` and `#attributes_for_routing_key` (if defined)
+- constructs headers using `headers_callable` and `#attributes_for_headers` (if defined)
+- publishes Rabbit message (uses attributes from queried/initialized object as data)
+- sends notification (if set up)
+
+### What it does (when uses `#publish_later`):
+- takes in the `original_attributes`, `object_class`, `debounce_time`, `event`
+- serializes the `original_attributes`, silently filters out unserializable keys/values
+- enqueues (or skips) the job with the `serialized_original_attributes` to be performed in time according to debounce
+- job (if enqueued) calls `TableSync::Publishing::Single#publish_now` with `serialized_original_attributes` and the same `object_class`, `debounce_time`, `event`
+
+### Serialization
+
+Currently allowed key/values are:
+  `NilClass`, `String`, `TrueClass`, `FalseClass`, `Numeric`, `Symbol`.
+
+### Job
+
+Job is defined in `TableSync.single_publishing_job_class_callable` as a proc. Read more in [Configuration](docs/publishing/configuration.md).
+
+### Example #1 (send right now)
+
+```ruby
+  TableSync::Publishing::Single.new(
+    object_class: "User",
+    original_attributes: { id: 1, name: "Mark" },
+    debounce_time: 60, # useless for #publish _now, can be skipped
+    event: :create,
+  ).publish_now
+```
+
+### Example #2 (enqueue job)
+
+```ruby
+  TableSync::Publishing::Single.new(
+    object_class: "User",
+    original_attributes: { id: 1, name: "Mark" }, # will be serialized!
+    debounce_time: 60,
+    event: :update,
+  ).publish_later
+```
+
+## Batch
+
+- `TableSync::Publishing::Batch` - sends a batch of rows with initialization.
+
+### Expected parameters:
+
+- `object_class` - class (model) used to initialize published objects
+- `original_attributes` - array of attributes used to initialize `object_class` with
+- `event` - type of event that happened to the published objects (`create`, `update`, `destroy`); `update` by default
+- `routing_key` - custom routing_key
+- `headers` - custom headers
+
+### What it does (when uses `#publish_now`):
+- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
+- constantizes `object_class`
+- extracts primary keys (`needles`) of the `object_class` from the array of `original_attributes`
+- queries the database for the objects with `needles` (for `update` and `create`) or initializes the `object_class` with `original_attributes` (for `destroy`)
+- constructs routing_key using `routing_key_callable` (ignores `#attributes_for_routing_key`) or uses `routing_key` if given
+- constructs headers using `headers_callable` (ignores `#attributes_for_headers`)  or uses `headers` if given
+- publishes Rabbit message (uses attributes from queried/initialized objects as data)
+- sends notification (if set up)
+
+### What it does (when uses `#publish_later`):
+- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
+- serializes the array of `original_attributes`, silently filters out unserializable keys/values
+- enqueues the job with the `serialized_original_attributes` 
+- job calls `TableSync::Publishing::Batch#publish_now` with `serialized_original_attributes` and the same `object_class`, `event`, `routing_key`, `headers`
+
+### Serialization
+
+Currently allowed key/values are:
+  `NilClass`, `String`, `TrueClass`, `FalseClass`, `Numeric`, `Symbol`.
+
+### Job
+
+Job is defined in `TableSync.batch_publishing_job_class_callable` as a proc. Read more in [Configuration](docs/publishing/configuration.md).
+
+### Example #1 (send right now)
+
+```ruby
+  TableSync::Publishing::Batch.new(
+    object_class: "User",
+    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
+    event: :create,
+    routing_key: :custom_key,   # optional
+    headers: { type: "admin" }, # optional
+  ).publish_now
+```
+
+### Example #2 (enqueue job)
+
+```ruby
+  TableSync::Publishing::Batch.new(
+    object_class: "User",
+    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
+    event: :create,
+    routing_key: :custom_key,   # optional
+    headers: { type: "admin" }, # optional
+  ).publish_later
+```
+
+## Raw
+- `TableSync::Publishing::Raw` - sends raw data without checks.
+
+Be carefull with this publisher. There are no checks for the data sent.
+You can send anything.
+
+### Expected parameters:
+
+- `object_class` - model
+- `original_attributes` - raw data that will be sent
+- `event` - type of event that happened to the published objects (`create`, `update`, `destroy`); `update` by default
+- `routing_key` - custom routing_key
+- `headers` - custom headers
+
+### What it does (when uses `#publish_now`):
+- takes in the `original_attributes`, `object_class`, `event`, `routing_key`, `headers`
+- constantizes `object_class`
+- constructs routing_key using `routing_key_callable` (ignores `#attributes_for_routing_key`) or uses `routing_key` if given
+- constructs headers using `headers_callable` (ignores `#attributes_for_headers`)  or uses `headers` if given
+- publishes Rabbit message (uses `original_attributes` as is)
+- sends notification (if set up)
+
+### Example
+
+```ruby
+  TableSync::Publishing::Raw.new(
+    object_class: "User",
+    original_attributes: [{ id: 1, name: "Mark" }, { id: 2, name: "Bob" }],
+    event: :create,
+    routing_key: :custom_key,   # optional
+    headers: { type: "admin" }, # optional
+  ).publish_now
+```

data/docs/publishing.md

@@ -1,147 +1,63 @@
-# Publishing changes
+# Publishing
 
-Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. `:if` and `:unless` are
-supported for Sequel and ActiveRecord
+TableSync can be used to send data using RabbitMQ.
 
-Functioning `Rails.cache` is required
+You can do in two ways. Automatic and manual.
+Each one has its own pros and cons.
 
-Example:
-
-```ruby
-class SomeModel < Sequel::Model
-    TableSync.sync(self, { if: -> (*) { some_code } })
-end
-```
-
-#### #attributes_for_sync
-
-Models can implement `#attributes_for_sync` to override which attributes are published. If not
-present, all attributes are published
+Automatic is used to publish changes in realtime, as soon as the tracked entity changes.
+Usually syncs one entity at a time.
 
-#### #attrs_for_routing_key
+Manual allows to sync a lot of entities per message.
+But demands greater amount of work and data preparation.
 
-Models can implement `#attrs_for_routing_key` to override which attributes are given to routing_key_callable. If not present, default attributes are given
+## Automatic
 
-#### #attrs_for_metadata
+Include `TableSync.sync(self)` into a Sequel or ActiveRecord model. 
 
-Models can implement `#attrs_for_metadata` to override which attributes are given to metadata_callable. If not present, default attributes are given
+Options:
 
-#### .table_sync_model_name
+- `if:` and `unless:` - Runs given proc in the scope of an instance. Skips sync on `false` for `if:` and on `true` for `unless:`.
+- `on:` - specify events (`create`, `update`, `destroy`) to trigger sync on. Triggered for all of them without this option.
+- `debounce_time` - min time period allowed between synchronizations.
 
-Models can implement `.table_sync_model_name` class method to override the model name used for
-publishing events. Default is model class name
+Functioning `Rails.cache` is required.
 
-#### .table_sync_destroy_attributes(original_attributes)
+How it works:
 
-Models can implement `.table_sync_destroy_attributes` class method to override the attributes
-used for publishing destroy events. Default is object's primary key
+- `TableSync.sync(self)` - registers new callbacks (for `create`, `update`, `destroy`) for ActiveRecord model, and defines `after_create`, `after_update` and `after_destroy` callback methods for Sequel model.
 
-## Configuration
-
-- `TableSync.publishing_job_class_callable` is a callable which should resolve to a ActiveJob
-subclass that calls TableSync back to actually publish changes (required)
+- Callbacks call `TableSync::Publishing::Single#publish_later` with given options and object attributes. It enqueues a job which then publishes a message.
 
 Example:
 
 ```ruby
-class TableSync::Job < ActiveJob::Base
-  def perform(*args)
-    TableSync::Publisher.new(*args).publish_now
-  end
+class SomeModel < Sequel::Model
+  TableSync.sync(self, { if: -> (*) { some_code }, unless: -> (*) { some_code }, on: [:create, :update] })
 end
-```
-
-- `TableSync.batch_publishing_job_class_callable` is a callable which should resolve to a ActiveJob
-subclass that calls TableSync batch publisher back to actually publish changes (required for batch publisher)
-
-- `TableSync.routing_key_callable` is a callable which resolves which routing key to use when
-publishing changes. It receives object class and attributes (required)
-
-Example:
-
-```ruby
-TableSync.routing_key_callable = -> (klass, attributes) { klass.gsub('::', '_').tableize }
-```
-
-- `TableSync.routing_metadata_callable` is a callable that adds RabbitMQ headers which can be
-used in routing (optional). One possible way of using it is defining a headers exchange and
-routing rules based on key-value pairs (which correspond to sent headers)
 
-Example:
-
-```ruby
-TableSync.routing_metadata_callable = -> (klass, attributes) { attributes.slice("project_id") }
+class SomeOtherModel < Sequel::Model
+  TableSync.sync(self)
+end
 ```
 
-- `TableSync.exchange_name` defines the exchange name used for publishing (optional, falls back
-to default Rabbit gem configuration).
-
-- `TableSync.notifier` is a module that provides publish and recieve notifications.
-
-# Manual publishing
-
-`TableSync::Publisher.new(object_class, original_attributes, confirm: true, state: :updated, debounce_time: 45)`
-where state is one of `:created / :updated / :destroyed` and `confirm` is Rabbit's confirm delivery flag and optional param `debounce_time` determines debounce time in seconds, 1 minute by default.
-
-# Manual publishing with batches
-
-You can use `TableSync::BatchPublisher` to publish changes in batches (array of hashes in `attributes`).
+## Manual
 
-When using `TableSync::BatchPublisher`,` TableSync.routing_key_callable` is called as follows:
-`TableSync.routing_key_callable.call(klass, {})`, i.e. empty hash is passed instead of attributes.
-And `TableSync.routing_metadata_callable` is not called at all: metadata is set to empty hash.
-
-`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, where `original_attributes_array` is an array with hash of attributes of published objects and `options` is a hash of options.
-
-`options` consists of:
-- `confirm`, which is a flag for RabbitMQ, `true` by default
-- `routing_key`, which is a custom key used (if given) to override one from `TableSync.routing_key_callable`, `nil` by default
-- `push_original_attributes` (default value is `false`), if this option is set to `true`,
-original_attributes_array will be pushed to Rabbit instead of fetching records from database and sending their mapped attributes.
-- `headers`, which is an option for custom headers (can be used for headers exchanges routes), `nil` by default
-- `event`, which is an option for event specification (`:destroy` or `:update`), `:update` by default
+Directly call one of the publishers. It's the best if you need to sync a lot of data.
+This way you don't even need for the changes to occur.
 
 Example:
 
 ```ruby
-TableSync::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
+  TableSync::Publishing::Batch.new(
+    object_class: "User",
+    original_attributes: [{ id: 1 }, { id: 2 }],
+    event: :update,
+  ).publish_now
 ```
 
-# Manual publishing with batches (Russian)
-
-С помощью класса `TableSync::BatchPublisher` вы можете опубликовать изменения батчами (массивом в `attributes`).
-
-При использовании `TableSync::BatchPublisher`, `TableSync.routing_key_callable` вызывается следующим образом:
-`TableSync.routing_key_callable.call(klass, {})`, то есть вместо аттрибутов передается пустой хэш.
-А `TableSync.routing_metadata_callable` не вызывается вовсе: в метадате устанавливается пустой хэш.
-
-`TableSync::BatchPublisher.new(object_class, original_attributes_array, **options)`, где `original_attributes_array` - массив с аттрибутами публикуемых объектов и `options`- это хэш с дополнительными опциями.
+## Read More
 
-`options` состоит из:
-- `confirm`, флаг для RabbitMQ, по умолчанию - `true`
-- `routing_key`, ключ, который (если указан) замещает ключ, получаемый из `TableSync.routing_key_callable`, по умолчанию - `nil`
-- `push_original_attributes` (значение по умолчанию `false`), если для этой опции задано значение true, в Rabbit будут отправлены original_attributes_array, вместо получения значений записей из базы непосредственно перед отправкой.
-- `headers`, опция для задания headers (можно использовать для задания маршрутов в headers exchange'ах), `nil` по умолчанию
-- `event`, опция для указания типа события (`:destroy` или `:update`), `:update` по умолчанию
-
-Example:
-
-```ruby
-TableSync::BatchPublisher.new(
-  "SomeClass",
-  [{ id: 1 }, { id: 2 }],
-  confirm: false,
-  routing_key: "custom_routing_key",
-  push_original_attributes: true,
-  headers: { key: :value },
-  event: :destroy,
-)
-```
+- [Publishers](publishing/publishers.md)
+- [Configuration](publishing/configuration.md)
+- [Manual Sync (examples)](publishing/manual.md)