pg_eventstore 2.0.0 → 3.0.0

data/docs/reading_streams.md

@@ -0,0 +1,50 @@
+# Reading streams
+
+`pg_eventstore` provides API to directly read a set of streams and detailed information of them.
+
+## Stream revision
+
+`Client#stream_revision` provides the best speed of retrieving current stream revision. Alternative way would be to read
+last event in the stream, but it is much slower than this and may raise exception if stream does not exist. Example:
+
+```ruby
+stream = PgEventstore::Stream.new(context: 'FooCtx', stream_name: 'Foo', stream_id: '1')
+PgEventstore.client.stream_revision(stream) 
+# => stream revision or Stream::NON_EXISTING_STREAM_REVISION (which is -1) if stream does not exist
+```
+
+## Streams list
+
+You can directly retrieve a list of streams using `#read_streams`. The list is ordered by the stream starting position.
+Starting position is a global position of the first event in the stream. Supported options are:
+
+- `:direction`. Read direction. Default is `:asc`
+- `:from_position`. Starting position to read from
+- `:max_count`. Max number of objects to return
+
+Examples:
+
+```ruby
+PgEventstore.client.read_streams
+PgEventstore.client.read_streams(options: { direction: :desc })
+PgEventstore.client.read_streams(options: { max_count: 10 })
+PgEventstore.client.read_streams(options: { from_position: 123 })
+PgEventstore.client.read_streams(options: { from_position: 123, direction: :desc })
+PgEventstore.client.read_streams(options: { from_position: 123, max_count: 123 })
+PgEventstore.client.read_streams(options: { from_position: 123, direction: :desc, max_count: 10 })
+```
+
+Unlike `Stream` object you can access via `Event#stream` of a persisted event - `Stream` objects from this API
+additionally load values of `#starting_position` and `#stream_revision` attributes.
+
+You can also paginate by streams. `#read_streams_paginated` yields an array of streams on each iteration. It accepts the
+same list of options as `#read_streams`. Example:
+
+```ruby
+PgEventstore.client.read_streams_paginated(options: { max_count: 10 }).each do |streams|
+  # yields up to :max_count objects
+  streams.each do |stream|
+    # do something with your stream
+  end
+end
+```

data/docs/replication.md

@@ -0,0 +1,99 @@
+# Database replication
+
+pg_eventstore can be replicated, but for different purposes you should choose different replication strategies. This is
+related to the complex architecture of pg_eventstore database(some tables are partitioned) and to the neediness to have
+only certain tables list in your replica node. For example, such tables as `subscriptions`, `subscriptions_set`, 
+`event_subscription_positions_unprocessed`, `subscription_commands`, `subscriptions_set_commands` must not present in
+the replica, because they are node-specific.
+
+Let's review a simple example and, based on it, we build a replica setup. So, let's say we have distributed
+architecture, and we target three regions - EU, US and Asia. We then want to have primary node along with two replica
+nodes of two other regions per region. We want to have a hot standby replica per region in case of emergency as well. We
+can summarize this description as follows:
+
+## EU Region
+
+- **EU primary**
+    - Replicates to: **EU hot standby**
+    - Replicates to: **US replica node**
+    - Replicates to: **Asia replica node**
+
+- **US replica**
+    - Source: async replication from **US primary**
+
+- **Asia replica**
+    - Source: async replication from **Asia primary**
+
+- **EU read models** built from:
+    - EU primary
+    - US replica
+    - Asia replica
+
+## US Region
+
+- **US primary**
+    - Replicates to: **US hot standby**
+    - Replicates to: **EU replica node**
+    - Replicates to: **Asia replica node** 
+
+- **EU replica**
+    - Source: async replication from **EU primary**
+
+- **Asia replica**
+    - Source: async replication from **Asia primary**
+
+- **US read models** built from:
+    - US primary
+    - EU replica
+    - Asia replica
+
+## Asia Region
+
+- **Asia primary**
+    - Replicates to: **Asia hot standby**
+    - Replicates to: **US replica node**
+    - Replicates to: **EU replica node**
+
+- **EU replica**
+    - Source: async replication from **EU primary**
+
+- **US replica**
+    - Source: async replication from **US primary**
+
+- **Asia read models** built from:
+    - Asia primary
+    - EU replica
+    - US replica
+
+From the example above:
+- applications, connecting to primary node should have `:primary` [role](configuration.md#eventstore-role)
+- applications, connecting to replica node should have `:replica` [role](configuration.md#eventstore-role)
+- hot standby replica should utilize [streaming WAL](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) replication. This will be a full copy of region's primary node
+- region replicas should utilize pg_eventstore replication via `#create_replication`. Example (from the perspective of
+  US region primary node):
+```ruby
+# config.rb
+PgEventstore.configure do |config|
+  config.pg_uri = 'postgresql://postgres:postgres@localhost:6432/eventstore_us'
+  config.eventstore_role = :primary
+end
+
+PgEventstore.configure(name: :eu_replica) do |config|
+  config.pg_uri = 'postgresql://postgres:postgres@localhost:6432/eventstore_us'
+  config.eventstore_role = :replica
+end
+
+PgEventstore.configure(name: :asia_replica) do |config|
+  config.pg_uri = 'postgresql://postgres:postgres@localhost:6432/eventstore_us'
+  config.eventstore_role = :replica
+end
+
+# subscriptions
+subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: "Replications")
+# Replicates US pg_eventstore into EU region pg_eventstore
+subscriptions_manager.create_replication('EU', :eu_replica)
+# Replicates US pg_eventstore into Asia region pg_eventstore
+subscriptions_manager.create_replication('Asia', :asia_replica)
+
+subscriptions_manager.start
+```

data/docs/subscriptions.md

@@ -1,27 +1,46 @@
 # Subscriptions
 
-In order to process new events in your microservices you have to have the ability to listen for them. `pg_eventstore` implements a subscription feature for this matter. It is implemented as a background thread that pulls new events according to your filters from time to time (see `subscription_pull_interval` setting option under [**Configuration**](configuration.md) chapter).
+In order to process new events in your microservices you have to have the ability to listen for them. `pg_eventstore`
+implements a subscription feature for this matter. It is implemented as a background thread that pulls new events
+according to your filters from time to time (see `subscription_pull_interval` setting option under [**Configuration
+**](configuration.md) chapter).
 
 ## PgEventstore::Subscription
 
-`pg_eventstore` stores various subscription information in the database. The corresponding object that describes the database records is the `PgEventstore::Subscription` object. It is used in the `config.subscription_restart_terminator` setting for example. You can find its attributes summary [here](https://rubydoc.info/gems/pg_eventstore/PgEventstore/Subscription).  
+`pg_eventstore` stores various subscription information in the database. The corresponding object that describes the
+database records is the `PgEventstore::Subscription` object. It is used in the `config.subscription_restart_terminator`
+setting for example. You can find its attributes
+summary [here](https://rubydoc.info/gems/pg_eventstore/PgEventstore/Subscription).
 
 ## PgEventstore::SubscriptionsSet
 
-`pg_eventstore` also stores information about which subscriptions are set. The corresponding object that describes the database records is `PgEventstore::SubscriptionsSet`. You can find its attributes summary [here](https://rubydoc.info/gems/pg_eventstore/PgEventstore/SubscriptionsSet).
+`pg_eventstore` also stores information about which subscriptions are set. The corresponding object that describes the
+database records is `PgEventstore::SubscriptionsSet`. You can find its attributes
+summary [here](https://rubydoc.info/gems/pg_eventstore/PgEventstore/SubscriptionsSet).
 
-This record is created when you start your subscriptions. All subscriptions created using a single subscriptions manager instance are locked using a single `PgEventstore::SubscriptionsSet`. When subscriptions are locked, they can't be managed anywhere else. When you stop your subscriptions, the `PgEventstore::SubscriptionsSet` is deleted, unlocking the subscriptions. The `SubscriptionSet` also holds information about the state, number of restarts, the restart interval and last error of the background runner which is responsible for pulling the subscription's events. You can set the max number of restarts and the restarts interval of your subscriptions set via `config.subscriptions_set_max_retries` and `config.subscriptions_set_retries_interval` settings. See [**Configuration**](configuration.md) chapter for more info.
+This record is created when you start your subscriptions. All subscriptions created using a single subscriptions manager
+instance are locked using a single `PgEventstore::SubscriptionsSet`. When subscriptions are locked, they can't be
+managed anywhere else. When you stop your subscriptions, the `PgEventstore::SubscriptionsSet` is deleted, unlocking the
+subscriptions. The `SubscriptionSet` also holds information about the state, number of restarts, the restart interval
+and last error of the background runner which is responsible for pulling the subscription's events. You can set the max
+number of restarts and the restarts interval of your subscriptions set via `config.subscriptions_set_max_retries` and
+`config.subscriptions_set_retries_interval` settings. See [**Configuration**](configuration.md) chapter for more info.
 
 ## Creating a subscription
 
-First step you need to do is to create a `PgEventstore::SubscriptionsManager` object and provide the `subscription_set` keyword argument. Optionally you can provide a config name to use, override the `config.subscriptions_set_max_retries` and `config.subscriptions_set_retries_interval` settings:
+First step you need to do is to create a `PgEventstore::SubscriptionsManager` object and provide the `subscription_set`
+keyword argument. Optionally you can provide a config name to use, override the `config.subscriptions_set_max_retries`
+and `config.subscriptions_set_retries_interval` settings:
 
 ```ruby
 subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'SubscriptionsOfMyAwesomeMicroservice')
-another_subscriptions_manager = PgEventstore.subscriptions_manager(:my_custom_config, subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', max_retries: 5, retries_interval: 2)
+another_subscriptions_manager = PgEventstore.subscriptions_manager(
+  :my_custom_config, subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', max_retries: 5, retries_interval: 2
+)
 ```
 
-The required `subscription_set` option groups your subscriptions into a set. For example, you could refer to your service's name in the subscription set name.
+The required `subscription_set` option groups your subscriptions into a set. For example, you could refer to your
+service's name in the subscription set name.
 
 Now we can use the `#subscribe` method to create the subscription:
 
@@ -29,19 +48,9 @@ Now we can use the `#subscribe` method to create the subscription:
 subscriptions_manager.subscribe('MyAwesomeSubscription', handler: proc { |event| puts event })
 ```
 
-First argument is the subscription's name. **It must be unique within the subscriptions set**. Second argument is your subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of arguments required to create the subscription. 
-
-In the given state it will be listening to all events from all streams. You can define various filters by providing the `:filter` key of `options` argument:
-
-```ruby
-subscriptions_manager.subscribe(
-  'MyAwesomeSubscription', 
-  handler: proc { |event| puts event }, 
-  options: { filter: { streams: [{ context: 'MyAwesomeContext' }], event_types: ['Foo', 'Bar'] } }
-)
-```
-
-`:filter` supports the same options as the `#read` method supports when reading from the `"all"` stream. See [*"all" stream filtering*](reading_events.md#all-stream-filtering) section of **Reading events** chapter.
+First argument is the subscription's name. **It must be unique within the subscriptions set**. Second argument is your
+subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of
+arguments required to create the subscription.
 
 After you added all necessary subscriptions, it is time to start them:
 
@@ -50,12 +59,17 @@ subscriptions_manager.start
 # => PgEventstore::BasicRunner
 ```
 
-After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its position, meaning the same event will be processed several times.
+After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other
+subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using
+different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its
+position, meaning the same event will be processed several times.
 
 If, for some reason, you want to lock already locked subscription - you can provide `force_lock: true`:
 
 ```ruby
-subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', force_lock: true)
+subscriptions_manager = PgEventstore.subscriptions_manager(
+  subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', force_lock: true
+)
 subscriptions_manager.start
 ```
 
@@ -63,27 +77,28 @@ A complete example of the subscription setup process looks like this:
 
 ```ruby
 PgEventstore.configure do |config|
-  config.pg_uri = ENV.fetch('PG_EVENTSTORE_URI') { 'postgresql://postgres:postgres@localhost:5532/eventstore' }  
+  config.pg_uri = ENV.fetch('PG_EVENTSTORE_URI') { 'postgresql://postgres:postgres@localhost:5532/eventstore' }
 end
 
 subscriptions_manager = PgEventstore.subscriptions_manager(
   subscription_set: 'MyAwesomeSubscriptions'
 )
 subscriptions_manager.subscribe(
-  'Foo events Subscription', 
-  handler: proc { |event| p "Foo events Subscription: #{event.inspect}" }, 
+  'Foo events Subscription',
+  handler: proc { |event| p "Foo events Subscription: #{event.inspect}" },
   options: { filter: { event_types: ['Foo'] } }
 )
 subscriptions_manager.subscribe(
   '"BarCtx" context Subscription',
-  handler: proc { |event| p "'BarCtx' context Subscription: #{event.inspect}" }, 
-  options: { filter: { streams: [{ context: 'BarCtx' }] } 
+  handler: proc { |event| p "'BarCtx' context Subscription: #{event.inspect}" },
+  options: { filter: { streams: [{ context: 'BarCtx' }] }
   }
 )
 subscriptions_manager.start
 ```
 
-Persist this script into a file(let's say `subscriptions.rb`). Now it is time to start the process which will be processing those subscriptions. `pg_eventstore` has CLI for that purpose:
+Persist this script into a file(let's say `subscriptions.rb`). Now it is time to start the process which will be
+processing those subscriptions. `pg_eventstore` has CLI for that purpose:
 
 ```bash
 # -r ./subscriptions.rb will load our subscriptions definitions
@@ -105,15 +120,93 @@ PgEventstore.client.append_to_stream(foo_stream, PgEventstore::Event.new(type: '
 PgEventstore.client.append_to_stream(bar_stream, PgEventstore::Event.new(type: 'Foo', data: { foo: :bar }))
 ```
 
-You will then see the output of your subscription handlers. To gracefully stop the subscriptions process, use `kill -TERM <pid>` command.
+You will then see the output of your subscription handlers. To gracefully stop the subscriptions process, use
+`kill -TERM <pid>` command.
+
+### Filtering events
+
+To listen to the specific events you have to supply `:filter` option. Example:
+
+```ruby
+subscriptions_manager.subscribe(
+  'MyAwesomeSubscription',
+  handler: proc { |event| puts event },
+  options: { filter: { streams: [{ context: 'FooCtx' }], event_types: %w[Foo Bar] } }
+)
+```
+
+Filtering events of a subscription is the same as filtering events when [reading from "all" stream](reading_events.md#all-stream-filtering),
+but with some additions:
+- there is no limitations on markers filter. The next is all allowed:
+```ruby
+# Listen to all events from 'FooCtx' context, marked with 'foo' marker
+subscriptions_manager.subscribe(
+  'Sub1',
+  handler: proc { |event| puts event },
+  options: { filter: { streams: [{ context: 'FooCtx' }], event_types: [{ markers: ['foo'] }] } }
+)
+# Listen to all events from 'FooCtx' context & 'Foo' stream name, marked with 'foo' marker
+subscriptions_manager.subscribe(
+  'Sub2',
+  handler: proc { |event| puts event },
+  options: { filter: { streams: [{ context: 'FooCtx', stream_name: 'Foo' }], event_types: [{ markers: ['foo'] }] } }
+)
+```
+- you can filter event types by prefixes. In the next example the subscription will catch all events with event types
+starting from `'Foo'`(e.g. `'FooBar'`, `'Foo1'`, `'Foosomethingelse'`, etc):
+```ruby
+# Listen to all events from 'FooCtx' context, marked with 'foo' marker
+subscriptions_manager.subscribe(
+  'Sub1',
+  handler: proc { |event| puts event },
+  options: { filter: { streams: [{ context: 'FooCtx' }], event_types: [{ prefix: 'Foo' }] } }
+)
+```
+
+### Subscription position
+
+Internally, each event has its own subscription position assigned. This happens in the background process with
+`config.events_subscription_position_update_interval` interval(in seconds) whenever you start any subscription. The
+algorithm of assigning of event subscription position is deterministic and events processing by subscriptions based on
+that value is idempotent. The event subscription position ordering may differ from `Event#global_position` ordering, but
+it correlates with a stream revision ordering inside a single stream, meaning that an event with stream revision `0`
+can't be processed earlier than event with stream revision `1` if they both belong to the same stream.
+
+Let's say you never run subscriptions, and you have now plenty of events without an assigned subscription position. What
+will happen if you start any subscription? The worker which is responsible for assigning those positions will need to
+process every existing event. It does so in batches of 100k records(current implementation), each batch takes 1-2
+seconds to finish(depending on your hardware). Thus, if your subscription targets some events farther from the beginning
+of your events history - it will require some time to get there.
+
+#### Setting starting position
+
+You can set the initial position of new subscription to start with:
+
+```ruby
+subscriptions_manager.subscribe(
+  'MyAwesomeSubscription',
+  handler: proc { |event| puts event },
+  options: { from_position: 100 }
+)
+```
+
+This allows to jump to the certain position to start with and skip unwanted events. Please note that **this is not** an
+`Event#global_position` value, but rather subscription position of the event. You can find corresponding subscription
+position of an event in admin web UI.
+
+Unlike `:from_position` of Read API - `:from_position` in Subscriptions API is **exclusive**. For example, if you have
+events with positions 100, 101 and 102, setting `:from_position` to 100 will result in processing event at position 101.
 
 ## Overriding Subscription config values
 
-You can override `subscription_pull_interval`, `subscription_max_retries`, `subscription_retries_interval`, `subscription_restart_terminator`, `failed_subscription_notifier` and `subscription_graceful_shutdown_timeout` config values (see [**Configuration**](configuration.md) chapter for details) for the specific subscription by providing the corresponding arguments. Example:
+You can override `subscription_pull_interval`, `subscription_max_retries`, `subscription_retries_interval`,
+`subscription_restart_terminator`, `failed_subscription_notifier` and `subscription_graceful_shutdown_timeout` config
+values (see [**Configuration**](configuration.md) chapter for details) for the specific subscription by providing the
+corresponding arguments. Example:
 
 ```ruby
 subscriptions_manager.subscribe(
-  'MyAwesomeSubscription', 
+  'MyAwesomeSubscription',
   handler: proc { |event| puts event },
   # overrides config.subscription_pull_interval
   pull_interval: 0.5,
@@ -126,13 +219,33 @@ subscriptions_manager.subscribe(
   # overrides config.failed_subscription_notifier
   failed_subscription_notifier: proc { |_subscription, err| p err },
   # overrides config.subscription_graceful_shutdown_timeout
-  graceful_shutdown_timeout: 20
+  graceful_shutdown_timeout: 20,
+  # Yield array of events into your handler instead a single event. See example bellow.
+  in_batches: true
+)
+```
+
+## Processing events in batches
+
+There is an ability to tell the subscription to yield an array of events it pulled last time. The number of events is
+the implementation specific(current implementation may yield up to 1k events) and can't be adjusted. Example:
+
+```ruby
+subscriptions_manager.subscribe(
+  'MyAwesomeSubscription',
+  handler: proc { |events| puts events }, # => outputs array of events
+  in_batches: true
 )
 ```
 
+In this case the subscription's position will be advanced to the position of the last event of the chunk. Please note
+that in case of failure the whole chunk will be yielded again during subscription retries phase.
+
 ## Middlewares
 
-If you would like to skip some of your registered middlewares from processing events after they are being pulled by the subscription, you should use the `:middlewares` argument which allows you to override the list of middlewares you would like to use.
+If you would like to skip some of your registered middlewares from processing events after they are being pulled by the
+subscription, you should use the `:middlewares` argument which allows you to override the list of middlewares you would
+like to use.
 
 Let's say you have these registered middlewares:
 
@@ -142,7 +255,8 @@ PgEventstore.configure do |config|
 end
 ```
 
-And you want to skip `FooMiddleware` and `BazMiddleware`. You simply have to provide an array of corresponding middleware keys you would like to use when creating the subscription:
+And you want to skip `FooMiddleware` and `BazMiddleware`. You simply have to provide an array of corresponding
+middleware keys you would like to use when creating the subscription:
 
 ```ruby
 subscriptions_manager.subscribe('MyAwesomeSubscription', handler: proc { |event| puts event }, middlewares: %i[bar])
@@ -152,4 +266,20 @@ See the [Writing middleware](writing_middleware.md) chapter for info about what
 
 ## How many subscriptions I should put in one process?
 
-It depends on the nature of your subscription handlers. If they spend more time on ruby code execution than on IO operations, you should limit the number of subscriptions per single process. This can be especially noticed when you rebuild the read models of your microservice, processing all events from the start.
+It depends on the nature of your subscription handlers. If they spend more time on ruby code execution than on IO
+operations, you should limit the number of subscriptions per single process. This can be especially noticed when you
+rebuild the read models of your microservice, processing all events from the start.
+
+## Few words about a subscription handler
+
+It is recommended your subscription handler be idempotent. The subscription implementation tries hard not to process the
+same event multiple times. The internal implementation is next:
+
+- fetch an event from the database
+- pass it into subscription's handler
+- advance the subscription's position
+
+These steps are not atomic, and anything can happen in between the second and the third steps. In this case the same
+event may be processed multiple times until the whole process succeeds. In this case the implementation guarantees the
+processed event will never be processed again after the subscription's position was persisted (unless you reset the
+subscription position of course).

data/docs/writing_middleware.md

@@ -1,8 +1,12 @@
 # Writing middleware
 
-Middlewares are objects that modify events before they are appended to a stream, or right after they are read from a stream. Middleware object must respond to `#serialize` and `#deserialize` methods. The `#serialize` method is called each time an event is going to be appended. The `#deserialize` method is called each time an event is read from a stream. There are two ways how you can define a middleware:
+Middlewares are objects that modify events before they are appended to a stream, or right after they are read from a
+stream. Middleware object must respond to `#serialize` and `#deserialize` methods. The `#serialize` method is called
+each time an event is going to be appended. The `#deserialize` method is called each time an event is read from a
+stream. There are two ways how you can define a middleware:
 
-- by defining your class and including `PgEventstore::Middleware` module in it. This way you can override only one of its methods, or both of them. Example:
+- by defining your class and including `PgEventstore::Middleware` module in it. This way you can override only one of
+  its methods, or both of them. Example:
 
 ```ruby
 # Override #serialize only to define your custom logic
@@ -38,11 +42,15 @@ end
 
 # Configure your middlewares
 PgEventstore.configure do |config|
-  config.middlewares = { my_awesome_serializer: MyAwesomeSerializer.new, my_awesome_deserializer: MyAwesomeDeserializer.new, my_awesome_middleware: MyAwesomeMiddleware.new }
+  config.middlewares = { 
+    my_awesome_serializer: MyAwesomeSerializer.new, 
+    my_awesome_deserializer: MyAwesomeDeserializer.new, 
+    my_awesome_middleware: MyAwesomeMiddleware.new 
+  }
 end
 ```
 
-- implement your own object that implements `#serialize` and `#deserialize` methods. Example: 
+- implement your own object that implements `#serialize` and `#deserialize` methods. Example:
 
 ```ruby
 require 'securerandom'
@@ -99,7 +107,7 @@ class ExtractLargePayload
     def resolve_large_payload(payload_key)
       JSON.parse(Faraday.get("https://my.awesome.api/api/large_payload", { payload_key: payload_key }).body)['value']
     end
-  end  
+  end
 end
 
 # Configure our middlewares
@@ -127,9 +135,13 @@ PgEventstore.client.read(stream).last
 
 ## Remarks
 
-It is important to know that `pg_eventstore` may retry commands. In that case `#serialize` and `#deserialize` methods may also be retried. You have to make sure that the implementation of `#serialize` and `#deserialize` always returns the same result for the same input, and it does not create duplications. Let's look at the `#serialize` implementation from the example above:
+It is important to know that `pg_eventstore` may retry commands. In that case `#serialize` and `#deserialize` methods
+may also be retried. You have to make sure that the implementation of `#serialize` and `#deserialize` always returns the
+same result for the same input, and it does not create duplications. Let's look at the `#serialize` implementation from
+the example above:
 
 ```ruby
+
 def serialize(event)
   return if event.fields_with_large_payloads.empty?
 
@@ -153,8 +165,13 @@ def extract_large_payload_async(field_name, value)
 end
 ```
 
-Private method `#extract_large_payload_async` should return the same result when passing the same `field_name` and `value` arguments values, and `POST https://my.awesome.api/api/extract_large_payload` may not want to produce duplicates when called multiple times with the same payload.
+Private method `#extract_large_payload_async` should return the same result when passing the same `field_name` and
+`value` arguments values, and `POST https://my.awesome.api/api/extract_large_payload` may not want to produce duplicates
+when called multiple times with the same payload.
 
 ## Async vs Sync implementation
 
-You may notice that the extracting of a large payload is asynchronous in the example above. It is recommended approach of the implementation of `#serialize` method to increase overall performance. But if it hard for you to guarantee the persistence of a payload value - you can go with sync approach, thus not allowing event to be persisted if a payload extraction request fails.
+You may notice that the extracting of a large payload is asynchronous in the example above. It is recommended approach
+of the implementation of `#serialize` method to increase overall performance. But if it hard for you to guarantee the
+persistence of a payload value - you can go with sync approach, thus not allowing event to be persisted if a payload
+extraction request fails.

data/ext/pg_eventstore_ext/extconf.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+require 'mkmf'
+
+create_makefile('pg_eventstore_ext')

data/ext/pg_eventstore_ext/pg_eventstore_ext.c

@@ -0,0 +1,74 @@
+#include "ruby.h"
+#include "ruby/st.h"
+
+/*
+  Loops through the given array of partition ids to find an index at which we have up to max_partitions_per_call unique
+  partitions. Ruby representation of this implementation looks like this:
+    def range_to_slice2(partition_ids, max_partitions_per_call)
+      return (0..) if partition_ids.size <= max_partitions_per_call
+
+      partitions_map = {}
+      latest_index = nil
+      partition_ids.each_with_index do |partition_id, index|
+        partitions_map[partition_id] = true
+        if partitions_map.size > max_partitions_per_call
+          latest_index = index - 1
+          break
+        end
+      end
+      0..latest_index
+    end
+  It was extracted into C extension because of performance of ruby loops - they are deadly slow comparing to C loops.
+  The performance gain is x5-x15 times faster comparing to ruby implementation. Because we need to do this operation on
+  each read command - it worth having it here.
+*/
+static VALUE
+range_to_slice(VALUE self, VALUE partition_ids, VALUE max_partitions_per_call)
+{
+    long partition_ids_size = RARRAY_LEN(partition_ids);
+    const VALUE *partition_id_values;
+    long max_partitions = NUM2LONG(max_partitions_per_call);
+    st_table *unique_partition_ids;
+    long unique_partitions = 0;
+    long index;
+    VALUE latest_index = Qnil;
+
+    if (partition_ids_size <= max_partitions) {
+        return rb_range_new(INT2FIX(0), Qnil, 0);
+    }
+
+    partition_id_values = RARRAY_CONST_PTR(partition_ids);
+    unique_partition_ids = st_init_numtable();
+
+    for (index = 0; index < partition_ids_size; index++) {
+        st_data_t partition_id = (st_data_t)partition_id_values[index];
+
+        if (!st_lookup(unique_partition_ids, partition_id, NULL)) {
+            st_insert(unique_partition_ids, partition_id, (st_data_t)1);
+            unique_partitions++;
+        }
+
+        if (unique_partitions > max_partitions) {
+            latest_index = LONG2NUM(index - 1);
+            break;
+        }
+    }
+
+    st_free_table(unique_partition_ids);
+
+    return rb_range_new(INT2FIX(0), latest_index, 0);
+}
+
+void
+Init_pg_eventstore_ext(void)
+{
+    VALUE pg_eventstore = rb_define_module("PgEventstore");
+    VALUE utils = rb_define_class_under(pg_eventstore, "Utils", rb_cObject);
+
+    rb_define_singleton_method(
+        utils,
+        "range_to_slice",
+        range_to_slice,
+        2
+    );
+}

data/lib/pg_eventstore/async_runner.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  class AsyncRunner
+    class Cancellation < StandardError
+    end
+
+    def initialize
+      @jobs = {}
+    end
+
+    # @return [Integer]
+    def jobs_size
+      @jobs.size
+    end
+
+    # @return [void]
+    def run
+      loop do
+        break if @jobs.empty?
+
+        run_once
+      end
+    end
+
+    def run_once
+      return if @jobs.empty?
+
+      @jobs.keys.each do |job|
+        @jobs[job] = true
+        job.resume
+        @jobs.delete(job) unless job.alive?
+      end
+    rescue
+      @jobs.each do |job, started|
+        next unless job.alive?
+
+        if started
+          begin
+            job.raise(Cancellation)
+          rescue StandardError
+            # Because the rescue mechanisms inside the terminating job can potentially raise - catch them here
+          end
+        end
+
+        begin
+          # Kill those jobs which survived our Cancellation exception
+          job.kill if job.alive?
+        rescue StandardError
+          # Ensure we handle any errors inside an exception handler(e.g. ensure block) of the given job
+        end
+      end
+
+      @jobs.clear
+      raise
+    end
+
+    # @return [void]
+    def async(&)
+      @jobs[Fiber.new(&)] = false
+    end
+  end
+end