pg_eventstore 1.5.0 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec41b398e1f38d66e73967447ea4f12896ed2cf57eb0dbad8b71acf4257f9cd4
-  data.tar.gz: ce571ef411adeeb18b7bae463cac339a053d96f1b36e3de0b1023b8551b03ac5
+  metadata.gz: ff514958069cbf4b819b2ad959eac80ca9f24445bca19690ed4cb3ff23989804
+  data.tar.gz: bd91be800c3739a43edf4c17c996156642f00f0d977446201c84784a1bcc6e74
 SHA512:
-  metadata.gz: 9273a4efb9855d219b1a92b5d9081e3d0d62f9de7f21c48e634949d218ed6a8278d81367f794ee8b70efa64541d7a4442ccf09e7b4cc3bcbd39398f5c7980ec2
-  data.tar.gz: ffb85f8c9063aa6e11b8f900e128b4b873e89db117b39e703de6603fc73c014811b63950ff0a0340b0316da11ed2562418f56f1faf42a04491fb0105d913fced
+  metadata.gz: 1d4946c58c314f0bfb04c0c20ca1308e4c8afe915adacee8722fcd703404b26dbc429bdda5bc5a44f94654cb366813065226b42c930620bf30fcd4a345f592b1
+  data.tar.gz: 89738f55a626cbcd4eab9d336b7f4b9ff2e3520fc30a7fc7d87dfac5e755cd6431392438d1522cec706188aa855942db4cfd6fa96bc83c5e0b67086d6e05cef8

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 ## [Unreleased]
 
+## [1.7.0]
+- Implement reading from `"$streams"` system stream
+- Disable Host authorization introduced in sinatra v4.1
+
+## [1.6.0]
+- Introduce subscriptions CLI. Type `pg-eventstore subscriptions --help` to see available commands. The main purpose of it is to provide the single way to start/stop subscription processes. Check [Subscriptions](docs/subscriptions.md#creating-a-subscription) docs about the new way to start and keep running a subscriptions process.
+
 ## [1.5.0]
 - Add ability to toggle link events in the admin UI
 - Mark linked events in the admin UI with "link" icon

data/README.md

@@ -49,6 +49,55 @@ Documentation chapters:
 - [How to make multiple commands atomic](docs/multiple_commands.md)
 - [Admin UI](docs/admin_ui.md)
 
+## CLI
+
+The gem is shipped with its own CLI. Use `pg-eventstore --help` to find out its capabilities.
+
+## RSpec
+
+### Clean up test db
+
+The gem provides a class to clean up your `pg_eventstore` test db between tests. Example usage(in your `spec/spec_helper.rb`:
+
+```ruby
+require 'pg_eventstore/rspec/test_helpers'
+
+RSpec.configure do |config|
+  config.before do
+    PgEventstore::TestHelpers.clean_up_db
+  end
+end
+```
+
+### RSpec matcher for OptionsExtension
+
+If you would like to be able to test the functional, provided by `PgEventstore::Extensions::OptionsExtension` extension - there is a rspec matcher. Load custom matcher in you `spec_helper.rb`:
+
+```ruby
+require 'pg_eventstore/rspec/has_option_matcher'
+```
+
+Let's say you have next class:
+```ruby
+class SomeClass
+  include PgEventstore::Extensions::OptionsExtension
+
+  option(:some_opt, metadata: { foo: :bar }) { '1' }
+end
+```
+
+To test that its instance has the proper option with the proper default value and proper metadata you can use this matcher:
+```ruby
+RSpec.describe SomeClass do
+  subject { described_class.new }
+
+  # Check that :some_opt is present
+  it { is_expected.to have_option(:some_opt) }
+  # Check that :some_opt is present and has the correct default value
+  it { is_expected.to have_option(:some_opt).with_default_value('1').with_metadata(foo: :bar) }
+end
+```
+
 ## Development
 
 After checking out the repo, run:

data/db/migrations/7_support_reading_streams_system_stream.sql

@@ -0,0 +1,2 @@
+CREATE INDEX idx_events_0_stream_revision_global_position ON events USING btree (stream_revision, global_position) WHERE stream_revision = 0;
+CREATE VIEW "$streams" AS SELECT * FROM events WHERE stream_revision = 0;

data/docs/reading_events.md

@@ -53,7 +53,7 @@ In case a stream with given name does not exist, a `PgEventstore::StreamNotFound
 ```ruby
 begin
   stream = PgEventstore::Stream.new(context: 'non-existing-context', stream_name: 'User', stream_id: 'f37b82f2-4152-424d-ab6b-0cc6f0a53aae')
-  PgEventstore.client.read(stream)
+  PgEventstore.client.read(stream, options: { max_count: 1 })
 rescue PgEventstore::StreamNotFoundError => e
   puts e.message # => Stream #<PgEventstore::Stream:0x01> does not exist.
   puts e.stream # => #<PgEventstore::Stream:0x01>
@@ -80,6 +80,15 @@ You can read from a specific position of the "all" stream. This is very similar
 PgEventstore.client.read(PgEventstore::Stream.all_stream, options: { from_position: 9023, direction: 'Backwards' })
 ```
 
+## Reading from "$streams" system stream
+
+`"$streams"` is a special stream which consists of events with `stream_revision == 0`. This allows you to effectively query all streams. Example:
+
+```ruby
+stream = PgEventstore::Stream.system_stream("$streams")
+PgEventstore.client.read(stream).map(&:stream) # => array of unique streams
+```
+
 ## Middlewares
 
 If you would like to skip some of your registered middlewares from processing events after they being read from a stream - you should use the `:middlewares` argument which allows you to override the list of middlewares you would like to use.
@@ -160,6 +169,12 @@ You can also mix filtering by stream's attributes and event types. The result wi
 PgEventstore.client.read(PgEventstore::Stream.all_stream, options: { filter: { streams: [{ context: 'MyAwesomeContext' }], event_types: %w[Foo Bar] } })
 ```
 
+### "$streams" stream filtering
+
+When reading from `"$streams"` same rules apply as when reading from "all" stream. For example, read all streams which have `context == "MyAwesomeContext"` and start from events with event type either `"Foo"` or `"Bar"`:
+```ruby
+PgEventstore.client.read(PgEventstore::Stream.system_stream("$streams"), options: { filter: { streams: [{ context: 'MyAwesomeContext' }], event_types: %w[Foo Bar] } })
+```
 
 ## Pagination
 

data/docs/subscriptions.md

@@ -29,7 +29,7 @@ Now we can use the `#subscribe` method to create the subscription:
 subscriptions_manager.subscribe('MyAwesomeSubscription', handler: proc { |event| puts event })
 ```
 
-First argument is the subscription's name. **It must be unique within the subscription set**. Second argument is your subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of arguments required to create the subscription. 
+First argument is the subscription's name. **It must be unique within the subscriptions set**. Second argument is your subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of arguments required to create the subscription. 
 
 In the given state it will be listening to all events from all streams. You can define various filters by providing the `:filter` key of `options` argument:
 
@@ -50,44 +50,25 @@ subscriptions_manager.start
 # => PgEventstore::BasicRunner
 ```
 
-After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its position, meaning the same event will be processed several times. In real world the lock attempt may still happen. This can be common scenario when using kubernetes which rolls out new deployment before shutting down old one. In this case `#start` will return `nil`. You can use this behavior to properly handle such cases. Example:
+After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its position, meaning the same event will be processed several times.
 
-```ruby
-timeout = 20 # 20 seconds
-deadline = Time.now + timeout
-loop do
-  break if subscriptions_manager.start
-  if Time.now > deadline
-    puts "Failed to acquire subscriptions lock within #{timeout} seconds. Exiting now."
-    exit
-  end
-  sleep 2
-end
-```
-
-To "unlock" the subscription you should gracefully stop the subscription manager:
-
-```ruby
-subscriptions_manager.stop
-```
-
-If you shut down the process which runs your subscriptions without calling the `#stop` method, subscriptions will remain locked, and the only way to unlock them will be to call the `#force_lock!` method before calling the `#start` method:
+If, for some reason, you want to lock already locked subscription - you can provide `force_lock: true`:
 
 ```ruby
-subscriptions_manager.force_lock!
+subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', force_lock: true)
 subscriptions_manager.start
 ```
 
 A complete example of the subscription setup process looks like this:
 
 ```ruby
-require 'pg_eventstore'
-
 PgEventstore.configure do |config|
   config.pg_uri = ENV.fetch('PG_EVENTSTORE_URI') { 'postgresql://postgres:postgres@localhost:5532/eventstore' }  
 end
 
-subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'MyAwesomeSubscriptions')
+subscriptions_manager = PgEventstore.subscriptions_manager(
+  subscription_set: 'MyAwesomeSubscriptions'
+)
 subscriptions_manager.subscribe(
   'Foo events Subscription', 
   handler: proc { |event| p "Foo events Subscription: #{event.inspect}" }, 
@@ -99,40 +80,17 @@ subscriptions_manager.subscribe(
   options: { filter: { streams: [{ context: 'BarCtx' }] } 
   }
 )
-subscriptions_manager.force_lock! if ENV['FORCE_LOCK'] == 'true'
-timeout = 20 # 20 seconds
-deadline = Time.now + timeout
-loop do
-  break if subscriptions_manager.start
-  if Time.now > deadline
-    puts "Failed to acquire subscriptions lock within #{timeout} seconds. Exiting now."
-    exit
-  end
-  sleep 2
-end
+subscriptions_manager.start
+```
 
-Kernel.trap('TERM') do
-  puts "Received TERM signal. Stopping Subscriptions Manager and exiting..."
-  # It is important to wrap subscriptions_manager.stop into another Thread, because it uses Thread::Mutex#synchronize
-  # internally, but its usage is not allowed inside Kernel.trap block
-  Thread.new { subscriptions_manager.stop }.join
-  exit
-end
+Persist this script into a file(let's say `subscriptions.rb`). Now it is time to start the process which will be processing those subscriptions. `pg_eventstore` has CLI for that purpose:
 
-loop do
-  sleep 5
-  subscriptions_manager.subscriptions.each do |subscription|
-    puts <<~TEXT
-      Subscription <<#{subscription.name.inspect}>> is at position #{subscription.current_position}. \
-      Events processed: #{subscription.total_processed_events}
-    TEXT
-  end
-  puts "Current SubscriptionsSet: #{subscriptions_manager.subscriptions_set}"
-  puts ""
-end
+```bash
+# -r ./subscriptions.rb will load our subscriptions definitions
+pg-eventstore subscriptions start -r ./subscriptions.rb
 ```
 
-You can save this script in `subscriptions.rb`, run it with `bundle exec ruby subscriptions.rb`, open another ruby console and test posting different events:
+After running that test subscriptions you can open another ruby console and test posting different events:
 
 ```ruby
 require 'pg_eventstore'

data/exe/pg-eventstore

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "pg_eventstore"
+require "pg_eventstore/cli"
+require "logger"
+
+logger = Logger.new(STDOUT)
+logger.level = :info
+logger.progname = "pg_eventstore"
+logger.formatter = proc do |severity, time, progname, msg|
+  "\e[36m#{progname} | \e[0m#{time.utc.strftime("%FT%TZ")} #{severity}: #{msg}\n"
+end
+
+PgEventstore.logger = logger
+Kernel.exit(PgEventstore::CLI.execute(ARGV))

data/lib/pg_eventstore/callbacks.rb

@@ -95,6 +95,22 @@ module PgEventstore
       result
     end
 
+    # @param action [Object]
+    # @param filter [Symbol]
+    # @param callback [#call]
+    # @return [void]
+    def remove_callback(action, filter, callback)
+      return unless @callbacks.dig(action, filter)
+
+      @callbacks[action][filter].delete(callback)
+    end
+
+    # Clear all defined callbacks
+    # @return [void]
+    def clear
+      @callbacks.clear
+    end
+
     private
 
     # @return [void]

data/lib/pg_eventstore/cli/commands/base_command.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class BaseCommand
+        module BaseCommandActions
+          # @return [Integer] exit code
+          def call
+            load_external_files
+            super
+          end
+
+          private
+
+          # @return [void]
+          def load_external_files
+            options.requires.each do |file_path|
+              require(file_path)
+            end
+          end
+        end
+
+        class << self
+          def inherited(klass)
+            super
+            klass.prepend BaseCommandActions
+          end
+        end
+
+        attr_reader :options
+
+        # @param options [PgEventstore::CLI::ParserOptions::BaseOptions]
+        def initialize(options)
+          @options = options
+        end
+
+        # @return [Integer] exit code
+        def call
+          raise NotImplementedError
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/callback_handlers/start_cmd_handlers.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      module CallbackHandlers
+        class StartCmdHandlers
+          include Extensions::CallbackHandlersExtension
+
+          class << self
+            # @param subscription_managers [Set<PgEventstore::SubscriptionsManager>]
+            # @param manager [PgEventstore::SubscriptionsManager]
+            # @return [void]
+            def register_managers(subscription_managers, manager)
+              subscription_managers.add(manager)
+            end
+
+            # @param action [Proc]
+            # @param manager [PgEventstore::SubscriptionsManager]
+            # @return [void]
+            def handle_start_up(action, manager)
+              action.call
+            rescue SubscriptionAlreadyLockedError => error
+              PgEventstore.logger&.error(
+                <<~TEXT
+                  Subscription #{error.name.inspect} from #{error.set.inspect} set is locked under \
+                  SubscriptionsSet##{error.lock_id}. Trying to unlock...
+                TEXT
+              )
+              raise unless TryUnlockSubscriptionsSet.try_unlock(manager.config_name, error.lock_id)
+              retry
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/help_command.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class HelpCommand
+        attr_reader :options
+
+        # @param options [PgEventstore::CLI::ParserOptions::BaseOptions]
+        def initialize(options)
+          @options = options
+        end
+
+        # @return [Integer] exit code
+        def call
+          puts options.help
+          ExitCodes::SUCCESS
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/start_subscriptions_command.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+require_relative 'callback_handlers/start_cmd_handlers'
+
+module PgEventstore
+  module CLI
+    module Commands
+      class StartSubscriptionsCommand < BaseCommand
+        # @return [Integer] seconds
+        KEEP_ALIVE_INTERVAL = 2
+
+        def initialize(...)
+          super
+          @subscription_managers = Set.new
+          @running = false
+          attach_callbacks
+        end
+
+        # @return [Integer] exit code
+        def call
+          return ExitCodes::ERROR unless running_subscriptions?
+
+          @running = true
+          setup_killsig
+          persist_pid
+          keep_process_alive
+          ExitCodes::SUCCESS
+        rescue SubscriptionAlreadyLockedError => error
+          PgEventstore.logger&.error(
+            "SubscriptionsSet##{error.lock_id} is still there. Are you stopping it at all?"
+          )
+          ExitCodes::ERROR
+        end
+
+        private
+
+        # @return [Boolean]
+        def running_subscriptions?
+          return true if @subscription_managers.any?(&:running?)
+
+          PgEventstore.logger&.warn("No subscriptions start ups were detected. Existing...")
+          false
+        end
+
+        # @return [void]
+        def setup_killsig
+          Kernel.trap('TERM') do
+            Thread.new do
+              PgEventstore.logger&.info("Received TERM signal, stopping subscriptions and exiting...")
+            end.join
+            # Because the implementation uses Mutex - wrap it into Thread to bypass the limitations of Kernel#trap
+            @subscription_managers.map do |manager|
+              Thread.new do
+                # Initiate graceful shutdown
+                manager.stop
+              end
+            end.each(&:join)
+            Utils.remove_file(options.pid_path)
+            @running = false
+          end
+        end
+
+        # @return [void]
+        def persist_pid
+          Utils.write_to_file(options.pid_path, Process.pid.to_s)
+        end
+
+        # @return [void]
+        def keep_process_alive
+          PgEventstore.logger&.info("Startup is successful. Processing subscriptions...")
+          loop do
+            # SubscriptionsManager#subscriptions_set becomes nil when everything gets stopped.
+            if @subscription_managers.all? { |manager| manager.subscriptions_set.nil? }
+              PgEventstore.logger&.info("All subscriptions were gracefully shut down. Exiting now...")
+              break
+            end
+            break unless @running
+            sleep KEEP_ALIVE_INTERVAL
+          end
+        end
+
+        # @return [void]
+        def attach_callbacks
+          CLI.callbacks.define_callback(
+            :start_manager, :before,
+            CallbackHandlers::StartCmdHandlers.setup_handler(:register_managers, @subscription_managers)
+          )
+          CLI.callbacks.define_callback(
+            :start_manager, :around,
+            CallbackHandlers::StartCmdHandlers.setup_handler(:handle_start_up)
+          )
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/stop_subscriptions_command.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class StopSubscriptionsCommand < BaseCommand
+        # @return [Integer] exit code
+        def call
+          pid = Utils.read_pid(options.pid_path)&.to_i
+          if pid && pid > 0
+            PgEventstore.logger&.info("Stopping process #{pid}.")
+            Process.kill('TERM', pid)
+            return ExitCodes::SUCCESS
+          end
+
+          PgEventstore.logger&.error("Pid file #{options.pid_path.inspect} does not exist or empty.")
+          ExitCodes::ERROR
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative 'commands/base_command'
+require_relative 'commands/help_command'
+require_relative 'commands/start_subscriptions_command'
+require_relative 'commands/stop_subscriptions_command'

data/lib/pg_eventstore/cli/exit_codes.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ExitCodes
+      # @return [Integer]
+      SUCCESS = 0
+      # @return [Integer]
+      ERROR = 1
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/base_options.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class BaseOptions
+        include Extensions::OptionsExtension
+
+        option(:help, metadata: Metadata.new(short: '-h', long: '--help', description: 'Prints this help'))
+        option(
+          :requires,
+          metadata: Metadata.new(
+            short: '-rFILE_PATH',
+            long: '--require=FILE_PATH',
+            description: 'Ruby files to load. You can provide this option multiple times to load more files.'
+          )
+        ) do
+          []
+        end
+
+        # @param parser [OptionParser]
+        # @return [void]
+        def attach_parser_handlers(parser)
+          parser.on(*to_parser_opts(:help)) do
+            self.help = parser.to_s
+          end
+          parser.on(*to_parser_opts(:requires)) do |path|
+            requires.push(path)
+          end
+        end
+
+        # @param option [Symbol]
+        # @return [Array<String>]
+        def to_parser_opts(option)
+          option(option).metadata.to_parser_opts
+        end
+
+        # @param option [Symbol]
+        # @return [PgEventstore::Extensions::OptionsExtension::Option]
+        def option(option)
+          self.class.options[option]
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/default_options.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class DefaultOptions < BaseOptions
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/metadata.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class Metadata
+        include Extensions::OptionsExtension
+
+        option(:short)
+        option(:long)
+        option(:description)
+
+        # @return [Array<String>]
+        def to_parser_opts
+          [short, long, description]
+        end
+
+        # @return [Integer]
+        def hash
+          to_parser_opts.hash
+        end
+
+        # @param another [Object]
+        # @return [Boolean]
+        def ==(another)
+          return false unless another.is_a?(Metadata)
+
+          to_parser_opts == another.to_parser_opts
+        end
+        alias eql? ==
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/subscription_options.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class SubscriptionOptions < BaseOptions
+        option(
+          :pid_path,
+          metadata: Metadata.new(
+            short: '-pFILE_PATH',
+            long: '--pid=FILE_PATH',
+            description: 'Defines pid file path. Defaults to /tmp/pg-es_subscriptions.pid'
+          )
+        ) do
+          '/tmp/pg-es_subscriptions.pid'
+        end
+
+        # @param parser [OptionParser]
+        # @return [void]
+        def attach_parser_handlers(parser)
+          super
+          %i[pid_path].each do |option|
+            parser.on(*to_parser_opts(option)) do |value|
+              public_send("#{option}=", value)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative 'parser_options/metadata'
+require_relative 'parser_options/base_options'
+require_relative 'parser_options/default_options'
+require_relative 'parser_options/subscription_options'