pg_eventstore 1.5.0 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec41b398e1f38d66e73967447ea4f12896ed2cf57eb0dbad8b71acf4257f9cd4
-  data.tar.gz: ce571ef411adeeb18b7bae463cac339a053d96f1b36e3de0b1023b8551b03ac5
+  metadata.gz: 337db5e3df424d6d72b3e08367a730af7c77ebfd907d38b309309e30a563e81e
+  data.tar.gz: 71c36ea38c4d756adea73939ab15e75a2b4500d17768416e0f37013160f0b696
 SHA512:
-  metadata.gz: 9273a4efb9855d219b1a92b5d9081e3d0d62f9de7f21c48e634949d218ed6a8278d81367f794ee8b70efa64541d7a4442ccf09e7b4cc3bcbd39398f5c7980ec2
-  data.tar.gz: ffb85f8c9063aa6e11b8f900e128b4b873e89db117b39e703de6603fc73c014811b63950ff0a0340b0316da11ed2562418f56f1faf42a04491fb0105d913fced
+  metadata.gz: 70699c2d05ddb69f64922c6e0e67d46a523b5ed2f70d9278134435a144bc002013fe0657a408a0657ccb9f0fe2d1f9cd7a1521a50e128375b1c1d8ec1e127584
+  data.tar.gz: c1125f8dc9af55a1ce1752b94a9bb3273e1153c70e726845867bb91cad9a4b407bcc63537d2bb5859bef20508ce5bfb4391ed56218a690bef2119cf1351597d7

data/CHANGELOG.md

@@ -1,5 +1,8 @@
 ## [Unreleased]
 
+## [1.6.0]
+- Introduce subscriptions CLI. Type `pg-eventstore subscriptions --help` to see available commands. The main purpose of it is to provide the single way to start/stop subscription processes. Check [Subscriptions](docs/subscriptions.md#creating-a-subscription) docs about the new way to start and keep running a subscriptions process.
+
 ## [1.5.0]
 - Add ability to toggle link events in the admin UI
 - Mark linked events in the admin UI with "link" icon

data/README.md

@@ -49,6 +49,10 @@ Documentation chapters:
 - [How to make multiple commands atomic](docs/multiple_commands.md)
 - [Admin UI](docs/admin_ui.md)
 
+## CLI
+
+The gem is shipped with its own CLI. Use `pg-eventstore --help` to find out its capabilities.
+
 ## Development
 
 After checking out the repo, run:

data/docs/subscriptions.md

@@ -29,7 +29,7 @@ Now we can use the `#subscribe` method to create the subscription:
 subscriptions_manager.subscribe('MyAwesomeSubscription', handler: proc { |event| puts event })
 ```
 
-First argument is the subscription's name. **It must be unique within the subscription set**. Second argument is your subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of arguments required to create the subscription. 
+First argument is the subscription's name. **It must be unique within the subscriptions set**. Second argument is your subscription's handler where you will be processing your events as they arrive. The example shows the minimum set of arguments required to create the subscription. 
 
 In the given state it will be listening to all events from all streams. You can define various filters by providing the `:filter` key of `options` argument:
 
@@ -50,44 +50,25 @@ subscriptions_manager.start
 # => PgEventstore::BasicRunner
 ```
 
-After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its position, meaning the same event will be processed several times. In real world the lock attempt may still happen. This can be common scenario when using kubernetes which rolls out new deployment before shutting down old one. In this case `#start` will return `nil`. You can use this behavior to properly handle such cases. Example:
+After calling `#start` all subscriptions are locked behind the given subscriptions set and can't be locked by any other subscriptions set. This measure is needed to prevent running the same subscription under the same subscription set using different processes/subscription managers. Such situation will lead to a malformed subscription state and will break its position, meaning the same event will be processed several times.
 
-```ruby
-timeout = 20 # 20 seconds
-deadline = Time.now + timeout
-loop do
-  break if subscriptions_manager.start
-  if Time.now > deadline
-    puts "Failed to acquire subscriptions lock within #{timeout} seconds. Exiting now."
-    exit
-  end
-  sleep 2
-end
-```
-
-To "unlock" the subscription you should gracefully stop the subscription manager:
-
-```ruby
-subscriptions_manager.stop
-```
-
-If you shut down the process which runs your subscriptions without calling the `#stop` method, subscriptions will remain locked, and the only way to unlock them will be to call the `#force_lock!` method before calling the `#start` method:
+If, for some reason, you want to lock already locked subscription - you can provide `force_lock: true`:
 
 ```ruby
-subscriptions_manager.force_lock!
+subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'SubscriptionsOfMyAwesomeMicroservice', force_lock: true)
 subscriptions_manager.start
 ```
 
 A complete example of the subscription setup process looks like this:
 
 ```ruby
-require 'pg_eventstore'
-
 PgEventstore.configure do |config|
   config.pg_uri = ENV.fetch('PG_EVENTSTORE_URI') { 'postgresql://postgres:postgres@localhost:5532/eventstore' }  
 end
 
-subscriptions_manager = PgEventstore.subscriptions_manager(subscription_set: 'MyAwesomeSubscriptions')
+subscriptions_manager = PgEventstore.subscriptions_manager(
+  subscription_set: 'MyAwesomeSubscriptions'
+)
 subscriptions_manager.subscribe(
   'Foo events Subscription', 
   handler: proc { |event| p "Foo events Subscription: #{event.inspect}" }, 
@@ -99,40 +80,17 @@ subscriptions_manager.subscribe(
   options: { filter: { streams: [{ context: 'BarCtx' }] } 
   }
 )
-subscriptions_manager.force_lock! if ENV['FORCE_LOCK'] == 'true'
-timeout = 20 # 20 seconds
-deadline = Time.now + timeout
-loop do
-  break if subscriptions_manager.start
-  if Time.now > deadline
-    puts "Failed to acquire subscriptions lock within #{timeout} seconds. Exiting now."
-    exit
-  end
-  sleep 2
-end
+subscriptions_manager.start
+```
 
-Kernel.trap('TERM') do
-  puts "Received TERM signal. Stopping Subscriptions Manager and exiting..."
-  # It is important to wrap subscriptions_manager.stop into another Thread, because it uses Thread::Mutex#synchronize
-  # internally, but its usage is not allowed inside Kernel.trap block
-  Thread.new { subscriptions_manager.stop }.join
-  exit
-end
+Persist this script into a file(let's say `subscriptions.rb`). Now it is time to start the process which will be processing those subscriptions. `pg_eventstore` has CLI for that purpose:
 
-loop do
-  sleep 5
-  subscriptions_manager.subscriptions.each do |subscription|
-    puts <<~TEXT
-      Subscription <<#{subscription.name.inspect}>> is at position #{subscription.current_position}. \
-      Events processed: #{subscription.total_processed_events}
-    TEXT
-  end
-  puts "Current SubscriptionsSet: #{subscriptions_manager.subscriptions_set}"
-  puts ""
-end
+```bash
+# -r ./subscriptions.rb will load our subscriptions definitions
+pg-eventstore subscriptions start -r ./subscriptions.rb
 ```
 
-You can save this script in `subscriptions.rb`, run it with `bundle exec ruby subscriptions.rb`, open another ruby console and test posting different events:
+After running that test subscriptions you can open another ruby console and test posting different events:
 
 ```ruby
 require 'pg_eventstore'

data/exe/pg-eventstore

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "pg_eventstore"
+require "pg_eventstore/cli"
+require "logger"
+
+logger = Logger.new(STDOUT)
+logger.level = :info
+logger.progname = "pg_eventstore"
+logger.formatter = proc do |severity, time, progname, msg|
+  "\e[36m#{progname} | \e[0m#{time.utc.strftime("%FT%TZ")} #{severity}: #{msg}\n"
+end
+
+PgEventstore.logger = logger
+Kernel.exit(PgEventstore::CLI.execute(ARGV))

data/lib/pg_eventstore/callbacks.rb

@@ -95,6 +95,22 @@ module PgEventstore
       result
     end
 
+    # @param action [Object]
+    # @param filter [Symbol]
+    # @param callback [#call]
+    # @return [void]
+    def remove_callback(action, filter, callback)
+      return unless @callbacks.dig(action, filter)
+
+      @callbacks[action][filter].delete(callback)
+    end
+
+    # Clear all defined callbacks
+    # @return [void]
+    def clear
+      @callbacks.clear
+    end
+
     private
 
     # @return [void]

data/lib/pg_eventstore/cli/commands/base_command.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class BaseCommand
+        module BaseCommandActions
+          # @return [Integer] exit code
+          def call
+            load_external_files
+            super
+          end
+
+          private
+
+          # @return [void]
+          def load_external_files
+            options.requires.each do |file_path|
+              require(file_path)
+            end
+          end
+        end
+
+        class << self
+          def inherited(klass)
+            super
+            klass.prepend BaseCommandActions
+          end
+        end
+
+        attr_reader :options
+
+        # @param options [PgEventstore::CLI::ParserOptions::BaseOptions]
+        def initialize(options)
+          @options = options
+        end
+
+        # @return [Integer] exit code
+        def call
+          raise NotImplementedError
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/callback_handlers/start_cmd_handlers.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      module CallbackHandlers
+        class StartCmdHandlers
+          include Extensions::CallbackHandlersExtension
+
+          class << self
+            # @param subscription_managers [Set<PgEventstore::SubscriptionsManager>]
+            # @param manager [PgEventstore::SubscriptionsManager]
+            # @return [void]
+            def register_managers(subscription_managers, manager)
+              subscription_managers.add(manager)
+            end
+
+            # @param action [Proc]
+            # @param manager [PgEventstore::SubscriptionsManager]
+            # @return [void]
+            def handle_start_up(action, manager)
+              action.call
+            rescue SubscriptionAlreadyLockedError => error
+              PgEventstore.logger&.error(
+                <<~TEXT
+                  Subscription #{error.name.inspect} from #{error.set.inspect} set is locked under \
+                  SubscriptionsSet##{error.lock_id}. Trying to unlock...
+                TEXT
+              )
+              raise unless TryUnlockSubscriptionsSet.try_unlock(manager.config_name, error.lock_id)
+              retry
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/help_command.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class HelpCommand
+        attr_reader :options
+
+        # @param options [PgEventstore::CLI::ParserOptions::BaseOptions]
+        def initialize(options)
+          @options = options
+        end
+
+        # @return [Integer] exit code
+        def call
+          puts options.help
+          ExitCodes::SUCCESS
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/start_subscriptions_command.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+require_relative 'callback_handlers/start_cmd_handlers'
+
+module PgEventstore
+  module CLI
+    module Commands
+      class StartSubscriptionsCommand < BaseCommand
+        # @return [Integer] seconds
+        KEEP_ALIVE_INTERVAL = 2
+
+        def initialize(...)
+          super
+          @subscription_managers = Set.new
+          @running = false
+          attach_callbacks
+        end
+
+        # @return [Integer] exit code
+        def call
+          return ExitCodes::ERROR unless running_subscriptions?
+
+          @running = true
+          setup_killsig
+          persist_pid
+          keep_process_alive
+          ExitCodes::SUCCESS
+        rescue SubscriptionAlreadyLockedError => error
+          PgEventstore.logger&.error(
+            "SubscriptionsSet##{error.lock_id} is still there. Are you stopping it at all?"
+          )
+          ExitCodes::ERROR
+        end
+
+        private
+
+        # @return [Boolean]
+        def running_subscriptions?
+          return true if @subscription_managers.any?(&:running?)
+
+          PgEventstore.logger&.warn("No subscriptions start ups were detected. Existing...")
+          false
+        end
+
+        # @return [void]
+        def setup_killsig
+          Kernel.trap('TERM') do
+            Thread.new do
+              PgEventstore.logger&.info("Received TERM signal, stopping subscriptions and exiting...")
+            end.join
+            # Because the implementation uses Mutex - wrap it into Thread to bypass the limitations of Kernel#trap
+            @subscription_managers.map do |manager|
+              Thread.new do
+                # Initiate graceful shutdown
+                manager.stop
+              end
+            end.each(&:join)
+            Utils.remove_file(options.pid_path)
+            @running = false
+          end
+        end
+
+        # @return [void]
+        def persist_pid
+          Utils.write_to_file(options.pid_path, Process.pid.to_s)
+        end
+
+        # @return [void]
+        def keep_process_alive
+          PgEventstore.logger&.info("Startup is successful. Processing subscriptions...")
+          loop do
+            # SubscriptionsManager#subscriptions_set becomes nil when everything gets stopped.
+            if @subscription_managers.all? { |manager| manager.subscriptions_set.nil? }
+              PgEventstore.logger&.info("All subscriptions were gracefully shut down. Exiting now...")
+              break
+            end
+            break unless @running
+            sleep KEEP_ALIVE_INTERVAL
+          end
+        end
+
+        # @return [void]
+        def attach_callbacks
+          CLI.callbacks.define_callback(
+            :start_manager, :before,
+            CallbackHandlers::StartCmdHandlers.setup_handler(:register_managers, @subscription_managers)
+          )
+          CLI.callbacks.define_callback(
+            :start_manager, :around,
+            CallbackHandlers::StartCmdHandlers.setup_handler(:handle_start_up)
+          )
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands/stop_subscriptions_command.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Commands
+      class StopSubscriptionsCommand < BaseCommand
+        # @return [Integer] exit code
+        def call
+          pid = Utils.read_pid(options.pid_path)&.to_i
+          if pid && pid > 0
+            PgEventstore.logger&.info("Stopping process #{pid}.")
+            Process.kill('TERM', pid)
+            return ExitCodes::SUCCESS
+          end
+
+          PgEventstore.logger&.error("Pid file #{options.pid_path.inspect} does not exist or empty.")
+          ExitCodes::ERROR
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/commands.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative 'commands/base_command'
+require_relative 'commands/help_command'
+require_relative 'commands/start_subscriptions_command'
+require_relative 'commands/stop_subscriptions_command'

data/lib/pg_eventstore/cli/exit_codes.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ExitCodes
+      # @return [Integer]
+      SUCCESS = 0
+      # @return [Integer]
+      ERROR = 1
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/base_options.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class BaseOptions
+        include Extensions::OptionsExtension
+
+        option(:help, metadata: Metadata.new(short: '-h', long: '--help', description: 'Prints this help'))
+        option(
+          :requires,
+          metadata: Metadata.new(
+            short: '-rFILE_PATH',
+            long: '--require=FILE_PATH',
+            description: 'Ruby files to load. You can provide this option multiple times to load more files.'
+          )
+        ) do
+          []
+        end
+
+        # @param parser [OptionParser]
+        # @return [void]
+        def attach_parser_handlers(parser)
+          parser.on(*to_parser_opts(:help)) do
+            self.help = parser.to_s
+          end
+          parser.on(*to_parser_opts(:requires)) do |path|
+            requires.push(path)
+          end
+        end
+
+        # @param option [Symbol]
+        # @return [Array<String>]
+        def to_parser_opts(option)
+          option(option).metadata.to_parser_opts
+        end
+
+        # @param option [Symbol]
+        # @return [PgEventstore::Extensions::OptionsExtension::Option]
+        def option(option)
+          self.class.options[option]
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/default_options.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class DefaultOptions < BaseOptions
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/metadata.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class Metadata
+        include Extensions::OptionsExtension
+
+        option(:short)
+        option(:long)
+        option(:description)
+
+        # @return [Array<String>]
+        def to_parser_opts
+          [short, long, description]
+        end
+
+        # @return [Integer]
+        def hash
+          to_parser_opts.hash
+        end
+
+        # @param another [Object]
+        # @return [Boolean]
+        def ==(another)
+          return false unless another.is_a?(Metadata)
+
+          to_parser_opts == another.to_parser_opts
+        end
+        alias eql? ==
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options/subscription_options.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module ParserOptions
+      class SubscriptionOptions < BaseOptions
+        option(
+          :pid_path,
+          metadata: Metadata.new(
+            short: '-pFILE_PATH',
+            long: '--pid=FILE_PATH',
+            description: 'Defines pid file path. Defaults to /tmp/pg-es_subscriptions.pid'
+          )
+        ) do
+          '/tmp/pg-es_subscriptions.pid'
+        end
+
+        # @param parser [OptionParser]
+        # @return [void]
+        def attach_parser_handlers(parser)
+          super
+          %i[pid_path].each do |option|
+            parser.on(*to_parser_opts(option)) do |value|
+              public_send("#{option}=", value)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parser_options.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative 'parser_options/metadata'
+require_relative 'parser_options/base_options'
+require_relative 'parser_options/default_options'
+require_relative 'parser_options/subscription_options'

data/lib/pg_eventstore/cli/parsers/base_parser.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Parsers
+      class BaseParser
+        class << self
+          # @return [String]
+          def banner
+            raise NotImplementedError
+          end
+        end
+
+        attr_reader :args, :options
+
+        # @param args [Array<String>]
+        # @param options [PgEventstore::CLI::ParserOptions::BaseOptions]
+        def initialize(args, options)
+          @args = args
+          @options = options
+          @parser = ::OptionParser.new(self.class.banner)
+        end
+
+        # @return [Array<Array<String>, PgEventstore::CLI::ParserOptions::BaseOptions>] list of commands and parsed
+        #   options
+        def parse
+          @options.attach_parser_handlers(@parser)
+          [@parser.parse(args), @options]
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parsers/default_parser.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Parsers
+      class DefaultParser < BaseParser
+        class << self
+          # @return [String]
+          def banner
+            <<~TEXT
+            Usage: pg-eventstore [options]
+                   pg-eventstore [command]
+        
+              Commands:
+                subscriptions     Start, stop subscriptions
+  
+              Options:
+          TEXT
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parsers/subscription_parser.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module PgEventstore
+  module CLI
+    module Parsers
+      class SubscriptionParser < BaseParser
+        class << self
+          # @return [String]
+          def banner
+            <<~TEXT
+              Usage: pg-eventstore subscriptions [command] [options]
+                
+                Commands:
+                  start     Start subscriptions. Example: pg-eventstore subscriptions start -r lib/my_subscriptions.rb 
+                  stop      Stop subscriptions. Example: pg-eventstore subscriptions stop
+
+                Options:
+            TEXT
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pg_eventstore/cli/parsers.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+require_relative 'parsers/base_parser'
+require_relative 'parsers/default_parser'
+require_relative 'parsers/subscription_parser'