blue_green_process 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 52701d853a73bdee8a68d84b64f395590122f426d429c16d02429ea39ee04872
-  data.tar.gz: b2b537c0446067c400e58a578baa3c6c2fab48db0222864c65abb8066bebd040
+  metadata.gz: 89de1bb6ab2d7f30cf4063f1f04411da882e1061550603ee8ca866839a62f9d0
+  data.tar.gz: dc64126db9c11c6e03dfa80be32aed24e0a7142f463efd04915729f335233c68
 SHA512:
-  metadata.gz: ae61b9553e9a7e4af24b2d0371ba8ed60273e177b62f6ecc92115e643852bf02d67c4f5b5c32ad4f849cb37fc8b0116004299988654bb80f7335075d216e7889
-  data.tar.gz: 9965a927dca526abc3143d7f471eb87ba5ffc987abe3c4d66a740c4621c5a62d431db249ce05eacf5129c0032e4cb81618ae179961a2db96844c6054605b6895
+  metadata.gz: '00874bdfeba757a3e8cc5bba6cebdf8a8bfa63d2ab8ff4779f26a9ce5bd8a5afd9ac933e4833b8a0042ce55cefda62d69f70077396d8f6df9bb64961d2e12f70'
+  data.tar.gz: c5de3504ad7ec738260fcf4969ea9cb1cba38539f582ad122c159f543e05666ac1cc3dc7d8f1111134a5d8ad8ae4392006db8d7c4893f4d1214ae9bcea98361f

data/.rubocop.yml

@@ -25,3 +25,18 @@ Metrics/AbcSize:
 
 Metrics/MethodLength:
   Enabled: false
+
+Metrics/CyclomaticComplexity:
+  Enabled: false
+
+Security/Eval:
+  Enabled: false
+
+Lint/MissingSuper:
+  Enabled: false
+
+Metrics/ClassLength:
+  Enabled: false
+
+Lint/AmbiguousBlockAssociation:
+  Enabled: false

data/.ruby-version

@@ -0,0 +1 @@
+3.0.2

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+## [0.1.3] - 2022-9-5
+* 単一プロセスでの実行を延長するとGC.startを実行しなくなりました
+    * 長時間にわたって延長する時は呼び出し側でGC.startを実行してください
+
+## [0.1.2] - 2022-9-5
+- プロセス間での値の共有で値の共有ができるようになりました
+- 単一プロセスでの実行を延長できるようになりました
+- プロセスをforkした時に実行するコールバックをblockで設定できるようになりました
+- プロセスの切り替え時にかかった時間を取得できるようになりました
+    - BlueGreenProcess.performance.process_switching_time_before_work
+
+## [0.1] - 2022-06-17
+
+- Initial release

data/Gemfile

@@ -6,6 +6,6 @@ source "https://rubygems.org"
 gemspec
 
 gem "pry"
-gem "rake", "~> 13.0"
-gem "rspec", "~> 3.0"
+gem "rake"
+gem "rspec"
 gem "rubocop", "~> 1.21"

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    blue_green_process (0.1.1)
+    blue_green_process (0.1.3)
 
 GEM
   remote: https://rubygems.org/
@@ -57,8 +57,8 @@ PLATFORMS
 DEPENDENCIES
   blue_green_process!
   pry
-  rake (~> 13.0)
-  rspec (~> 3.0)
+  rake
+  rspec
   rubocop (~> 1.21)
 
 BUNDLED WITH

data/README.md

@@ -16,7 +16,7 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
 ```ruby
 BlueGreenProcess.configure do |config|
-  config.after_fork = ->{ puts 'forked!'  }
+  config.after_fork = ->{ puts 'forked!' }
 end
 
 process = BlueGreenProcess.new(
@@ -31,14 +31,98 @@ end
 sleep(1)
 
 process.shutdown
-loop do
-  result = Process.waitall
-  if result.empty?
-    break
-  else
-    sleep(0.01)
+Process.waitall
+```
+
+### プロセス間での値の共有
+* Hashが入っている'BlueGreenProcess::SharedVariable.data' の値はmaster process, work processで共有します.
+* 共有するHashのキーは `config.shared_variables` で許可する必要があります
+* プロセスを入れ替えるタイミングで値の復元とダンプを行います
+* JSONでシリアライズしているので共有できるオブジェクトはプリミティブ型に限定されます
+* GCの時間を軽減するために整数型だけを共有するとパフォーマンスに良さそう
+* `config.shared_variables` に最初から入っている `extend_run_on_this_process` は消すことができません
+
+```ruby
+BlueGreenProcess.configure do |config|
+  config.shared_variables = [:count]
+end
+
+worker_class = Class.new(BlueGreenProcess::BaseWorker) do
+  def initialize(*); end
+
+  def work(label)
+    BlueGreenProcess::SharedVariable.data['count'] += 1
+    puts "#{label}'s data['count'] is #{BlueGreenProcess::SharedVariable.data['count']}"
   end
 end
+
+BlueGreenProcess::SharedVariable.data['count'] = 0
+process = BlueGreenProcess.new(worker_instance: worker_class.new, max_work: 3)
+process.work # blue
+process.work # green
+process.work # blue
+BlueGreenProcess::SharedVariable.data['count']
+```
+
+outputs
+
+```
+blue's data['count'] is 1
+blue's data['count'] is 2
+blue's data['count'] is 3
+green's data['count'] is 4
+green's data['count'] is 5
+green's data['count'] is 6
+blue's data['count'] is 7
+blue's data['count'] is 8
+blue's data['count'] is 9
+9
+```
+
+### 単一プロセスでの実行を延長する
+* workerクラスの中で、`BlueGreenProcess::SharedVariable.extend_run_on_this_process`にtrueをセットするともう一度同じプロセスで処理を行います
+    * 次の実行でtrueを明示しない限りはプロセスを切り替えます
+* 単一プロセスでの実行を延長するとGC.startを実行しなくなります
+    * 長時間にわたって延長する時は呼び出し側でGC.startを実行してください
+
+```ruby
+BlueGreenProcess.configure do |config|
+  config.shared_variables = [:count]
+end
+
+worker_class = Class.new(BlueGreenProcess::BaseWorker) do
+  def initialize(*); end
+
+  def work(label)
+    BlueGreenProcess::SharedVariable.data['count'] += 1
+    BlueGreenProcess::SharedVariable.extend_run_on_this_process = true
+    puts "#{label}'s data['count'] is #{BlueGreenProcess::SharedVariable.data['count']}"
+  end
+end
+
+BlueGreenProcess::SharedVariable.data['count'] = 0
+process = BlueGreenProcess.new(worker_instance: worker_class.new, max_work: 3)
+process.work # blue
+process.work # blue
+process.work # blue
+process.work # blue
+```
+
+### Metrics
+パフォーマンスの解析に使えます
+
+##### BlueGreenProcess.performance.process_switching_time_before_work
+* プロセスを最後に入れ替えた時にかかった時間を返す
+
+### Callbacks
+#### after_fork
+
+プロセスをforkした時に実行する
+
+```ruby
+BlueGreenProcess.configure do |config|
+  config.after_fork = ->{ puts 'forked!' }
+end
 ```
 
 ## Development
@@ -56,14 +140,14 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/[USERN
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 ## NOTE
-* 前回の処理が終わっていないのにqueuingしない
-* workerの処理内容は固定
+* 処理は直列で行う
+* processが行う処理内容は引数を含めて固定
+* A processがinactiveになった時に行うGC.startに時間がかかると、次にA processがbe_activeになったらレスポンスが遅れる. 構造上の仕様.
+  * これが起きる場合はオブジェクトの生成を減らすとか、blue, greenではなくプロセスのプールを作ってプロセスがGCに時間を費やせるようにする
+* workerプロセスでエラーが起きたらmasterプロセスにそのエラーが伝わり、workerプロセスは終了します
 
 ## TODO
+* shutdownしないでプロセスを停止したときにSIGINTを受け取りたい
 * runしている間にsignalをもらったらすぐにdieを送りたい
-* プロセスを入れ替えるときに変数を受け渡しをする
-* queueしてからのdequeueするまでの時間を測定したい
-    * webサーバでよくあるqueued timeみたいな扱い
-    * これが伸びると致命的なのでチューニングできるようにしたいため
 * inactiveからactiveへの切り替えになる時間を測定したい
   * GCが長引いてactiveプロセスが処理開始に時間がかかるので

data/lib/blue_green_process/config.rb

@@ -1,7 +1,11 @@
 # frozen_string_literal: true
 
+require "logger"
+
 module BlueGreenProcess
   class Config
+    attr_writer :logger
+
     def after_fork=(block)
       @after_fork_block = block
     end
@@ -10,8 +14,20 @@ module BlueGreenProcess
       @after_fork_block || -> {}
     end
 
-    def reset
-      @after_fork_block = nil
+    def logger
+      @logger ||= Logger.new("/dev/null")
+    end
+
+    def shared_variables
+      @shared_variables ||= []
+      @shared_variables.push(:extend_run_on_this_process)
+      @shared_variables.uniq
+    end
+
+    def shared_variables=(value)
+      @shared_variables = value.map(&:to_s)
+      @shared_variables.push("extend_run_on_this_process")
+      @shared_variables.uniq
     end
   end
 end

data/lib/blue_green_process/master_process.rb

@@ -1,6 +1,15 @@
 # frozen_string_literal: true
 
 module BlueGreenProcess
+  class ErrorWrapper < StandardError
+    attr_accessor :error_class, :message
+
+    def initialize(error_class, error_message)
+      self.error_class = error_class
+      self.message = error_message
+    end
+  end
+
   class MasterProcess
     def initialize(worker_instance:, max_work:)
       blue = fork_process(label: :blue, worker_instance: worker_instance)
@@ -15,12 +24,17 @@ module BlueGreenProcess
       @max_work = max_work
     end
 
+    def pids
+      @processes.map(&:pid)
+    end
+
     def fork_process(label:, worker_instance:)
       child_read, parent_write = IO.pipe
       parent_read, child_write = IO.pipe
 
       pid = fork do
         BlueGreenProcess.config.after_fork.call
+        ::GC.disable
 
         parent_write.close
         parent_read.close
@@ -28,29 +42,35 @@ module BlueGreenProcess
 
         loop do
           data = child_read.gets&.strip
-          case data
+          json = JSON.parse(data)
+          command = json["c"]
+          case command
           when BlueGreenProcess::PROCESS_COMMAND_DIE, nil, ""
-            BlueGreenProcess.debug_log "#{label}'ll die(#{$PROCESS_ID})"
+            BlueGreenProcess.logger.debug "[BLUE_GREEN_PROCESS] #{label} will die(#{$PROCESS_ID})"
             exit 0
           when BlueGreenProcess::PROCESS_COMMAND_BE_ACTIVE
             process_status = BlueGreenProcess::PROCESS_STATUS_ACTIVE
-            BlueGreenProcess.debug_log "#{label}'ll be active(#{$PROCESS_ID})"
-            child_write.puts BlueGreenProcess::PROCESS_RESPONSE
-            ::GC.disable
+            BlueGreenProcess::SharedVariable.instance.restore(json["data"])
+            BlueGreenProcess.logger.debug "[BLUE_GREEN_PROCESS] #{label} has become active(#{$PROCESS_ID})"
+            child_write.puts({ c: BlueGreenProcess::RESPONSE_OK }.to_json)
           when BlueGreenProcess::PROCESS_COMMAND_BE_INACTIVE
             process_status = BlueGreenProcess::PROCESS_STATUS_INACTIVE
-            BlueGreenProcess.debug_log "#{label}'ll be inactive(#{$PROCESS_ID})"
-            child_write.puts BlueGreenProcess::PROCESS_RESPONSE
-            ::GC.enable
-            ::GC.start
+            BlueGreenProcess.logger.debug "[BLUE_GREEN_PROCESS] #{label} has become inactive(#{$PROCESS_ID})"
+            child_write.puts({ c: BlueGreenProcess::RESPONSE_OK,
+                               data: BlueGreenProcess::SharedVariable.data }.to_json)
+            ::GC.start unless BlueGreenProcess::SharedVariable.extend_run_on_this_process
           when BlueGreenProcess::PROCESS_COMMAND_WORK
             if process_status == BlueGreenProcess::PROCESS_STATUS_INACTIVE
               warn "Should not be able to run in this status"
             end
-            # too verbose
-            # BlueGreenProcess.debug_log "#{label}'ll work(#{$PROCESS_ID})"
-            worker_instance.work(*label)
-            child_write.puts BlueGreenProcess::PROCESS_RESPONSE
+
+            begin
+              worker_instance.work(*label)
+              child_write.puts({ c: BlueGreenProcess::RESPONSE_OK }.to_json)
+            rescue StandardError => e
+              child_write.puts({ c: BlueGreenProcess::RESPONSE_ERROR, err_class: e.class.name,
+                                 err_message: e.message }.to_json)
+            end
           else
             child_write.puts "NG"
             puts "unknown. from #{label}(#{$PROCESS_ID})"
@@ -73,23 +93,38 @@ module BlueGreenProcess
           process.work
         end
       end
-
-      true
+    rescue BlueGreenProcess::ErrorWrapper => e
+      shutdown
+      BlueGreenProcess.logger.error "[BLUE_GREEN_PROCESS] #{e.error_class}: #{e.message}"
+      raise eval(e.error_class), e.message
     end
 
     def shutdown
-      @processes.each do |process|
-        process.wpipe.puts(BlueGreenProcess::PROCESS_COMMAND_DIE)
-      end
+      @processes.each(&:shutdown)
+      Process.waitall
     end
 
     private
 
     def active_process
-      active_process = @stage[@stage_state].be_active
+      active_process = nil
       @stage[!@stage_state].be_inactive
+      process_switching_time = Benchmark.realtime do
+        active_process = @stage[@stage_state].be_active
+      end
+      BlueGreenProcess.performance.process_switching_time_before_work = process_switching_time
+
       yield(active_process)
-      @stage_state = !@stage_state
+
+      active_process.be_inactive
+      if BlueGreenProcess::SharedVariable.extend_run_on_this_process
+        BlueGreenProcess::SharedVariable.extend_run_on_this_process = false
+        active_process.be_active
+      else
+        @stage_state = !@stage_state
+      end
+
+      true
     end
   end
 end

data/lib/blue_green_process/performance.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module BlueGreenProcess
+  class Performance
+    attr_writer :process_switching_time_before_work
+
+    def process_switching_time_before_work
+      @process_switching_time_before_work || 0
+    end
+  end
+end

data/lib/blue_green_process/shared_variable.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require "singleton"
+
+module BlueGreenProcess
+  class SharedVariable
+    include Singleton
+
+    attr_writer :data
+
+    def self.data
+      instance.data
+    end
+
+    def self.data=(value)
+      instance.data = value
+    end
+
+    def self.extend_run_on_this_process
+      instance.extend_run_on_this_process
+    end
+
+    def self.extend_run_on_this_process=(value)
+      instance.extend_run_on_this_process = (value)
+    end
+
+    # @return [Hash]
+    def data
+      @data ||= {}
+    end
+
+    # @return [Boolean]
+    def extend_run_on_this_process
+      @data["extend_run_on_this_process"] ||= false
+    end
+
+    # @return [Boolean]
+    def extend_run_on_this_process=(value)
+      @data["extend_run_on_this_process"] = value
+    end
+
+    # @return [Hash]
+    def restore(json)
+      return if json.nil?
+
+      self.data = json.slice(*BlueGreenProcess.config.shared_variables)
+    end
+
+    # @return [Hash]
+    def dump
+      data.slice(*BlueGreenProcess.config.shared_variables)
+    end
+
+    # @return [NilClass]
+    def reset
+      @data = nil
+    end
+  end
+end

data/lib/blue_green_process/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module BlueGreenProcess
-  VERSION = "0.1.1"
+  VERSION = "0.1.3"
 end

data/lib/blue_green_process/worker_process.rb

@@ -15,7 +15,8 @@ module BlueGreenProcess
     def be_active
       return self if status == BlueGreenProcess::PROCESS_STATUS_ACTIVE
 
-      write_and_await_until_read(BlueGreenProcess::PROCESS_COMMAND_BE_ACTIVE)
+      write_and_await_until_read(BlueGreenProcess::PROCESS_COMMAND_BE_ACTIVE,
+                                 { data: BlueGreenProcess::SharedVariable.data })
       self.status = BlueGreenProcess::PROCESS_STATUS_ACTIVE
       self
     end
@@ -34,24 +35,36 @@ module BlueGreenProcess
       write_and_await_until_read(BlueGreenProcess::PROCESS_COMMAND_WORK)
     end
 
+    def shutdown
+      write(BlueGreenProcess::PROCESS_COMMAND_DIE)
+    end
+
     private
 
-    def write_and_await_until_read(command)
-      write(command)
+    def write_and_await_until_read(command, args = {})
+      write(command, args)
       wait_response
     end
 
     def wait_response
-      response = read
-      raise "invalid response." unless response == BlueGreenProcess::PROCESS_RESPONSE
+      response = JSON.parse(read)
+      BlueGreenProcess::SharedVariable.instance.restore(response["data"])
+      case response["c"]
+      when BlueGreenProcess::RESPONSE_OK
+        [BlueGreenProcess::SharedVariable.data, response]
+      when BlueGreenProcess::RESPONSE_ERROR
+        raise BlueGreenProcess::ErrorWrapper.new(response["err_class"], response["err_message"])
+      else
+        raise "invalid response."
+      end
     end
 
     def read
       rpipe.gets.strip
     end
 
-    def write(token)
-      wpipe.puts token
+    def write(token, args = {})
+      wpipe.puts({ c: token }.merge!(args).to_json)
     end
 
     def enforce_to_be_active

data/lib/blue_green_process.rb

@@ -2,10 +2,15 @@
 
 require "English"
 require_relative "blue_green_process/version"
-require "blue_green_process/master_process"
-require "blue_green_process/worker_process"
-require "blue_green_process/base_worker"
-require "blue_green_process/config"
+require_relative  "blue_green_process/master_process"
+require_relative  "blue_green_process/worker_process"
+require_relative  "blue_green_process/base_worker"
+require_relative  "blue_green_process/config"
+require_relative  "blue_green_process/performance"
+require_relative  "blue_green_process/shared_variable"
+require "benchmark"
+require "json"
+require "singleton"
 
 module BlueGreenProcess
   PROCESS_STATUS_ACTIVE = :active
@@ -13,21 +18,16 @@ module BlueGreenProcess
 
   PROCESS_COMMAND_DIE = "die"
   PROCESS_COMMAND_BE_ACTIVE = "be_active"
-  PROCESS_COMMAND_BE_INACTIVE = "work"
-  PROCESS_COMMAND_WORK = "be_inactive"
+  PROCESS_COMMAND_BE_INACTIVE = "be_inactive"
+  PROCESS_COMMAND_WORK = "work"
 
-  PROCESS_RESPONSE = "ACK"
+  RESPONSE_OK = "OK"
+  RESPONSE_ERROR = "ERR"
 
   def self.new(worker_instance:, max_work:)
     BlueGreenProcess::MasterProcess.new(worker_instance: worker_instance, max_work: max_work)
   end
 
-  def self.debug_log(message)
-    return unless ENV["VERBOSE"]
-
-    puts message
-  end
-
   def self.configure
     @config = Config.new
     yield(@config)
@@ -37,4 +37,17 @@ module BlueGreenProcess
   def self.config
     @config ||= Config.new
   end
+
+  def self.logger
+    config.logger
+  end
+
+  def self.performance
+    @performance ||= Performance.new
+  end
+
+  def self.reset
+    @config = Config.new
+    @performance = Performance.new
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: blue_green_process
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - jiikko
-autorequire:
+autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-08-02 00:00:00.000000000 Z
+date: 2022-09-05 00:00:00.000000000 Z
 dependencies: []
 description: A library that solves GC bottlenecks with multi-process.
 email:
@@ -19,16 +19,19 @@ extra_rdoc_files: []
 files:
 - ".rspec"
 - ".rubocop.yml"
+- ".ruby-version"
+- CHANGELOG.md
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
 - README.md
 - Rakefile
-- blue_green_process.gemspec
 - lib/blue_green_process.rb
 - lib/blue_green_process/base_worker.rb
 - lib/blue_green_process/config.rb
 - lib/blue_green_process/master_process.rb
+- lib/blue_green_process/performance.rb
+- lib/blue_green_process/shared_variable.rb
 - lib/blue_green_process/version.rb
 - lib/blue_green_process/worker_process.rb
 - sig/blue_green_process.rbs
@@ -38,7 +41,7 @@ licenses:
 metadata:
   homepage_uri: https://github.com/splaplapla/blue_green_process
   source_code_uri: https://github.com/splaplapla/blue_green_process
-post_install_message:
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -53,8 +56,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.19
-signing_key:
+rubygems_version: 3.0.3.1
+signing_key: 
 specification_version: 4
 summary: A library that solves GC bottlenecks with multi-process.
 test_files: []

data/blue_green_process.gemspec

@@ -1,36 +0,0 @@
-# frozen_string_literal: true
-
-require_relative "lib/blue_green_process/version"
-
-Gem::Specification.new do |spec|
-  spec.name = "blue_green_process"
-  spec.version = BlueGreenProcess::VERSION
-  spec.authors = ["jiikko"]
-  spec.email = ["n905i.1214@gmail.com"]
-
-  spec.summary = "A library that solves GC bottlenecks with multi-process."
-  spec.description = spec.summary
-  spec.homepage = "https://github.com/splaplapla/blue_green_process"
-  spec.license = "MIT"
-  spec.required_ruby_version = ">= 2.5"
-
-  spec.metadata["homepage_uri"] = spec.homepage
-  spec.metadata["source_code_uri"] = "https://github.com/splaplapla/blue_green_process"
-
-  # Specify which files should be added to the gem when it is released.
-  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
-  spec.files = Dir.chdir(__dir__) do
-    `git ls-files -z`.split("\x0").reject do |f|
-      (f == __FILE__) || f.match(%r{\A(?:(?:bin|test|spec|features)/|\.(?:git|travis|circleci)|appveyor)})
-    end
-  end
-  spec.bindir = "exe"
-  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
-  spec.require_paths = ["lib"]
-
-  # Uncomment to register a new dependency of your gem
-  # spec.add_dependency "example-gem", "~> 1.0"
-
-  # For more information and examples about making a new gem, check out our
-  # guide at: https://bundler.io/guides/creating_gem.html
-end