brow 0.1.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8f08f2a47a1034332966c9ef42acdecc5629052fd8255857a61cdaa8483be502
-  data.tar.gz: f42dac7bdfe22a48b7e7d5e0ef5ecb91407505f9ef0fb91393a263b194a8845c
+  metadata.gz: 1e823ebd67a0230133814bbdd5adc0463f2a6e2fe730b54ee2b55ecb925d347f
+  data.tar.gz: 5168d92d78eae8958098cacee8b0c4a25b140a52a6a3b08f7bd1e8e9388056ba
 SHA512:
-  metadata.gz: abb687ce5fe388f7c87826752255ff1227dd275365ed599d3e78519f1402a2bf4a6010e9f6da307f8f7265be8a69e6f26baea43e30b7bd9adc2cd9f3253ca447
-  data.tar.gz: 57722a879c3fa49461ffbf8334a5674333fce12bb30dc2037b7dcb09f65d3e2335e57d4abb47359a7ed2b032154e1e9494e92c3a5f1a9e00a99cd0627feb1488
+  metadata.gz: aeaa195ffd0be0945d088219d0dd60451d5ef445bee52f35c3f6d379dc2fbbc7ec5784fe0cf0cfd32a771235e87ae607b85173ef264cb4107311026c613befae
+  data.tar.gz: '09f9ae0b7030d6daab14014d31e705d2df68d099ea5fe66a217dcde23fba3b9fededab0f25663651f354174efd2b0e23aa01cbc8faa0e0c6394ab1805ef1567a'

data/CHANGELOG.md

@@ -1,5 +1,57 @@
-## [Unreleased]
+# Changelog
 
-## [0.1.0] - 2021-10-14
+All notable changes to this project will be documented in this file.
 
-- Initial release
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.4.1] - 2021-11-05
+
+- Move progname to logg message so it always appears (5ad1596df79f14d06d7b6dc508b1d5a9282aeebb and 531a999067d398daadc4b13d501e7044bcd3b709)
+- Fix echo server in examples (53eea5af37a888666e8740eac196832b3f67dfc7).
+
+## [0.4.0] - 2021-11-04
+
+### Added
+
+- Allow configuring most options from ENV variables by default (fb7819b0237a81e573677f3050446a4f41e8fb47).
+- Extra early return to avoid mutex lock if thread is alive (ac7dcfe54ee83b18e0df5ab3778a077584c843bd).
+- Validation on many of the configuration options (c50b11a2917272a87937f8aa86007816a87c63a2 and 07e2581397f870249a347d4d68e4fce172d33cef).
+
+### Changed
+
+- Stop stringifying keys. Just enqueue whatever is passed and let JSON do the rest (2e63d5328e048f0fad9fc41ca0935f97fb5ada2f).
+- A bunch of test stuff to make them faster and less flaky.
+
+## [0.3.0] - 2021-10-29
+
+https://github.com/jnunemaker/brow/pull/4
+
+### Fixed
+
+- Fixed thread churn. Upon digging in, I realized that the previous code was creating a bunch of threads. Basically one for each batch, which seems far from ideal. I'm surprised it worked that way. This changes it to be one worker thread that just sits there forever in a loop. When a batch is full, it transports it. When shutdown happens, a shutdown message is enqueued and the worker breaks the loop.
+- Moved worker thread management to `Worker` from `Client`.
+- Back off policy is now reset after `Transport#send_batch` completes. Previously it wasn't, which meant the next interval would get to the max and stay there.
+
+### Changed
+
+- Switched to stringify data keys instead of symbolize. Old versions of ruby didn't gc symbols so that was a memory leak. Might be fixed now, but strings are fine here so lets roll with them.
+- Removed test mode and test queue. I didn't like this implementation and neither did @bkeepers. We'll come up with something new and better soon like Brow::Clients::Memory.new or something.
+
+## [0.2.0] - 2021-10-25
+
+### Changed
+
+- [c25dce](https://github.com/jnunemaker/brow/commit/c25dcedcab2b75cfe28a561e80e537fefae6cc52) `record` is now `push`.
+
+### Fixed
+
+- [eceb02](https://github.com/jnunemaker/brow/commit/eceb02f810cc5ace7d7540c957fc1cf924849629) Fixed problems with shutdown (required a flush to get whatever batches were in progress) and forking (caused queue to not get worked off).
+
+### Added
+
+- [c7f7e4](https://github.com/jnunemaker/brow/commit/c7f7e42b0d6bfa9fa96bac58fda0ef94f93d223d) `BackoffPolicy` now gets `options` so you can pass those to `Client` and they'll make it all the way through.
+
+## [0.1.0] - 2021-10-20
+
+- Initial release. Let's face it I just wanted to squat on the gem name.

data/Gemfile

@@ -5,8 +5,10 @@ gemspec
 
 gem "rake", "~> 13.0"
 gem "minitest", "~> 5.0"
+gem "maxitest", "~> 4.1"
 gem "minitest-heat", "~> 0.0"
 gem "webmock", "~> 3.10.0"
+gem "climate_control", "~> 0.2.0"
 
 group(:guard) do
   gem "guard", "~> 2.18.0"

data/Guardfile

@@ -32,6 +32,7 @@ guard :minitest do
   watch(%r{^test/(.*)\/?(.*)_test\.rb$})
   watch(%r{^lib/(.*/)?([^/]+)\.rb$})     { |m| "test/#{m[1]}#{m[2]}_test.rb" }
   watch(%r{^test/test_helper\.rb$})      { 'test' }
+  watch(%r{^test/support/fake_server\.rb$})      { 'test' }
 
   # with Minitest::Spec
   # watch(%r{^spec/(.*)_spec\.rb$})

data/README.md

@@ -1,6 +1,6 @@
 # Brow
 
-A generic background thread worker for shipping events via https to some API backend.
+A generic background thread worker for shipping events via https to some API backend. It'll get events to your API by the sweat of its brow.
 
 I've been wanting to build something like this for a while. This might be a terrible start. But its a start.
 
@@ -36,14 +36,13 @@ client = Brow::Client.new({
 })
 
 50.times do |n|
-  client.record({
+  client.push({
     number: n,
     now: Time.now.utc,
   })
 end
 
 # batch of 50 events sent to api url above as json
-client.flush
 ```
 
 ## Development

data/examples/basic.rb

@@ -1,14 +1,14 @@
 require_relative "../lib/brow"
+require_relative "echo_server"
 
 client = Brow::Client.new({
-  url: "https://requestbin.net/r/rna67for",
+  url: "http://localhost:#{EchoServer.instance.port}",
+  batch_size: 10,
 })
 
-50.times do |n|
-  client.record({
-    number: n,
+5.times do |n|
+  client.push({
+    n: n,
     now: Time.now.utc,
   })
 end
-
-client.flush

data/examples/echo_server.rb

@@ -0,0 +1,51 @@
+# Usage: bundle exec ruby examples/echo_server.rb
+#
+# By default this starts in thread that other example scripts can use.
+#
+# By setting FOREGROUND=1, this will run in the foreground instead of
+# background thread.
+#
+# FOREGROUND=1 bundle exec ruby examples/echo_server.rb
+require "socket"
+require "thread"
+require "logger"
+require "json"
+require "singleton"
+require "webrick"
+
+class EchoServer
+  include Singleton
+
+  attr_reader :port, :thread
+
+  def initialize
+    @logger = Logger.new(STDOUT)
+    @logger.level = Logger::INFO
+    @port = ENV.fetch("PORT", 9999)
+    @started = false
+
+    @server = WEBrick::HTTPServer.new({
+      Port: @port,
+      StartCallback: -> { @started = true },
+      Logger: WEBrick::Log.new(@logger, WEBrick::Log::INFO),
+      AccessLog: [
+        [@logger, WEBrick::AccessLog::COMMON_LOG_FORMAT],
+      ],
+    })
+
+    @server.mount_proc '/' do |request, response|
+      @logger.debug JSON.parse(request.body).inspect
+      response.header["Content-Type"] = "application/json"
+      response.body = "{}"
+    end
+
+    @thread = Thread.new { @server.start }
+    Timeout.timeout(10) { :wait until @started }
+  end
+end
+
+EchoServer.instance
+
+if ENV.fetch("FOREGROUND", "0") == "1"
+  EchoServer.instance.thread.join
+end

data/examples/forked.rb

@@ -0,0 +1,20 @@
+require_relative "../lib/brow"
+require_relative "echo_server"
+
+client = Brow::Client.new({
+  url: "http://localhost:#{EchoServer.instance.port}",
+  batch_size: 10,
+})
+
+client.push({
+  now: Time.now.utc,
+  parent: true,
+})
+
+pid = fork {
+  client.push({
+    now: Time.now.utc,
+    child: true,
+  })
+}
+Process.waitpid pid, 0

data/examples/long_running.rb

@@ -0,0 +1,32 @@
+require_relative "../lib/brow"
+
+port = ENV.fetch("PORT") { 9999 }
+
+if ENV.fetch("START_SERVER", "1") == "1"
+  require_relative "echo_server"
+  port = EchoServer.instance.port
+end
+
+Brow.logger = Logger.new(STDOUT)
+Brow.logger.level = Logger::INFO
+
+client = Brow::Client.new({
+  url: "http://localhost:#{port}",
+  batch_size: 1_000,
+})
+
+running = true
+
+trap("INT") {
+  puts "Shutting down"
+  running = false
+}
+
+while running
+  rand(10_000).times { client.push("foo" => "bar") }
+
+  puts "Queue size: #{client.worker.queue.size}"
+
+  # Pretend to work
+  sleep(rand)
+end

data/lib/brow/backoff_policy.rb

@@ -6,7 +6,7 @@ module Brow
     MIN_TIMEOUT_MS = 100
 
     # Private: The default maximum timeout between intervals in milliseconds.
-    MAX_TIMEOUT_MS = 10000
+    MAX_TIMEOUT_MS = 10_000
 
     # Private: The value to multiply the current interval with for each
     # retry attempt.
@@ -16,6 +16,12 @@ module Brow
     # retry interval.
     RANDOMIZATION_FACTOR = 0.5
 
+    # Private
+    attr_reader :min_timeout_ms, :max_timeout_ms, :multiplier, :randomization_factor
+
+    # Private
+    attr_reader :attempts
+
     # Public: Create new instance of backoff policy.
     #
     # options - The Hash of options.
@@ -26,10 +32,30 @@ module Brow
     #   :randomization_factor - The randomization factor to use to create a range
     #                           around the retry interval.
     def initialize(options = {})
-      @min_timeout_ms = options[:min_timeout_ms] || MIN_TIMEOUT_MS
-      @max_timeout_ms = options[:max_timeout_ms] || MAX_TIMEOUT_MS
-      @multiplier = options[:multiplier] || MULTIPLIER
-      @randomization_factor = options[:randomization_factor] || RANDOMIZATION_FACTOR
+      @min_timeout_ms = options.fetch(:min_timeout_ms) {
+        ENV.fetch("BROW_BACKOFF_MIN_TIMEOUT_MS", MIN_TIMEOUT_MS).to_i
+      }
+      @max_timeout_ms = options.fetch(:max_timeout_ms) {
+        ENV.fetch("BROW_BACKOFF_MAX_TIMEOUT_MS", MAX_TIMEOUT_MS).to_i
+      }
+      @multiplier = options.fetch(:multiplier) {
+        ENV.fetch("BROW_BACKOFF_MULTIPLIER", MULTIPLIER).to_f
+      }
+      @randomization_factor = options.fetch(:randomization_factor) {
+        ENV.fetch("BROW_BACKOFF_RANDOMIZATION_FACTOR", RANDOMIZATION_FACTOR).to_f
+      }
+
+      unless @min_timeout_ms >= 0
+        raise ArgumentError, ":min_timeout_ms must be >= 0 but was #{@min_timeout_ms.inspect}"
+      end
+
+      unless @max_timeout_ms >= 0
+        raise ArgumentError, ":max_timeout_ms must be >= 0 but was #{@max_timeout_ms.inspect}"
+      end
+
+      unless @min_timeout_ms <= max_timeout_ms
+        raise ArgumentError, ":min_timeout_ms (#{@min_timeout_ms.inspect}) must be <= :max_timeout_ms (#{@max_timeout_ms.inspect})"
+      end
 
       @attempts = 0
     end
@@ -44,6 +70,10 @@ module Brow
       [interval, @max_timeout_ms].min
     end
 
+    def reset
+      @attempts = 0
+    end
+
     private
 
     def add_jitter(base, randomization_factor)

data/lib/brow/client.rb

@@ -1,110 +1,69 @@
 # frozen_string_literal: true
 
-require 'thread'
 require 'time'
 
 require_relative 'utils'
 require_relative 'worker'
-require_relative 'test_queue'
 
 module Brow
   class Client
-    # Private: Default # of items that can be in queue before we start dropping data.
-    MAX_QUEUE_SIZE = 10_000
-
     # Public: Create a new instance of a client.
     #
     # options - The Hash of options.
+    #   :url - The URL where all batches of data should be transported.
     #   :max_queue_size - The maximum number of calls to be remain queued.
+    #   :logger - The Logger to use to log useful information about what is
+    #             going on.
+    #   :queue - The Queue to use to store data until it can be batched up and
+    #            transported to the API.
+    #   :worker - The Worker that will pop items off the queue, batch them up
+    #             and transport them to the API.
+    #   :transport - The Transport to use to transport batches to the API.
+    #   :headers - The Hash of headers to include when transporting batches to
+    #              the API. These could be used for auth or whatever.
+    #   :retries - The Integer number of times the transport should retry a call
+    #              before giving up.
+    #   :read_timeout - The number of seconds to wait when reading data before
+    #                   giving up.
+    #   :open_timeout - The number of seconds to wait when opening a connection
+    #                   to the API.
+    #   :backoff_policy - The BackoffPolicy to use to determine when the next
+    #                     retry should occur when the transport fails to send a
+    #                     batch of data to the API.
+    #   :min_timeout_ms - The minimum number of milliseconds to wait before
+    #                     retrying a failed call to the API.
+    #   :max_timeout_ms - The maximum number of milliseconds to wait before
+    #                     retrying a failed call to the API.
+    #   :multiplier - The value to multily the current interval with for each
+    #                 retry attempt.
+    #   :randomization_factor - The value to use to create a range of jitter
+    #                 around the retry interval.
+    #   :batch - The MessageBatch used to batch up several events to be
+    #            transported in one call to the API.
+    #   :shutdown_timeout - The number of seconds to wait for the worker thread
+    #                       to join when shutting down.
+    #   :shutdown_automatically - Should the worker shutdown automatically or
+    #                             manually. If true, shutdown is automatic. If
+    #                             false, you'll need to handle this on your own.
+    #   :max_size - The maximum number of items a batch can contain before it
+    #               should be transported to the API. Only used if not :batch
+    #               is provided.
     #   :on_error - The Proc that handles error calls from the API.
     def initialize(options = {})
       options = Brow::Utils.symbolize_keys(options)
-
-      @worker_thread = nil
-      @worker_mutex = Mutex.new
-      @test = options[:test]
-      @max_queue_size = options[:max_queue_size] || MAX_QUEUE_SIZE
-      @logger = options.fetch(:logger) { Brow.logger }
-      @queue = options.fetch(:queue) { Queue.new }
-      @worker = options.fetch(:worker) { Worker.new(@queue, options) }
-
-      at_exit { @worker_thread && @worker_thread[:should_exit] = true }
+      @worker = options.fetch(:worker) { Worker.new(options) }
     end
 
-    # Public: Synchronously waits until the worker has flushed the queue.
-    #
-    # Use only for scripts which are not long-running, and will
-    # specifically exit.
-    def flush
-      while !@queue.empty? || @worker.requesting?
-        ensure_worker_running
-        sleep(0.1)
-      end
-    end
+    # Private
+    attr_reader :worker
 
-    # Public: Enqueues the event.
+    # Public: Enqueues an event to eventually be transported to backend service.
     #
-    # event - The Hash of event data.
+    # data - The Hash of data.
     #
-    # Returns Boolean of whether the item was added to the queue.
-    def record(event)
-      raise ArgumentError, "event must be a Hash" unless event.is_a?(Hash)
-
-      event = Brow::Utils.symbolize_keys(event)
-      event = Brow::Utils.isoify_dates(event)
-      enqueue event
-    end
-
-    # Public: Returns the number of messages in the queue.
-    def queued_messages
-      @queue.length
-    end
-
-    # Public: For test purposes only. If test: true is passed to #initialize
-    # then all recording of events will go to test queue in memory so they can
-    # be verified with assertions.
-    def test_queue
-      unless @test
-        raise 'Test queue only available when setting :test to true.'
-      end
-
-      @test_queue ||= TestQueue.new
-    end
-
-    private
-
-    # Private: Enqueues the event.
-    #
-    # Returns Boolean of whether the item was added to the queue.
-    def enqueue(action)
-      if @test
-        test_queue << action
-        return true
-      end
-
-      if @queue.length < @max_queue_size
-        @queue << action
-        ensure_worker_running
-
-        true
-      else
-        @logger.warn 'Queue is full, dropping events. The :max_queue_size configuration parameter can be increased to prevent this from happening.'
-        false
-      end
-    end
-
-    def ensure_worker_running
-      return if worker_running?
-      @worker_mutex.synchronize do
-        return if worker_running?
-        @worker_thread = Thread.new do
-          @worker.run
-        end
-      end
-    end
-
-    def worker_running?
-      @worker_thread && @worker_thread.alive?
+    # Returns Boolean of whether the data was added to the queue.
+    def push(data)
+      worker.push(data)
     end
   end
 end

data/lib/brow/message_batch.rb

@@ -22,12 +22,21 @@ module Brow
 
     def_delegators :@messages, :empty?
     def_delegators :@messages, :length
+    def_delegators :@messages, :size
+    def_delegators :@messages, :count
 
-    attr_reader :uuid, :json_size
+    attr_reader :uuid, :json_size, :max_size
 
     def initialize(options = {})
       clear
-      @max_size = options[:max_size] || MAX_SIZE
+      @max_size = options.fetch(:max_size) {
+        ENV.fetch("BROW_BATCH_SIZE", MAX_SIZE).to_i
+      }
+
+      unless @max_size > 0
+        raise ArgumentError, ":max_size must be > 0 but was #{@max_size.inspect}"
+      end
+
       @logger = options.fetch(:logger) { Brow.logger }
     end
 
@@ -41,7 +50,7 @@ module Brow
       message_json_size = message_json.bytesize
 
       if message_too_big?(message_json_size)
-        @logger.error('a message exceeded the maximum allowed size')
+        @logger.error { "#{LOG_PREFIX} a message exceeded the maximum allowed size" }
       else
         @messages << message
         @json_size += message_json_size + 1 # One byte for the comma

data/lib/brow/transport.rb

@@ -3,94 +3,130 @@
 require 'net/http'
 require 'net/https'
 require 'json'
+require 'set'
 
 require_relative 'response'
 require_relative 'backoff_policy'
 
 module Brow
   class Transport
+    # Private: Default number of times to retry request.
     RETRIES = 10
-    HEADERS = {
-      "Accept" => "application/json",
-      "Content-Type" => "application/json",
-      "User-Agent" => "brow-ruby/#{Brow::VERSION}",
-      "Client-Language" => "ruby",
-      "Client-Language-Version" => "#{RUBY_VERSION} p#{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE})",
-      "Client-Platform" => RUBY_PLATFORM,
-      "Client-Engine" => defined?(RUBY_ENGINE) ? RUBY_ENGINE : "",
-      "Client-Pid" => Process.pid.to_s,
-      "Client-Thread" => Thread.current.object_id.to_s,
-      "Client-Hostname" => Socket.gethostname,
-    }
-
-    attr_reader :url
+
+    # Private: Default read timeout on requests.
+    READ_TIMEOUT = 8
+
+    # Private: Default open timeout on requests.
+    OPEN_TIMEOUT = 4
+
+    # Private: Default write timeout on requests.
+    WRITE_TIMEOUT = 4
+
+    # Private: URL schemes that this transport supports.
+    VALID_HTTP_SCHEMES = Set["http", "https"].freeze
+
+    # Private
+    attr_reader :url, :headers, :retries, :logger, :backoff_policy, :http
 
     def initialize(options = {})
-      @url = options[:url] || raise(ArgumentError, ":url is required to be present so we know where to send batches")
+      @url = options.fetch(:url) {
+        ENV.fetch("BROW_URL") {
+          raise ArgumentError, ":url is required to be present so we know where to send batches"
+        }
+      }
       @uri = URI.parse(@url)
 
+      unless VALID_HTTP_SCHEMES.include?(@uri.scheme)
+        raise ArgumentError, ":url was must be http(s) scheme but was #{@uri.scheme.inspect}"
+      end
+
       # Default path if people forget a slash.
       if @uri.path.nil? || @uri.path.empty?
         @uri.path = "/"
       end
 
-      @headers = HEADERS.merge(options[:headers] || {})
-      @retries = options[:retries] || RETRIES
+      @headers = options[:headers] || {}
+      @retries = options.fetch(:retries) {
+        ENV.fetch("BROW_RETRIES", RETRIES).to_i
+      }
+
+      unless @retries >= 0
+        raise ArgumentError, ":retries must be >= to 0 but was #{@retries.inspect}"
+      end
 
       @logger = options.fetch(:logger) { Brow.logger }
       @backoff_policy = options.fetch(:backoff_policy) {
-        Brow::BackoffPolicy.new
+        Brow::BackoffPolicy.new(options)
       }
 
       @http = Net::HTTP.new(@uri.host, @uri.port)
       @http.use_ssl = @uri.scheme == "https"
-      @http.read_timeout = options[:read_timeout] || 8
-      @http.open_timeout = options[:open_timeout] || 4
+
+      read_timeout = options.fetch(:read_timeout) {
+        ENV.fetch("BROW_READ_TIMEOUT", READ_TIMEOUT).to_f
+      }
+      @http.read_timeout = read_timeout if read_timeout
+
+      open_timeout = options.fetch(:open_timeout) {
+        ENV.fetch("BROW_OPEN_TIMEOUT", OPEN_TIMEOUT).to_f
+      }
+      @http.open_timeout = open_timeout if open_timeout
+
+      if RUBY_VERSION >= '2.6.0'
+        write_timeout = options.fetch(:write_timeout) {
+          ENV.fetch("BROW_WRITE_TIMEOUT", WRITE_TIMEOUT).to_f
+        }
+        @http.write_timeout = write_timeout if write_timeout
+      else
+        Kernel.warn("Warning: option :write_timeout requires Ruby version 2.6.0 or later")
+      end
     end
 
     # Sends a batch of messages to the API
     #
     # @return [Response] API response
     def send_batch(batch)
-      @logger.debug("Sending request for #{batch.length} items")
+      logger.debug { "#{LOG_PREFIX} Sending request for #{batch.length} items" }
 
-      last_response, exception = retry_with_backoff(@retries) do
+      last_response, exception = retry_with_backoff(retries) do
         response = send_request(batch)
-        status_code = response.code.to_i
-        should_retry = should_retry_request?(status_code, response.body)
-        @logger.debug("Response status code: #{status_code}")
-
-        [Response.new(status_code, nil), should_retry]
+        logger.debug { "#{LOG_PREFIX} Response: status=#{response.code}, body=#{response.body}" }
+        [Response.new(response.code.to_i, nil), retry?(response)]
       end
 
       if exception
-        @logger.error(exception.message)
-        exception.backtrace.each { |line| @logger.error(line) }
+        logger.error { "#{LOG_PREFIX} #{exception.message}" }
+        exception.backtrace.each { |line| logger.error(line) }
         Response.new(-1, exception.to_s)
       else
         last_response
       end
+    ensure
+      backoff_policy.reset
+      batch.clear
     end
 
     # Closes a persistent connection if it exists
     def shutdown
+      logger.info { "#{LOG_PREFIX} Transport shutting down" }
       @http.finish if @http.started?
     end
 
     private
 
-    def should_retry_request?(status_code, body)
+    def retry?(response)
+      status_code = response.code.to_i
       if status_code >= 500
         # Server error. Retry and log.
-        @logger.info("Server error: status=#{status_code}, body=#{body}")
+        logger.info { "#{LOG_PREFIX} Server error: status=#{status_code}, body=#{response.body}" }
         true
       elsif status_code == 429
-        # Rate limited
-        @logger.info "Rate limit error"
+        # Rate limited. Retry and log.
+        logger.info { "#{LOG_PREFIX} Rate limit error: body=#{response.body}" }
         true
       elsif status_code >= 400
         # Client error. Do not retry, but log.
-        @logger.error("Client error: status=#{status_code}, body=#{body}")
+        logger.error { "#{LOG_PREFIX} Client error: status=#{status_code}, body=#{response.body}" }
         false
       else
         false
@@ -112,13 +148,13 @@ module Brow
         result, should_retry = yield
         return [result, nil] unless should_retry
       rescue StandardError => error
-        @logger.debug "Request error: #{error}"
+        logger.debug { "#{LOG_PREFIX} Request error: #{error}" }
         should_retry = true
         caught_exception = error
       end
 
       if should_retry && (retries_remaining > 1)
-        @logger.debug("Retrying request, #{retries_remaining} retries left")
+        logger.debug { "#{LOG_PREFIX} Retrying request, #{retries_remaining} retries left" }
         sleep(@backoff_policy.next_interval.to_f / 1000)
         retry_with_backoff(retries_remaining - 1, &block)
       else
@@ -126,12 +162,23 @@ module Brow
       end
     end
 
-    # Sends a request for the batch, returns [status_code, body]
     def send_request(batch)
-      payload = batch.to_json
-      @http.start unless @http.started? # Maintain a persistent connection
-      request = Net::HTTP::Post.new(@uri.path, @headers)
-      @http.request(request, payload)
+      headers = {
+        "Accept" => "application/json",
+        "Content-Type" => "application/json",
+        "User-Agent" => "Brow v#{Brow::VERSION}",
+        "Client-Language" => "ruby",
+        "Client-Language-Version" => "#{RUBY_VERSION} p#{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE})",
+        "Client-Platform" => RUBY_PLATFORM,
+        "Client-Engine" => defined?(RUBY_ENGINE) ? RUBY_ENGINE : "",
+        "Client-Hostname" => Socket.gethostname,
+        "Client-Pid" => Process.pid.to_s,
+        "Client-Thread" => Thread.current.object_id.to_s,
+      }.merge(@headers)
+
+      @http.start unless @http.started?
+      request = Net::HTTP::Post.new(@uri.path, headers)
+      @http.request(request, batch.to_json)
     end
   end
 end

data/lib/brow/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Brow
-  VERSION = "0.1.0"
+  VERSION = "0.4.1"
 end

data/lib/brow/worker.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'thread'
+
 require_relative 'message_batch'
 require_relative 'transport'
 require_relative 'utils'
@@ -7,60 +9,186 @@ require_relative 'utils'
 module Brow
   # Internal: The Worker to pull items off the queue and put them
   class Worker
+    # Private: Noop default on error proc.
     DEFAULT_ON_ERROR = proc { |response| }
 
+    # Private: Object to enqueue to signal shutdown for worker.
+    SHUTDOWN = :__ಠ_ಠ__
+
+    # Private: Default number of seconds to wait to shutdown worker thread.
+    SHUTDOWN_TIMEOUT = 5
+
+    # Private: Default # of items that can be in queue before we start dropping data.
+    MAX_QUEUE_SIZE = 10_000
+
+    # Private
+    attr_reader :thread, :queue, :pid, :mutex, :on_error, :batch_size, :max_queue_size
+
+    # Private
+    attr_reader :logger, :transport, :shutdown_timeout
+
     # Internal: Creates a new worker
     #
     # The worker continuously takes messages off the queue and makes requests to
     # the api.
     #
-    # queue   - Queue synchronized between client and worker
     # options - The Hash of worker options.
-    #           batch_size - Fixnum of how many items to send in a batch.
-    #           on_error - Proc of what to do on an error.
-    #           transport - The Transport object to deliver batches.
-    #           logger - The Logger object for all log messages.
-    #           batch - The MessageBatch to collect messages and deliver batches
-    #                   via Transport.
-    def initialize(queue, options = {})
-      @queue = queue
-      @lock = Mutex.new
+    #   :queue - Queue synchronized between client and worker
+    #   :on_error - Proc of what to do on an error.
+    #   :batch_size - Fixnum of how many items to send in a batch.
+    #   :transport - The Transport object to deliver batches.
+    #   :logger - The Logger object for all log messages.
+    #   :batch - The MessageBatch to collect messages and deliver batches
+    #            via Transport.
+    #   :shutdown_timeout - The number of seconds to wait for the worker thread
+    #                       to join when shutting down.
+    #   :start_automatically - Should the client start the worker thread
+    #                          automatically and keep it running.
+    #   :shutdown_automatically - Should the client shutdown automatically or
+    #                             manually. If true, shutdown is automatic. If
+    #                             false, you'll need to handle this on your own.
+    def initialize(options = {})
+      @thread = nil
+      @queue = options.fetch(:queue) { Queue.new }
+      @pid = Process.pid
+      @mutex = Mutex.new
       options = Brow::Utils.symbolize_keys(options)
       @on_error = options[:on_error] || DEFAULT_ON_ERROR
-      @transport = options.fetch(:transport) { Transport.new(options) }
       @logger = options.fetch(:logger) { Brow.logger }
-      @batch = options.fetch(:batch) { MessageBatch.new(max_size: options[:batch_size]) }
+      @transport = options.fetch(:transport) { Transport.new(options) }
+
+      @batch_size = options.fetch(:batch_size) {
+        ENV.fetch("BROW_BATCH_SIZE", MessageBatch::MAX_SIZE).to_i
+      }
+      @max_queue_size = options.fetch(:max_queue_size) {
+        ENV.fetch("BROW_MAX_QUEUE_SIZE", MAX_QUEUE_SIZE).to_i
+      }
+      @shutdown_timeout = options.fetch(:shutdown_timeout) {
+        ENV.fetch("BROW_SHUTDOWN_TIMEOUT", SHUTDOWN_TIMEOUT).to_f
+      }
+
+      if @batch_size <= 0
+        raise ArgumentError, ":batch_size must be greater than 0"
+      end
+
+      if @max_queue_size <= 0
+        raise ArgumentError, ":max_queue_size must be greater than 0"
+      end
+
+      if @shutdown_timeout <= 0
+        raise ArgumentError, ":shutdown_timeout must be greater than 0"
+      end
+
+      @start_automatically = options.fetch(:start_automatically, true)
+
+      if options.fetch(:shutdown_automatically, true)
+        at_exit { stop }
+      end
+    end
+
+    def push(data)
+      raise ArgumentError, "data must be a Hash" unless data.is_a?(Hash)
+      start if @start_automatically
+
+      data = Utils.isoify_dates(data)
+
+      if queue.length < max_queue_size
+        queue << data
+        true
+      else
+        logger.warn { "#{LOG_PREFIX} Queue is full, dropping events. The :max_queue_size configuration parameter can be increased to prevent this from happening." }
+        false
+      end
+    end
+
+    def start
+      reset if forked?
+      ensure_worker_running
+    end
+
+    def stop
+      queue << SHUTDOWN
+
+      if @thread
+        begin
+          if @thread.join(shutdown_timeout)
+            logger.info { "#{LOG_PREFIX} Worker thread [#{@thread.object_id}] joined sucessfully" }
+          else
+            logger.info { "#{LOG_PREFIX} Worker thread [#{@thread.object_id}] did not join successfully" }
+          end
+        rescue => error
+          logger.info { "#{LOG_PREFIX} Worker thread [#{@thread.object_id}] error shutting down: #{error.inspect}" }
+        end
+      end
     end
 
     # Internal: Continuously runs the loop to check for new events
     def run
-      until Thread.current[:should_exit]
-        return if @queue.empty?
+      batch = MessageBatch.new(max_size: batch_size)
 
-        @lock.synchronize do
-          consume_message_from_queue! until @batch.full? || @queue.empty?
-        end
+      loop do
+        message = queue.pop
 
-        response = @transport.send_batch @batch
-        @on_error.call(response) unless response.status == 200
+        case message
+        when SHUTDOWN
+          logger.info { "#{LOG_PREFIX} Worker shutting down" }
+          send_batch(batch) unless batch.empty?
+          break
+        else
+          begin
+            batch << message
+          rescue MessageBatch::JSONGenerationError => error
+            on_error.call(Response.new(-1, error))
+          end
 
-        @lock.synchronize { @batch.clear }
+          send_batch(batch) if batch.full?
+        end
       end
     ensure
-      @transport.shutdown
+      transport.shutdown
     end
 
-    # Internal: Check whether we have outstanding requests.
-    def requesting?
-      @lock.synchronize { !@batch.empty? }
+    private
+
+    def forked?
+      pid != Process.pid
     end
 
-    private
+    def ensure_worker_running
+      # Return early if thread is alive and avoid the mutex lock and unlock.
+      return if thread_alive?
+
+      # If another thread is starting worker thread, then return early so this
+      # thread can enqueue and move on with life.
+      return unless mutex.try_lock
+
+      begin
+        return if thread_alive?
+        @thread = Thread.new { run }
+        logger.debug { "#{LOG_PREFIX} Worker thread [#{@thread.object_id}] started" }
+      ensure
+        mutex.unlock
+      end
+    end
+
+    def thread_alive?
+      @thread && @thread.alive?
+    end
+
+    def reset
+      @pid = Process.pid
+      mutex.unlock if mutex.locked?
+      queue.clear
+    end
+
+    def send_batch(batch)
+      response = transport.send_batch(batch)
+
+      unless response.status == 200
+        on_error.call(response)
+      end
 
-    def consume_message_from_queue!
-      @batch << @queue.pop
-    rescue MessageBatch::JSONGenerationError => error
-      @on_error.call(Response.new(-1, error))
+      response
     end
   end
 end

data/lib/brow.rb

@@ -6,17 +6,18 @@ require "logger"
 module Brow
   class Error < StandardError; end
 
+  # Private
+  LOG_PREFIX = "[brow]"
+
   # Public: Returns the logger instance to use for logging of things.
   def self.logger
     return @logger if @logger
 
-    base_logger = if defined?(Rails)
+    @logger = if defined?(Rails)
       Rails.logger
     else
       Logger.new(STDOUT)
     end
-
-    @logger = PrefixedLogger.new(base_logger, "[brow]")
   end
 
   # Public: Sets the logger instance to use for logging things.
@@ -26,4 +27,3 @@ module Brow
 end
 
 require_relative "brow/client"
-require_relative "brow/prefixed_logger"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: brow
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.4.1
 platform: ruby
 authors:
 - John Nunemaker
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-10-20 00:00:00.000000000 Z
+date: 2021-11-06 00:00:00.000000000 Z
 dependencies: []
 description: 
 email:
@@ -26,13 +26,14 @@ files:
 - bin/console
 - bin/setup
 - examples/basic.rb
+- examples/echo_server.rb
+- examples/forked.rb
+- examples/long_running.rb
 - lib/brow.rb
 - lib/brow/backoff_policy.rb
 - lib/brow/client.rb
 - lib/brow/message_batch.rb
-- lib/brow/prefixed_logger.rb
 - lib/brow/response.rb
-- lib/brow/test_queue.rb
 - lib/brow/transport.rb
 - lib/brow/utils.rb
 - lib/brow/version.rb

data/lib/brow/prefixed_logger.rb

@@ -1,25 +0,0 @@
-module Brow
-  # Internal: Wraps an existing logger and adds a prefix to all messages.
-  class PrefixedLogger
-    def initialize(logger, prefix)
-      @logger = logger
-      @prefix = prefix
-    end
-
-    def debug(message)
-      @logger.debug("#{@prefix} #{message}")
-    end
-
-    def info(message)
-      @logger.info("#{@prefix} #{message}")
-    end
-
-    def warn(message)
-      @logger.warn("#{@prefix} #{message}")
-    end
-
-    def error(message)
-      @logger.error("#{@prefix} #{message}")
-    end
-  end
-end

data/lib/brow/test_queue.rb

@@ -1,29 +0,0 @@
-# frozen_string_literal: true
-
-module Brow
-  # Public: The test queue to use if the `Client` is in test mode. Keeps all
-  # messages in an array so you can add assertions.
-  #
-  # Be sure to reset before each test case.
-  class TestQueue
-    attr_reader :messages
-
-    def initialize
-      reset
-    end
-
-    def count
-      messages.count
-    end
-    alias_method :size, :count
-    alias_method :length, :count
-
-    def <<(message)
-      messages << message
-    end
-
-    def reset
-      @messages = []
-    end
-  end
-end