bc-prometheus-ruby 0.2.1 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf312395a8013ec6faae2f338f74480cdd6b419ea117f0e872fd161c4da54c94
-  data.tar.gz: f149b260bae32001429d95ed39a55f40060feb47412fc8df205ddf2974259374
+  metadata.gz: 005ae1dcfa809b70f6842f4448a99101da4218b176b7cb2a7efd575cda266961
+  data.tar.gz: a9f8f5b8c0601294f1a6b4d123a73eaef140724b55adbfca0d03adbb9037c942
 SHA512:
-  metadata.gz: 42f9133f0ef55104154f4da0524895f57f9ebbbaf79c03e3af959c5cd1603782a9b27d92d20f1c349b4119c204b0f23ab37d51ae669055a4aa01a170f9db72ff
-  data.tar.gz: 99163c006b2e1beaa5a06612ed59bb437bfb123022cfc2f214d9a3974ebf15a283fd2e3596cbf1b676ebc8c625da09dbac5738074fa675f68b93e3023c9440d3
+  metadata.gz: 41fd284c235b974405840ed5c95321442d37f0f3f84d451b8ce988247ddfa8c3139a70de74a1a3734e4d78de53671787efcfbc9223b51997432e7b0f66f22251
+  data.tar.gz: f782538242bb23e53404700a31ab727fec937ced0231207ecf1ae9f2dc11a35b043c2a5b8b482515abed8fe3eb5484659192a2377571825e09657671d2d69286

data/CHANGELOG.md

@@ -2,6 +2,33 @@ Changelog for the bc-prometheus-ruby gem.
 
 ### Pending Release
 
+### 0.3.1
+
+- Update prometheus_exporter dependency to ~> 0.5 to fix memory leaks
+
+### 0.3.0
+
+- Support for only Ruby 2.6+ going forward
+- Updated README around custom metrics and collectors
+- Add ability to pass custom resque and hutch Collectors/TypeCollectors
+- Add ENV support for all configuration elements
+- Fix issue where base collector did not use Bigcommerce::Prometheus.client
+- Expose new `push` method for Collectors::Base to ease use of custom ad hoc metrics 
+
+### 0.2.4
+
+- Fix cant modify frozen array error when using bc-prometheus-ruby outside a web process 
+  but within rails
+  
+### 0.2.3
+
+- Set default STDOUT logger to INFO level
+- Fix bug with resque type collector
+
+### 0.2.2
+
+- Fix missing inheritance for resque collector
+ 
 ### 0.2.1
 
 - Prevent starting of Puma integration if Puma is not loaded

data/README.md

@@ -49,6 +49,127 @@ After requiring the main file, you can further configure with:
 | server_port | The port to run the exporter on | 9394 |
 | process_name | What the current process name is. Used in logging. | `ENV['PROCESS']` | 
 
+## Custom Collectors
+
+To create custom metrics and collectors, simply create two files: a collector (the class that runs and collects metrics),
+and the type collector, which runs on the threaded prometheus server and 
+
+### Type Collector
+
+First, create a type collector. Note that the "type" of this will be the full name of the class, with `TypeCollector`
+stripped. This is important later. Our example here will have a "type" of "app".
+
+```ruby
+class AppTypeCollector < ::Bigcommerce::Prometheus::TypeCollectors::Base
+  def build_metrics
+    {
+      honks: PrometheusExporter::Metric::Counter.new('honks', 'Running counter of honks'),
+      points: PrometheusExporter::Metric::Gauge.new('points', 'Current amount of points')
+    }
+  end
+
+  def collect_metrics(data:, labels: {})
+    metric(:points).observe(data.fetch('points', 0))
+    metric(:honks).observe(1, labels) if data.fetch('honks', 0).to_i.positive?
+  end
+end
+```
+
+There are two important methods here: `build_metrics`, which registers the different metrics you want to measure, and
+`collect_metrics`, which actually takes in the metrics and prepares them to be rendered so that Prometheus can scrape
+them.
+
+Note also in the example the different ways of observing Gauges vs Counters. 
+
+### Collector
+
+Next, create a collector. Your "type" of the Collector must match the type collector above, so that bc-prometheus-ruby
+knows how to map the metrics to the right TypeCollector. This is inferred from the class name. Here, it is "app":
+
+```ruby
+class AppCollector < ::Bigcommerce::Prometheus::Collectors::Base
+  def honk!
+    push(
+      honks: 1,
+      custom_labels: {
+        volume: 'loud'
+      }
+    )
+  end
+
+  def collect(metrics)
+    metrics[:points] = rand(1..100)
+    metrics
+  end
+end
+```
+
+There are two types of metrics here: on-demand, and polled. Let's look at the first:
+
+#### On-Demand Metrics
+
+To issue an on-demand metric (usually a counter) that then automatically updates, in your application code, you would
+then run:
+
+```ruby
+app_collector = AppCollector.new
+app_collector.honk!
+```
+
+This will "push" the metrics to our `AppTypeCollector` instance, which will render them as:
+
+```
+# HELP ruby_honks Running counter of honks
+# TYPE ruby_honks counter
+ruby_honks{volume="loud"} 2
+```
+
+As you can see this will respect any custom labels we push in as well.
+
+### Polling Metrics
+
+Using our same AppCollector, if you note the `collect` method: this method will run on a 15 second polled basis
+(the frequency of which is configurable in the initializer of the AppCollector). Here we're just spitting out random
+points, so it'll look something like this:
+
+```
+# HELP ruby_points Current amount of points
+# TYPE ruby_points gauge
+ruby_points 42
+```
+
+### Registering Our Collectors
+
+Each different type of integration will need to have the collectors passed into them, where appropriate. For example,
+if we want these collectors to run on our web, resque, and hutch processes, we'll need to:
+
+```ruby
+::Bigcommerce::Prometheus.configure do |c|
+  c.web_collectors = [AppCollector]
+  c.web_type_collectors = [AppTypeCollector.new]
+  c.resque_collectors = [AppCollector]
+  c.resque_type_collectors = [AppTypeCollector.new]
+  c.hutch_collectors = [AppCollector]
+  c.hutch_type_collectors = [AppTypeCollector.new]
+end
+```
+
+#### Custom Server Integrations
+
+For custom integrations that initialize their own server, you'll need to pass your TypeCollector instance via the 
+`.add_type_collector` method on the prometheus server instance before starting it:
+
+```ruby
+server = ::Bigcommerce::Prometheus::Server.new
+Bigcommerce::Prometheus.web_type_collectors.each do |tc|
+  server.add_type_collector(tc)
+end
+
+# and for polling:
+
+AppCollector.start
+```
+
 ## License
 
 Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved 

data/bc-prometheus-ruby.gemspec

@@ -31,6 +31,7 @@ Gem::Specification.new do |spec|
 
   spec.files         = Dir['README.md', 'CHANGELOG.md', 'CODE_OF_CONDUCT.md', 'lib/**/*', 'bc-prometheus-ruby.gemspec']
   spec.require_paths = ['lib']
+  spec.required_ruby_version = '>= 2.6'
 
   spec.add_development_dependency 'rake', '>= 10.0'
   spec.add_development_dependency 'rspec', '>= 3.8'
@@ -42,6 +43,6 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'simplecov', '>= 0.16'
 
   spec.add_runtime_dependency 'bigcommerce-multitrap', '~> 0.1'
-  spec.add_runtime_dependency 'prometheus_exporter', '~> 0.4'
+  spec.add_runtime_dependency 'prometheus_exporter', '~> 0.5'
   spec.add_runtime_dependency 'thin', '~> 1.7'
 end

data/lib/bigcommerce/prometheus/client.rb

@@ -24,16 +24,16 @@ module Bigcommerce
       include Singleton
       include Loggable
 
-      def initialize
+      def initialize(host: nil, port: nil, max_queue_size: nil, thread_sleep: nil, custom_labels: nil, process_name: nil)
         super(
-          host: Bigcommerce::Prometheus.server_host,
-          port: Bigcommerce::Prometheus.server_port,
-          max_queue_size: Bigcommerce::Prometheus.client_max_queue_size,
-          thread_sleep: Bigcommerce::Prometheus.client_thread_sleep,
-          custom_labels: Bigcommerce::Prometheus.client_custom_labels
+          host: host || Bigcommerce::Prometheus.server_host,
+          port: port || Bigcommerce::Prometheus.server_port,
+          max_queue_size: max_queue_size || Bigcommerce::Prometheus.client_max_queue_size,
+          thread_sleep: thread_sleep || Bigcommerce::Prometheus.client_thread_sleep,
+          custom_labels: custom_labels || Bigcommerce::Prometheus.client_custom_labels
         )
         PrometheusExporter::Client.default = self
-        @process_name = ::Bigcommerce::Prometheus.process_name
+        @process_name = process_name || ::Bigcommerce::Prometheus.process_name
       end
 
       ##
@@ -55,7 +55,7 @@ module Bigcommerce
 
       ##
       # @param [String] path
-      # @return [URI]
+      # @return [Module<URI>]
       #
       def uri_path(path)
         URI("http://#{@host}:#{@port}#{path}")

data/lib/bigcommerce/prometheus/collectors/base.rb

@@ -49,15 +49,15 @@ module Bigcommerce
         end
 
         ##
-        # @param [PrometheusExporter::Client] client
+        # @param [Bigcommerce::Prometheus::Client] client
         # @param [String] type
         # @param [Integer] frequency
         # @param [Hash] options
         #
         def initialize(client: nil, type: nil, frequency: nil, options: nil)
-          @client = client || PrometheusExporter::Client.default
+          @client = client || Bigcommerce::Prometheus.client
           @type = type || self.class.to_s.downcase.gsub('::', '_').gsub('collector', '')
-          @frequency = frequency || 15
+          @frequency = frequency || Bigcommerce::Prometheus.collector_collection_frequency
           @options = options || {}
           @logger = Bigcommerce::Prometheus.logger
         end
@@ -66,12 +66,9 @@ module Bigcommerce
         # Run the collector and send stats
         #
         def run
-          metrics = { type: @type }
+          metrics = {}
           metrics = collect(metrics)
-          @logger.debug "[bigcommerce-prometheus] Pushing #{@type} metrics to type collector: #{metrics.inspect}"
-          @client.send_json(metrics)
-        rescue StandardError => e
-          @logger.error("Prometheus Exporter Failed To Collect #{@type} Stats #{e}")
+          push(metrics)
         ensure
           sleep @frequency
         end
@@ -85,6 +82,19 @@ module Bigcommerce
         def collect(metrics = {})
           metrics
         end
+
+        private
+
+        ##
+        # @param [Hash] metric
+        #
+        def push(metric)
+          metric[:type] = @type unless metric.key?(:type)
+          @logger.debug("[bigcommerce-prometheus] Pushing #{metric[:type]} metrics to type collector: #{metric.inspect}")
+          @client.send_json(metric)
+        rescue StandardError => e
+          @logger.error("[bigcommerce-prometheus] Prometheus Exporter failed to send #{metric[:type]} stats to type collector #{e}")
+        end
       end
     end
   end

data/lib/bigcommerce/prometheus/collectors/resque.rb

@@ -21,7 +21,7 @@ module Bigcommerce
       ##
       # Collect metrics to push to the server type collector
       #
-      class Resque
+      class Resque < Base
         ##
         # @param [Hash] metrics
         # @return [Hash]

data/lib/bigcommerce/prometheus/configuration.rb

@@ -23,18 +23,32 @@ module Bigcommerce
     module Configuration
       VALID_CONFIG_KEYS = {
         logger: nil,
+        enabled: ENV.fetch('PROMETHEUS_ENABLED', 1).to_i.positive?,
+
+        # Client configuration
         client_custom_labels: nil,
-        client_max_queue_size: 10_000,
-        client_thread_sleep: 0.5,
-        enabled: true,
-        puma_collection_frequency: 30,
-        puma_process_label: 'web',
-        resque_collection_frequency: 30,
-        resque_process_label: 'resque',
-        server_host: '0.0.0.0',
-        server_port: PrometheusExporter::DEFAULT_PORT,
-        server_timeout: PrometheusExporter::DEFAULT_TIMEOUT,
-        server_prefix: PrometheusExporter::DEFAULT_PREFIX,
+        client_max_queue_size: ENV.fetch('PROMETHEUS_CLIENT_MAX_QUEUE_SIZE', 10_000).to_i,
+        client_thread_sleep: ENV.fetch('PROMETHEUS_CLIENT_THREAD_SLEEP', 0.5).to_f,
+
+        # Integration configuration
+        puma_collection_frequency: ENV.fetch('PROMETHEUS_PUMA_COLLECTION_FREQUENCY', 30).to_i,
+        puma_process_label: ENV.fetch('PROMETHEUS_PUMA_PROCESS_LABEL', 'web').to_s,
+        resque_collection_frequency: ENV.fetch('PROMETHEUS_RESQUE_COLLECTION_FREQUENCY', 30).to_i,
+        resque_process_label: ENV.fetch('PROMETHEUS_REQUEST_PROCESS_LABEL', 'resque').to_s,
+
+        # Server configuration
+        not_found_body: ENV.fetch('PROMETHEUS_SERVER_NOT_FOUND_BODY', 'Not Found! The Prometheus Ruby Exporter only listens on /metrics and /send-metrics').to_s,
+        server_host: ENV.fetch('PROMETHEUS_SERVER_HOST', '0.0.0.0').to_s,
+        server_port: ENV.fetch('PROMETHEUS_SERVER_PORT', PrometheusExporter::DEFAULT_PORT).to_i,
+        server_timeout: ENV.fetch('PROMETHEUS_DEFAULT_TIMEOUT', PrometheusExporter::DEFAULT_TIMEOUT).to_i,
+        server_prefix: ENV.fetch('PROMETHEUS_DEFAULT_PREFIX', PrometheusExporter::DEFAULT_PREFIX).to_s,
+
+        # Custom collector configuration
+        collector_collection_frequency: ENV.fetch('PROMETHEUS_DEFAULT_COLLECTOR_COLLECTION_FREQUENCY_SEC', 15).to_i,
+        hutch_collectors: [],
+        hutch_type_collectors: [],
+        resque_collectors: [],
+        resque_type_collectors: [],
         web_collectors: [],
         web_type_collectors: []
       }.freeze
@@ -85,12 +99,7 @@ module Bigcommerce
           send("#{k}=".to_sym, v)
         end
         determine_logger
-        self.enabled = ENV.fetch('PROMETHEUS_ENABLED', 1).to_i.positive?
-        self.server_host = ENV.fetch('PROMETHEUS_SERVER_HOST', '0.0.0.0').to_s
-        self.server_port = ENV.fetch('PROMETHEUS_SERVER_PORT', PrometheusExporter::DEFAULT_PORT).to_i
 
-        self.puma_process_label = ENV.fetch('PROMETHEUS_PUMA_PROCESS_LABEL', 'web').to_s
-        self.puma_collection_frequency = ENV.fetch('PROMETHEUS_PUMA_COLLECTION_FREQUENCY', 30).to_i
         self.web_type_collectors = []
       end
 
@@ -110,6 +119,7 @@ module Bigcommerce
         else
           require 'logger'
           self.logger = ::Logger.new(STDOUT)
+          self.logger.level = ::Logger::Severity::INFO
         end
       end
 

data/lib/bigcommerce/prometheus/instrumentors/hutch.rb

@@ -31,6 +31,8 @@ module Bigcommerce
           @server_port = Bigcommerce::Prometheus.server_port
           @server_timeout = Bigcommerce::Prometheus.server_timeout
           @server_prefix = Bigcommerce::Prometheus.server_prefix
+          @collectors = Bigcommerce::Prometheus.hutch_collectors || []
+          @type_collectors = Bigcommerce::Prometheus.hutch_type_collectors || []
         end
 
         ##
@@ -44,6 +46,9 @@ module Bigcommerce
 
           server.add_type_collector(PrometheusExporter::Server::ActiveRecordCollector.new)
           server.add_type_collector(PrometheusExporter::Server::HutchCollector.new)
+          @type_collectors.each do |tc|
+            server.add_type_collector(tc)
+          end
           server.start
           setup_middleware
         rescue StandardError => e
@@ -65,6 +70,9 @@ module Bigcommerce
           require 'hutch'
           ::Hutch::Config.set(:tracer, PrometheusExporter::Instrumentation::Hutch)
           @app.middleware.unshift(PrometheusExporter::Middleware, client: Bigcommerce::Prometheus.client)
+          @collectors.each(&:start)
+        rescue StandardError => e
+          logger.warn "[bigcommerce-prometheus][#{@process_name}] Failed to setup hutch prometheus middleware: #{e.message}"
         end
       end
     end

data/lib/bigcommerce/prometheus/instrumentors/resque.rb

@@ -31,6 +31,8 @@ module Bigcommerce
           @server_port = Bigcommerce::Prometheus.server_port
           @server_timeout = Bigcommerce::Prometheus.server_timeout
           @server_prefix = Bigcommerce::Prometheus.server_prefix
+          @collectors = Bigcommerce::Prometheus.resque_collectors || []
+          @type_collectors = Bigcommerce::Prometheus.resque_type_collectors || []
         end
 
         ##
@@ -44,6 +46,9 @@ module Bigcommerce
 
           server.add_type_collector(PrometheusExporter::Server::ActiveRecordCollector.new)
           server.add_type_collector(Bigcommerce::Prometheus::TypeCollectors::Resque.new)
+          @type_collectors.each do |tc|
+            server.add_type_collector(tc)
+          end
           server.start
           setup_middleware
         rescue StandardError => e
@@ -63,7 +68,8 @@ module Bigcommerce
         def setup_middleware
           logger.info "[bigcommerce-prometheus][#{@process_name}] Setting up resque prometheus middleware"
           ::Resque.before_first_fork do
-            ::Bigcommerce::Prometheus::Integrations::Resque.start
+            ::Bigcommerce::Prometheus::Integrations::Resque.start(client: Bigcommerce::Prometheus.client)
+            @collectors.each(&:start)
           end
         end
       end

data/lib/bigcommerce/prometheus/instrumentors/web.rb

@@ -77,13 +77,15 @@ module Bigcommerce
         def setup_after_fork
           @app.config.after_fork_callbacks = [] unless @app.config.after_fork_callbacks
           @app.config.after_fork_callbacks << lambda do
-            ::Bigcommerce::Prometheus::Integrations::Puma.start
+            ::Bigcommerce::Prometheus::Integrations::Puma.start(client: Bigcommerce::Prometheus.client)
             @collectors.each(&:start)
           end
         end
 
         def setup_middleware
           @app.middleware.unshift(PrometheusExporter::Middleware, client: Bigcommerce::Prometheus.client)
+        rescue StandardError => e
+          logger.warn "[bc-prometheus-ruby] Failed to attach app middleware in web instrumentor: #{e.message}"
         end
       end
     end

data/lib/bigcommerce/prometheus/integrations/railtie.rb

@@ -22,7 +22,7 @@ module Bigcommerce
       # Railtie for automatic configuration of Rails environments
       #
       class Railtie < ::Rails::Railtie
-        config.after_initialize do |app|
+        initializer 'zzz.bc_prometheus_ruby.configure_rails_initialization' do |app|
           Bigcommerce::Prometheus::Instrumentors::Web.new(app: app).start
         end
       end

data/lib/bigcommerce/prometheus/integrations/resque.rb

@@ -25,13 +25,13 @@ module Bigcommerce
         ##
         # Start the resque integration
         #
-        def self.start
+        def self.start(client: nil)
           ::PrometheusExporter::Instrumentation::Process.start(
-            client: ::Bigcommerce::Prometheus.client,
+            client: client || ::Bigcommerce::Prometheus.client,
             type: ::Bigcommerce::Prometheus.resque_process_label
           )
           ::Bigcommerce::Prometheus::Collectors::Resque.start(
-            client: ::Bigcommerce::Prometheus.client,
+            client: client || ::Bigcommerce::Prometheus.client,
             frequency: ::Bigcommerce::Prometheus.resque_collection_frequency
           )
         end

data/lib/bigcommerce/prometheus/servers/thin/controllers/not_found_controller.rb

@@ -26,7 +26,7 @@ module Bigcommerce
           class NotFoundController < BaseController
             def call
               @response.status = 404
-              @response.body << 'Not Found! The Prometheus Ruby Exporter only listens on /metrics and /send-metrics'
+              @response.body << Bigcommerce::Prometheus.not_found_body
             end
           end
         end

data/lib/bigcommerce/prometheus/servers/thin/server.rb

@@ -24,9 +24,9 @@ module Bigcommerce
         #
         class Server < ::Thin::Server
           def initialize(port:, host: nil, timeout: nil, logger: nil)
-            @port = port || ::PrometheusExporter::DEFAULT_PORT
-            @host = host || '0.0.0.0'
-            @timeout = timeout || ::PrometheusExporter::DEFAULT_TIMEOUT
+            @port = port || ::Bigcommerce::Prometheus.server_port
+            @host = host || ::Bigcommerce::Prometheus.server_host
+            @timeout = timeout || ::Bigcommerce::Prometheus.server_timeout
             @logger = logger || ::Bigcommerce::Prometheus.logger
             @rack_app = ::Bigcommerce::Prometheus::Servers::Thin::RackApp.new(timeout: timeout, logger: logger)
             super(@host, @port, @rack_app)

data/lib/bigcommerce/prometheus/type_collectors/resque.rb

@@ -40,11 +40,11 @@ module Bigcommerce
         # Collect resque metrics from input data
         #
         def collect_metrics(data:, labels: {})
-          metric(:workers_total).observe(obj['workers_total'], labels)
-          metric(:jobs_failed_total).observe(obj['jobs_failed_total'], labels)
-          metric(:jobs_pending_total).observe(obj['jobs_pending_total'], labels)
-          metric(:jobs_processed_total).observe(obj['jobs_processed_total'], labels)
-          metric(:queues_total).observe(obj['queues_total'], labels)
+          metric(:workers_total).observe(data['workers_total'], labels)
+          metric(:jobs_failed_total).observe(data['jobs_failed_total'], labels)
+          metric(:jobs_pending_total).observe(data['jobs_pending_total'], labels)
+          metric(:jobs_processed_total).observe(data['jobs_processed_total'], labels)
+          metric(:queues_total).observe(data['queues_total'], labels)
 
           data['queues'].each do |name, size|
             metric(:queue_sizes).observe(size, labels.merge(queue: name))

data/lib/bigcommerce/prometheus/version.rb

@@ -17,6 +17,6 @@
 #
 module Bigcommerce
   module Prometheus
-    VERSION = '0.2.1'
+    VERSION = '0.3.1'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bc-prometheus-ruby
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.1
 platform: ruby
 authors:
 - Shaun McCormick
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-01-03 00:00:00.000000000 Z
+date: 2020-08-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -142,14 +142,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
 - !ruby/object:Gem::Dependency
   name: thin
   requirement: !ruby/object:Gem::Requirement
@@ -203,7 +203,7 @@ homepage: https://github.com/bigcommerce/bc-prometheus-ruby
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -211,7 +211,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.6'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
@@ -219,7 +219,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.0.6
-signing_key: 
+signing_key:
 specification_version: 4
 summary: Simple integration of ruby and puma servers with prometheus
 test_files: []