bc-prometheus-ruby 0.2.4 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 777810436bcae4dfe552e0376be8e174b7d751497e4e9fc7b5ada857d606cc50
-  data.tar.gz: 720817317a900dee51a5c8c3e418f3012e1695150326cce6b23d031b5f5922cb
+  metadata.gz: 31329daa503c8b88c1c31df265199e8fd417ddd66b10d64d3329246e43e0f215
+  data.tar.gz: f1a9a94e4a3ca36f5bbf4e94fe916865c661b3eabc8e9edf038c3a8eb0fbcbdc
 SHA512:
-  metadata.gz: 68dce4e18b32c60a9a857050934754bbc659242ad34928414c5a437dd5990d2cc46eca24aa7e7a8f5ec44743856109332a6b65c9deef4e64bc2b6cedfc77ea56
-  data.tar.gz: 84f41b8ab19e3b650e34f09868fa8d9c6f46f355ad96898cab37fe9724d21ee78a77072ca0e944a74d374f04e227a1632e9b46ce68e69a0fc112f2102f4b5212
+  metadata.gz: '08ed46adf2df0d5d3ec9e6e97e05598bb8e9cdf467b119369427aa0f1aae5174fdac8ef6782845578df2c4f8c202ba0cb21089e7f3d55d09fde590e701cd6644'
+  data.tar.gz: 56b31060bba9d40eeb82e787798c2b7114b2d8869fd9ac8ba68f6cc7f123a534b7b91df17cf06ea5af980b2e6d120f80a3ee394ae9fefdb78a7ec38b9a8f367a

data/CHANGELOG.md

@@ -2,6 +2,15 @@ Changelog for the bc-prometheus-ruby gem.
 
 ### Pending Release
 
+### 0.3.0
+
+- Support for only Ruby 2.6+ going forward
+- Updated README around custom metrics and collectors
+- Add ability to pass custom resque and hutch Collectors/TypeCollectors
+- Add ENV support for all configuration elements
+- Fix issue where base collector did not use Bigcommerce::Prometheus.client
+- Expose new `push` method for Collectors::Base to ease use of custom ad hoc metrics 
+
 ### 0.2.4
 
 - Fix cant modify frozen array error when using bc-prometheus-ruby outside a web process 

data/README.md

@@ -49,6 +49,127 @@ After requiring the main file, you can further configure with:
 | server_port | The port to run the exporter on | 9394 |
 | process_name | What the current process name is. Used in logging. | `ENV['PROCESS']` | 
 
+## Custom Collectors
+
+To create custom metrics and collectors, simply create two files: a collector (the class that runs and collects metrics),
+and the type collector, which runs on the threaded prometheus server and 
+
+### Type Collector
+
+First, create a type collector. Note that the "type" of this will be the full name of the class, with `TypeCollector`
+stripped. This is important later. Our example here will have a "type" of "app".
+
+```ruby
+class AppTypeCollector < ::Bigcommerce::Prometheus::TypeCollectors::Base
+  def build_metrics
+    {
+      honks: PrometheusExporter::Metric::Counter.new('honks', 'Running counter of honks'),
+      points: PrometheusExporter::Metric::Gauge.new('points', 'Current amount of points')
+    }
+  end
+
+  def collect_metrics(data:, labels: {})
+    metric(:points).observe(data.fetch('points', 0))
+    metric(:honks).observe(1, labels) if data.fetch('honks', 0).to_i.positive?
+  end
+end
+```
+
+There are two important methods here: `build_metrics`, which registers the different metrics you want to measure, and
+`collect_metrics`, which actually takes in the metrics and prepares them to be rendered so that Prometheus can scrape
+them.
+
+Note also in the example the different ways of observing Gauges vs Counters. 
+
+### Collector
+
+Next, create a collector. Your "type" of the Collector must match the type collector above, so that bc-prometheus-ruby
+knows how to map the metrics to the right TypeCollector. This is inferred from the class name. Here, it is "app":
+
+```ruby
+class AppCollector < ::Bigcommerce::Prometheus::Collectors::Base
+  def honk!
+    push(
+      honks: 1,
+      custom_labels: {
+        volume: 'loud'
+      }
+    )
+  end
+
+  def collect(metrics)
+    metrics[:points] = rand(1..100)
+    metrics
+  end
+end
+```
+
+There are two types of metrics here: on-demand, and polled. Let's look at the first:
+
+#### On-Demand Metrics
+
+To issue an on-demand metric (usually a counter) that then automatically updates, in your application code, you would
+then run:
+
+```ruby
+app_collector = AppCollector.new
+app_collector.honk!
+```
+
+This will "push" the metrics to our `AppTypeCollector` instance, which will render them as:
+
+```
+# HELP ruby_honks Running counter of honks
+# TYPE ruby_honks counter
+ruby_honks{volume="loud"} 2
+```
+
+As you can see this will respect any custom labels we push in as well.
+
+### Polling Metrics
+
+Using our same AppCollector, if you note the `collect` method: this method will run on a 15 second polled basis
+(the frequency of which is configurable in the initializer of the AppCollector). Here we're just spitting out random
+points, so it'll look something like this:
+
+```
+# HELP ruby_points Current amount of points
+# TYPE ruby_points gauge
+ruby_points 42
+```
+
+### Registering Our Collectors
+
+Each different type of integration will need to have the collectors passed into them, where appropriate. For example,
+if we want these collectors to run on our web, resque, and hutch processes, we'll need to:
+
+```ruby
+::Bigcommerce::Prometheus.configure do |c|
+  c.web_collectors = [AppCollector]
+  c.web_type_collectors = [AppTypeCollector.new]
+  c.resque_collectors = [AppCollector]
+  c.resque_type_collectors = [AppTypeCollector.new]
+  c.hutch_collectors = [AppCollector]
+  c.hutch_type_collectors = [AppTypeCollector.new]
+end
+```
+
+#### Custom Server Integrations
+
+For custom integrations that initialize their own server, you'll need to pass your TypeCollector instance via the 
+`.add_type_collector` method on the prometheus server instance before starting it:
+
+```ruby
+server = ::Bigcommerce::Prometheus::Server.new
+Bigcommerce::Prometheus.web_type_collectors.each do |tc|
+  server.add_type_collector(tc)
+end
+
+# and for polling:
+
+AppCollector.start
+```
+
 ## License
 
 Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved 

data/bc-prometheus-ruby.gemspec

@@ -31,6 +31,7 @@ Gem::Specification.new do |spec|
 
   spec.files         = Dir['README.md', 'CHANGELOG.md', 'CODE_OF_CONDUCT.md', 'lib/**/*', 'bc-prometheus-ruby.gemspec']
   spec.require_paths = ['lib']
+  spec.required_ruby_version = '>= 2.6'
 
   spec.add_development_dependency 'rake', '>= 10.0'
   spec.add_development_dependency 'rspec', '>= 3.8'

data/lib/bigcommerce/prometheus/client.rb

@@ -24,16 +24,16 @@ module Bigcommerce
       include Singleton
       include Loggable
 
-      def initialize
+      def initialize(host: nil, port: nil, max_queue_size: nil, thread_sleep: nil, custom_labels: nil, process_name: nil)
         super(
-          host: Bigcommerce::Prometheus.server_host,
-          port: Bigcommerce::Prometheus.server_port,
-          max_queue_size: Bigcommerce::Prometheus.client_max_queue_size,
-          thread_sleep: Bigcommerce::Prometheus.client_thread_sleep,
-          custom_labels: Bigcommerce::Prometheus.client_custom_labels
+          host: host || Bigcommerce::Prometheus.server_host,
+          port: port || Bigcommerce::Prometheus.server_port,
+          max_queue_size: max_queue_size || Bigcommerce::Prometheus.client_max_queue_size,
+          thread_sleep: thread_sleep || Bigcommerce::Prometheus.client_thread_sleep,
+          custom_labels: custom_labels || Bigcommerce::Prometheus.client_custom_labels
         )
         PrometheusExporter::Client.default = self
-        @process_name = ::Bigcommerce::Prometheus.process_name
+        @process_name = process_name || ::Bigcommerce::Prometheus.process_name
       end
 
       ##
@@ -55,7 +55,7 @@ module Bigcommerce
 
       ##
       # @param [String] path
-      # @return [URI]
+      # @return [Module<URI>]
       #
       def uri_path(path)
         URI("http://#{@host}:#{@port}#{path}")

data/lib/bigcommerce/prometheus/collectors/base.rb

@@ -49,15 +49,15 @@ module Bigcommerce
         end
 
         ##
-        # @param [PrometheusExporter::Client] client
+        # @param [Bigcommerce::Prometheus::Client] client
         # @param [String] type
         # @param [Integer] frequency
         # @param [Hash] options
         #
         def initialize(client: nil, type: nil, frequency: nil, options: nil)
-          @client = client || PrometheusExporter::Client.default
+          @client = client || Bigcommerce::Prometheus.client
           @type = type || self.class.to_s.downcase.gsub('::', '_').gsub('collector', '')
-          @frequency = frequency || 15
+          @frequency = frequency || Bigcommerce::Prometheus.collector_collection_frequency
           @options = options || {}
           @logger = Bigcommerce::Prometheus.logger
         end
@@ -66,12 +66,9 @@ module Bigcommerce
         # Run the collector and send stats
         #
         def run
-          metrics = { type: @type }
+          metrics = {}
           metrics = collect(metrics)
-          @logger.debug "[bigcommerce-prometheus] Pushing #{@type} metrics to type collector: #{metrics.inspect}"
-          @client.send_json(metrics)
-        rescue StandardError => e
-          @logger.error("Prometheus Exporter Failed To Collect #{@type} Stats #{e}")
+          push(metrics)
         ensure
           sleep @frequency
         end
@@ -85,6 +82,19 @@ module Bigcommerce
         def collect(metrics = {})
           metrics
         end
+
+        private
+
+        ##
+        # @param [Hash] metric
+        #
+        def push(metric)
+          metric[:type] = @type unless metric.key?(:type)
+          @logger.debug("[bigcommerce-prometheus] Pushing #{metric[:type]} metrics to type collector: #{metric.inspect}")
+          @client.send_json(metric)
+        rescue StandardError => e
+          @logger.error("[bigcommerce-prometheus] Prometheus Exporter failed to send #{metric[:type]} stats to type collector #{e}")
+        end
       end
     end
   end

data/lib/bigcommerce/prometheus/configuration.rb

@@ -23,18 +23,32 @@ module Bigcommerce
     module Configuration
       VALID_CONFIG_KEYS = {
         logger: nil,
+        enabled: ENV.fetch('PROMETHEUS_ENABLED', 1).to_i.positive?,
+
+        # Client configuration
         client_custom_labels: nil,
-        client_max_queue_size: 10_000,
-        client_thread_sleep: 0.5,
-        enabled: true,
-        puma_collection_frequency: 30,
-        puma_process_label: 'web',
-        resque_collection_frequency: 30,
-        resque_process_label: 'resque',
-        server_host: '0.0.0.0',
-        server_port: PrometheusExporter::DEFAULT_PORT,
-        server_timeout: PrometheusExporter::DEFAULT_TIMEOUT,
-        server_prefix: PrometheusExporter::DEFAULT_PREFIX,
+        client_max_queue_size: ENV.fetch('PROMETHEUS_CLIENT_MAX_QUEUE_SIZE', 10_000).to_i,
+        client_thread_sleep: ENV.fetch('PROMETHEUS_CLIENT_THREAD_SLEEP', 0.5).to_f,
+
+        # Integration configuration
+        puma_collection_frequency: ENV.fetch('PROMETHEUS_PUMA_COLLECTION_FREQUENCY', 30).to_i,
+        puma_process_label: ENV.fetch('PROMETHEUS_PUMA_PROCESS_LABEL', 'web').to_s,
+        resque_collection_frequency: ENV.fetch('PROMETHEUS_RESQUE_COLLECTION_FREQUENCY', 30).to_i,
+        resque_process_label: ENV.fetch('PROMETHEUS_REQUEST_PROCESS_LABEL', 'resque').to_s,
+
+        # Server configuration
+        not_found_body: ENV.fetch('PROMETHEUS_SERVER_NOT_FOUND_BODY', 'Not Found! The Prometheus Ruby Exporter only listens on /metrics and /send-metrics').to_s,
+        server_host: ENV.fetch('PROMETHEUS_SERVER_HOST', '0.0.0.0').to_s,
+        server_port: ENV.fetch('PROMETHEUS_SERVER_PORT', PrometheusExporter::DEFAULT_PORT).to_i,
+        server_timeout: ENV.fetch('PROMETHEUS_DEFAULT_TIMEOUT', PrometheusExporter::DEFAULT_TIMEOUT).to_i,
+        server_prefix: ENV.fetch('PROMETHEUS_DEFAULT_PREFIX', PrometheusExporter::DEFAULT_PREFIX).to_s,
+
+        # Custom collector configuration
+        collector_collection_frequency: ENV.fetch('PROMETHEUS_DEFAULT_COLLECTOR_COLLECTION_FREQUENCY_SEC', 15).to_i,
+        hutch_collectors: [],
+        hutch_type_collectors: [],
+        resque_collectors: [],
+        resque_type_collectors: [],
         web_collectors: [],
         web_type_collectors: []
       }.freeze
@@ -85,12 +99,7 @@ module Bigcommerce
           send("#{k}=".to_sym, v)
         end
         determine_logger
-        self.enabled = ENV.fetch('PROMETHEUS_ENABLED', 1).to_i.positive?
-        self.server_host = ENV.fetch('PROMETHEUS_SERVER_HOST', '0.0.0.0').to_s
-        self.server_port = ENV.fetch('PROMETHEUS_SERVER_PORT', PrometheusExporter::DEFAULT_PORT).to_i
 
-        self.puma_process_label = ENV.fetch('PROMETHEUS_PUMA_PROCESS_LABEL', 'web').to_s
-        self.puma_collection_frequency = ENV.fetch('PROMETHEUS_PUMA_COLLECTION_FREQUENCY', 30).to_i
         self.web_type_collectors = []
       end
 

data/lib/bigcommerce/prometheus/instrumentors/hutch.rb

@@ -31,6 +31,8 @@ module Bigcommerce
           @server_port = Bigcommerce::Prometheus.server_port
           @server_timeout = Bigcommerce::Prometheus.server_timeout
           @server_prefix = Bigcommerce::Prometheus.server_prefix
+          @collectors = Bigcommerce::Prometheus.hutch_collectors || []
+          @type_collectors = Bigcommerce::Prometheus.hutch_type_collectors || []
         end
 
         ##
@@ -44,6 +46,9 @@ module Bigcommerce
 
           server.add_type_collector(PrometheusExporter::Server::ActiveRecordCollector.new)
           server.add_type_collector(PrometheusExporter::Server::HutchCollector.new)
+          @type_collectors.each do |tc|
+            server.add_type_collector(tc)
+          end
           server.start
           setup_middleware
         rescue StandardError => e
@@ -65,6 +70,9 @@ module Bigcommerce
           require 'hutch'
           ::Hutch::Config.set(:tracer, PrometheusExporter::Instrumentation::Hutch)
           @app.middleware.unshift(PrometheusExporter::Middleware, client: Bigcommerce::Prometheus.client)
+          @collectors.each(&:start)
+        rescue StandardError => e
+          logger.warn "[bigcommerce-prometheus][#{@process_name}] Failed to setup hutch prometheus middleware: #{e.message}"
         end
       end
     end

data/lib/bigcommerce/prometheus/instrumentors/resque.rb

@@ -31,6 +31,8 @@ module Bigcommerce
           @server_port = Bigcommerce::Prometheus.server_port
           @server_timeout = Bigcommerce::Prometheus.server_timeout
           @server_prefix = Bigcommerce::Prometheus.server_prefix
+          @collectors = Bigcommerce::Prometheus.resque_collectors || []
+          @type_collectors = Bigcommerce::Prometheus.resque_type_collectors || []
         end
 
         ##
@@ -44,6 +46,9 @@ module Bigcommerce
 
           server.add_type_collector(PrometheusExporter::Server::ActiveRecordCollector.new)
           server.add_type_collector(Bigcommerce::Prometheus::TypeCollectors::Resque.new)
+          @type_collectors.each do |tc|
+            server.add_type_collector(tc)
+          end
           server.start
           setup_middleware
         rescue StandardError => e
@@ -63,7 +68,8 @@ module Bigcommerce
         def setup_middleware
           logger.info "[bigcommerce-prometheus][#{@process_name}] Setting up resque prometheus middleware"
           ::Resque.before_first_fork do
-            ::Bigcommerce::Prometheus::Integrations::Resque.start
+            ::Bigcommerce::Prometheus::Integrations::Resque.start(client: Bigcommerce::Prometheus.client)
+            @collectors.each(&:start)
           end
         end
       end

data/lib/bigcommerce/prometheus/instrumentors/web.rb

@@ -77,7 +77,7 @@ module Bigcommerce
         def setup_after_fork
           @app.config.after_fork_callbacks = [] unless @app.config.after_fork_callbacks
           @app.config.after_fork_callbacks << lambda do
-            ::Bigcommerce::Prometheus::Integrations::Puma.start
+            ::Bigcommerce::Prometheus::Integrations::Puma.start(client: Bigcommerce::Prometheus.client)
             @collectors.each(&:start)
           end
         end

data/lib/bigcommerce/prometheus/integrations/resque.rb

@@ -25,13 +25,13 @@ module Bigcommerce
         ##
         # Start the resque integration
         #
-        def self.start
+        def self.start(client: nil)
           ::PrometheusExporter::Instrumentation::Process.start(
-            client: ::Bigcommerce::Prometheus.client,
+            client: client || ::Bigcommerce::Prometheus.client,
             type: ::Bigcommerce::Prometheus.resque_process_label
           )
           ::Bigcommerce::Prometheus::Collectors::Resque.start(
-            client: ::Bigcommerce::Prometheus.client,
+            client: client || ::Bigcommerce::Prometheus.client,
             frequency: ::Bigcommerce::Prometheus.resque_collection_frequency
           )
         end

data/lib/bigcommerce/prometheus/servers/thin/controllers/not_found_controller.rb

@@ -26,7 +26,7 @@ module Bigcommerce
           class NotFoundController < BaseController
             def call
               @response.status = 404
-              @response.body << 'Not Found! The Prometheus Ruby Exporter only listens on /metrics and /send-metrics'
+              @response.body << Bigcommerce::Prometheus.not_found_body
             end
           end
         end

data/lib/bigcommerce/prometheus/servers/thin/server.rb

@@ -24,9 +24,9 @@ module Bigcommerce
         #
         class Server < ::Thin::Server
           def initialize(port:, host: nil, timeout: nil, logger: nil)
-            @port = port || ::PrometheusExporter::DEFAULT_PORT
-            @host = host || '0.0.0.0'
-            @timeout = timeout || ::PrometheusExporter::DEFAULT_TIMEOUT
+            @port = port || ::Bigcommerce::Prometheus.server_port
+            @host = host || ::Bigcommerce::Prometheus.server_host
+            @timeout = timeout || ::Bigcommerce::Prometheus.server_timeout
             @logger = logger || ::Bigcommerce::Prometheus.logger
             @rack_app = ::Bigcommerce::Prometheus::Servers::Thin::RackApp.new(timeout: timeout, logger: logger)
             super(@host, @port, @rack_app)

data/lib/bigcommerce/prometheus/version.rb

@@ -17,6 +17,6 @@
 #
 module Bigcommerce
   module Prometheus
-    VERSION = '0.2.4'
+    VERSION = '0.3.0'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bc-prometheus-ruby
 version: !ruby/object:Gem::Version
-  version: 0.2.4
+  version: 0.3.0
 platform: ruby
 authors:
 - Shaun McCormick
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-04-22 00:00:00.000000000 Z
+date: 2020-04-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -211,7 +211,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.6'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="