gruf-prometheus 1.0.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94521bc11036957fc917248b31bcf494ad3f722623d830eff57150ad66c713ff
-  data.tar.gz: 8ebd7717b4f6203ecd8ab80d15504536dab599cfd4db0da41a4bd73a25dcd795
+  metadata.gz: bde4487ef4bd6038fe38578ed17becfa9d8b72e50cc2ed1593d83285e02abf7f
+  data.tar.gz: a5934b48753295219ff703cb5b5d53dcfb3cf4575c6a693444026695f19731db
 SHA512:
-  metadata.gz: feba751d2044a89a2d950892d4cbbb94101a3d753eb953a30210f5020f6552871a981720e8052c7461b0298b9e0897536bfae709df9e7b7a4c157c612e1990fb
-  data.tar.gz: 588f16d8a45586d71adf4223ba308678cd08495177d23483068c4de22c3e6b8c1d140dd5d960f3b4f7a76dca632e410ed809fae6b8a9b0a19d85acd7e4b54a67
+  metadata.gz: 7bedc3d3375e23ba127364f3c3a1824c0a11403c26e52aa81d7c2a30266e4dfaede052c686020bb4bb66d24f224bdf20668c702e3bc52ec0614e145871798b2d
+  data.tar.gz: 294cd6cc3dd8330fde2af43d86281695e0aa18dabd6c430ba889d05e3b7a2135edc45258bb41b3572ed2b80623c11f77890fcddcc627ee97f3f4c961202c579a

data/CHANGELOG.md

@@ -2,6 +2,30 @@ Changelog for the gruf-prometheus gem.
 
 ### Pending Release
 
+- Add server interceptor for measuring counters/histograms for server metrics
+- Add client interceptor for measuring counters/histograms for client metrics
+- Bump Rubocop to 1.1, remove development dependency on null_logger
+
+### 1.3.0
+
+- Drop Ruby < 2.6 support
+- Bump bc-prometheus-ruby dependency to 0.3
+- Adds support for Ruby 2.7
+- Adds help script for testing locally
+
+### 1.2.0
+
+- Add the ability to have custom collectors and type collectors
+
+### 1.1.0
+
+- Refactor collector/type collector to utilize new base abstractions
+- Bump bc-prometheus-ruby dependency
+
+### 1.0.2
+
+- Bump bc-prometheus-ruby dependency
+
 ### 1.0.1
 
 - Bump bc-prometheus-ruby dependency

data/README.md

@@ -1,6 +1,6 @@
 # gruf-prometheus - Prometheus support for gruf
 
-[![CircleCI](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/master.svg?style=svg)](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/master)  [![Gem Version](https://badge.fury.io/rb/gruf-prometheus.svg)](https://badge.fury.io/rb/gruf-prometheus) [![Documentation](https://inch-ci.org/github/bigcommerce/gruf-prometheus.svg?branch=master)](https://inch-ci.org/github/bigcommerce/gruf-prometheus?branch=master)
+[![CircleCI](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/main.svg?style=svg)](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/main)  [![Gem Version](https://badge.fury.io/rb/gruf-prometheus.svg)](https://badge.fury.io/rb/gruf-prometheus) [![Documentation](https://inch-ci.org/github/bigcommerce/gruf-prometheus.svg?branch=main)](https://inch-ci.org/github/bigcommerce/gruf-prometheus?branch=main)
 
 Adds Prometheus support for [gruf](https://github.com/bigcommerce/gruf) 2.7.0+.
 
@@ -24,16 +24,118 @@ Then `bundle exec gruf` and you'll automatically have prometheus metrics for you
 
 The gruf server will by default run on port 9394, and can be scraped at `/metrics`.
 
+## Integrations
+
+### System Metrics
+
+The gem comes with general system metrics out of the box that illustrate server health/statistics:
+
+|Name|Type|Description|
+|---|---|---|
+|ruby_grpc_pool_jobs_waiting_total|gauge|Number of jobs in the gRPC thread pool that are actively waiting|
+|ruby_grpc_pool_ready_workers_total|gauge|The amount of non-busy workers in the thread pool|
+|ruby_grpc_pool_workers_total|gauge|Number of workers in the gRPC thread pool|
+|ruby_grpc_pool_initial_size|gauge|Initial size of the gRPC thread pool|
+|ruby_grpc_poll_period|gauge|Polling period for the gRPC thread pool|
+
+### Server Metrics
+
+Furthermore, the server interceptor measures general counts (and optionally, latencies), and can be setup via:
+
+```ruby
+::Gruf.interceptors.use(::Gruf::Prometheus::Server::Interceptor)
+```
+
+This will output the following metrics:
+
+|Name|Type|Description|
+|---|---|---|
+|ruby_grpc_server_started_total|counter|Total number of RPCs started on the server|
+|ruby_grpc_server_handled_total|counter|Total number of RPCs completed on the server, regardless of success or failure|
+|ruby_grpc_server_handled_latency_seconds|histogram|Histogram of response latency of RPCs handled by the server, in seconds|
+
+Note that the histogram is disabled by default - you'll have to turn it on either through the `server_measure_latency`
+configuration yielded in `Gruf::Prometheus.configure`, or through the `PROMETHEUS_SERVER_MEASURE_LATENCY` environment
+variable. Also, the `measure_latency: true` option can be passed as a second argument to `Gruf.interceptors.use` to 
+configure this directly in the interceptor.
+
+The precedence order for this is, from first to last, with last taking precedence:
+1) `measure_latency: true` passed into the interceptor 
+2) `Gruf::Prometheus.configure` explicit setting globally
+3) `PROMETHEUS_SERVER_MEASURE_LATENCY` ENV var globally. This is the only value set by default - to `false` - and will
+   be the default unless other methods are invoked.
+
+### Client Metrics
+
+gruf-prometheus can also measure gruf client timings, via the interceptor:
+
+```ruby
+Gruf::Client.new(
+  service: MyService,
+  client_options: {
+    interceptors: [Gruf::Prometheus::Client::Interceptor.new]
+  }
+)
+``` 
+
+|Name|Type|Description|
+|---|---|---|
+|ruby_grpc_client_started_total|counter|Total number of RPCs started by the client|
+|ruby_grpc_client_completed|counter|Total number of RPCs completed by the client, regardless of success or failure|
+|ruby_grpc_client_completed_latency_seconds|histogram|Histogram of response latency of RPCs completed by the client, in seconds|
+
+Note that the histogram is disabled by default - you'll have to turn it on either through the `client_measure_latency`
+configuration yielded in `Gruf::Prometheus.configure`, or through the `PROMETHEUS_CLIENT_MEASURE_LATENCY` environment
+variable. Optionally, you can pass in `measure_latency: true` into the Interceptor directly as an option argument in the 
+initializer. 
+
+The precedence order for this is, from first to last, with last taking precedence:
+1) `measure_latency: true` passed into the interceptor 
+2) `Gruf::Prometheus.configure` explicit setting globally
+3) `PROMETHEUS_CLIENT_MEASURE_LATENCY` ENV var globally. This is the only value set by default - to `false` - and will
+   be the default unless other methods are invoked. 
+
+### Running the Client Interceptor in Non-gRPC Processes
+
+One caveat is that you _must_ have the appropriate Type Collector setup in whatever process you are running in. If
+you are already doing this in a gruf gRPC service that is using the hook provided by this gem above, no further 
+configuration is needed. Otherwise, in whatever bc-prometheus-ruby configuration you have setup, you'll need to ensure
+the type collector is loaded: 
+
+```ruby
+# prometheus_server is whatever `::Bigcommerce::Prometheus::Server` instance you are using in the current process
+# Often hooks into these are exposed as configuration options, e.g. `web_collectors`, `resque_collectors`, etc
+prometheus_server.add_type_collector(::Gruf::Prometheus::Client::TypeCollector.new)  
+```
+
+Note that you don't need to do this for the `Gruf::Prometheus::Client::Collector`, as it is an on-demand collector
+that does not run in a threaded loop.  
+
+See [bc-prometheus-ruby](https://github.com/bigcommerce/bc-prometheus-ruby#custom-server-integrations)'s documentation
+on custom server integrations for more information.
+
 ## Configuration
 
-You can further configure via bc-prometheus-ruby with:
+You can further configure `Gruf::Prometheus` globally using the block syntax:
+
+```ruby
+Gruf::Prometheus.configure do |config|
+  # config here
+end
+```
+
+where the options available are:
+
+| Option | Description | Default | ENV Name |
+| ------ | ----------- | ------- | -------- |
+| process_label | The label to use for metric prefixing | grpc | PROMETHEUS_PROCESS_LABEL |
+| process_name | Label to use for process name in logging | grpc | PROMETHEUS_PROCESS_NAME | 	
+| collection_frequency | The period in seconds in which to collect metrics | 30 | PROMETHEUS_COLLECTION_FREQUENCY |
+| collectors | Any collectors you would like to start with the server. Passed as a hash of collector class => options | {} | |
+| type_collectors | Any type collectors you would like to start with the server. Passed as an array of collector objects | [] | |
+| server_measure_latency| Whether or not to measure latency as a histogram for servers | 0 | PROMETHEUS_SERVER_MEASURE_LATENCY |
+| client_measure_latency| Whether or not to measure latency as a histogram for clients | 0 | PROMETHEUS_CLIENT_MEASURE_LATENCY |
 
-| Option | Description | Default |
-| ------ | ----------- | ------- |
-| process_label | The label to use for metric prefixing | grpc |
-| process_name | Label to use for process name in logging | grpc | 	
-| collection_frequency | The period in seconds in which to collect metrics | 30 |
-        
 ## License
 
 Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved 

data/gruf-prometheus.gemspec

@@ -15,7 +15,7 @@
 # COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
 # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 #
-$:.push File.expand_path('../lib', __FILE__)
+$LOAD_PATH.push File.expand_path('lib', __dir__)
 require 'gruf/prometheus/version'
 
 Gem::Specification.new do |spec|
@@ -31,16 +31,21 @@ Gem::Specification.new do |spec|
 
   spec.files         = Dir['README.md', 'CHANGELOG.md', 'CODE_OF_CONDUCT.md', 'lib/**/*', 'gruf-prometheus.gemspec']
   spec.require_paths = ['lib']
+  spec.required_ruby_version = '>= 2.6'
 
-  spec.add_development_dependency 'rake', '>= 10.0'
-  spec.add_development_dependency 'rspec', '>= 3.8'
-  spec.add_development_dependency 'rspec_junit_formatter', '>= 0.4'
-  spec.add_development_dependency 'bundler-audit', '>= 0.6'
-  spec.add_development_dependency 'rubocop', '>= 0.68'
-  spec.add_development_dependency 'simplecov', '>= 0.16'
-  spec.add_development_dependency 'null-logger', '>= 0.1'
-  spec.add_development_dependency 'pry', '>= 0.12'
-
+  # Runtime dependencies
+  spec.add_runtime_dependency 'bc-prometheus-ruby', '~> 0.3'
   spec.add_runtime_dependency 'gruf', '>= 2.7'
-  spec.add_runtime_dependency 'bc-prometheus-ruby', '~> 0.1.4'
+
+  # Development dependencies
+  spec.add_development_dependency 'bundler-audit', '>= 0.6'
+  spec.add_development_dependency 'pry', '>= 0.13'
+  spec.add_development_dependency 'rake', '>= 13.0'
+  spec.add_development_dependency 'rspec', '>= 3.10'
+  spec.add_development_dependency 'rspec_junit_formatter', '>= 0.4'
+  spec.add_development_dependency 'rubocop', '>= 1.1'
+  spec.add_development_dependency 'rubocop-packaging', '~> 0.5'
+  spec.add_development_dependency 'rubocop-performance', '~> 1.8'
+  spec.add_development_dependency 'rubocop-thread_safety', '~> 0.3'
+  spec.add_development_dependency 'simplecov', '>= 0.19'
 end

data/lib/gruf/prometheus.rb

@@ -27,8 +27,15 @@ require 'bigcommerce/prometheus'
 
 require_relative 'prometheus/version'
 require_relative 'prometheus/configuration'
-require_relative 'prometheus/collectors/grpc'
-require_relative 'prometheus/type_collectors/grpc'
+require_relative 'prometheus/request_types'
+require_relative 'prometheus/server/collector'
+require_relative 'prometheus/server/interceptor'
+require_relative 'prometheus/server/type_collector'
+require_relative 'prometheus/client/collector'
+require_relative 'prometheus/client/interceptor'
+require_relative 'prometheus/client/type_collector'
+require_relative 'prometheus/collector'
+require_relative 'prometheus/type_collector'
 require_relative 'prometheus/hook'
 
 module Gruf

data/lib/gruf/prometheus/client/collector.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Gruf
+  module Prometheus
+    module Client
+      ##
+      # Prometheus instrumentor for gRPC clients
+      #
+      class Collector < Bigcommerce::Prometheus::Collectors::Base
+        RESPONSE_CODE_OK = 'OK'
+
+        ##
+        # @param [Gruf::Outbound::RequestContext] request_context
+        #
+        def started_total(request_context:)
+          push(
+            grpc_client_started_total: 1,
+            custom_labels: custom_labels(request_context: request_context)
+          )
+        end
+
+        ##
+        # @param [Gruf::Controller::RequestContext] request_context
+        # @param [Gruf::Interceptors::Timer::Result] result
+        #
+        def completed(request_context:, result:)
+          push(
+            grpc_client_completed: 1,
+            custom_labels: custom_labels(request_context: request_context, result: result)
+          )
+        end
+
+        ##
+        # @param [Gruf::Outbound::RequestContext] request_context
+        # @param [Gruf::Interceptors::Timer::Result] result
+        #
+        def completed_latency_seconds(request_context:, result:)
+          push(
+            grpc_client_completed_latency_seconds: result.elapsed.to_f,
+            custom_labels: custom_labels(request_context: request_context, result: result)
+          )
+        end
+
+        private
+
+        ##
+        # @param [Gruf::Outbound::RequestContext] request_context
+        # @param [Gruf::Interceptors::Timer::Result|NilClass] result
+        # @return [Hash]
+        #
+        def custom_labels(request_context:, result: nil)
+          labels = {
+            grpc_service: format_service_name(request_context.method.to_s),
+            grpc_method: request_context.method_name,
+            grpc_type: determine_type(request_context)
+          }
+          if result
+            labels[:grpc_code] = result.successful? ? RESPONSE_CODE_OK : result.message_class_name.split('::').last
+          end
+          labels
+        end
+
+        ##
+        # Format the service name as `path.to.Service` (from `/path.to.Service/MethodName`)
+        #
+        # @param [String] name
+        # @return [String]
+        #
+        def format_service_name(name)
+          name.split('/').reject(&:empty?).first
+        end
+
+        ##
+        # @param [Gruf::Outbound::RequestContext] request_context
+        # @return [String]
+        #
+        def determine_type(request_context)
+          case request_context.type.to_sym
+          when :client_streamer
+            Gruf::Prometheus::RequestTypes::CLIENT_STREAM
+          when :server_streamer
+            Gruf::Prometheus::RequestTypes::SERVER_STREAM
+          when :bidi_streamer
+            Gruf::Prometheus::RequestTypes::BIDI_STREAM
+          else
+            Gruf::Prometheus::RequestTypes::UNARY
+          end
+        end
+      end
+    end
+  end
+end

data/lib/gruf/prometheus/client/interceptor.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Gruf
+  module Prometheus
+    module Client
+      ##
+      # Server interceptor for measuring counter/timers for gRPC inbound requests
+      #
+      class Interceptor < ::Gruf::Interceptors::ClientInterceptor
+        ##
+        # Intercept the call and send metrics
+        #
+        def call(request_context:, &block)
+          result = ::Gruf::Interceptors::Timer.time(&block)
+
+          send_metrics(request_context: request_context, result: result)
+
+          raise result.message unless result.successful?
+
+          result.message
+        end
+
+        private
+
+        ##
+        # @param [Gruf::Outbound::RequestContext] request_context
+        # @param [Gruf::Interceptors::Timer::Result] result
+        #
+        def send_metrics(request_context:, result:)
+          prometheus_collector.started_total(request_context: request_context)
+          prometheus_collector.completed(request_context: request_context, result: result)
+          prometheus_collector.completed_latency_seconds(request_context: request_context, result: result) if measure_latency?
+        rescue StandardError => e
+          # we don't want this to affect actual RPC execution, so just log an error and move on
+          Gruf.logger.error "Failed registering metric to prometheus type collector: #{e.message} - #{e.class.name}"
+        end
+
+        ##
+        # @return [::Gruf::Prometheus::Client::Collector]
+        #
+        def prometheus_collector
+          @prometheus_collector ||= ::Gruf::Prometheus::Client::Collector.new(type: 'grpc_client')
+        end
+
+        ##
+        # @return [Boolean]
+        #
+        def measure_latency?
+          @measure_latency ||= @options.fetch(:measure_latency, ::Gruf::Prometheus.client_measure_latency)
+        end
+      end
+    end
+  end
+end

data/lib/gruf/prometheus/client/type_collector.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Gruf
+  module Prometheus
+    module Client
+      ##
+      # Type Collector for grpc client metrics
+      #
+      class TypeCollector < Bigcommerce::Prometheus::TypeCollectors::Base
+        def type
+          'grpc_client'
+        end
+
+        private
+
+        ##
+        # Initialize the collector
+        #
+        def build_metrics
+          metrics = {
+            grpc_client_started_total: PrometheusExporter::Metric::Counter.new('grpc_client_started_total', 'Total number of RPCs started by the client'),
+            grpc_client_completed: PrometheusExporter::Metric::Counter.new('grpc_client_completed', 'Total number of RPCs completed by the client, regardless of success or failure')
+          }
+          metrics[:grpc_client_completed_latency_seconds] = PrometheusExporter::Metric::Histogram.new('grpc_client_completed_latency_seconds', 'Histogram of response latency of RPCs completed by the client, in seconds') if measure_latency?
+          metrics
+        end
+
+        ##
+        # Collect the object into the buffer
+        #
+        def collect_metrics(data: {}, labels: {})
+          metric(:grpc_client_started_total)&.observe(data['grpc_client_started_total'].to_i, labels)
+          metric(:grpc_client_completed)&.observe(data['grpc_client_completed'].to_i, labels)
+          metric(:grpc_client_completed_latency_seconds)&.observe(data['grpc_client_completed_latency_seconds'].to_f, labels) if measure_latency?
+        end
+
+        ##
+        # @return [Boolean]
+        #
+        def measure_latency?
+          @measure_latency ||= ::Gruf::Prometheus.client_measure_latency
+        end
+      end
+    end
+  end
+end