bc-prometheus-ruby 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bc4204f3e256d97a75520de62e6e9861561fc348d2d560f5bb79c2c25e098bb9
+  data.tar.gz: bc6ecb116fc1d36cd0199f8a0b7549d8c6c2077beb891bc6055f3c0143298bee
+SHA512:
+  metadata.gz: 6bda9f2a9613145633bd5c7eb74f6426aad79842017c9be4b97b30303ab52a6dae61544e5774450f33bdbd98082671d2fa254e1b1d968c38463f9d6e90500da7
+  data.tar.gz: a9ee4863cca42c2a90ee9db303530f6f6eeaf47e893a7747db64183afd92e794894467409e247362e4ffdeee113a5277dc8bfb0ebe71a3f7d18e6fd5fff6833b

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+Changelog for the bc-prometheus-ruby gem.
+
+h3. Pending Release
+
+h3. 0.1.0
+
+- Replace WEBrick server from PrometheusExporter with Thin server implementation to reduce memory leakage
+- Utilize NET::HTTP instead of direct sockets to prevent bad socket errors
+
+h3. 0.0.5
+
+- Add resque instrumentation
+
+h3. 0.0.4
+
+- Properly handle SIGINT/SIGTERM to shutdown prometheus exporter
+- Add process names to log output for easier debugging
+
+h3. 0.0.3
+
+- Add hutch instrumentor for hutch / rmq support
+
+h3. 0.0.2
+
+- Better support for older Rails / Puma versions
+- Adds basic support for non-Rails applications
+
+h3. 0.0.1
+
+- Initial public release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at splittingred@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/README.md

@@ -0,0 +1,67 @@
+# bc-prometheus-ruby - Drop-in Prometheus metrics
+
+[![CircleCI](https://circleci.com/gh/bigcommerce/bc-prometheus-ruby.svg?style=svg&circle-token=fc3e2c4405a1f53a31e298f0ef981c2d0dfdee90)](https://circleci.com/gh/bigcommerce/bc-prometheus-ruby)
+
+## Installation
+
+```ruby
+gem 'bc-prometheus-ruby'
+```
+
+Then in your `application.rb`, prior to extending `Rails::Application` or any initializers:
+
+```ruby
+require 'bigcommerce/prometheus'
+```
+
+You can then view your metrics at: http://0.0.0.0:9394/metrics
+
+## Puma
+
+For extra Puma metrics, add this to `config/puma.rb`:
+
+```ruby
+after_worker_fork do
+  Rails.application.config.after_fork_callbacks.each(&:call)
+end
+```
+
+## Resque
+
+In your `task 'resque:setup'` rake task, do: 
+
+```ruby
+require 'bigcommerce/prometheus'
+Bigcommerce::Prometheus::Instrumentors::Resque.new(app: Rails.application).start
+```
+
+## Configuration
+
+After requiring the main file, you can further configure with:
+
+| Option | Description | Default |
+| ------ | ----------- | ------- |
+| client_custom_labels | A hash of custom labels to send with each client request | `{}` |
+| client_max_queue_size | The max amount of metrics to send before flushing | 10000 |
+| client_thread_sleep | How often to sleep the worker thread that manages the client buffer (seconds) | 0.5 |
+| puma_collection_frequency | How often to poll puma collection metrics (seconds) | 30 |
+| server_host | The host to run the exporter on | 0.0.0.0 |
+| server_port | The port to run the exporter on | 9394 |
+| process_name | What the current process name is. Used in logging. | `ENV['PROCESS']` | 
+
+## License
+
+Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated 
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation the 
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit 
+persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the 
+Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bc-prometheus-ruby.gemspec

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+$:.push File.expand_path('../lib', __FILE__)
+require 'bigcommerce/prometheus/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'bc-prometheus-ruby'
+  spec.version       = Bigcommerce::Prometheus::VERSION
+  spec.authors       = ['Shaun McCormick']
+  spec.email         = ['shaun.mccormick@bigcommerce.com']
+
+  spec.summary       = 'Simple integration of ruby and puma servers with prometheus'
+  spec.description   = spec.summary
+  spec.homepage      = 'https://github.com/bigcommerce/bc-prometheus-ruby'
+  spec.license       = 'MIT'
+
+  spec.files         = Dir['README.md', 'CHANGELOG.md', 'CODE_OF_CONDUCT.md', 'lib/**/*', 'bc-prometheus-ruby.gemspec']
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'rake', '>= 10.0'
+  spec.add_development_dependency 'rspec', '>= 3.8'
+  spec.add_development_dependency 'rspec_junit_formatter', '>= 0.4'
+  spec.add_development_dependency 'bundler-audit', '>= 0.6'
+  spec.add_development_dependency 'null-logger', '>= 0.1'
+  spec.add_development_dependency 'pry', '>= 0.12'
+  spec.add_development_dependency 'rubocop', '>= 0.74'
+  spec.add_development_dependency 'simplecov', '>= 0.16'
+
+  spec.add_runtime_dependency 'prometheus_exporter', '~> 0.4'
+  spec.add_runtime_dependency 'thin', '~> 1.7'
+end

data/lib/bigcommerce/prometheus.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+require 'net/http'
+require 'prometheus_exporter'
+require 'prometheus_exporter/server'
+require 'prometheus_exporter/client'
+require 'prometheus_exporter/middleware'
+require 'prometheus_exporter/instrumentation'
+require 'thin'
+
+require_relative 'prometheus/version'
+require_relative 'prometheus/loggable'
+require_relative 'prometheus/configuration'
+require_relative 'prometheus/server'
+require_relative 'prometheus/client'
+
+require_relative 'prometheus/collectors/resque'
+require_relative 'prometheus/type_collectors/resque'
+
+require_relative 'prometheus/instrumentors/web'
+require_relative 'prometheus/instrumentors/hutch'
+require_relative 'prometheus/instrumentors/resque'
+require_relative 'prometheus/integrations/railtie' if defined?(Rails)
+require_relative 'prometheus/integrations/puma'
+require_relative 'prometheus/integrations/resque'
+
+require_relative 'prometheus/servers/thin/server'
+require_relative 'prometheus/servers/thin/rack_app'
+require_relative 'prometheus/servers/thin/server_metrics'
+require_relative 'prometheus/servers/thin/controllers/base_controller'
+require_relative 'prometheus/servers/thin/controllers/error_controller'
+require_relative 'prometheus/servers/thin/controllers/metrics_controller'
+require_relative 'prometheus/servers/thin/controllers/not_found_controller'
+require_relative 'prometheus/servers/thin/controllers/send_metrics_controller'
+
+module Bigcommerce
+  ##
+  # Base top-level prometheus module
+  #
+  module Prometheus
+    extend Configuration
+
+    def self.client
+      Client.instance
+    end
+  end
+end

data/lib/bigcommerce/prometheus/client.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Bigcommerce
+  module Prometheus
+    ##
+    # Client implementation for Prometheus
+    #
+    class Client < ::PrometheusExporter::Client
+      include Singleton
+      include Loggable
+
+      def initialize
+        super(
+          host: Bigcommerce::Prometheus.server_host,
+          port: Bigcommerce::Prometheus.server_port,
+          max_queue_size: Bigcommerce::Prometheus.client_max_queue_size,
+          thread_sleep: Bigcommerce::Prometheus.client_thread_sleep,
+          custom_labels: Bigcommerce::Prometheus.client_custom_labels
+        )
+        PrometheusExporter::Client.default = self
+        @process_name = ::Bigcommerce::Prometheus.process_name
+      end
+
+      ##
+      # Patch the worker loop to make it more resilient
+      #
+      def worker_loop
+        close_socket_if_old!
+        process_queue
+      rescue StandardError => e
+        logger.warn "[bigcommerce-prometheus][#{@process_name}] Prometheus client failed to send message to #{@host}:#{@port} #{e} - #{e.backtrace[0..5].join("\n")}"
+      end
+
+      ##
+      # Patch the close socket command to handle when @socket_started is nil
+      #
+      def close_socket_if_old!
+        close_socket! if @socket && ((@socket_started.to_i + MAX_SOCKET_AGE) < Time.now.to_f)
+      end
+
+      ##
+      # @param [String] path
+      # @return [URI]
+      #
+      def uri_path(path)
+        URI("http://#{@host}:#{@port}#{path}")
+      end
+
+      ##
+      # Process the current queue and flush to the collector
+      #
+      def process_queue
+        while @queue.length.to_i.positive?
+          begin
+            message = @queue.pop
+            Net::HTTP.post(uri_path('/send-metrics'), message)
+          rescue StandardError => e
+            logger.warn "[bigcommerce-prometheus][#{@process_name}] Prometheus Exporter is dropping a message tp #{uri_path('/send-metrics')}: #{e}"
+            raise
+          end
+        end
+      end
+    end
+  end
+end

data/lib/bigcommerce/prometheus/collectors/resque.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Bigcommerce
+  module Prometheus
+    module Collectors
+      ##
+      # Collect metrics to push to the server type collector
+      #
+      class Resque
+        include Bigcommerce::Prometheus::Loggable
+
+        ##
+        # @param [Bigcommerce::Prometheus::Client] client
+        # @param [Integer] frequency
+        #
+        def initialize(client:, frequency: nil)
+          @client = client || Bigcommerce::Prometheus.client
+          @frequency = frequency || 15
+        end
+
+        ##
+        # Start the collector
+        #
+        def self.start(client: nil, frequency: nil)
+          collector = new(client: client, frequency: frequency)
+          Thread.new do
+            loop do
+              collector.run
+            end
+          end
+        end
+
+        def run
+          metric = collect
+          logger.debug "[bigcommerce-prometheus] Pushing resque metrics to type collector: #{metric.inspect}"
+          @client.send_json metric
+        rescue StandardError => e
+          logger.error "[bigcommerce-prometheus] Failed to collect resque prometheus stats: #{e.message}"
+        ensure
+          sleep @frequency
+        end
+
+        private
+
+        def collect
+          info = ::Resque.info
+
+          metric = {}
+          metric[:type] = 'resque'
+          metric[:environment] = info[:environment].to_s
+          metric[:workers_total] = info[:workers].to_i
+          metric[:jobs_failed_total] = info[:failed].to_i
+          metric[:jobs_pending_total] = info[:pending].to_i
+          metric[:jobs_processed_total] = info[:processed].to_i
+          metric[:queues_total] = info[:queues].to_i
+          metric[:queues] = queue_sizes
+          metric
+        end
+
+        def queue_sizes
+          queues = {}
+          ::Resque.queues.each do |queue|
+            queues[queue.to_sym] = ::Resque.size(queue)
+          end
+          queues
+        end
+      end
+    end
+  end
+end

data/lib/bigcommerce/prometheus/configuration.rb

@@ -0,0 +1,127 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+module Bigcommerce
+  module Prometheus
+    ##
+    # General configuration for prometheus integration
+    #
+    module Configuration
+      VALID_CONFIG_KEYS = {
+        logger: nil,
+        client_custom_labels: nil,
+        client_max_queue_size: 10_000,
+        client_thread_sleep: 0.5,
+        enabled: true,
+        puma_collection_frequency: 30,
+        puma_process_label: 'web',
+        resque_collection_frequency: 30,
+        resque_process_label: 'resque',
+        server_host: '0.0.0.0',
+        server_port: PrometheusExporter::DEFAULT_PORT,
+        server_timeout: PrometheusExporter::DEFAULT_TIMEOUT,
+        server_prefix: PrometheusExporter::DEFAULT_PREFIX
+      }.freeze
+
+      attr_accessor *VALID_CONFIG_KEYS.keys
+
+      ##
+      # Whenever this is extended into a class, setup the defaults
+      #
+      def self.extended(base)
+        if defined?(Rails)
+          Bigcommerce::Prometheus::Integrations::Railtie.config.before_initialize { base.reset }
+        else
+          base.reset
+        end
+      end
+
+      ##
+      # Yield self for ruby-style initialization
+      #
+      # @yields [Bigcommerce::Prometheus::Configuration]
+      # @return [Bigcommerce::Prometheus::Configuration]
+      #
+      def configure
+        reset unless @configured
+        yield self
+        @configured = true
+      end
+
+      ##
+      # Return the current configuration options as a Hash
+      #
+      # @return [Hash]
+      #
+      def options
+        opts = {}
+        VALID_CONFIG_KEYS.each_key do |k|
+          opts.merge!(k => send(k))
+        end
+        opts
+      end
+
+      ##
+      # Set the default configuration onto the extended class
+      #
+      def reset
+        VALID_CONFIG_KEYS.each do |k, v|
+          send("#{k}=".to_sym, v)
+        end
+        determine_logger
+        self.enabled = ENV.fetch('PROMETHEUS_ENABLED', 1).to_i.positive?
+        self.server_host = ENV.fetch('PROMETHEUS_SERVER_HOST', '0.0.0.0').to_s
+        self.server_port = ENV.fetch('PROMETHEUS_SERVER_PORT', PrometheusExporter::DEFAULT_PORT).to_i
+
+        self.puma_process_label = ENV.fetch('PROMETHEUS_PUMA_PROCESS_LABEL', 'web').to_s
+        self.puma_collection_frequency = ENV.fetch('PROMETHEUS_PUMA_COLLECTION_FREQUENCY', 30).to_i
+      end
+
+      ##
+      # @return [String]
+      def process_name
+        @process_name ||= ENV.fetch('PROCESS', 'unknown')
+      end
+
+      private
+
+      def determine_logger
+        if defined?(Rails) && Rails.logger
+          self.logger = Rails.logger
+        elsif defined?(Application) && Application.respond_to?(:logger)
+          self.logger = Application.logger
+        else
+          require 'logger'
+          self.logger = ::Logger.new(STDOUT)
+        end
+      end
+
+      ##
+      # Automatically determine environment
+      #
+      # @return [String] The current Ruby environment
+      #
+      def environment
+        if defined?(Rails)
+          Rails.env.to_s
+        else
+          (ENV['RACK_ENV'] || ENV['RAILS_ENV'] || 'development').to_s
+        end
+      end
+    end
+  end
+end