prometheus_exporter 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c77ede3528c6008266620498f698c8a0e9e17bfa0c0691fedf5a3c4c3bdb8ebd
-  data.tar.gz: 62f8b159f68036f05b785fea4eebef48df66606b083136eda5c37e4d3c294132
+  metadata.gz: 2f949348bbafc06b8f7a3dd3927693de8fef54090d85a7889afbfbf7dcb167b6
+  data.tar.gz: adeca8000ebb59c7225c1453135900aeab5c1760902edd5b4fecf9fa4b879a57
 SHA512:
-  metadata.gz: 2c0699014753102b514f1bd6fa965cd82f6d83c34084daf4a3b07537b9224295a1df02aedce3082588e44f4c001e71ea83178456bd7efd8a8eae836b2fc6e9a7
-  data.tar.gz: df738517c22422ba9e4e2e7cfd6ea6bbcf7542175a827b1632e7ff69af4651d4acf45bcec9693c4dd899b9b305b952fac989999ebea23013f3ddad0d3c396e00
+  metadata.gz: 7d3829ad57f03a19f6081c3444f9b13906aedd4fa1c5b99237d454096e0d993c27c2df0993482c421b6034a58a55c9603620bcdf7698248e4ac23a5063d63718
+  data.tar.gz: fb915716acbbb4f24a152c8ebf9eb35bdc97185a7144eb986e50aafad07a36d5cc83cbbbf7c16e8b399207e9c66d90973d2e6f9ddcdc8a17cab797840d13ed8b

data/.github/workflows/ci.yml

@@ -0,0 +1,42 @@
+name: Test Exporter
+
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: '0 0 * * 0' # weekly
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    name: Ruby ${{ matrix.ruby }}
+    strategy:
+      matrix:
+        ruby: ["2.7", "2.6", "2.5"]
+    steps:
+      - uses: actions/checkout@master
+        with:
+          fetch-depth: 1
+      - uses: actions/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+      - uses: actions/cache@v2
+        with:
+          path: vendor/bundle
+          key: ${{ runner.os }}-${{ matrix.ruby }}-gems-v2-${{ hashFiles('**/Gemfile.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-${{ matrix.ruby }}-gems-v2-
+      - name: Setup gems
+        run: |
+          gem install bundler
+          # for Ruby <= 2.6 , details https://github.com/rubygems/rubygems/issues/3284
+          gem update --system 3.0.8 && gem update --system
+          bundle config path vendor/bundle
+          bundle install --jobs 4
+          bundle exec appraisal install
+      - name: Rubocop
+        run: bundle exec rubocop
+      - name: install gems
+        run: bundle exec appraisal bundle
+      - name: Run tests
+        run: bundle exec appraisal rake

data/.gitignore

@@ -7,5 +7,7 @@
 /spec/reports/
 /tmp/
 Gemfile.lock
+/gemfiles/*.gemfile.lock
+
 
 .rubocop-https---raw-githubusercontent-com-discourse-discourse-master--rubocop-yml

data/.rubocop.yml

@@ -1 +1,7 @@
-inherit_from: https://raw.githubusercontent.com/discourse/discourse/master/.rubocop.yml
+inherit_gem:
+  rubocop-discourse: default.yml
+
+AllCops:
+  Exclude:
+    - 'gemfiles/**/*'
+    - 'vendor/**/*'

data/Appraisals

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+appraise "ar-60" do
+  # we are using this version as default in gemspec
+  # gem "activerecord", "~> 6.0.0"
+end
+
+appraise "ar-61" do
+  gem "activerecord", "~> 6.1.0.rc2"
+end

data/CHANGELOG

@@ -1,9 +1,34 @@
-0.5.0 - 14-02-2019
+0.7.0 - 29-12-2020
+
+- Dev: Removed support from EOL rubies, only 2.5, 2.6, 2.7 and 3.0 are supported now.
+- Dev: Better support for Ruby 3.0, explicitly depending on webrick
+- Dev: Rails 6.1 instrumentation support
+- FEATURE: clean pattern for overriding middleware labels was introduced (in README)
+- Fix: Better support for forking
+
+0.6.0 - 17-11-2020
+
+- FEATURE: add support for basic-auth in the prometheus_exporter web server
+
+0.5.3 - 29-07-2020
+
+- FEATURE: added #remove to all metric types so users can remove specific labels if needed
+
+0.5.2 - 01-07-2020
+
+- FEATURE: expanded instrumentation for sidekiq
+- FEATURE: configurable default labels
+
+0.5.1 - 25-02-2020
+
+- FEATURE: Allow configuring the default client's host and port via environment variables
+
+0.5.0 - 14-02-2020
 
 - Breaking change: listen only to localhost by default to prevent unintended insecure configuration
 - FIX: Avoid calling `hostname` aggressively, instead cache it on the exporter instance
 
-0.4.17 - 13-01-2019
+0.4.17 - 13-01-2020
 
 - FEATURE: add support for `to_h` on all metrics which can be used to query existing key/values
 

data/README.md

@@ -24,6 +24,7 @@ To learn more see [Instrumenting Rails with Prometheus](https://samsaffron.com/a
   * [GraphQL support](#graphql-support)
   * [Metrics default prefix / labels](#metrics-default-prefix--labels)
   * [Client default labels](#client-default-labels)
+  * [Client default host](#client-default-host)
 * [Transport concerns](#transport-concerns)
 * [JSON generation and parsing](#json-generation-and-parsing)
 * [Contributing](#contributing)
@@ -32,7 +33,7 @@ To learn more see [Instrumenting Rails with Prometheus](https://samsaffron.com/a
 
 ## Requirements
 
-Minimum Ruby of version 2.3.0 is required, Ruby 2.2.0 is EOL as of 2018-03-31
+Minimum Ruby of version 2.5.0 is required, Ruby 2.4.0 is EOL as of 2020-04-05
 
 ## Installation
 
@@ -189,13 +190,71 @@ Ensure you run the exporter in a monitored background process:
 $ bundle exec prometheus_exporter
 ```
 
+#### Metrics collected by Rails integration middleware
+
+| Type    | Name                            | Description                                                 |
+| ---     | ---                             | ---                                                         |
+| Counter | `http_requests_total`           | Total HTTP requests from web app                            |
+| Summary | `http_duration_seconds`         | Time spent in HTTP reqs in seconds                          |
+| Summary | `http_redis_duration_seconds`¹  | Time spent in HTTP reqs in Redis, in seconds                |
+| Summary | `http_sql_duration_seconds`²    | Time spent in HTTP reqs in SQL in seconds                   |
+| Summary | `http_queue_duration_seconds`³  | Time spent queueing the request in load balancer in seconds |
+
+All metrics have a `controller` and an `action` label.
+`http_requests_total` additionally has a (HTTP response) `status` label.
+
+To add your own labels to the default metrics, create a subclass of `PrometheusExporter::Middleware`, override `custom_labels`, and use it in your initializer.
+```ruby
+class MyMiddleware < PrometheusExporter::Middleware
+  def custom_labels(env)
+    labels = {}
+
+    if env['HTTP_X_PLATFORM']
+      labels['platform'] = env['HTTP_X_PLATFORM']
+    end
+
+    labels
+  end
+end
+```
+
+If you're not using Rails like framework, you can extend `PrometheusExporter::Middleware#default_labels` in a way to add more relevant labels.
+For example you can mimic [prometheus-client](https://github.com/prometheus/client_ruby) labels with code like this:
+```ruby
+class MyMiddleware < PrometheusExporter::Middleware
+  def default_labels(env, result)
+    status = (result && result[0]) || -1
+    path = [env["SCRIPT_NAME"], env["PATH_INFO"]].join
+    {
+      path: strip_ids_from_path(path),
+      method: env["REQUEST_METHOD"],
+      status: status
+    }
+  end
+
+  def strip_ids_from_path(path)
+    path
+      .gsub(%r{/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}(/|$)}, '/:uuid\\1')
+      .gsub(%r{/\d+(/|$)}, '/:id\\1')
+  end
+end
+```
+That way you won't have all metrics labeled with `controller=other` and `action=other`, but have labels such as
+```
+ruby_http_duration_seconds{path="/api/v1/teams/:id",method="GET",status="200",quantile="0.99"} 0.009880661998977303
+```
+
+¹) Only available when Redis is used.
+²) Only available when Mysql or PostgreSQL are used.
+³) Only available when [Instrumenting Request Queueing Time](#instrumenting-request-queueing-time) is set up.
+
 #### Activerecord Connection Pool Metrics
 
 This collects activerecord connection pool metrics.
 
 It supports injection of custom labels and the connection config options (`username`, `database`, `host`, `port`) as labels.
 
-For Puma single mode 
+For Puma single mode
 ```ruby
 #in puma.rb
 require 'prometheus_exporter/instrumentation'
@@ -243,6 +302,19 @@ Sidekiq.configure_server do |config|
 end
 ```
 
+##### Metrics collected by ActiveRecord Instrumentation
+
+| Type  | Name                                        | Description                           |
+| ---   | ---                                         | ---                                   |
+| Gauge | `active_record_connection_pool_connections` | Total connections in pool             |
+| Gauge | `active_record_connection_pool_busy`        | Connections in use in pool            |
+| Gauge | `active_record_connection_pool_dead`        | Dead connections in pool              |
+| Gauge | `active_record_connection_pool_idle`        | Idle connections in pool              |
+| Gauge | `active_record_connection_pool_waiting`     | Connection requests waiting           |
+| Gauge | `active_record_connection_pool_size`        | Maximum allowed connection pool size  |
+
+All metrics collected by the ActiveRecord integration include at least the following labels: `pid` (of the process the stats where collected in), `pool_name`, any labels included in the `config_labels` option (prefixed with `dbconfig_`, example: `dbconfig_host`), and all custom labels provided with the `custom_labels` option.
+
 #### Per-process stats
 
 You may also be interested in per-process stats. This collects memory and GC stats:
@@ -259,11 +331,30 @@ end
 # in unicorn/puma/passenger be sure to run a new process instrumenter after fork
 after_fork do
   require 'prometheus_exporter/instrumentation'
-  PrometheusExporter::Instrumentation::Process.start(type:"web")
+  PrometheusExporter::Instrumentation::Process.start(type: "web")
 end
 
 ```
 
+##### Metrics collected by Process Instrumentation
+
+| Type    | Name                      | Description                                  |
+| ---     | ---                       | ---                                          |
+| Gauge   | `heap_free_slots`         | Free ruby heap slots                         |
+| Gauge   | `heap_live_slots`         | Used ruby heap slots                         |
+| Gauge   | `v8_heap_size`*           | Total JavaScript V8 heap size (bytes)        |
+| Gauge   | `v8_used_heap_size`*      | Total used JavaScript V8 heap size (bytes)   |
+| Gauge   | `v8_physical_size`*       | Physical size consumed by V8 heaps           |
+| Gauge   | `v8_heap_count`*          | Number of V8 contexts running                |
+| Gauge   | `rss`                     | Total RSS used by process                    |
+| Counter | `major_gc_ops_total`      | Major GC operations by process               |
+| Counter | `minor_gc_ops_total`      | Minor GC operations by process               |
+| Counter | `allocated_objects_total` | Total number of allocated objects by process |
+
+_Metrics marked with * are only collected when `MiniRacer` is defined._
+
+Metrics collected by Process instrumentation include labels `type` (as given with the `type` option), `pid` (of the process the stats where collected in), and any custom labels given to `Process.start` with the `labels` option.
+
 #### Sidekiq metrics
 
 Including Sidekiq metrics (how many jobs ran? how many failed? how long did they take? how many are dead? how many were restarted?)
@@ -278,6 +369,17 @@ Sidekiq.configure_server do |config|
 end
 ```
 
+To monitor Queue size and latency:
+
+```ruby
+Sidekiq.configure_server do |config|
+  config.on :startup do
+    require 'prometheus_exporter/instrumentation'
+    PrometheusExporter::Instrumentation::SidekiqQueue.start
+  end
+end
+```
+
 To monitor Sidekiq process info:
 
 ```ruby
@@ -299,6 +401,35 @@ Sometimes the Sidekiq server shuts down before it can send metrics, that were ge
   end
 ```
 
+##### Metrics collected by Sidekiq Instrumentation
+
+**PrometheusExporter::Instrumentation::Sidekiq**
+| Type    | Name                           | Description                                                                  |
+| ---     | ---                            | ---                                                                          |
+| Summary | `sidekiq_job_duration_seconds` | Time spent in sidekiq jobs                                                   |
+| Counter | `sidekiq_jobs_total`           | Total number of sidekiq jobs executed                                        |
+| Counter | `sidekiq_restarted_jobs_total` | Total number of sidekiq jobs that we restarted because of a sidekiq shutdown |
+| Counter | `sidekiq_failed_jobs_total`    | Total number of failed sidekiq jobs                                          |
+
+All metrics have a `job_name` label and a `queue` label.
+
+**PrometheusExporter::Instrumentation::Sidekiq.death_handler**
+| Type    | Name                      | Description                       |
+| ---     | ---                       | ---                               |
+| Counter | `sidekiq_dead_jobs_total` | Total number of dead sidekiq jobs |
+
+This metric has a `job_name` label and a `queue` label.
+
+**PrometheusExporter::Instrumentation::SidekiqQueue**
+| Type  | Name                            | Description                  |
+| ---   | ---                             | ---                          |
+| Gauge | `sidekiq_queue_backlog_total`   | Size of the sidekiq queue    |
+| Gauge | `sidekiq_queue_latency_seconds` | Latency of the sidekiq queue |
+
+Both metrics will have a `queue` label with the name of the queue.
+
+_See [Metrics collected by Process Instrumentation](#metrics-collected-by-process-instrumentation) for a list of metrics the Process instrumentation will produce._
+
 #### Shoryuken metrics
 
 For Shoryuken metrics (how many jobs ran? how many failed? how long did they take? how many were restarted?)
@@ -312,6 +443,17 @@ Shoryuken.configure_server do |config|
 end
 ```
 
+##### Metrics collected by Shoryuken Instrumentation
+
+| Type    | Name                             | Description                                                                      |
+| ---     | ---                              | ---                                                                              |
+| Counter | `shoryuken_job_duration_seconds` | Total time spent in shoryuken jobs                                               |
+| Counter | `shoryuken_jobs_total`           | Total number of shoryuken jobs executed                                          |
+| Counter | `shoryuken_restarted_jobs_total` | Total number of shoryuken jobs that we restarted because of a shoryuken shutdown |
+| Counter | `shoryuken_failed_jobs_total`    | Total number of failed shoryuken jobs                                            |
+
+All metrics have labels for `job_name` and `queue_name`.
+
 #### Delayed Job plugin
 
 In an initializer:
@@ -323,6 +465,19 @@ unless Rails.env == "test"
 end
 ```
 
+##### Metrics collected by Delayed Job Instrumentation
+
+| Type    | Name                                      | Description                                                        | Labels     |
+| ---     | ---                                       | ---                                                                | ---        |
+| Counter | `delayed_job_duration_seconds`            | Total time spent in delayed jobs                                   | `job_name` |
+| Counter | `delayed_jobs_total`                      | Total number of delayed jobs executed                              | `job_name` |
+| Gauge   | `delayed_jobs_enqueued`                   | Number of enqueued delayed jobs                                    | -          |
+| Gauge   | `delayed_jobs_pending`                    | Number of pending delayed jobs                                     | -          |
+| Counter | `delayed_failed_jobs_total`               | Total number failed delayed jobs executed                          | `job_name` |
+| Counter | `delayed_jobs_max_attempts_reached_total` | Total number of delayed jobs that reached max attempts             | -          |
+| Summary | `delayed_job_duration_seconds_summary`    | Summary of the time it takes jobs to execute                       | `status`   |
+| Summary | `delayed_job_attempts_summary`            | Summary of the amount of attempts it takes delayed jobs to succeed | -          |
+
 #### Hutch Message Processing Tracer
 
 Capture [Hutch](https://github.com/gocardless/hutch) metrics (how many jobs ran? how many failed? how long did they take?)
@@ -334,6 +489,16 @@ unless Rails.env == "test"
 end
 ```
 
+##### Metrics collected by Hutch Instrumentation
+
+| Type    | Name                         | Description                             |
+| ---     | ---                          | ---                                     |
+| Counter | `hutch_job_duration_seconds` | Total time spent in hutch jobs          |
+| Counter | `hutch_jobs_total`           | Total number of hutch jobs executed     |
+| Counter | `hutch_failed_jobs_total`    | Total number failed hutch jobs executed |
+
+All metrics have a `job_name` label.
+
 #### Instrumenting Request Queueing Time
 
 Request Queueing is defined as the time it takes for a request to reach your application (instrumented by this `prometheus_exporter`) from farther upstream (as your load balancer). A high queueing time usually means that your backend cannot handle all the incoming requests in time, so they queue up (= you should see if you need to add more capacity).
@@ -358,6 +523,20 @@ after_worker_boot do
 end
 ```
 
+#### Metrics collected by Puma Instrumentation
+
+| Type  | Name                              | Description                                                 |
+| ---   | ---                               | ---                                                         |
+| Gauge | `puma_workers_total`              | Number of puma workers                                      |
+| Gauge | `puma_booted_workers_total`       | Number of puma workers booted                               |
+| Gauge | `puma_old_workers_total`          | Number of old puma workers                                  |
+| Gauge | `puma_running_threads_total`      | Number of puma threads currently running                    |
+| Gauge | `puma_request_backlog_total`      | Number of requests waiting to be processed by a puma thread |
+| Gauge | `puma_thread_pool_capacity_total` | Number of puma threads available at current scale           |
+| Gauge | `puma_max_threads_total`          | Number of puma threads at available at max scale            |
+
+All metrics may have a `phase` label.
+
 ### Unicorn process metrics
 
 In order to gather metrics from unicorn processes, we use `rainbows`, which exposes `Rainbows::Linux.tcp_listener_stats` to gather information about active workers and queued requests. To start monitoring your unicorn processes, you'll need to know both the path to unicorn PID file and the listen address (`pid_file` and `listen` in your unicorn config file)
@@ -373,6 +552,14 @@ prometheus_exporter --unicorn-master /var/run/unicorn.pid --unicorn-listen-addre
 
 Note: You must install the `raindrops` gem in your `Gemfile` or locally.
 
+#### Metrics collected by Unicorn Instrumentation
+
+| Type  | Name                            | Description                                                    |
+| ---   | ---                             | ---                                                            |
+| Gauge | `unicorn_workers_total`         | Number of unicorn workers                                      |
+| Gauge | `unicorn_active_workers_total`  | Number of active unicorn workers                               |
+| Gauge | `unicorn_request_backlog_total` | Number of requests waiting to be processed by a unicorn worker |
+
 ### Custom type collectors
 
 In some cases you may have custom metrics you want to ship the collector in a batch. In this case you may still be interested in the base collector behavior, but would like to add your own special messages.
@@ -431,8 +618,8 @@ Then you can collect the metrics you need on demand:
 
 ```ruby
 def metrics
-  user_count_gague = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
-  user_count_gague.observe User.count
+  user_count_gauge = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
+  user_count_gauge.observe User.count
   [user_count_gauge]
 end
 ```
@@ -541,6 +728,68 @@ ruby_web_requests{hostname="app-server-01",route="test/route"} 1
 ruby_web_requests{hostname="app-server-01"} 1
 ```
 
+### Exporter Process Configuration
+
+When running the process for `prometheus_exporter` using `bin/prometheus_exporter`, there are several configurations that
+can be passed in:
+
+```
+Usage: prometheus_exporter [options]
+    -p, --port INTEGER               Port exporter should listen on (default: 9394)
+    -b, --bind STRING                IP address exporter should listen on (default: localhost)
+    -t, --timeout INTEGER            Timeout in seconds for metrics endpoint (default: 2)
+        --prefix METRIC_PREFIX       Prefix to apply to all metrics (default: ruby_)
+        --label METRIC_LABEL         Label to apply to all metrics (default: {})
+    -c, --collector FILE             (optional) Custom collector to run
+    -a, --type-collector FILE        (optional) Custom type collectors to run in main collector
+    -v, --verbose
+        --auth FILE                  (optional) enable basic authentication using a htpasswd FILE
+        --realm REALM                (optional) Use REALM for basic authentication (default: "Prometheus Exporter")
+        --unicorn-listen-address ADDRESS
+                                     (optional) Address where unicorn listens on (unix or TCP address)
+        --unicorn-master PID_FILE    (optional) PID file of unicorn master process to monitor unicorn
+```
+
+#### Example
+
+The following will run the process at
+- Port `8080` (default `9394`)
+- Bind to `0.0.0.0` (default `localhost`)
+- Timeout in `1 second` for metrics endpoint (default `2 seconds`)
+- Metric prefix as `foo_` (default `ruby_`)
+- Default labels as `{environment: "integration", foo: "bar"}`
+
+```bash
+prometheus_exporter -p 8080 \
+                    -b 0.0.0.0 \
+                    -t 1 \
+                    --label '{"environment": "integration", "foo": "bar"}' \
+                    --prefix 'foo_'
+```
+
+#### Enabling Basic Authentication
+
+If you desire authentication on your `/metrics` route, you can enable basic authentication with the `--auth` option.
+
+```
+$ prometheus_exporter --auth my-htpasswd-file
+```
+
+Additionally, the `--realm` option may be used to provide a customized realm for the challenge request.
+
+Notes:
+
+* You will need to create a `htpasswd` formatted file before hand which contains one or more user:password entries
+* Only the basic `crypt` encryption is currently supported
+
+A simple `htpasswd` file can be created with the Apache `htpasswd` utility; e.g:
+
+```
+$ htpasswd -cdb my-htpasswd-file my-user my-unencrypted-password
+```
+
+This will create a file named `my-htpasswd-file` which is suitable for use the `--auth` option.
+
 ### Client default labels
 
 You can specify a default label for instrumentation metrics sent by a specific client. For example:
@@ -560,6 +809,9 @@ Will result in:
 http_requests_total{controller="home","action"="index",service="app-server-01",app_name="app-01"} 2
 http_requests_total{service="app-server-01",app_name="app-01"} 1
 ```
+### Client default host
+
+By default, `PrometheusExporter::Client.default` connects to `localhost:9394`. If your setup requires this (e.g. when using `docker-compose`), you can change the default host and port by setting the environment variables `PROMETHEUS_EXPORTER_HOST` and `PROMETHEUS_EXPORTER_PORT`.
 
 ## Transport concerns