prometheus_exporter 0.4.17 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ca59b198973bbfb5df6e5204f7a74138df05759f6a44839a60eaabd1d0d58697
-  data.tar.gz: 384a5a138d1eb8c24ccc47984d6f6437bcced89f058cc55460bf3dcb60f326fb
+  metadata.gz: f516c39448418a2216851149f0479f3f9e0f5feadae5a5468e7953fc0551d318
+  data.tar.gz: dcc937b79e05d4cd74a64ab2135425c75c4bba1be053bec99e8db0695a3ac998
 SHA512:
-  metadata.gz: 7406bf4ed7f65ee3dd243ac01294737292b721b9cd04f34a93b84200ad3259bc18ee8326aa8f5bea438a6ada13dab42453dd380e0fd2d34008378ecd8f581504
-  data.tar.gz: 8834124a60121673f01023a2e564b3afb55f3977d2d0214cccc2b7fb201788c2b15650992c49402180e6468a7676278ce963a0bded8f2fcf6ccc9984937a9199
+  metadata.gz: 5664f24c1c4a1520bafe789e5df8a9f1c40b118933cab9b73baa953f633ec6e81c3a11bbc9fc63097a2a2acdd2390daeade9449e6451cc83207fc2162575b359
+  data.tar.gz: ca4aeedbc6e211818569257e8a3e9fde93be363a2da246abae45749bc9cc232052ac52f0850cd54d0f600904bed193a10db1f76e4fc9567b837b20a78264158b

data/.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: Test Exporter
+
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: '0 0 * * 0' # weekly
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    name: Ruby ${{ matrix.ruby }}
+    strategy:
+      matrix:
+        ruby: ["2.7", "2.6", "2.5", "2.4"]
+    steps:
+      - uses: actions/checkout@master
+        with:
+          fetch-depth: 1
+      - uses: actions/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+      - uses: actions/cache@v2
+        with:
+          path: vendor/bundle
+          key: ${{ runner.os }}-${{ matrix.ruby }}-gems-${{ hashFiles('**/Gemfile.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-${{ matrix.ruby }}-gems-
+      - name: Setup gems
+        run: |
+          bundle config path vendor/bundle
+          bundle install --jobs 4
+      - name: Rubocop
+        run: bundle exec rubocop
+      - name: Run tests
+        run: bundle exec rake

data/.rubocop.yml

@@ -1 +1,2 @@
-inherit_from: https://raw.githubusercontent.com/discourse/discourse/master/.rubocop.yml
+inherit_gem:
+  rubocop-discourse: default.yml

data/CHANGELOG

@@ -1,4 +1,26 @@
-0.4.17 - 13-01-2019
+0.6.0 - 10-11-2020
+
+- FEATURE: add support for basic-auth in the prometheus_exporter web server
+
+0.5.3 - 29-07-2020
+
+- FEATURE: added #remove to all metric types so users can remove specific labels if needed
+
+0.5.2 - 01-07-2020
+
+- FEATURE: expanded instrumentation for sidekiq
+- FEATURE: configurable default labels
+
+0.5.1 - 25-02-2020
+
+- FEATURE: Allow configuring the default client's host and port via environment variables
+
+0.5.0 - 14-02-2020
+
+- Breaking change: listen only to localhost by default to prevent unintended insecure configuration
+- FIX: Avoid calling `hostname` aggressively, instead cache it on the exporter instance
+
+0.4.17 - 13-01-2020
 
 - FEATURE: add support for `to_h` on all metrics which can be used to query existing key/values
 

data/README.md

@@ -13,6 +13,7 @@ To learn more see [Instrumenting Rails with Prometheus](https://samsaffron.com/a
   * [Rails integration](#rails-integration)
     * [Per-process stats](#per-process-stats)
     * [Sidekiq metrics](#sidekiq-metrics)
+    * [Shoryuken metrics](#shoryuken-metrics)
     * [ActiveRecord Connection Pool Metrics](#activerecord-connection-pool-metrics)
     * [Delayed Job plugin](#delayed-job-plugin)
     * [Hutch metrics](#hutch-message-processing-tracer)
@@ -23,6 +24,7 @@ To learn more see [Instrumenting Rails with Prometheus](https://samsaffron.com/a
   * [GraphQL support](#graphql-support)
   * [Metrics default prefix / labels](#metrics-default-prefix--labels)
   * [Client default labels](#client-default-labels)
+  * [Client default host](#client-default-host)
 * [Transport concerns](#transport-concerns)
 * [JSON generation and parsing](#json-generation-and-parsing)
 * [Contributing](#contributing)
@@ -62,8 +64,9 @@ require 'prometheus_exporter/server'
 require 'prometheus_exporter/client'
 require 'prometheus_exporter/instrumentation'
 
+# bind is the address, on which the webserver will listen
 # port is the port that will provide the /metrics route
-server = PrometheusExporter::Server::WebServer.new port: 12345
+server = PrometheusExporter::Server::WebServer.new bind: 'localhost', port: 12345
 server.start
 
 # wire up a default local client
@@ -115,7 +118,7 @@ In some cases (for example, unicorn or puma clusters) you may want to aggregate
 
 Simplest way to achieve this is to use the built-in collector.
 
-First, run an exporter on your desired port (we use the default port of 9394):
+First, run an exporter on your desired port (we use the default bind to localhost and port of 9394):
 
 ```
 $ prometheus_exporter
@@ -187,13 +190,45 @@ Ensure you run the exporter in a monitored background process:
 $ bundle exec prometheus_exporter
 ```
 
+#### Metrics collected by Rails integration middleware
+
+| Type    | Name                            | Description                                                 |
+| ---     | ---                             | ---                                                         |
+| Counter | `http_requests_total`           | Total HTTP requests from web app                            |
+| Summary | `http_duration_seconds`         | Time spent in HTTP reqs in seconds                          |
+| Summary | `http_redis_duration_seconds`¹  | Time spent in HTTP reqs in Redis, in seconds                |
+| Summary | `http_sql_duration_seconds`²    | Time spent in HTTP reqs in SQL in seconds                   |
+| Summary | `http_queue_duration_seconds`³  | Time spent queueing the request in load balancer in seconds |
+
+All metrics have a `controller` and an `action` label.
+`http_requests_total` additionally has a (HTTP response) `status` label.
+
+To add your own labels to the default metrics, create a subclass of `PrometheusExporter::Middleware`, override `custom_labels`, and use it in your initializer.
+```ruby
+class MyMiddleware < PrometheusExporter::Middleware
+  def custom_labels(env)
+    labels = {}
+
+    if env['HTTP_X_PLATFORM']
+      labels['platform'] = env['HTTP_X_PLATFORM']
+    end
+
+    labels
+  end
+end
+```
+
+¹) Only available when Redis is used.
+²) Only available when Mysql or PostgreSQL are used.
+³) Only available when [Instrumenting Request Queueing Time](#instrumenting-request-queueing-time) is set up.
+
 #### Activerecord Connection Pool Metrics
 
 This collects activerecord connection pool metrics.
 
 It supports injection of custom labels and the connection config options (`username`, `database`, `host`, `port`) as labels.
 
-For Puma single mode 
+For Puma single mode
 ```ruby
 #in puma.rb
 require 'prometheus_exporter/instrumentation'
@@ -219,7 +254,7 @@ end
 For Unicorn / Passenger
 
 ```ruby
-after_fork do 
+after_fork do |_server, _worker|
   require 'prometheus_exporter/instrumentation'
   PrometheusExporter::Instrumentation::ActiveRecord.start(
     custom_labels: { type: "unicorn_worker" }, #optional params
@@ -241,6 +276,19 @@ Sidekiq.configure_server do |config|
 end
 ```
 
+##### Metrics collected by ActiveRecord Instrumentation
+
+| Type  | Name                                        | Description                           |
+| ---   | ---                                         | ---                                   |
+| Gauge | `active_record_connection_pool_connections` | Total connections in pool             |
+| Gauge | `active_record_connection_pool_busy`        | Connections in use in pool            |
+| Gauge | `active_record_connection_pool_dead`        | Dead connections in pool              |
+| Gauge | `active_record_connection_pool_idle`        | Idle connections in pool              |
+| Gauge | `active_record_connection_pool_waiting`     | Connection requests waiting           |
+| Gauge | `active_record_connection_pool_size`        | Maximum allowed connection pool size  |
+
+All metrics collected by the ActiveRecord integration include at least the following labels: `pid` (of the process the stats where collected in), `pool_name`, any labels included in the `config_labels` option (prefixed with `dbconfig_`, example: `dbconfig_host`), and all custom labels provided with the `custom_labels` option.
+
 #### Per-process stats
 
 You may also be interested in per-process stats. This collects memory and GC stats:
@@ -257,11 +305,30 @@ end
 # in unicorn/puma/passenger be sure to run a new process instrumenter after fork
 after_fork do
   require 'prometheus_exporter/instrumentation'
-  PrometheusExporter::Instrumentation::Process.start(type:"web")
+  PrometheusExporter::Instrumentation::Process.start(type: "web")
 end
 
 ```
 
+##### Metrics collected by Process Instrumentation
+
+| Type    | Name                      | Description                                  |
+| ---     | ---                       | ---                                          |
+| Gauge   | `heap_free_slots`         | Free ruby heap slots                         |
+| Gauge   | `heap_live_slots`         | Used ruby heap slots                         |
+| Gauge   | `v8_heap_size`*           | Total JavaScript V8 heap size (bytes)        |
+| Gauge   | `v8_used_heap_size`*      | Total used JavaScript V8 heap size (bytes)   |
+| Gauge   | `v8_physical_size`*       | Physical size consumed by V8 heaps           |
+| Gauge   | `v8_heap_count`*          | Number of V8 contexts running                |
+| Gauge   | `rss`                     | Total RSS used by process                    |
+| Counter | `major_gc_ops_total`      | Major GC operations by process               |
+| Counter | `minor_gc_ops_total`      | Minor GC operations by process               |
+| Counter | `allocated_objects_total` | Total number of allocated objects by process |
+
+_Metrics marked with * are only collected when `MiniRacer` is defined._
+
+Metrics collected by Process instrumentation include labels `type` (as given with the `type` option), `pid` (of the process the stats where collected in), and any custom labels given to `Process.start` with the `labels` option.
+
 #### Sidekiq metrics
 
 Including Sidekiq metrics (how many jobs ran? how many failed? how long did they take? how many are dead? how many were restarted?)
@@ -276,6 +343,17 @@ Sidekiq.configure_server do |config|
 end
 ```
 
+To monitor Queue size and latency:
+
+```ruby
+Sidekiq.configure_server do |config|
+  config.on :startup do
+    require 'prometheus_exporter/instrumentation'
+    PrometheusExporter::Instrumentation::SidekiqQueue.start
+  end
+end
+```
+
 To monitor Sidekiq process info:
 
 ```ruby
@@ -297,6 +375,59 @@ Sometimes the Sidekiq server shuts down before it can send metrics, that were ge
   end
 ```
 
+##### Metrics collected by Sidekiq Instrumentation
+
+**PrometheusExporter::Instrumentation::Sidekiq**
+| Type    | Name                           | Description                                                                  |
+| ---     | ---                            | ---                                                                          |
+| Summary | `sidekiq_job_duration_seconds` | Time spent in sidekiq jobs                                                   |
+| Counter | `sidekiq_jobs_total`           | Total number of sidekiq jobs executed                                        |
+| Counter | `sidekiq_restarted_jobs_total` | Total number of sidekiq jobs that we restarted because of a sidekiq shutdown |
+| Counter | `sidekiq_failed_jobs_total`    | Total number of failed sidekiq jobs                                          |
+
+All metrics have a `job_name` label and a `queue` label.
+
+**PrometheusExporter::Instrumentation::Sidekiq.death_handler**
+| Type    | Name                      | Description                       |
+| ---     | ---                       | ---                               |
+| Counter | `sidekiq_dead_jobs_total` | Total number of dead sidekiq jobs |
+
+This metric has a `job_name` label and a `queue` label.
+
+**PrometheusExporter::Instrumentation::SidekiqQueue**
+| Type  | Name                            | Description                  |
+| ---   | ---                             | ---                          |
+| Gauge | `sidekiq_queue_backlog_total`   | Size of the sidekiq queue    |
+| Gauge | `sidekiq_queue_latency_seconds` | Latency of the sidekiq queue |
+
+Both metrics will have a `queue` label with the name of the queue.
+
+_See [Metrics collected by Process Instrumentation](#metrics-collected-by-process-instrumentation) for a list of metrics the Process instrumentation will produce._
+
+#### Shoryuken metrics
+
+For Shoryuken metrics (how many jobs ran? how many failed? how long did they take? how many were restarted?)
+
+```ruby
+Shoryuken.configure_server do |config|
+  config.server_middleware do |chain|
+    require 'prometheus_exporter/instrumentation'
+    chain.add PrometheusExporter::Instrumentation::Shoryuken
+  end
+end
+```
+
+##### Metrics collected by Shoryuken Instrumentation
+
+| Type    | Name                             | Description                                                                      |
+| ---     | ---                              | ---                                                                              |
+| Counter | `shoryuken_job_duration_seconds` | Total time spent in shoryuken jobs                                               |
+| Counter | `shoryuken_jobs_total`           | Total number of shoryuken jobs executed                                          |
+| Counter | `shoryuken_restarted_jobs_total` | Total number of shoryuken jobs that we restarted because of a shoryuken shutdown |
+| Counter | `shoryuken_failed_jobs_total`    | Total number of failed shoryuken jobs                                            |
+
+All metrics have labels for `job_name` and `queue_name`.
+
 #### Delayed Job plugin
 
 In an initializer:
@@ -308,6 +439,19 @@ unless Rails.env == "test"
 end
 ```
 
+##### Metrics collected by Delayed Job Instrumentation
+
+| Type    | Name                                      | Description                                                        | Labels     |
+| ---     | ---                                       | ---                                                                | ---        |
+| Counter | `delayed_job_duration_seconds`            | Total time spent in delayed jobs                                   | `job_name` |
+| Counter | `delayed_jobs_total`                      | Total number of delayed jobs executed                              | `job_name` |
+| Gauge   | `delayed_jobs_enqueued`                   | Number of enqueued delayed jobs                                    | -          |
+| Gauge   | `delayed_jobs_pending`                    | Number of pending delayed jobs                                     | -          |
+| Counter | `delayed_failed_jobs_total`               | Total number failed delayed jobs executed                          | `job_name` |
+| Counter | `delayed_jobs_max_attempts_reached_total` | Total number of delayed jobs that reached max attempts             | -          |
+| Summary | `delayed_job_duration_seconds_summary`    | Summary of the time it takes jobs to execute                       | `status`   |
+| Summary | `delayed_job_attempts_summary`            | Summary of the amount of attempts it takes delayed jobs to succeed | -          |
+
 #### Hutch Message Processing Tracer
 
 Capture [Hutch](https://github.com/gocardless/hutch) metrics (how many jobs ran? how many failed? how long did they take?)
@@ -319,6 +463,16 @@ unless Rails.env == "test"
 end
 ```
 
+##### Metrics collected by Hutch Instrumentation
+
+| Type    | Name                         | Description                             |
+| ---     | ---                          | ---                                     |
+| Counter | `hutch_job_duration_seconds` | Total time spent in hutch jobs          |
+| Counter | `hutch_jobs_total`           | Total number of hutch jobs executed     |
+| Counter | `hutch_failed_jobs_total`    | Total number failed hutch jobs executed |
+
+All metrics have a `job_name` label.
+
 #### Instrumenting Request Queueing Time
 
 Request Queueing is defined as the time it takes for a request to reach your application (instrumented by this `prometheus_exporter`) from farther upstream (as your load balancer). A high queueing time usually means that your backend cannot handle all the incoming requests in time, so they queue up (= you should see if you need to add more capacity).
@@ -343,6 +497,20 @@ after_worker_boot do
 end
 ```
 
+#### Metrics collected by Puma Instrumentation
+
+| Type  | Name                              | Description                                                 |
+| ---   | ---                               | ---                                                         |
+| Gauge | `puma_workers_total`              | Number of puma workers                                      |
+| Gauge | `puma_booted_workers_total`       | Number of puma workers booted                               |
+| Gauge | `puma_old_workers_total`          | Number of old puma workers                                  |
+| Gauge | `puma_running_threads_total`      | Number of puma threads currently running                    |
+| Gauge | `puma_request_backlog_total`      | Number of requests waiting to be processed by a puma thread |
+| Gauge | `puma_thread_pool_capacity_total` | Number of puma threads available at current scale           |
+| Gauge | `puma_max_threads_total`          | Number of puma threads at available at max scale            |
+
+All metrics may have a `phase` label.
+
 ### Unicorn process metrics
 
 In order to gather metrics from unicorn processes, we use `rainbows`, which exposes `Rainbows::Linux.tcp_listener_stats` to gather information about active workers and queued requests. To start monitoring your unicorn processes, you'll need to know both the path to unicorn PID file and the listen address (`pid_file` and `listen` in your unicorn config file)
@@ -358,6 +526,14 @@ prometheus_exporter --unicorn-master /var/run/unicorn.pid --unicorn-listen-addre
 
 Note: You must install the `raindrops` gem in your `Gemfile` or locally.
 
+#### Metrics collected by Unicorn Instrumentation
+
+| Type  | Name                            | Description                                                    |
+| ---   | ---                             | ---                                                            |
+| Gauge | `unicorn_workers_total`         | Number of unicorn workers                                      |
+| Gauge | `unicorn_active_workers_total`  | Number of active unicorn workers                               |
+| Gauge | `unicorn_request_backlog_total` | Number of requests waiting to be processed by a unicorn worker |
+
 ### Custom type collectors
 
 In some cases you may have custom metrics you want to ship the collector in a batch. In this case you may still be interested in the base collector behavior, but would like to add your own special messages.
@@ -416,8 +592,8 @@ Then you can collect the metrics you need on demand:
 
 ```ruby
 def metrics
-  user_count_gague = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
-  user_count_gague.observe User.count
+  user_count_gauge = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
+  user_count_gauge.observe User.count
   [user_count_gauge]
 end
 ```
@@ -526,6 +702,68 @@ ruby_web_requests{hostname="app-server-01",route="test/route"} 1
 ruby_web_requests{hostname="app-server-01"} 1
 ```
 
+### Exporter Process Configuration
+
+When running the process for `prometheus_exporter` using `bin/prometheus_exporter`, there are several configurations that
+can be passed in:
+
+```
+Usage: prometheus_exporter [options]
+    -p, --port INTEGER               Port exporter should listen on (default: 9394)
+    -b, --bind STRING                IP address exporter should listen on (default: localhost)
+    -t, --timeout INTEGER            Timeout in seconds for metrics endpoint (default: 2)
+        --prefix METRIC_PREFIX       Prefix to apply to all metrics (default: ruby_)
+        --label METRIC_LABEL         Label to apply to all metrics (default: {})
+    -c, --collector FILE             (optional) Custom collector to run
+    -a, --type-collector FILE        (optional) Custom type collectors to run in main collector
+    -v, --verbose
+        --auth FILE                  (optional) enable basic authentication using a htpasswd FILE
+        --realm REALM                (optional) Use REALM for basic authentication (default: "Prometheus Exporter")
+        --unicorn-listen-address ADDRESS
+                                     (optional) Address where unicorn listens on (unix or TCP address)
+        --unicorn-master PID_FILE    (optional) PID file of unicorn master process to monitor unicorn
+```
+
+#### Example
+
+The following will run the process at
+- Port `8080` (default `9394`)
+- Bind to `0.0.0.0` (default `localhost`)
+- Timeout in `1 second` for metrics endpoint (default `2 seconds`)
+- Metric prefix as `foo_` (default `ruby_`)
+- Default labels as `{environment: "integration", foo: "bar"}`
+
+```bash
+prometheus_exporter -p 8080 \
+                    -b 0.0.0.0 \
+                    -t 1 \
+                    --label '{"environment": "integration", "foo": "bar"}' \
+                    --prefix 'foo_'
+```
+
+#### Enabling Basic Authentication
+
+If you desire authentication on your `/metrics` route, you can enable basic authentication with the `--auth` option.
+
+```
+$ prometheus_exporter --auth my-htpasswd-file
+```
+
+Additionally, the `--realm` option may be used to provide a customized realm for the challenge request.
+
+Notes:
+
+* You will need to create a `htpasswd` formatted file before hand which contains one or more user:password entries
+* Only the basic `crypt` encryption is currently supported
+
+A simple `htpasswd` file can be created with the Apache `htpasswd` utility; e.g:
+
+```
+$ htpasswd -cdb my-htpasswd-file my-user my-unencrypted-password
+```
+
+This will create a file named `my-htpasswd-file` which is suitable for use the `--auth` option.
+
 ### Client default labels
 
 You can specify a default label for instrumentation metrics sent by a specific client. For example:
@@ -545,6 +783,9 @@ Will result in:
 http_requests_total{controller="home","action"="index",service="app-server-01",app_name="app-01"} 2
 http_requests_total{service="app-server-01",app_name="app-01"} 1
 ```
+### Client default host
+
+By default, `PrometheusExporter::Client.default` connects to `localhost:9394`. If your setup requires this (e.g. when using `docker-compose`), you can change the default host and port by setting the environment variables `PROMETHEUS_EXPORTER_HOST` and `PROMETHEUS_EXPORTER_PORT`.
 
 ## Transport concerns