ezmetrics 1.2.2 → 3.0.0

data/README.md

@@ -10,70 +10,29 @@ Simple, lightweight and fast metrics aggregation for Rails.
 gem 'ezmetrics'
 ```
 
-## Usage
-
-### Getting started
-
-This tool captures and aggregates Rails application metrics such as
-
-- `duration`
-- `views`
-- `db`
-- `queries`
-- `status`
-
-and stores them for the timeframe you specified, 60 seconds by default.
-
-You can change the timeframe according to your needs and save the metrics by calling `log` method:
-
-```ruby
-# Store the metrics for 60 seconds (default behaviour)
-EZmetrics.new.log(
-  duration: 100.5,
-  views:    40.7,
-  db:       59.8,
-  queries:  4,
-  status:   200
-)
-```
+## Available metrics
 
-```ruby
-# Store the metrics for 10 minutes
-EZmetrics.new(10.minutes).log(
-  duration: 100.5,
-  views:    40.7,
-  db:       59.8,
-  queries:  4,
-  status:   200
-)
-```
-
----
-
-For displaying metrics you need to call `show` method:
-
-```ruby
-# Aggregate and show metrics for last 60 seconds (default behaviour)
-EZmetrics.new.show
-```
+|    Type    |        Aggregate functions        |
+| :--------: | :-------------------------------: |
+| `duration` |    `avg`, `max`, `percentile`     |
+|  `views`   |    `avg`, `max`, `percentile`     |
+|    `db`    |    `avg`, `max`, `percentile`     |
+| `queries`  |    `avg`, `max`, `percentile`     |
+| `requests` | `all`, `2xx`, `3xx`, `4xx`, `5xx` |
 
-```ruby
-# Aggregate and show metrics for last 10 minutes
-EZmetrics.new(10.minutes).show
-```
+## Usage
 
-You can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
 
 ### Capture metrics
 
-Just add an initializer to your application:
+Add an initializer to your application:
 
 ```ruby
 # config/initializers/ezmetrics.rb
 
 ActiveSupport::Notifications.subscribe("sql.active_record") do |*args|
   event = ActiveSupport::Notifications::Event.new(*args)
-  unless event.payload[:name] == "SCHEMA"
+  unless event.payload[:name] == "SCHEMA" || event.payload[:sql] =~ /\ABEGIN|COMMIT|ROLLBACK\z/
     Thread.current[:queries] ||= 0
     Thread.current[:queries] += 1
   end
@@ -81,22 +40,35 @@ end
 
 ActiveSupport::Notifications.subscribe("process_action.action_controller") do |*args|
   event = ActiveSupport::Notifications::Event.new(*args)
-  EZmetrics.new.log(
+  Ezmetrics::Storage.new(24.hours).log(
     duration: event.duration.to_f,
     views:    event.payload[:view_runtime].to_f,
     db:       event.payload[:db_runtime].to_f,
     status:   event.payload[:status].to_i || 500,
     queries:  Thread.current[:queries].to_i,
   )
+  Thread.current[:queries] = 0
 end
 ```
 
 ### Display metrics
 
+#### 1. Dashboard
+
+```ruby
+# add the following line to 'config/routes.rb'
+
+mount Dashboard::Ezmetrics, at: "/dashboard", as: "dashboard"
+```
+
+![Dashboard](https://user-images.githubusercontent.com/1847948/73551868-e26b2800-444f-11ea-83b9-fc81c8d05c07.png)
+
+#### 2. Directly
+
 As simple as:
 
 ```ruby
-EZmetrics.new.show
+Ezmetrics::Storage.new.show
 ```
 
 This will return a hash with the following structure:
@@ -136,7 +108,7 @@ This will return a hash with the following structure:
 If you prefer a single level object - you can change the default output structure by calling `.flatten` before `.show`
 
 ```ruby
-EZmetrics.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
+Ezmetrics::Storage.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
 ```
 
 ```ruby
@@ -152,7 +124,7 @@ EZmetrics.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
 Same for [partitioned aggregation](#partitioning)
 
 ```ruby
-EZmetrics.new(1.hour).partition_by(:minute).flatten.show(db: :avg, duration: [:avg, :max])
+Ezmetrics::Storage.new(1.hour).partition_by(:minute).flatten.show(db: :avg, duration: [:avg, :max])
 ```
 
 ```ruby
@@ -179,7 +151,7 @@ The aggregation can be easily configured by specifying aggregation options as in
 **1. Single**
 
 ```ruby
-EZmetrics.new.show(duration: :max)
+Ezmetrics::Storage.new.show(duration: :max)
 ```
 
 ```ruby
@@ -195,7 +167,7 @@ EZmetrics.new.show(duration: :max)
 **2. Multiple**
 
 ```ruby
-EZmetrics.new.show(queries: [:max, :avg])
+Ezmetrics::Storage.new.show(queries: [:max, :avg])
 ```
 
 ```ruby
@@ -212,7 +184,7 @@ EZmetrics.new.show(queries: [:max, :avg])
 **3. Requests**
 
 ```ruby
-EZmetrics.new.show(requests: true)
+Ezmetrics::Storage.new.show(requests: true)
 ```
 
 ```ruby
@@ -234,7 +206,7 @@ EZmetrics.new.show(requests: true)
 **4. Combined**
 
 ```ruby
-EZmetrics.new.show(views: :avg, :db: [:avg, :max], requests: true)
+Ezmetrics::Storage.new.show(views: :avg, :db: [:avg, :max], requests: true)
 ```
 
 ```ruby
@@ -258,6 +230,70 @@ EZmetrics.new.show(views: :avg, :db: [:avg, :max], requests: true)
 }
 ```
 
+---
+
+**5. Percentile**
+
+This feature is available since version `2.0.0`.
+
+By default percentile aggregation is turned off because it requires to store each value of all metrics.
+
+To enable this feature - you need to set `store_each_value: true` when saving the metrics:
+
+```ruby
+Ezmetrics::Storage.new.log(
+  duration:         100.5,
+  views:            40.7,
+  db:               59.8,
+  queries:          4,
+  status:           200,
+  store_each_value: true
+)
+```
+
+The aggregation syntax has the following format `metrics_type: :percentile_{number}` where `number` is any integer in the 1..99 range.
+
+```ruby
+Ezmetrics::Storage.new.show(db: [:avg, :percentile_90, :percentile_95], duration: :percentile_99)
+```
+
+```ruby
+{
+  db: {
+    avg: 155,
+    percentile_90: 205,
+    percentile_95: 215
+  },
+  duration: {
+    percentile_99: 236
+  }
+}
+```
+
+**6. Percentile distribution**
+
+If you want to visualize percentile distribution (from 1% to 99%):
+
+```ruby
+Ezmetrics::Storage.new.show(duration: :percentile_distribution)
+```
+
+```ruby
+{
+  duration: {
+    percentile_distribution: {
+      1: 12,
+      2: 15,
+      3: 19,
+      #...
+      97: 6540,
+      98: 6682,
+      99: 6730
+    }
+  }
+}
+```
+
 ### Partitioning
 
 If you want to visualize your metrics by using a **line chart**, you will need to use partitioning.
@@ -266,7 +302,7 @@ To aggregate metrics, partitioned by a unit of time you need to call `.partition
 
 ```ruby
 # Aggregate metrics for last hour, partition by minute
-EZmetrics.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
+Ezmetrics::Storage.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
 ```
 
 This will return an array of objects with the following structure:
@@ -313,38 +349,38 @@ like in the example below:
 
 Available time units for partitioning: `second`, `minute`, `hour`, `day`. Default: `minute`.
 
-### Performance
+## Performance
 
 The aggregation speed relies on the performance of **Redis** (data storage) and **Oj** (json serialization/parsing).
 
-You can check the **aggregation** time by running:
+You can check the **aggregation** time (in seconds) by running:
 
 ```ruby
-EZmetrics::Benchmark.new.measure_aggregation
-```
+# 1. Simple
+Ezmetrics::Benchmark.new.measure_aggregation
 
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.04        |
-| 12 hours |        0.49        |
-| 24 hours |        1.51        |
-| 48 hours |        3.48        |
-
----
+# 2. Partitioned
+Ezmetrics::Benchmark.new.measure_aggregation(:minute)
 
-To check the **partitioned aggregation** time you need to run:
+# 3. Percentile
+Ezmetrics::Benchmark.new(true).measure_aggregation
 
-```ruby
-EZmetrics::Benchmark.new.measure_aggregation(:minute)
+# 4. Percentile (partitioned)
+Ezmetrics::Benchmark.new(true).measure_aggregation(:minute)
 ```
 
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.04        |
-| 12 hours |        0.53        |
-| 24 hours |        1.59        |
-| 48 hours |        3.51        |
+| Interval | Simple aggregation | Partitioned | Percentile | Percentile (partitioned) |
+| :------: | :----------------: | :---------: | :--------: | :----------------------: |
+| 1 minute |        0.0         |     0.0     |    0.0     |           0.0            |
+|  1 hour  |        0.02        |    0.02     |    0.14    |           0.16           |
+| 12 hours |        0.22        |    0.25     |    2.11    |           1.97           |
+| 24 hours |        0.61        |    0.78     |    5.85    |           5.85           |
+| 48 hours |        1.42        |    1.75     |    14.1    |           13.9           |
 
 The benchmarks above were run on a _2017 Macbook Pro 2.9 GHz Intel Core i7 with 16 GB of RAM_
+
+## [Changelog](CHANGELOG.md)
+
+## License
+
+ezmetrics is released under the [MIT License](https://opensource.org/licenses/MIT).

data/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-tactile

data/ezmetrics.gemspec

@@ -0,0 +1,21 @@
+require_relative "lib/ezmetrics/version"
+
+Gem::Specification.new do |gem|
+  gem.name                  = "ezmetrics"
+  gem.version               = Ezmetrics::VERSION
+  gem.date                  = "2020-02-01"
+  gem.summary               = "Rails metrics aggregation tool."
+  gem.description           = "Simple, lightweight and fast metrics aggregation for Rails."
+  gem.authors               = ["Nicolae Rotaru"]
+  gem.email                 = "nyku.rn@gmail.com"
+  gem.homepage              = "https://github.com/nyku/ezmetrics"
+  gem.license               = "MIT"
+  gem.files                 = `git ls-files | grep -Ev '^(spec)'`.split("\n")
+  gem.require_paths         = ["lib"]
+  gem.required_ruby_version = ">= 2.4.0"
+  gem.add_dependency "redis", ["~> 4.0"]
+  gem.add_dependency "hiredis", ["~> 0.6.3"]
+  gem.add_dependency "oj", ["~> 3.10"]
+
+  gem.add_development_dependency "rspec", "~> 3.5"
+end

data/lib/ezmetrics.rb

@@ -2,205 +2,12 @@ require "redis"
 require "redis/connection/hiredis"
 require "oj"
 
-class EZmetrics
-  METRICS               = [:duration, :views, :db, :queries].freeze
-  AGGREGATION_FUNCTIONS = [:max, :avg].freeze
-  PARTITION_UNITS       = [:second, :minute, :hour, :day].freeze
+module Ezmetrics
+  require_relative "ezmetrics/version"
+  require_relative "ezmetrics/storage"
+  require_relative "ezmetrics/benchmark"
 
-  def initialize(interval_seconds=60)
-    @interval_seconds = interval_seconds.to_i
-    @redis            = Redis.new
+  if defined?(Rails)
+    require_relative "ezmetrics/dashboard/lib/dashboard"
   end
-
-  def log(payload={duration: 0.0, views: 0.0, db: 0.0, queries: 0, status: 200})
-    @safe_payload = {
-      duration: payload[:duration].to_f,
-      views:    payload[:views].to_f,
-      db:       payload[:db].to_f,
-      queries:  payload[:queries].to_i,
-      status:   payload[:status].to_i
-    }
-
-    this_second          = Time.now.to_i
-    status_group         = "#{payload[:status].to_s[0]}xx"
-    @this_second_metrics = redis.get(this_second)
-
-    if this_second_metrics
-      @this_second_metrics = Oj.load(this_second_metrics)
-
-      METRICS.each do |metrics_type|
-        update_sum(metrics_type)
-        update_max(metrics_type)
-      end
-
-      this_second_metrics["statuses"]["all"]        += 1
-      this_second_metrics["statuses"][status_group] += 1
-    else
-      @this_second_metrics = {
-        "second"       => this_second,
-        "duration_sum" => safe_payload[:duration],
-        "duration_max" => safe_payload[:duration],
-        "views_sum"    => safe_payload[:views],
-        "views_max"    => safe_payload[:views],
-        "db_sum"       => safe_payload[:db],
-        "db_max"       => safe_payload[:db],
-        "queries_sum"  => safe_payload[:queries],
-        "queries_max"  => safe_payload[:queries],
-        "statuses"     => { "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0, "all" => 1 }
-      }
-
-      this_second_metrics["statuses"][status_group] = 1
-    end
-
-    redis.setex(this_second, interval_seconds, Oj.dump(this_second_metrics))
-    true
-  rescue => error
-    formatted_error(error)
-  end
-
-  def show(options=nil)
-    @options = options || default_options
-    partitioned_metrics ? aggregate_partitioned_data : aggregate_data
-  end
-
-  def flatten
-    @flat = true
-    self
-  end
-
-  def partition_by(time_unit=:minute)
-    time_unit = PARTITION_UNITS.include?(time_unit) ? time_unit : :minute
-    @partitioned_metrics = interval_metrics.group_by { |h| second_to_partition_unit(time_unit, h["second"]) }
-    self
-  end
-
-  private
-
-  attr_reader :redis, :interval_seconds, :interval_metrics, :requests, :flat,
-    :storage_key, :safe_payload, :this_second_metrics, :partitioned_metrics, :options
-
-  def aggregate_data
-    return {} unless interval_metrics.any?
-    @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
-    build_result
-  rescue
-    {}
-  end
-
-  def aggregate_partitioned_data
-    partitioned_metrics.map do |partition, metrics|
-      @interval_metrics = metrics
-      @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
-      flat ? { timestamp: partition, **build_result } : { timestamp: partition, data: build_result }
-    end
-  rescue
-    new(options)
-  end
-
-  def build_result
-    result = {}
-
-    if options[:requests]
-      append_requests_to_result(result, { all: requests, grouped: count_all_status_groups })
-    end
-
-    options.each do |metrics, aggregation_functions|
-      next unless METRICS.include?(metrics)
-      aggregation_functions = [aggregation_functions] unless aggregation_functions.is_a?(Array)
-      next unless aggregation_functions.any?
-
-      aggregation_functions.each do |aggregation_function|
-        aggregated_metrics = aggregate(metrics, aggregation_function)
-        append_metrics_to_result(result, metrics, aggregation_function, aggregated_metrics)
-      end
-    end
-    result
-  ensure
-    result
-  end
-
-  def append_requests_to_result(result, aggregated_requests)
-    return result[:requests] = aggregated_requests unless flat
-
-    result[:requests_all] = aggregated_requests[:all]
-    aggregated_requests[:grouped].each do |group, counter|
-      result[:"requests_#{group}"] = counter
-    end
-  end
-
-  def append_metrics_to_result(result, metrics, aggregation_function, aggregated_metrics)
-    return result[:"#{metrics}_#{aggregation_function}"] = aggregated_metrics if flat
-
-    result[metrics] ||= {}
-    result[metrics][aggregation_function] = aggregated_metrics
-  end
-
-  def second_to_partition_unit(time_unit, second)
-    return second if time_unit == :second
-    time = Time.at(second)
-    return (time - time.sec - time.min * 60 - time.hour * 3600).to_i if time_unit == :day
-    return (time - time.sec - time.min * 60).to_i                    if time_unit == :hour
-    (time - time.sec).to_i
-  end
-
-  def interval_metrics
-    @interval_metrics ||= begin
-      interval_start    = Time.now.to_i - interval_seconds
-      interval_keys     = (interval_start..Time.now.to_i).to_a
-      redis.mget(interval_keys).compact.map { |hash| Oj.load(hash) }
-    end
-  end
-
-  def aggregate(metrics, aggregation_function)
-    return unless AGGREGATION_FUNCTIONS.include?(aggregation_function)
-    return avg("#{metrics}_sum") if aggregation_function == :avg
-    return max("#{metrics}_max") if aggregation_function == :max
-  end
-
-  def update_sum(metrics)
-    this_second_metrics["#{metrics}_sum"] += safe_payload[metrics]
-  end
-
-  def update_max(metrics)
-    max_value = [safe_payload[metrics], this_second_metrics["#{metrics}_max"]].max
-    this_second_metrics["#{metrics}_max"] = max_value
-  end
-
-  def avg(metrics)
-    (interval_metrics.sum { |h| h[metrics] }.to_f / requests).round
-  end
-
-  def max(metrics)
-    interval_metrics.max { |h| h[metrics] }[metrics].round
-  end
-
-  def count_all_status_groups
-    interval_metrics.inject({ "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0 }) do |result, h|
-      result["2xx"] += h["statuses"]["2xx"]
-      result["3xx"] += h["statuses"]["3xx"]
-      result["4xx"] += h["statuses"]["4xx"]
-      result["5xx"] += h["statuses"]["5xx"]
-      result
-    end
-  end
-
-  def default_options
-    {
-      duration: AGGREGATION_FUNCTIONS,
-      views:    AGGREGATION_FUNCTIONS,
-      db:       AGGREGATION_FUNCTIONS,
-      queries:  AGGREGATION_FUNCTIONS,
-      requests: true
-    }
-  end
-
-  def formatted_error(error)
-    {
-      error:     error.class.name,
-      message:   error.message,
-      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
-    }
-  end
-end
-
-require "ezmetrics/benchmark"
+end