ezmetrics 2.0.0 → 3.0.1

data/README.md

@@ -12,78 +12,27 @@ gem 'ezmetrics'
 
 ## Available metrics
 
-|    Type    |          Aggregate functions      |
-|:----------:|:---------------------------------:|
-| `duration` |  `avg`, `max`, `percentile`       |
-|   `views`  |  `avg`, `max`, `percentile`       |
-|    `db`    |  `avg`, `max`, `percentile`       |
-|  `queries` |  `avg`, `max`, `percentile`       |
+|    Type    |        Aggregate functions        |
+| :--------: | :-------------------------------: |
+| `duration` |    `avg`, `max`, `percentile`     |
+|  `views`   |    `avg`, `max`, `percentile`     |
+|    `db`    |    `avg`, `max`, `percentile`     |
+| `queries`  |    `avg`, `max`, `percentile`     |
 | `requests` | `all`, `2xx`, `3xx`, `4xx`, `5xx` |
 
 ## Usage
 
-### Getting started
-
-This tool captures and aggregates Rails application metrics such as
-
-- `duration`
-- `views`
-- `db`
-- `queries`
-- `status`
-
-and stores them for the timeframe you specified, 60 seconds by default.
-
-You can change the timeframe according to your needs and save the metrics by calling `log` method:
-
-```ruby
-# Store the metrics for 60 seconds (default behaviour)
-EZmetrics.new.log(
-  duration: 100.5,
-  views:    40.7,
-  db:       59.8,
-  queries:  4,
-  status:   200
-)
-```
-
-```ruby
-# Store the metrics for 10 minutes
-EZmetrics.new(10.minutes).log(
-  duration: 100.5,
-  views:    40.7,
-  db:       59.8,
-  queries:  4,
-  status:   200
-)
-```
-
----
-
-For displaying metrics you need to call `show` method:
-
-```ruby
-# Aggregate and show metrics for last 60 seconds (default behaviour)
-EZmetrics.new.show
-```
-
-```ruby
-# Aggregate and show metrics for last 10 minutes
-EZmetrics.new(10.minutes).show
-```
-
-You can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
 
 ### Capture metrics
 
-Just add an initializer to your application:
+Add an initializer to your application:
 
 ```ruby
 # config/initializers/ezmetrics.rb
 
 ActiveSupport::Notifications.subscribe("sql.active_record") do |*args|
   event = ActiveSupport::Notifications::Event.new(*args)
-  unless event.payload[:name] == "SCHEMA"
+  unless event.payload[:name] == "SCHEMA" || event.payload[:sql] =~ /\ABEGIN|COMMIT|ROLLBACK\z/
     Thread.current[:queries] ||= 0
     Thread.current[:queries] += 1
   end
@@ -91,22 +40,35 @@ end
 
 ActiveSupport::Notifications.subscribe("process_action.action_controller") do |*args|
   event = ActiveSupport::Notifications::Event.new(*args)
-  EZmetrics.new.log(
+  Ezmetrics::Storage.new(24.hours).log(
     duration: event.duration.to_f,
     views:    event.payload[:view_runtime].to_f,
     db:       event.payload[:db_runtime].to_f,
     status:   event.payload[:status].to_i || 500,
     queries:  Thread.current[:queries].to_i,
   )
+  Thread.current[:queries] = 0
 end
 ```
 
 ### Display metrics
 
+#### 1. Dashboard
+
+```ruby
+# add the following line to 'config/routes.rb'
+
+mount Dashboard::Ezmetrics, at: "/dashboard", as: "dashboard"
+```
+
+![Dashboard](https://user-images.githubusercontent.com/1847948/73551868-e26b2800-444f-11ea-83b9-fc81c8d05c07.png)
+
+#### 2. Directly
+
 As simple as:
 
 ```ruby
-EZmetrics.new.show
+Ezmetrics::Storage.new.show
 ```
 
 This will return a hash with the following structure:
@@ -146,7 +108,7 @@ This will return a hash with the following structure:
 If you prefer a single level object - you can change the default output structure by calling `.flatten` before `.show`
 
 ```ruby
-EZmetrics.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
+Ezmetrics::Storage.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
 ```
 
 ```ruby
@@ -162,7 +124,7 @@ EZmetrics.new(1.hour).flatten.show(db: :avg, duration: [:avg, :max])
 Same for [partitioned aggregation](#partitioning)
 
 ```ruby
-EZmetrics.new(1.hour).partition_by(:minute).flatten.show(db: :avg, duration: [:avg, :max])
+Ezmetrics::Storage.new(1.hour).partition_by(:minute).flatten.show(db: :avg, duration: [:avg, :max])
 ```
 
 ```ruby
@@ -189,7 +151,7 @@ The aggregation can be easily configured by specifying aggregation options as in
 **1. Single**
 
 ```ruby
-EZmetrics.new.show(duration: :max)
+Ezmetrics::Storage.new.show(duration: :max)
 ```
 
 ```ruby
@@ -205,7 +167,7 @@ EZmetrics.new.show(duration: :max)
 **2. Multiple**
 
 ```ruby
-EZmetrics.new.show(queries: [:max, :avg])
+Ezmetrics::Storage.new.show(queries: [:max, :avg])
 ```
 
 ```ruby
@@ -222,7 +184,7 @@ EZmetrics.new.show(queries: [:max, :avg])
 **3. Requests**
 
 ```ruby
-EZmetrics.new.show(requests: true)
+Ezmetrics::Storage.new.show(requests: true)
 ```
 
 ```ruby
@@ -244,7 +206,7 @@ EZmetrics.new.show(requests: true)
 **4. Combined**
 
 ```ruby
-EZmetrics.new.show(views: :avg, :db: [:avg, :max], requests: true)
+Ezmetrics::Storage.new.show(views: :avg, :db: [:avg, :max], requests: true)
 ```
 
 ```ruby
@@ -279,7 +241,7 @@ By default percentile aggregation is turned off because it requires to store eac
 To enable this feature - you need to set `store_each_value: true` when saving the metrics:
 
 ```ruby
-EZmetrics.new.log(
+Ezmetrics::Storage.new.log(
   duration:         100.5,
   views:            40.7,
   db:               59.8,
@@ -291,9 +253,8 @@ EZmetrics.new.log(
 
 The aggregation syntax has the following format `metrics_type: :percentile_{number}` where `number` is any integer in the 1..99 range.
 
-
 ```ruby
-EZmetrics.new.show(db: [:avg, :percentile_90, :percentile_95], duration: :percentile_99)
+Ezmetrics::Storage.new.show(db: [:avg, :percentile_90, :percentile_95], duration: :percentile_99)
 ```
 
 ```ruby
@@ -309,6 +270,29 @@ EZmetrics.new.show(db: [:avg, :percentile_90, :percentile_95], duration: :percen
 }
 ```
 
+**6. Percentile distribution**
+
+If you want to visualize percentile distribution (from 1% to 99%):
+
+```ruby
+Ezmetrics::Storage.new.show(duration: :percentile_distribution)
+```
+
+```ruby
+{
+  duration: {
+    percentile_distribution: {
+      1: 12,
+      2: 15,
+      3: 19,
+      #...
+      97: 6540,
+      98: 6682,
+      99: 6730
+    }
+  }
+}
+```
 
 ### Partitioning
 
@@ -318,7 +302,7 @@ To aggregate metrics, partitioned by a unit of time you need to call `.partition
 
 ```ruby
 # Aggregate metrics for last hour, partition by minute
-EZmetrics.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
+Ezmetrics::Storage.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
 ```
 
 This will return an array of objects with the following structure:
@@ -365,75 +349,38 @@ like in the example below:
 
 Available time units for partitioning: `second`, `minute`, `hour`, `day`. Default: `minute`.
 
-### Performance
+## Performance
 
 The aggregation speed relies on the performance of **Redis** (data storage) and **Oj** (json serialization/parsing).
 
-#### Simple aggregation
-
-You can check the **aggregation** time by running:
+You can check the **aggregation** time (in seconds) by running:
 
 ```ruby
-EZmetrics::Benchmark.new.measure_aggregation
-```
-
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.02        |
-| 12 hours |        0.22        |
-| 24 hours |        0.61        |
-| 48 hours |        1.42        |
+# 1. Simple
+Ezmetrics::Benchmark.new.measure_aggregation
 
----
+# 2. Partitioned
+Ezmetrics::Benchmark.new.measure_aggregation(:minute)
 
-To check the **partitioned aggregation** time you need to run:
+# 3. Percentile
+Ezmetrics::Benchmark.new(true).measure_aggregation
 
-```ruby
-EZmetrics::Benchmark.new.measure_aggregation(:minute)
+# 4. Percentile (partitioned)
+Ezmetrics::Benchmark.new(true).measure_aggregation(:minute)
 ```
 
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.02        |
-| 12 hours |        0.25        |
-| 24 hours |        0.78        |
-| 48 hours |        1.75        |
+| Interval | Simple aggregation | Partitioned | Percentile | Percentile (partitioned) |
+| :------: | :----------------: | :---------: | :--------: | :----------------------: |
+| 1 minute |        0.0         |     0.0     |    0.0     |           0.0            |
+|  1 hour  |        0.02        |    0.02     |    0.14    |           0.16           |
+| 12 hours |        0.22        |    0.25     |    2.11    |           1.97           |
+| 24 hours |        0.61        |    0.78     |    5.85    |           5.85           |
+| 48 hours |        1.42        |    1.75     |    14.1    |           13.9           |
 
----
-
-#### Percentile aggregation
-
-You can check the **percentile aggregation** time by running:
-
-```ruby
-EZmetrics::Benchmark.new(true).measure_aggregation
-```
-
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.14        |
-| 12 hours |        2.11        |
-| 24 hours |        5.85        |
-| 48 hours |        14.1        |
-
----
-
-To check the **partitioned aggregation** time for percentile you need to run:
-
-```ruby
-EZmetrics::Benchmark.new(true).measure_aggregation(:minute)
-```
+The benchmarks above were run on a _2017 Macbook Pro 2.9 GHz Intel Core i7 with 16 GB of RAM_
 
-| Interval | Duration (seconds) |
-| :------: | :----------------: |
-| 1 minute |        0.0         |
-|  1 hour  |        0.16        |
-| 12 hours |        1.97        |
-| 24 hours |        5.85        |
-| 48 hours |        13.9        |
+## [Changelog](CHANGELOG.md)
 
+## License
 
-The benchmarks above were run on a _2017 Macbook Pro 2.9 GHz Intel Core i7 with 16 GB of RAM_
+ezmetrics is released under the [MIT License](https://opensource.org/licenses/MIT).

data/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-tactile

data/ezmetrics.gemspec

@@ -0,0 +1,21 @@
+require_relative "lib/ezmetrics/version"
+
+Gem::Specification.new do |gem|
+  gem.name                  = "ezmetrics"
+  gem.version               = Ezmetrics::VERSION
+  gem.date                  = "2020-02-01"
+  gem.summary               = "Rails metrics aggregation tool."
+  gem.description           = "Simple, lightweight and fast metrics aggregation for Rails."
+  gem.authors               = ["Nicolae Rotaru"]
+  gem.email                 = "nyku.rn@gmail.com"
+  gem.homepage              = "https://github.com/nyku/ezmetrics"
+  gem.license               = "MIT"
+  gem.files                 = `git ls-files | grep -Ev '^(spec)'`.split("\n")
+  gem.require_paths         = ["lib"]
+  gem.required_ruby_version = ">= 2.4.0"
+  gem.add_dependency "redis", ["~> 4.0"]
+  gem.add_dependency "hiredis", ["~> 0.6.3"]
+  gem.add_dependency "oj", ["~> 3.10"]
+
+  gem.add_development_dependency "rspec", "~> 3.5"
+end

data/lib/ezmetrics.rb

@@ -2,286 +2,12 @@ require "redis"
 require "redis/connection/hiredis"
 require "oj"
 
-class EZmetrics
-  METRICS               = [:duration, :views, :db, :queries].freeze
-  AGGREGATION_FUNCTIONS = [:max, :avg].freeze
-  PARTITION_UNITS       = [:second, :minute, :hour, :day].freeze
+module Ezmetrics
+  require_relative "ezmetrics/version"
+  require_relative "ezmetrics/storage"
+  require_relative "ezmetrics/benchmark"
 
-  def initialize(interval_seconds=60)
-    @interval_seconds = interval_seconds.to_i
-    @redis            = Redis.new(driver: :hiredis)
-    @schema           = redis_schema
+  if defined?(Rails)
+    require_relative "ezmetrics/dashboard/lib/dashboard"
   end
-
-  def log(payload={duration: 0.0, views: 0.0, db: 0.0, queries: 0, status: 200, store_each_value: false})
-    @safe_payload = {
-      duration:         payload[:duration].to_f,
-      views:            payload[:views].to_f,
-      db:               payload[:db].to_f,
-      queries:          payload[:queries].to_i,
-      status:           payload[:status].to_i,
-      store_each_value: payload[:store_each_value].to_s == "true"
-    }
-
-    this_second          = Time.now.to_i
-    status_group         = "#{payload[:status].to_s[0]}xx"
-    @this_second_metrics = redis.get(this_second)
-
-    if this_second_metrics
-      @this_second_metrics = Oj.load(this_second_metrics)
-
-      METRICS.each do |metrics_type|
-        update_sum(metrics_type)
-        update_max(metrics_type)
-        store_value(metrics_type) if safe_payload[:store_each_value]
-      end
-
-      this_second_metrics[schema["all"]]        += 1
-      this_second_metrics[schema[status_group]] += 1
-    else
-      @this_second_metrics = {
-        "second"       => this_second,
-        "duration_sum" => safe_payload[:duration],
-        "duration_max" => safe_payload[:duration],
-        "views_sum"    => safe_payload[:views],
-        "views_max"    => safe_payload[:views],
-        "db_sum"       => safe_payload[:db],
-        "db_max"       => safe_payload[:db],
-        "queries_sum"  => safe_payload[:queries],
-        "queries_max"  => safe_payload[:queries],
-        "2xx"          => 0,
-        "3xx"          => 0,
-        "4xx"          => 0,
-        "5xx"          => 0,
-        "all"          => 1
-      }
-
-      if safe_payload[:store_each_value]
-        this_second_metrics.merge!(
-          "duration_values" => [safe_payload[:duration]],
-          "views_values"    => [safe_payload[:views]],
-          "db_values"       => [safe_payload[:db]],
-          "queries_values"  => [safe_payload[:queries]]
-        )
-      end
-
-      this_second_metrics[status_group] = 1
-
-      @this_second_metrics = this_second_metrics.values
-    end
-
-    redis.setex(this_second, interval_seconds, Oj.dump(this_second_metrics))
-    true
-  rescue => error
-    formatted_error(error)
-  end
-
-  def show(options=nil)
-    @options = options || default_options
-    partitioned_metrics ? aggregate_partitioned_data : aggregate_data
-  end
-
-  def flatten
-    @flat = true
-    self
-  end
-
-  def partition_by(time_unit=:minute)
-    time_unit = PARTITION_UNITS.include?(time_unit) ? time_unit : :minute
-    @partitioned_metrics = interval_metrics.group_by { |array| second_to_partition_unit(time_unit, array[schema["second"]]) }
-    self
-  end
-
-  private
-
-  attr_reader :redis, :interval_seconds, :interval_metrics, :requests, :flat, :schema,
-    :storage_key, :safe_payload, :this_second_metrics, :partitioned_metrics, :options
-
-  def aggregate_data
-    return {} unless interval_metrics.any?
-    @requests = interval_metrics.sum { |array| array[schema["all"]] }
-    build_result
-  rescue
-    {}
-  end
-
-  def aggregate_partitioned_data
-    partitioned_metrics.map do |partition, metrics|
-      @interval_metrics = metrics
-      @requests = interval_metrics.sum { |array| array[schema["all"]] }
-      METRICS.each { |metrics_type| instance_variable_set("@sorted_#{metrics_type}_values", nil) }
-      flat ? { timestamp: partition, **build_result } : { timestamp: partition, data: build_result }
-    end
-  rescue
-    self
-  end
-
-  def build_result
-    result = {}
-
-    if options[:requests]
-      append_requests_to_result(result, { all: requests, grouped: count_all_status_groups })
-    end
-
-    options.each do |metrics, aggregation_functions|
-      next unless METRICS.include?(metrics)
-      aggregation_functions = [aggregation_functions] unless aggregation_functions.is_a?(Array)
-      next unless aggregation_functions.any?
-
-      aggregation_functions.each do |aggregation_function|
-        aggregated_metrics = aggregate(metrics, aggregation_function)
-        append_metrics_to_result(result, metrics, aggregation_function, aggregated_metrics)
-      end
-    end
-    result
-  ensure
-    result
-  end
-
-  def append_requests_to_result(result, aggregated_requests)
-    return result[:requests] = aggregated_requests unless flat
-
-    result[:requests_all] = aggregated_requests[:all]
-    aggregated_requests[:grouped].each do |group, counter|
-      result[:"requests_#{group}"] = counter
-    end
-  end
-
-  def append_metrics_to_result(result, metrics, aggregation_function, aggregated_metrics)
-    return result[:"#{metrics}_#{aggregation_function}"] = aggregated_metrics if flat
-
-    result[metrics] ||= {}
-    result[metrics][aggregation_function] = aggregated_metrics
-  end
-
-  def second_to_partition_unit(time_unit, second)
-    return second if time_unit == :second
-    time = Time.at(second)
-    return (time - time.sec - time.min * 60 - time.hour * 3600).to_i if time_unit == :day
-    return (time - time.sec - time.min * 60).to_i                    if time_unit == :hour
-    (time - time.sec).to_i
-  end
-
-  def interval_metrics
-    @interval_metrics ||= begin
-      interval_start    = Time.now.to_i - interval_seconds
-      interval_keys     = (interval_start..Time.now.to_i).to_a
-      redis.mget(interval_keys).compact.map { |array| Oj.load(array) }
-    end
-  end
-
-  def aggregate(metrics, aggregation_function)
-    return avg("#{metrics}_sum") if aggregation_function == :avg
-    return max("#{metrics}_max") if aggregation_function == :max
-
-    percentile = aggregation_function.match(/percentile_(?<value>\d+)/)
-
-    if percentile && percentile["value"]
-      sorted_values = send("sorted_#{metrics}_values")
-      percentile(sorted_values, percentile["value"].to_i)&.round
-    end
-  end
-
-  METRICS.each do |metrics|
-    define_method "sorted_#{metrics}_values" do
-      instance_variable_get("@sorted_#{metrics}_values") || instance_variable_set(
-        "@sorted_#{metrics}_values", interval_metrics.map { |array| array[schema["#{metrics}_values"]] }.flatten.compact
-      )
-    end
-  end
-
-  def redis_schema
-    [
-      "second",
-      "duration_sum",
-      "duration_max",
-      "views_sum",
-      "views_max",
-      "db_sum",
-      "db_max",
-      "queries_sum",
-      "queries_max",
-      "2xx",
-      "3xx",
-      "4xx",
-      "5xx",
-      "all",
-      "duration_values",
-      "views_values",
-      "db_values",
-      "queries_values"
-    ].each_with_index.inject({}){ |result, pair| result[pair[0]] = pair[1] ; result }
-  end
-
-  def update_sum(metrics)
-    this_second_metrics[schema["#{metrics}_sum"]] += safe_payload[metrics]
-  end
-
-  def store_value(metrics)
-    this_second_metrics[schema["#{metrics}_values"]] << safe_payload[metrics]
-  end
-
-  def update_max(metrics)
-    max_value = [safe_payload[metrics], this_second_metrics[schema["#{metrics}_max"]]].max
-    this_second_metrics[schema["#{metrics}_max"]] = max_value
-  end
-
-  def avg(metrics)
-    (interval_metrics.sum { |array| array[schema[metrics]] }.to_f / requests).round
-  end
-
-  def max(metrics)
-    interval_metrics.max { |array| array[schema[metrics]] }[schema[metrics]].round
-  end
-
-  def percentile(array, pcnt)
-    sorted_array = array.sort
-
-    return nil if array.length == 0
-
-    rank  = (pcnt.to_f / 100) * (array.length + 1)
-    whole = rank.truncate
-
-    # if has fractional part
-    if whole != rank
-      s0 = sorted_array[whole - 1]
-      s1 = sorted_array[whole]
-
-      f = (rank - rank.truncate).abs
-
-      return (f * (s1 - s0)) + s0
-    else
-      return sorted_array[whole - 1]
-    end
-  end
-
-  def count_all_status_groups
-    interval_metrics.inject({ "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0 }) do |result, array|
-      result["2xx"] += array[schema["2xx"]]
-      result["3xx"] += array[schema["3xx"]]
-      result["4xx"] += array[schema["4xx"]]
-      result["5xx"] += array[schema["5xx"]]
-      result
-    end
-  end
-
-  def default_options
-    {
-      duration: AGGREGATION_FUNCTIONS,
-      views:    AGGREGATION_FUNCTIONS,
-      db:       AGGREGATION_FUNCTIONS,
-      queries:  AGGREGATION_FUNCTIONS,
-      requests: true
-    }
-  end
-
-  def formatted_error(error)
-    {
-      error:     error.class.name,
-      message:   error.message,
-      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
-    }
-  end
-end
-
-require "ezmetrics/benchmark"
+end