ezmetrics 1.0.4 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 55610b1357968e974ca2e525a6d09bd2245c8f5e
-  data.tar.gz: efef7c5d22cc0c4d9eb27e91b2149a23e3eb4832
+SHA256:
+  metadata.gz: 0affbdfcda841ffe4eec9105ded5526b13d209ccf0aa564e1b84c91651079d0e
+  data.tar.gz: 8c0f3879ee0f2cedeba7f6b0526f679ef9962e2270c7639837821f226a0f3580
 SHA512:
-  metadata.gz: a161d7d0715d82958a89131e31c3344dcd4553d78909406bedd0041c0d91e05486be64b2c7c382783b540ca837a57bc97763cc9fc1a88a821f7e19083b22e0ce
-  data.tar.gz: 32073d20b467a1c2e98b0efe561f8513d7e2de23dd75740c42cc3628a62112b1ee69658d533f2b91c99453c375556815dda4515d159b90d7ccf1b7b9cba602e8
+  metadata.gz: 28d6883722e140005b24071ab637104d6d0e16e22a75fffac734b242fafbb88c1ecdbfd4d4d44a11439a8cbac5d0ad5dcfba705ed26a706f6abb83f55f3f5513
+  data.tar.gz: b02ddcabf2669f9458a21dd37e0495218eedff6f15047c923b81d6fbcd311ce34ed2d27c7ea664a97aebaf806eac653dd4ec09fb3ee526607da796ec4c3a54f2

data/README.md

@@ -2,8 +2,7 @@
 
 [![Gem Version](https://badge.fury.io/rb/ezmetrics.svg)](https://badge.fury.io/rb/ezmetrics)
 
-A simple tool for capturing and displaying Rails metrics.
-
+Simple, lightweight and fast metrics aggregation for Rails.
 
 ## Installation
 
@@ -15,45 +14,59 @@ gem 'ezmetrics'
 
 ### Getting started
 
-This tool captures and aggregates metrics such as 
-- `status`
+This tool captures and aggregates Rails application metrics such as
+
 - `duration`
-- `queries`
+- `views`
 - `db`
+- `queries`
+- `status`
 
-for a 60 seconds timeframe by default.
+and stores them for the timeframe you specified, 60 seconds by default.
 
-You can change the timeframe according to your needs and save the metrics by calling `log` method: 
+You can change the timeframe according to your needs and save the metrics by calling `log` method:
 
 ```ruby
   # Store the metrics for 60 seconds (default behaviour)
-  EZmetrics.new.log(status: 200, db: 300.45, duration: 320.45, queries: 5)
+  EZmetrics.new.log(
+    duration: 100.5,
+    views:    40.7,
+    db:       59.8,
+    queries:  4,
+    status:   200
+  )
 ```
-or
 
 ```ruby
   # Store the metrics for 10 minutes
-  EZmetrics.new(10.minutes).log(status: 200, db: 300.45, duration: 320.45, queries: 5) 
+  EZmetrics.new(10.minutes).log(
+    duration: 100.5,
+    views:    40.7,
+    db:       59.8,
+    queries:  4,
+    status:   200
+  )
 ```
 
-For displaying metrics you need call `show` method:
+---
+
+For displaying metrics you need to call `show` method:
 
 ```ruby
   # Aggregate and show metrics for last 60 seconds (default behaviour)
-  EZmetrics.new.show 
+  EZmetrics.new.show
 ```
 
-or
-
 ```ruby
   # Aggregate and show metrics for last 10 minutes
   EZmetrics.new(10.minutes).show
 ```
 
-> Please note that you can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
+You can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
 
+### Capture metrics
 
-### Add an initializer to your Rails application
+Just add an initializer to your application:
 
 ```ruby
 # config/initializers/ezmetrics.rb
@@ -69,16 +82,19 @@ end
 ActiveSupport::Notifications.subscribe("process_action.action_controller") do |*args|
   event = ActiveSupport::Notifications::Event.new(*args)
   EZmetrics.new.log(
-    queries:  Thread.current[:queries].to_i,
-    db:       event.payload[:db_runtime].to_f,
     duration: event.duration.to_f,
-    status:   event.payload[:status].to_i || 500
+    views:    event.payload[:view_runtime].to_f,
+    db:       event.payload[:db_runtime].to_f,
+    status:   event.payload[:status].to_i || 500,
+    queries:  Thread.current[:queries].to_i,
   )
 end
 ```
 
 ### Display metrics
 
+As simple as:
+
 ```ruby
 EZmetrics.new.show
 ```
@@ -91,6 +107,10 @@ This will return a hash with the following structure:
     avg: 5569,
     max: 9675
   },
+  views: {
+    avg: 12,
+    max: 240
+  },
   db: {
     avg: 155,
     max: 4382
@@ -111,3 +131,177 @@ This will return a hash with the following structure:
 }
 ```
 
+### Aggregation
+
+The aggregation can be easily configured by specifying aggregation options as in the following examples:
+
+**1. Single**
+
+```ruby
+EZmetrics.new.show(duration: :max)
+```
+
+```ruby
+{
+  duration: {
+    max: 9675
+  }
+}
+```
+
+---
+
+**2. Multiple**
+
+```ruby
+EZmetrics.new.show(queries: [:max, :avg])
+```
+
+```ruby
+{
+  queries: {
+    max: 76,
+    avg: 26
+  }
+}
+```
+
+---
+
+**3. Requests**
+
+```ruby
+EZmetrics.new.show(requests: true)
+```
+
+```ruby
+{
+  requests: {
+    all: 2000,
+    grouped: {
+      "2xx" => 1900,
+      "3xx" => 15,
+      "4xx" => 80,
+      "5xx" => 5
+    }
+  }
+}
+```
+
+---
+
+**4. Combined**
+
+```ruby
+EZmetrics.new.show(views: :avg, :db: [:avg, :max], requests: true)
+```
+
+```ruby
+{
+  views: {
+    avg: 12
+  },
+  db: {
+    avg: 155,
+    max: 4382
+  },
+  requests: {
+    all: 2000,
+    grouped: {
+      "2xx" => 1900,
+      "3xx" => 15,
+      "4xx" => 80,
+      "5xx" => 5
+    }
+  }
+}
+```
+
+### Partitioning
+
+To aggregate metrics, partitioned by a unit of time you need to call `partition_by({time_unit})` before calling `show`
+
+```ruby
+  # Aggregate metrics for last hour, partition by minute
+  EZmetrics.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
+```
+
+This will return an array of objects with the following structure:
+
+```ruby
+[
+  {
+    timestamp: # UNIX timestamp
+    data:      # a hash with aggregated metrics
+  }
+]
+```
+
+like in the example below:
+
+```ruby
+[
+  {
+    timestamp: 1575242880,
+    data: {
+      duration: {
+        avg: 477,
+        max: 8566
+      },
+      db: {
+        avg: 387
+      }
+    }
+  },
+  {
+    timestamp: 1575242940,
+    data: {
+      duration: {
+        avg: 234,
+        max: 3675
+      },
+      db: {
+        avg: 123
+      }
+    }
+  }
+]
+```
+
+Available time units for partitioning: `second`, `minute`, `hour`, `day`. Default: `minute`.
+
+### Performance
+
+The aggregation speed relies on the performance of **Redis** (data storage) and **Oj** (json serialization/parsing).
+
+You can check the **aggregation** time by running:
+
+```ruby
+EZmetrics::Benchmark.new.measure_aggregation
+```
+
+| Interval | Duration (seconds) |
+| :------: | :----------------: |
+| 1 minute |        0.0         |
+|  1 hour  |        0.04        |
+| 12 hours |        0.49        |
+| 24 hours |        1.51        |
+| 48 hours |        3.48        |
+
+---
+
+To check the **partitioned aggregation** time you need to run:
+
+```ruby
+EZmetrics::Benchmark.new.measure_aggregation(:minute)
+```
+
+| Interval | Duration (seconds) |
+| :------: | :----------------: |
+| 1 minute |        0.0         |
+|  1 hour  |        0.05        |
+| 12 hours |        0.74        |
+| 24 hours |        2.12        |
+| 48 hours |        4.85        |
+
+The benchmarks above were run on a _2017 Macbook Pro 2.9 GHz Intel Core i7 with 16 GB of RAM_

data/lib/ezmetrics.rb

@@ -1,135 +1,183 @@
-require "redis" unless defined?(Redis)
-require "json"  unless defined?(JSON)
+require "redis"
+require "redis/connection/hiredis"
+require "oj"
 
 class EZmetrics
+  METRICS               = [:duration, :views, :db, :queries].freeze
+  AGGREGATION_FUNCTIONS = [:max, :avg].freeze
+  PARTITION_UNITS       = [:second, :minute, :hour, :day].freeze
+
   def initialize(interval_seconds=60)
     @interval_seconds = interval_seconds.to_i
     @redis            = Redis.new
-    @storage_key      = "ez-metrics"
   end
 
-  def log(payload={db: 0.0, duration: 0.0, queries: 0, status: 200})
-    payload = {
+  def log(payload={duration: 0.0, views: 0.0, db: 0.0, queries: 0, status: 200})
+    @safe_payload = {
+      duration: payload[:duration].to_f,
+      views:    payload[:views].to_f,
       db:       payload[:db].to_f,
       queries:  payload[:queries].to_i,
-      duration: payload[:duration].to_f,
       status:   payload[:status].to_i
     }
 
-    this_second         = Time.now.to_i
-    status_group        = "#{payload[:status].to_s[0]}xx"
-    this_second_metrics = redis.get("#{storage_key}:#{this_second}")
+    this_second          = Time.now.to_i
+    status_group         = "#{payload[:status].to_s[0]}xx"
+    @this_second_metrics = redis.get(this_second)
 
     if this_second_metrics
-      this_second_metrics = JSON.parse(this_second_metrics)
-      this_second_metrics["db_sum"]                 += payload[:db]
-      this_second_metrics["queries_sum"]            += payload[:queries]
-      this_second_metrics["duration_sum"]           += payload[:duration]
+      @this_second_metrics = Oj.load(this_second_metrics)
+
+      METRICS.each do |metrics_type|
+        update_sum(metrics_type)
+        update_max(metrics_type)
+      end
+
       this_second_metrics["statuses"]["all"]        += 1
       this_second_metrics["statuses"][status_group] += 1
-      this_second_metrics["db_max"]                 = [payload[:db], this_second_metrics["db_max"]].max
-      this_second_metrics["queries_max"]            = [payload[:queries], this_second_metrics["queries_max"]].max
-      this_second_metrics["duration_max"]           = [payload[:duration], this_second_metrics["duration_max"]].max
     else
-      this_second_metrics = {
-        "db_sum"       => payload[:db],
-        "db_max"       => payload[:db],
-        "queries_sum"  => payload[:queries],
-        "queries_max"  => payload[:queries],
-        "duration_sum" => payload[:duration],
-        "duration_max" => payload[:duration],
+      @this_second_metrics = {
+        "second"       => this_second,
+        "duration_sum" => safe_payload[:duration],
+        "duration_max" => safe_payload[:duration],
+        "views_sum"    => safe_payload[:views],
+        "views_max"    => safe_payload[:views],
+        "db_sum"       => safe_payload[:db],
+        "db_max"       => safe_payload[:db],
+        "queries_sum"  => safe_payload[:queries],
+        "queries_max"  => safe_payload[:queries],
         "statuses"     => { "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0, "all" => 1 }
       }
 
       this_second_metrics["statuses"][status_group] = 1
     end
 
-    redis.setex("#{storage_key}:#{this_second}", interval_seconds, JSON.generate(this_second_metrics))
-
+    redis.setex(this_second, interval_seconds, Oj.dump(this_second_metrics))
     true
   rescue => error
     formatted_error(error)
   end
 
-  def show
-    interval_start    = Time.now.to_i - interval_seconds
-    interval_keys     = (interval_start..Time.now.to_i).to_a.map { |second| "#{storage_key}:#{second}" }
-    @interval_metrics = redis.mget(interval_keys).compact.map { |hash| JSON.parse(hash) }
+  def show(options=nil)
+    @options = options || default_options
+    partitioned_metrics ? aggregate_partitioned_data : aggregate_data
+  end
 
-    return empty_metrics_object unless interval_metrics.any?
+  def partition_by(time_unit=:minute)
+    time_unit = PARTITION_UNITS.include?(time_unit) ? time_unit : :minute
+    @partitioned_metrics = interval_metrics.group_by { |h| second_to_partition_unit(time_unit, h["second"]) }
+    self
+  end
 
-    @requests = interval_metrics.map { |hash| hash["statuses"]["all"] }.compact.sum
+  private
 
-    metrics_object
+  attr_reader :redis, :interval_seconds, :interval_metrics, :requests,
+    :storage_key, :safe_payload, :this_second_metrics, :partitioned_metrics, :options
+
+  def aggregate_data
+    return {} unless interval_metrics.any?
+    @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
+    build_result
   rescue
-    empty_metrics_object
+    {}
   end
 
-  private
+  def aggregate_partitioned_data
+    partitioned_metrics.map do |partition, metrics|
+      @interval_metrics = metrics
+      @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
+      { timestamp: partition, data: build_result }
+    end
+  rescue
+    new(options)
+  end
 
-  attr_reader :redis, :interval_seconds, :interval_metrics, :requests, :storage_key
+  def build_result
+    result = {}
+
+    result[:requests] = { all: requests, grouped: count_all_status_groups } if options[:requests]
+
+    options.each do |metrics, aggregation_functions|
+      next unless METRICS.include?(metrics)
+      aggregation_functions = [aggregation_functions] unless aggregation_functions.is_a?(Array)
+      next unless aggregation_functions.any?
+
+      aggregation_functions.each do |aggregation_function|
+        result[metrics] ||= {}
+        result[metrics][aggregation_function] = aggregate(metrics, aggregation_function)
+      end
+    end
+    result
+  ensure
+    result
+  end
+
+  def second_to_partition_unit(time_unit, second)
+    return second if time_unit == :second
+    time_unit_depth = { minute: 4, hour: 3, day: 2 }
+    reset_depth     = time_unit_depth[time_unit]
+    time_to_array   = Time.at(second).to_a[0..5].reverse
+    Time.new(*time_to_array[0..reset_depth]).to_i
+  end
+
+  def interval_metrics
+    @interval_metrics ||= begin
+      interval_start    = Time.now.to_i - interval_seconds
+      interval_keys     = (interval_start..Time.now.to_i).to_a
+      redis.mget(interval_keys).compact.map { |hash| Oj.load(hash) }
+    end
+  end
+
+  def aggregate(metrics, aggregation_function)
+    return unless AGGREGATION_FUNCTIONS.include?(aggregation_function)
+    return avg("#{metrics}_sum") if aggregation_function == :avg
+    return max("#{metrics}_max") if aggregation_function == :max
+  end
+
+  def update_sum(metrics)
+    this_second_metrics["#{metrics}_sum"] += safe_payload[metrics]
+  end
+
+  def update_max(metrics)
+    max_value = [safe_payload[metrics], this_second_metrics["#{metrics}_max"]].max
+    this_second_metrics["#{metrics}_max"] = max_value
+  end
 
   def avg(metrics)
-    (interval_metrics.map { |h| h[metrics.to_s] }.sum.to_f / requests).round
+    (interval_metrics.sum { |h| h[metrics] }.to_f / requests).round
   end
 
   def max(metrics)
-    interval_metrics.map { |h| h[metrics.to_s] }.max.round
+    interval_metrics.max { |h| h[metrics] }[metrics].round
   end
 
-  def count(group)
-    interval_metrics.map { |h| h["statuses"][group.to_s] }.sum
+  def count_all_status_groups
+    interval_metrics.inject({ "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0 }) do |result, h|
+      result["2xx"] += h["statuses"]["2xx"]
+      result["3xx"] += h["statuses"]["3xx"]
+      result["4xx"] += h["statuses"]["4xx"]
+      result["5xx"] += h["statuses"]["5xx"]
+      result
+    end
   end
 
-  def formatted_error(error)
+  def default_options
     {
-      error:     error.class.name,
-      message:   error.message,
-      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
+      duration: AGGREGATION_FUNCTIONS,
+      views:    AGGREGATION_FUNCTIONS,
+      db:       AGGREGATION_FUNCTIONS,
+      queries:  AGGREGATION_FUNCTIONS,
+      requests: true
     }
   end
 
-  def metrics_object
+  def formatted_error(error)
     {
-      duration: {
-        avg: avg(:duration_sum),
-        max: max(:duration_max)
-      },
-      db: {
-        avg: avg(:db_sum),
-        max: max(:db_max)
-      },
-      queries: {
-        avg: avg(:queries_sum),
-        max: max(:queries_max)
-      },
-      requests: {
-        all:     requests,
-        grouped: {
-          "2xx" => count("2xx"),
-          "3xx" => count("3xx"),
-          "4xx" => count("4xx"),
-          "5xx" => count("5xx")
-        }
-      }
+      error:     error.class.name,
+      message:   error.message,
+      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
     }
   end
+end
 
-  def empty_metrics_object
-    {
-      duration: {
-        avg: 0,
-        max: 0
-      },
-      db: {
-        avg: 0,
-        max: 0
-      },
-      queries: {
-        avg: 0,
-        max: 0
-      },
-      requests: {}
-    }
-  end
-end
+require "ezmetrics/benchmark"

data/lib/ezmetrics/benchmark.rb

@@ -0,0 +1,95 @@
+require "benchmark"
+
+class EZmetrics::Benchmark
+
+  def initialize
+    @start      = Time.now.to_i
+    @redis      = Redis.new
+    @durations  = []
+    @iterations = 3
+    @intervals  = {
+      "1.minute" => 60,
+      "1.hour  " => 3600,
+      "12.hours" => 43200,
+      "24.hours" => 86400,
+      "48.hours" => 172800
+    }
+  end
+
+  def measure_aggregation(partition_by=nil)
+    write_metrics
+    print_header
+    intervals.each do |interval, seconds|
+      result = measure_aggregation_time(interval, seconds, partition_by)
+      print_row(result)
+    end
+    cleanup_metrics
+    print_footer
+  end
+
+  private
+
+  attr_reader :start, :redis, :durations, :intervals, :iterations
+
+  def write_metrics
+    seconds = intervals.values.max
+    seconds.times do |i|
+      second = start - i
+      payload = {
+        "second"          => second,
+        "duration_sum"    => rand(10000),
+        "duration_max"    => rand(10000),
+        "views_sum"       => rand(1000),
+        "views_max"       => rand(1000),
+        "db_sum"          => rand(8000),
+        "db_max"          => rand(8000),
+        "queries_sum"     => rand(100),
+        "queries_max"     => rand(100),
+        "statuses"        => {
+          "2xx" => rand(1..10),
+          "3xx" => rand(1..10),
+          "4xx" => rand(1..10),
+          "5xx" => rand(1..10),
+          "all" => rand(1..40)
+        }
+      }
+      redis.setex(second, seconds, Oj.dump(payload))
+    end
+    nil
+  end
+
+  def cleanup_metrics
+    interval_start = Time.now.to_i - intervals.values.max - 100
+    interval_keys  = (interval_start..Time.now.to_i).to_a
+    redis.del(interval_keys)
+  end
+
+  def measure_aggregation_time(interval, seconds, partition_by)
+    iterations.times do
+      durations << ::Benchmark.measure do
+        if partition_by
+          EZmetrics.new(seconds).partition_by(partition_by).show
+        else
+          EZmetrics.new(seconds).show
+        end
+      end.real
+    end
+
+    return {
+      interval: interval.gsub(".", " "),
+      duration: (durations.sum.to_f / iterations).round(2)
+    }
+  end
+
+  def print_header
+    print "\n#{'─'*31}\n| Interval | Duration (seconds)\n#{'─'*31}\n"
+  end
+
+  def print_row(result)
+    print "| #{result[:interval]} | #{result[:duration]}\n"
+  end
+
+  def print_footer
+    print "#{'─'*31}\n"
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ezmetrics
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.2.0
 platform: ruby
 authors:
 - Nicolae Rotaru
@@ -24,6 +24,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: hiredis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.6.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.6.3
+- !ruby/object:Gem::Dependency
+  name: oj
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -38,7 +66,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.5'
-description: A simple tool for capturing and displaying Rails metrics.
+description: Simple, lightweight and fast metrics aggregation for Rails.
 email: nyku.rn@gmail.com
 executables: []
 extensions: []
@@ -47,10 +75,12 @@ files:
 - LICENSE
 - README.md
 - lib/ezmetrics.rb
-homepage: https://github.com/nyku/ezmetrics
+- lib/ezmetrics/benchmark.rb
+homepage: https://nyku.github.io/ezmetrics
 licenses:
 - GPL-3.0
-metadata: {}
+metadata:
+  source_code_uri: https://github.com/nyku/ezmetrics
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -66,8 +96,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 3.0.6
 signing_key: 
 specification_version: 4
 summary: Rails metrics aggregation tool.