ezmetrics 1.0.5 → 1.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 1bd4e2f35cc2cf1b50efbac72d107041dc68129e
-  data.tar.gz: 179ed567aac638d1ab0d3a4ed86eaad458c55fd1
+SHA256:
+  metadata.gz: 61d26fa5d7a779917ac2ba154e82f29745a22b64261442117d15b286d0e06276
+  data.tar.gz: 44ff087c54cf0e5fdf8763b88deaa59f3ccdfa1e11c12ab5c1da2eebdeea7764
 SHA512:
-  metadata.gz: e805073975368174e1ce2fd62ce3ddeece44a50bb0141f4a2d7fb9a1b8392c1083667656f29217a1beef6f9add360af0bbba3fe4016263dfa20421af247e1bbf
-  data.tar.gz: 7364788497b995a02cb3385220a7f9e861cc4d7db6377f29850c6d22e25d99b25b6fdbaaa00a6ffb224573739262219136d9739e3676f65af6b4ba03a72c6606
+  metadata.gz: 6bf28edd6c644648198905bcdf50a77a0083c53b6eba312b709e9c1cb951adba1c407cbc28ac3ae0588b15185b5aa0132ef1f568ff30af0158b770d388460885
+  data.tar.gz: a08685fea3025b23e5dca7439e9b7f4f2313933363b01a716e68eba800e4b96fce0d976cf25111c0e7a36f655abb6497e3481d39caba881b2ee706e49958b365

data/README.md

@@ -2,8 +2,7 @@
 
 [![Gem Version](https://badge.fury.io/rb/ezmetrics.svg)](https://badge.fury.io/rb/ezmetrics)
 
-A simple tool for capturing and displaying Rails metrics.
-
+Simple, lightweight and fast metrics aggregation for Rails.
 
 ## Installation
 
@@ -15,46 +14,59 @@ gem 'ezmetrics'
 
 ### Getting started
 
-This tool captures and aggregates metrics such as
+This tool captures and aggregates Rails application metrics such as
+
 - `duration`
 - `views`
 - `db`
 - `queries`
 - `status`
 
-for a 60 seconds timeframe by default.
+and stores them for the timeframe you specified, 60 seconds by default.
 
 You can change the timeframe according to your needs and save the metrics by calling `log` method:
 
 ```ruby
   # Store the metrics for 60 seconds (default behaviour)
-  EZmetrics.new.log(duration: 100.5, views: 40.7, db: 59.8, queries: 4, status: 200)
+  EZmetrics.new.log(
+    duration: 100.5,
+    views:    40.7,
+    db:       59.8,
+    queries:  4,
+    status:   200
+  )
 ```
-or
 
 ```ruby
   # Store the metrics for 10 minutes
-  EZmetrics.new(10.minutes).log(duration: 100.5, views: 40.7, db: 59.8, queries: 4, status: 200)
+  EZmetrics.new(10.minutes).log(
+    duration: 100.5,
+    views:    40.7,
+    db:       59.8,
+    queries:  4,
+    status:   200
+  )
 ```
 
-For displaying metrics you need call `show` method:
+---
+
+For displaying metrics you need to call `show` method:
 
 ```ruby
   # Aggregate and show metrics for last 60 seconds (default behaviour)
   EZmetrics.new.show
 ```
 
-or
-
 ```ruby
   # Aggregate and show metrics for last 10 minutes
   EZmetrics.new(10.minutes).show
 ```
 
-> Please note that you can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
+You can combine these timeframes, for example - store for 10 minutes, display for 5 minutes.
 
+### Capture metrics
 
-### Add an initializer to your Rails application
+Just add an initializer to your application:
 
 ```ruby
 # config/initializers/ezmetrics.rb
@@ -81,6 +93,8 @@ end
 
 ### Display metrics
 
+As simple as:
+
 ```ruby
 EZmetrics.new.show
 ```
@@ -117,3 +131,179 @@ This will return a hash with the following structure:
 }
 ```
 
+### Aggregation
+
+The aggregation can be easily configured by specifying aggregation options as in the following examples:
+
+**1. Single**
+
+```ruby
+EZmetrics.new.show(duration: :max)
+```
+
+```ruby
+{
+  duration: {
+    max: 9675
+  }
+}
+```
+
+---
+
+**2. Multiple**
+
+```ruby
+EZmetrics.new.show(queries: [:max, :avg])
+```
+
+```ruby
+{
+  queries: {
+    max: 76,
+    avg: 26
+  }
+}
+```
+
+---
+
+**3. Requests**
+
+```ruby
+EZmetrics.new.show(requests: true)
+```
+
+```ruby
+{
+  requests: {
+    all: 2000,
+    grouped: {
+      "2xx" => 1900,
+      "3xx" => 15,
+      "4xx" => 80,
+      "5xx" => 5
+    }
+  }
+}
+```
+
+---
+
+**4. Combined**
+
+```ruby
+EZmetrics.new.show(views: :avg, :db: [:avg, :max], requests: true)
+```
+
+```ruby
+{
+  views: {
+    avg: 12
+  },
+  db: {
+    avg: 155,
+    max: 4382
+  },
+  requests: {
+    all: 2000,
+    grouped: {
+      "2xx" => 1900,
+      "3xx" => 15,
+      "4xx" => 80,
+      "5xx" => 5
+    }
+  }
+}
+```
+
+### Partitioning
+
+If you want to visualize your metrics by using a **line chart**, you will need to use partitioning.
+
+To aggregate metrics, partitioned by a unit of time you need to call `partition_by({time_unit})` before calling `show`
+
+```ruby
+  # Aggregate metrics for last hour, partition by minute
+  EZmetrics.new(1.hour).partition_by(:minute).show(duration: [:avg, :max], db: :avg)
+```
+
+This will return an array of objects with the following structure:
+
+```ruby
+[
+  {
+    timestamp: # UNIX timestamp
+    data:      # a hash with aggregated metrics
+  }
+]
+```
+
+like in the example below:
+
+```ruby
+[
+  {
+    timestamp: 1575242880,
+    data: {
+      duration: {
+        avg: 477,
+        max: 8566
+      },
+      db: {
+        avg: 387
+      }
+    }
+  },
+  {
+    timestamp: 1575242940,
+    data: {
+      duration: {
+        avg: 234,
+        max: 3675
+      },
+      db: {
+        avg: 123
+      }
+    }
+  }
+]
+```
+
+Available time units for partitioning: `second`, `minute`, `hour`, `day`. Default: `minute`.
+
+### Performance
+
+The aggregation speed relies on the performance of **Redis** (data storage) and **Oj** (json serialization/parsing).
+
+You can check the **aggregation** time by running:
+
+```ruby
+EZmetrics::Benchmark.new.measure_aggregation
+```
+
+| Interval | Duration (seconds) |
+| :------: | :----------------: |
+| 1 minute |        0.0         |
+|  1 hour  |        0.04        |
+| 12 hours |        0.49        |
+| 24 hours |        1.51        |
+| 48 hours |        3.48        |
+
+---
+
+To check the **partitioned aggregation** time you need to run:
+
+```ruby
+EZmetrics::Benchmark.new.measure_aggregation(:minute)
+```
+
+| Interval | Duration (seconds) |
+| :------: | :----------------: |
+| 1 minute |        0.0         |
+|  1 hour  |        0.04        |
+| 12 hours |        0.53        |
+| 24 hours |        1.59        |
+| 48 hours |        3.51        |
+
+The benchmarks above were run on a _2017 Macbook Pro 2.9 GHz Intel Core i7 with 16 GB of RAM_

data/lib/ezmetrics.rb

@@ -1,11 +1,15 @@
-require "redis" unless defined?(Redis)
-require "json"  unless defined?(JSON)
+require "redis"
+require "redis/connection/hiredis"
+require "oj"
 
 class EZmetrics
+  METRICS               = [:duration, :views, :db, :queries].freeze
+  AGGREGATION_FUNCTIONS = [:max, :avg].freeze
+  PARTITION_UNITS       = [:second, :minute, :hour, :day].freeze
+
   def initialize(interval_seconds=60)
     @interval_seconds = interval_seconds.to_i
     @redis            = Redis.new
-    @storage_key      = "ez-metrics"
   end
 
   def log(payload={duration: 0.0, views: 0.0, db: 0.0, queries: 0, status: 200})
@@ -19,12 +23,12 @@ class EZmetrics
 
     this_second          = Time.now.to_i
     status_group         = "#{payload[:status].to_s[0]}xx"
-    @this_second_metrics = redis.get("#{storage_key}:#{this_second}")
+    @this_second_metrics = redis.get(this_second)
 
     if this_second_metrics
-      @this_second_metrics = JSON.parse(this_second_metrics)
+      @this_second_metrics = Oj.load(this_second_metrics)
 
-      [:duration, :views, :db, :queries].each do |metrics_type|
+      METRICS.each do |metrics_type|
         update_sum(metrics_type)
         update_max(metrics_type)
       end
@@ -33,6 +37,7 @@ class EZmetrics
       this_second_metrics["statuses"][status_group] += 1
     else
       @this_second_metrics = {
+        "second"       => this_second,
         "duration_sum" => safe_payload[:duration],
         "duration_max" => safe_payload[:duration],
         "views_sum"    => safe_payload[:views],
@@ -47,110 +52,132 @@ class EZmetrics
       this_second_metrics["statuses"][status_group] = 1
     end
 
-    redis.setex("#{storage_key}:#{this_second}", interval_seconds, JSON.generate(this_second_metrics))
-
+    redis.setex(this_second, interval_seconds, Oj.dump(this_second_metrics))
     true
   rescue => error
     formatted_error(error)
   end
 
-  def show
-    interval_start    = Time.now.to_i - interval_seconds
-    interval_keys     = (interval_start..Time.now.to_i).to_a.map { |second| "#{storage_key}:#{second}" }
-    @interval_metrics = redis.mget(interval_keys).compact.map { |hash| JSON.parse(hash) }
+  def show(options=nil)
+    @options = options || default_options
+    partitioned_metrics ? aggregate_partitioned_data : aggregate_data
+  end
 
-    return empty_metrics_object unless interval_metrics.any?
+  def partition_by(time_unit=:minute)
+    time_unit = PARTITION_UNITS.include?(time_unit) ? time_unit : :minute
+    @partitioned_metrics = interval_metrics.group_by { |h| second_to_partition_unit(time_unit, h["second"]) }
+    self
+  end
 
-    @requests = interval_metrics.map { |hash| hash["statuses"]["all"] }.compact.sum
+  private
 
-    metrics_object
+  attr_reader :redis, :interval_seconds, :interval_metrics, :requests,
+    :storage_key, :safe_payload, :this_second_metrics, :partitioned_metrics, :options
+
+  def aggregate_data
+    return {} unless interval_metrics.any?
+    @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
+    build_result
   rescue
-    empty_metrics_object
+    {}
   end
 
-  private
+  def aggregate_partitioned_data
+    partitioned_metrics.map do |partition, metrics|
+      @interval_metrics = metrics
+      @requests = interval_metrics.sum { |hash| hash["statuses"]["all"] }
+      { timestamp: partition, data: build_result }
+    end
+  rescue
+    new(options)
+  end
+
+  def build_result
+    result = {}
+
+    result[:requests] = { all: requests, grouped: count_all_status_groups } if options[:requests]
+
+    options.each do |metrics, aggregation_functions|
+      next unless METRICS.include?(metrics)
+      aggregation_functions = [aggregation_functions] unless aggregation_functions.is_a?(Array)
+      next unless aggregation_functions.any?
+
+      aggregation_functions.each do |aggregation_function|
+        result[metrics] ||= {}
+        result[metrics][aggregation_function] = aggregate(metrics, aggregation_function)
+      end
+    end
+    result
+  ensure
+    result
+  end
+
+  def second_to_partition_unit(time_unit, second)
+    return second if time_unit == :second
+    time = Time.at(second)
+    return (time - time.sec - time.min * 60 - time.hour * 3600).to_i if time_unit == :day
+    return (time - time.sec - time.min * 60).to_i                    if time_unit == :hour
+    (time - time.sec).to_i
+  end
 
-  attr_reader :redis, :interval_seconds, :interval_metrics, :requests, :storage_key,
-    :safe_payload, :this_second_metrics
+  def interval_metrics
+    @interval_metrics ||= begin
+      interval_start    = Time.now.to_i - interval_seconds
+      interval_keys     = (interval_start..Time.now.to_i).to_a
+      redis.mget(interval_keys).compact.map { |hash| Oj.load(hash) }
+    end
+  end
+
+  def aggregate(metrics, aggregation_function)
+    return unless AGGREGATION_FUNCTIONS.include?(aggregation_function)
+    return avg("#{metrics}_sum") if aggregation_function == :avg
+    return max("#{metrics}_max") if aggregation_function == :max
+  end
 
   def update_sum(metrics)
-    this_second_metrics["#{metrics}_sum"] += safe_payload[metrics.to_sym]
+    this_second_metrics["#{metrics}_sum"] += safe_payload[metrics]
   end
 
   def update_max(metrics)
-    max_value = [safe_payload[metrics.to_sym], this_second_metrics["#{metrics}_max"]].max
+    max_value = [safe_payload[metrics], this_second_metrics["#{metrics}_max"]].max
     this_second_metrics["#{metrics}_max"] = max_value
   end
 
   def avg(metrics)
-    (interval_metrics.map { |h| h[metrics.to_s] }.sum.to_f / requests).round
+    (interval_metrics.sum { |h| h[metrics] }.to_f / requests).round
   end
 
   def max(metrics)
-    interval_metrics.map { |h| h[metrics.to_s] }.max.round
+    interval_metrics.max { |h| h[metrics] }[metrics].round
   end
 
-  def count(group)
-    interval_metrics.map { |h| h["statuses"][group.to_s] }.sum
+  def count_all_status_groups
+    interval_metrics.inject({ "2xx" => 0, "3xx" => 0, "4xx" => 0, "5xx" => 0 }) do |result, h|
+      result["2xx"] += h["statuses"]["2xx"]
+      result["3xx"] += h["statuses"]["3xx"]
+      result["4xx"] += h["statuses"]["4xx"]
+      result["5xx"] += h["statuses"]["5xx"]
+      result
+    end
   end
 
-  def formatted_error(error)
+  def default_options
     {
-      error:     error.class.name,
-      message:   error.message,
-      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
+      duration: AGGREGATION_FUNCTIONS,
+      views:    AGGREGATION_FUNCTIONS,
+      db:       AGGREGATION_FUNCTIONS,
+      queries:  AGGREGATION_FUNCTIONS,
+      requests: true
     }
   end
 
-  def metrics_object
+  def formatted_error(error)
     {
-      duration: {
-        avg: avg(:duration_sum),
-        max: max(:duration_max)
-      },
-      views: {
-        avg: avg(:views_sum),
-        max: max(:views_max)
-      },
-      db: {
-        avg: avg(:db_sum),
-        max: max(:db_max)
-      },
-      queries: {
-        avg: avg(:queries_sum),
-        max: max(:queries_max)
-      },
-      requests: {
-        all:     requests,
-        grouped: {
-          "2xx" => count("2xx"),
-          "3xx" => count("3xx"),
-          "4xx" => count("4xx"),
-          "5xx" => count("5xx")
-        }
-      }
+      error:     error.class.name,
+      message:   error.message,
+      backtrace: error.backtrace.reject { |line| line.match(/ruby|gems/) }
     }
   end
+end
 
-  def empty_metrics_object
-    {
-      duration: {
-        avg: 0,
-        max: 0
-      },
-      views: {
-        avg: 0,
-        max: 0
-      },
-      db: {
-        avg: 0,
-        max: 0
-      },
-      queries: {
-        avg: 0,
-        max: 0
-      },
-      requests: {}
-    }
-  end
-end
+require "ezmetrics/benchmark"

data/lib/ezmetrics/benchmark.rb

@@ -0,0 +1,95 @@
+require "benchmark"
+
+class EZmetrics::Benchmark
+
+  def initialize
+    @start      = Time.now.to_i
+    @redis      = Redis.new
+    @durations  = []
+    @iterations = 3
+    @intervals  = {
+      "1.minute" => 60,
+      "1.hour  " => 3600,
+      "12.hours" => 43200,
+      "24.hours" => 86400,
+      "48.hours" => 172800
+    }
+  end
+
+  def measure_aggregation(partition_by=nil)
+    write_metrics
+    print_header
+    intervals.each do |interval, seconds|
+      result = measure_aggregation_time(interval, seconds, partition_by)
+      print_row(result)
+    end
+    cleanup_metrics
+    print_footer
+  end
+
+  private
+
+  attr_reader :start, :redis, :durations, :intervals, :iterations
+
+  def write_metrics
+    seconds = intervals.values.max
+    seconds.times do |i|
+      second = start - i
+      payload = {
+        "second"          => second,
+        "duration_sum"    => rand(10000),
+        "duration_max"    => rand(10000),
+        "views_sum"       => rand(1000),
+        "views_max"       => rand(1000),
+        "db_sum"          => rand(8000),
+        "db_max"          => rand(8000),
+        "queries_sum"     => rand(100),
+        "queries_max"     => rand(100),
+        "statuses"        => {
+          "2xx" => rand(1..10),
+          "3xx" => rand(1..10),
+          "4xx" => rand(1..10),
+          "5xx" => rand(1..10),
+          "all" => rand(1..40)
+        }
+      }
+      redis.setex(second, seconds, Oj.dump(payload))
+    end
+    nil
+  end
+
+  def cleanup_metrics
+    interval_start = Time.now.to_i - intervals.values.max - 100
+    interval_keys  = (interval_start..Time.now.to_i).to_a
+    redis.del(interval_keys)
+  end
+
+  def measure_aggregation_time(interval, seconds, partition_by)
+    iterations.times do
+      durations << ::Benchmark.measure do
+        if partition_by
+          EZmetrics.new(seconds).partition_by(partition_by).show
+        else
+          EZmetrics.new(seconds).show
+        end
+      end.real
+    end
+
+    return {
+      interval: interval.gsub(".", " "),
+      duration: (durations.sum.to_f / iterations).round(2)
+    }
+  end
+
+  def print_header
+    print "\n#{'─'*31}\n| Interval | Duration (seconds)\n#{'─'*31}\n"
+  end
+
+  def print_row(result)
+    print "| #{result[:interval]} | #{result[:duration]}\n"
+  end
+
+  def print_footer
+    print "#{'─'*31}\n"
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ezmetrics
 version: !ruby/object:Gem::Version
-  version: 1.0.5
+  version: 1.2.1
 platform: ruby
 authors:
 - Nicolae Rotaru
@@ -24,6 +24,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: hiredis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.6.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.6.3
+- !ruby/object:Gem::Dependency
+  name: oj
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -38,7 +66,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.5'
-description: A simple tool for capturing and displaying Rails metrics.
+description: Simple, lightweight and fast metrics aggregation for Rails.
 email: nyku.rn@gmail.com
 executables: []
 extensions: []
@@ -47,6 +75,7 @@ files:
 - LICENSE
 - README.md
 - lib/ezmetrics.rb
+- lib/ezmetrics/benchmark.rb
 homepage: https://github.com/nyku/ezmetrics
 licenses:
 - GPL-3.0
@@ -66,8 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 3.0.6
 signing_key: 
 specification_version: 4
 summary: Rails metrics aggregation tool.