sensu-plugins-influxdb-metrics-checker 0.3.4 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 86fe26cfb281849f4b73356b47011b08bf1a7960
-  data.tar.gz: dca6df28cc658d7faff6274090ecb5b050b5654a
+  metadata.gz: ed89e4ad1b835c997279a3afc967efc91e6579fa
+  data.tar.gz: 5813fe0fa863a0e278620862bad7aeeae75651be
 SHA512:
-  metadata.gz: 861427b6a1b16058eb9bfc51c0186eed75d765942325762932933ec8d67efb81f0192c2804be9c4c443acd73545ce8a134ae6f5e96de219504a23ebda7b32524
-  data.tar.gz: 74d9af1f5b6de576c991bcece54f24041ca43d2b7128c60e7c5f7f825b4ad90eb6e1dfb49e83f711b3130c95d6c799328b35c694aab67cd7319593a2e82d1f3e
+  metadata.gz: b8638d8b0b9a343b0233a5e39e2425cb5c238c58a97124d81452dc0da5478ae7b3886393b507b4be6a8ac4aa8680fe6faf02e41bd03e955fb96f6724dbb6e139
+  data.tar.gz: 0cacd9d9ee2257904253403682edb271ed8f7431ca6b4bdeb0d17d3d847baae81c682e57647f9d0b616eb89a60de9d1fc2382becea3df2c96ec5bd710486d9b4

data/CHANGELOG.md

@@ -3,6 +3,11 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 This CHANGELOG follows the format listed at [Keep A Changelog](http://keepachangelog.com/)
 
+# [0.4.0] - 2017-01-20
+- seventh release
+New feature: Triangulation. Added the ability to get percentage of metric A, get percentage of metric B, and compare the distance between them. Useful when the metrics are related together by some business rule.
+Improved feedback when returning to customer.
+
 # [0.3.4] - 2017-01-17
 - sixth release
 Allow the usage of regex expressions that we can identify as "/^[your_regex]$". I'll strongly recommend to use this only for exceptions, and always aim for zero-exceptions, or it wouldn't be accurate. At the moment it will fire when the number of exception today is bigger than the number of exceptions yesterday.

data/README.md

@@ -11,10 +11,10 @@ We chose to do it as a Sensu plugin because it comes with Handlers that will all
 The result is that now we are able to experiment with our metrics and alerts, giving us a better understanding of whats going on in our systems.
 
 ## What it does
-The script will compare the values of yesterday at this time minus 10 minutes, with the values of today at this time minus 10 minus.
+The script will compare the values of yesterday at this time minus 25 minutes, with the values of today at this time minus 25 minus.
 It will calculate the percentage of difference and will act on that.
 You will be able to set a threshold of warning and critical values where your program will act.
-It will also leave it 5 minutes to aggregate the data in influxdb, so we are more precise.
+It will also leave it 10 minutes to aggregate the data in influxdb, so we are more precise.
 
 ## Components
 There is just one script that you can find at
@@ -22,8 +22,7 @@ There is just one script that you can find at
 
 ## Getting started
 
-At the moment there is just one script
-**check-influxdb-metrics** which you can run in a bash doing:
+Once we go to **check-influxdb-metrics** you can run it in a bash doing:
 
 ```
 ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086 --user=admin --password=password -c -3 -w -10 --db=statsd_metrics --metric=api.request.counter
@@ -48,9 +47,13 @@ ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086
 
 ## Advanced Queries
 
-You can use Regex in your metrics. The spirit behind this feature is to gather information about exceptions only, always aiming for a zero exception policy. So I'll advise against using it for other purposes.
+**Regex**
+
+You can use it in your metrics. The spirit behind this feature is to gather information about exceptions only, beware that this could gather all your metrics inside your influxdb cluster, which may produce some unintended pain, so always aiming for querying exceptions, and ideally a zero exception policy.
+I'll strongly advise against using it for other purposes.
 
 **How it works**
+
 1. It will understand that is a regex only when the metric name contains '/'. In the bash you'll need to include your metric inside double quotes.
 2. It will compare the number of metrics gathered today vs the number of metrics gathered yesterday.
 3. If today we read more than yesterday, it will blow up as **Critical**.
@@ -61,6 +64,38 @@ ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086
 
 ```
 
+**Triangulation**
+
+In trigonometry and geometry, triangulation is the process of determining the location of a point by forming triangles to it from known points. This feature of the script is inspired in that idea.
+
+
+[![triangulation_01.png](https://s24.postimg.org/kjihvilvp/triangulation_01.png (2KB))](https://postimg.org/image/hcnybw1fl/)
+
+Once we have a given metric A (ex: messages.sent), we'll normally compare that to yesterday's weather A', we'll get the percentage of difference and according to our threshold we'll fire an alert. Cool. Now let's go one step further.
+We may have a metric B (ex: sessions.generated), that has a business dependency on B. And if we dig further in our metrics, we may even relate that to, let's say, an average of 5 metrics A for each metric B. (In this example, you'll need 5 messages sent to build a session).
+
+If we could say that every 5 As relates to 1 B, then the % of difference for A and B will always be the same. Realistically, it's not always like that in production applications, sometimes you may need 7 messages, others only 4, so your average would be something around 5.333. Therefore, we can't say that the % in difference will always be the same, but once we look at the *distance* between these percentages, we'll see that they are pretty close. And that's the spirit of this feature, the ability to diagnose when the distance is higher than expected.
+
+Let's say that the system that sends items has an increase of 150%, and you are using this tool to verify that, therefore you don't get any exceptions because there is no drop in the metrics, but the system that process sessions keeps in the same 2% increase, which is a big distance up to 148. We clearly have a problem here. Maybe some bottleneck is happening somewhere, maybe some messages are lost due to this huge increase, and hopefully this feature will allow you to identify that something fussy is going on.
+
+**How it works**
+
+This query will get the distance between "messages.counter" % vs "sessions.generated" %. By default it's set to fire an alert if that turns out to be bigger than 2.
+
+```
+ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086 --user=admin --password=password -c -30 -w -10 --db=statsd_metrics --metric=messages.counter --triangulate=sessions.generated
+```
+If you want to increase the distance, you will just need --distance.
+
+```
+ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086 --user=admin --password=password -c -30 -w -10 --db=statsd_metrics --metric=messages.counter --triangulate=sessions.generated --distance=10
+```
+
+If you want to apply some tags and filters you can do it as you'll do normally, just bear in mind that by default they will not apply to both metrics, only tot he first one. If you want to apply them to the second metric you'll just need to add --applyfilterbothqueries=yes
+
+```
+ruby check-influxdb-metrics.rb --host=metrics-influxdb.internal.com --port=8086 --user=admin --password=password -c -30 -w -10 --db=statsd_metrics --metric=messages.counter --tag=datacenter --filter=pro-westeurope --triangulate=sessions.generated --distance=10 --applyfilterbothqueries=yes
+```
 
 ## Lessons learnt
 The InfluxDb query language that we used is not the latest, you can find it here:
@@ -72,6 +107,11 @@ What matters for this program is that:
 ```
 -24h will turn into bad request
 - 24h good
+
+- session.certified will turn into a bad request
+- "session.certified" is good. Notice that when you use grafana or influx db console you don't need the quotes
+ but when you query through the script you'll need them.
+ When using regex both with and without quotes will work, because what matters is `/^[metric]$/`
 ```
 
 **When passing parameters**

data/bin/check-influxdb-metrics.rb

@@ -83,22 +83,39 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
          long: '--period=VALUE',
          description: 'Filter by a given day period in minutes',
          proc: proc { |l| l.to_i },
-         default: 10
+         default: 25
 
-  def filter_by_environment_when_needed
-    config[:tag].nil? && config[:filter].nil? ? '' : " AND \"#{config[:tag]}\" =~ /#{config[:filter]}/"
+  option :triangulate,
+         long: '--triangulate=VALUE',
+         description: 'Triangulate with this metric'
+
+  option :applyfilterbothqueries,
+         long: '--applyfilterbothqueries=VALUE',
+         description: 'Set if you want to apply tag and filter also for the query that you are about to triangulate with'
+
+  option :distance,
+         long: '--distance=VALUE',
+         description: 'Set the distance threshold to alert in case of triangulation',
+         default: 2
+
+  BASE_QUERY = 'SELECT sum("value") from '.freeze
+  TODAY_START_PERIOD = 10
+  YESTERDAY_START_PERIOD = 1455 # starts counting 1455 minutes before now() [ yesetrday - 10 minutes] to match with today_query_for_a_period start_period
+
+  def yesterday_end_period
+    config[:period] + YESTERDAY_START_PERIOD
   end
 
-  def base_query
-    'SELECT sum("value") from '
+  def today_end_period
+    config[:period] + TODAY_START_PERIOD
   end
 
-  def base_query_with_metricname
-    base_query + clean_quotes_when_regex
+  def base_query_with_metricname(metric)
+    BASE_QUERY + clean_quotes_when_regex(metric)
   end
 
-  def clean_quotes_when_regex
-    metric = " \"#{config[:metric]}\""
+  def clean_quotes_when_regex(metric_to_clean)
+    metric = ' "' + metric_to_clean + '"'
     clean_metric = ''
     if metric.include?('/')
       clean_metric = metric.tr '\"', ''
@@ -110,23 +127,36 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
     clean_metric
   end
 
-  def today_query_for_a_period
-    start_period = 5; # starts counting 5 minutes before now() to let influxdb time to aggregate the data
-    end_period = config[:period] + 5; # adds 5 minutes to match with start_period
-    query = query_for_a_period(start_period, end_period)
-    query + filter_by_environment_when_needed
+  def filter_by_environment_when_needed
+    config[:tag].nil? && config[:filter].nil? ? '' : " AND \"#{config[:tag]}\" =~ /#{config[:filter]}/"
+  end
+
+  def filter_for_triangulate_when_needed
+    config[:applyfilterbothqueries].nil? ? '' : " AND \"#{config[:tag]}\" =~ /#{config[:filter]}/"
+  end
+
+  def query_for_a_period(metric, start_period, end_period, istriangulated)
+    query = base_query_with_metricname(metric) + ' WHERE time > now() - ' + end_period.to_s + 'm AND time < now() - ' + start_period.to_s + 'm'
+    query + add_filter_when_needed(istriangulated)
+  end
+
+  def add_filter_when_needed(istriangulated)
+    if istriangulated == true
+      filter_for_triangulate_when_needed
+    else
+      filter_by_environment_when_needed
+    end
   end
 
-  def yesterday_query_for_a_period
-    start_period = 1445; # starts counting 1445 minutes before now() [ yesetrday - 5 minutes] to match with today_query_for_a_period start_period
-    end_period = config[:period] + 1445; # adds 1445 minutes to match with start_period
-    query = query_for_a_period(start_period, end_period)
-    query + filter_by_environment_when_needed
+  def query_encoded_for_a_period(metric, start_period, end_period, istriangulated)
+    query = query_for_a_period(metric, start_period, end_period, istriangulated)
+    encode_parameters(query)
   end
 
-  def query_for_a_period(start_period, end_period)
-    query = base_query_with_metricname + ' WHERE time > now() - ' + end_period.to_s + 'm AND time < now() - ' + start_period.to_s + 'm'
-    query + filter_by_environment_when_needed
+  def metrics(metric, start_period, end_period, istriangulated)
+    query = query_encoded_for_a_period(metric, start_period, end_period, istriangulated)
+    response = request(query)
+    parse_json(response)
   end
 
   def encode_parameters(parameters)
@@ -141,46 +171,48 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
     "#{config[:db]}&q=" + encode_for_regex
   end
 
-  def yesterday_query_encoded
-    query = yesterday_query_for_a_period
-    encode_parameters(query)
-  end
-
-  def today_query_encoded
-    query = today_query_for_a_period
-    encode_parameters(query)
-  end
-
-  def today_value
-    response = request(today_query_encoded)
-    metrics = parse_json(response)
-    @today_metric_count = validate_metrics_and_count(metrics)
+  def today_metrics
+    today_info = metrics(config[:metric], TODAY_START_PERIOD, today_end_period, false)
+    @today_metric_count = validate_metrics_and_count(today_info)
     value = if @today_metric_count > 0
-              series = read_series_from_metrics(metrics)
+              series = read_series_from_metrics(today_info)
               @today_metrics = store_metrics(series)
               read_value_from_series(series)
             end
     value
   end
 
-  def yesterday_value
-    response = request(yesterday_query_encoded)
-    metrics = parse_json(response)
-    @yesterday_metric_count = validate_metrics_and_count(metrics)
-    value = if @today_metric_count > 0
-              series = read_series_from_metrics(metrics)
+  def yesterday_metrics
+    yesterday_info = metrics(config[:metric], YESTERDAY_START_PERIOD, yesterday_end_period, false)
+    @yesterday_metric_count = validate_metrics_and_count(yesterday_info)
+    value = if @yesterday_metric_count > 0
+              series = read_series_from_metrics(yesterday_info)
               @yesterday_metrics = store_metrics(series)
               read_value_from_series(series)
             end
     value
   end
 
-  def metric_bigger_than_zero?(metric)
-    metric > 0
+  def today_triangulated_metrics
+    today_triangulated_info = metrics(config[:triangulate], TODAY_START_PERIOD, today_end_period, true)
+    @today_triangulated_metric_count = validate_metrics_and_count(today_triangulated_info)
+    value = if @today_triangulated_metric_count > 0
+              series = read_series_from_metrics(today_triangulated_info)
+              @today_triangulated_metrics = store_metrics(series)
+              read_value_from_series(series)
+            end
+    value
   end
 
-  def using_regex?(using_regex)
-    using_regex == true
+  def yesterday_triangulated_metrics
+    yesterday_triangulated_info = metrics(config[:triangulate], YESTERDAY_START_PERIOD, yesterday_end_period, true)
+    @yesterday_triangulated_metric_count = validate_metrics_and_count(yesterday_triangulated_info)
+    value = if @yesterday_triangulated_metric_count > 0
+              series = read_series_from_metrics(yesterday_triangulated_info)
+              @yesterday_triangulated_metrics = store_metrics(series)
+              read_value_from_series(series)
+            end
+    value
   end
 
   def read_series_from_metrics(metrics)
@@ -234,14 +266,19 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
       ok 'no metrics found'
     elsif @today_metric_count > @yesterday_metric_count
       display_metrics
-      critical "For \"#{config[:metric]}\" more metrics were tracked today than yesterday. Check them out above"
+      critical 'For ' + config[:metric] + ' more metrics tracked today (' + @today_metric_count + ') than yesterday (' + @yesterday_metric_count + ') See above'
     elsif @today_metric_count == @yesterday_metric_count
       compare_each_metric_in_regex
     else
-      ok 'regex seems ok ' + @today_metric_count.to_s + ' metrics found today vs ' + @yesterday_metric_count.to_s + ' metrics found yesterday'
+      ok 'regex seems ok! Today metrics dropped. Yesterday (' + @yesterday_metric_count.to_s + ') vs (' + @today_metric_count.to_s + ') found today.'
     end
   end
 
+  def difference_for_standard_queries(today, yesterday)
+    difference = difference_between_two_metrics(today, yesterday)
+    evaluate_percentage_and_notify(difference)
+  end
+
   def compare_each_metric_in_regex
     @today_metrics.each do |today_key, today_value|
       @yesterday_metrics.each do |yesterday_key, yesterday_value|
@@ -271,7 +308,7 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
   end
 
   def evaluate_percentage_and_notify(difference)
-    puts 'Difference of: ' + difference.round(4).to_s + ' %  for a period of ' + config[:period].to_s + 'm'
+    puts 'Difference of: ' + difference.round(3).to_s + ' %  for a period of ' + config[:period].to_s + 'm'
     if difference < config[:crit]
       critical "\"#{config[:metric]}\" difference is below allowed minimum of #{config[:crit]} %"
     elsif difference < config[:warn]
@@ -281,33 +318,84 @@ class CheckInfluxDbMetrics < Sensu::Plugin::Check::CLI
     end
   end
 
+  def evaluate_distance_and_notify(distance)
+    if distance > config[:distance].to_f
+      critical config[:metric] + ' vs ' + config[:triangulate] + ' distance is greater than allowed minimum of ' + config[:distance]
+    else
+      ok 'distance ok'
+    end
+  end
+
   def calculate_difference_and_display_result(today, yesterday)
-    difference = if @is_using_regex
-                   difference_for_regex_and_notify
-                 else
-                   difference_for_standard_queries(today, yesterday)
-                 end
-    difference
+    if @is_using_regex
+      difference_for_regex_and_notify
+    else
+      difference_between_two_metrics(today, yesterday)
+    end
   end
 
-  def difference_for_standard_queries(today, yesterday)
-    difference = difference_between_two_metrics(today, yesterday)
-    evaluate_percentage_and_notify(difference)
+  def difference_between_percentages_of_two_metrics
+    validate_base_metrics
+    validate_triangulated_metrics
+    base = difference_between_two_metrics(today_metrics, yesterday_metrics)
+    triangulated = difference_between_two_metrics(today_triangulated_metrics, yesterday_triangulated_metrics)
+    puts 'difference for ' + config[:metric] + ' ' + base.round(3).to_s + '% vs ' + config[:triangulate] + ' ' + triangulated.round(3).to_s + '%'
+    distance = distance_between_two_numbers(base, triangulated)
+    evaluate_distance_and_notify(distance)
+  end
+
+  def distance_between_two_numbers(a, b)
+    (a - b).abs
+  end
+
+  def validate_triangulated_metrics
+    today = today_triangulated_metrics
+    yesterday = yesterday_triangulated_metrics
+    if today.nil? && yesterday.nil?
+      puts 'No metrics found to triangulate'
+      exit
+    else
+      0
+    end
+  end
+
+  def validate_base_metrics
+    today = today_metrics
+    yesterday = yesterday_metrics
+    if today.nil? && yesterday.nil?
+      puts 'No metrics found in base to triangulate'
+      exit
+    else
+      0
+    end
   end
 
-  def difference_in_metrics
-    today = today_value
-    yesterday = yesterday_value
+  def difference_between_metrics
+    today = today_metrics
+    yesterday = yesterday_metrics
     if today.nil? && yesterday.nil?
       puts 'No results coming from InfluxDB either for Today nor Yesterday. Please check your query or try again'
     else
-      calculate_difference_and_display_result(today, yesterday)
+      difference = calculate_difference_and_display_result(today, yesterday)
+      evaluate_percentage_and_notify(difference)
     end
     exit
   end
 
+  def triangulation?
+    config[:triangulate].nil?
+  end
+
+  def check_metrics_in_influxdb
+    if triangulation?
+      difference_between_metrics
+    else
+      difference_between_percentages_of_two_metrics
+    end
+  end
+
   def run
-    difference_in_metrics
+    check_metrics_in_influxdb
 
   rescue Errno::ECONNREFUSED => e
     critical 'InfluxDB is not responding' + e.message

data/lib/sensu-plugins-influxdb-metrics-checker/version.rb

@@ -1,8 +1,8 @@
 module SensuPluginsInfluxDbMetricsChecker
   module Version
     MAJOR = 0
-    MINOR = 3
-    PATCH = 4
+    MINOR = 4
+    PATCH = 0
 
     VER_STRING = [MAJOR, MINOR, PATCH].compact.join('.')
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sensu-plugins-influxdb-metrics-checker
 version: !ruby/object:Gem::Version
-  version: 0.3.4
+  version: 0.4.0
 platform: ruby
 authors:
 - Juanjo Guerrero Cerezuela
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-01-17 00:00:00.000000000 Z
+date: 2017-01-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sensu-plugin