kennel 1.72.1 → 1.75.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9dec3dd3c3595176bc3340b7fcfd0227cca53a09ab56db9911da90546f8cc125
-  data.tar.gz: '019ee1e8e9c7e37d3218f77790ab5cc5ef08504fd0a603ee710a4623e36f72fa'
+  metadata.gz: e6a61329e4c2b2ccec0021103dbca60ec7fb9658e3a45a0c4212a08e63ea1395
+  data.tar.gz: 50af562a677393894101f495b150be434f7af67851e774109aa6b4605bedffab
 SHA512:
-  metadata.gz: a49dda9dcff0d0a8d330a68f2c9a6e69fb20624067bc24b9fae57c3e7f9cfe94ea9f3cb412acc456a4743ca5b61643a7494e5599833d296e92893372ee39be9a
-  data.tar.gz: 11a86599302eb683e8b7f37df641af3d0965ac52ace5ee00375d2a3644cc35e0b4844c31f1cdd71a98a9eec5e91002a7b20d23a611dd51ce902de5c93df462cf
+  metadata.gz: c884d5d811aa4ed5df99e90382d2311391035570185149bae41f0ac0c080df6132877c4dcd9ffbba88a83e829203557353b405ed71b373e997337223192ed5b4
+  data.tar.gz: 4e453ebad3dae75cd38f1901ab6ea50d22f886d2b3004ff3d25c3ab64881f852d7592b73a4efd0eabe244179ee39570f0028a62f3a1aa1bb0c47765c131fb097

data/Readme.md

@@ -1,17 +1,58 @@
-# Kennel
-
 ![](template/github/cage.jpg?raw=true)
 
-Manage datadog monitors/dashboards/slos as code
+Manage Datadog Monitors / Dashboards / Slos as code
 
- - Documented, reusable, and searchable
- - Changes are PR reviewed and auditable
+ - DRY, searchable, audited, documented
+ - Changes are PR reviewed and applied on merge
  - Updating shows diff before applying
- - Automated import of existing monitors/dashboards/slos
+ - Automated import of existing resources
+ - Resources are grouped into projects that belong to teams and inherit tags
+ - No copy-pasting of ids to create new resources
+ - Automated cleanup when removing code
+ - [Helpers](#helpers) for automating common tasks
+ 
+### Applying changes
 
 ![](template/github/screen.png?raw=true)
+
+### Example code
+
+```Ruby
+# teams/foo.rb
+module Teams
+  class Foo < Kennel::Models::Team
+    defaults(mention: -> { "@slack-my-team" })
+  end
+end
+
+# projects/bar.rb
+class Bar < Kennel::Models::Project
+  defaults(
+    team: -> { Teams::Foo.new }, # use mention and tags from the team
+    parts: -> {
+      [
+        Kennel::Models::Monitor.new(
+          self, # the current project
+          type: -> { "query alert" },
+          kennel_id: -> { "load-too-high" }, # pick a unique name
+          name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
+          message: -> {
+            <<~TEXT
+              This is bad!
+              #{super()} # inserts mention from team
+            TEXT
+          },
+          query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
+          critical: -> { 20 }
+        )
+      ]
+    }
+  )
+end
+```
+
 <!-- NOT IN template/Readme.md  -->
-## Install
+## Installation
 
  - create a new private `kennel` repo for your organization (do not fork this repo)
  - use the template folder as starting point:
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
     cd kennel && git add . && git commit -m 'initial'
     ```
  - add a basic projects and teams so others can copy-paste to get started
- - setup travis build for your repo
- - uncomment `.travis.yml` section for automated PR planing and datadog updates on merge
+ - setup CI build for your repo (travis and Github Actions supported)
+ - uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)
  - follow `Setup` in your repos Readme.md
 <!-- NOT IN -->
 
@@ -109,7 +150,7 @@ end
  - alternatively: `bundle exec rake generate` to only locally update the generated `json` files
  - review changes then `git commit`
  - make a PR ... get reviewed ... merge
- - datadog is updated by travis
+ - datadog is updated by CI
 
 ### Adding a new dashboard
  - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
  - figure out project name by converting the class name to snake-case
  - run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
 
-### Listing un-muted alerts
-
-Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
-
-### Validating mentions work
-
-`rake kennel:validate_mentions` should run as part of CI
-
-### Grepping through all of datadog
-
-`TYPE=monitor rake kennel:dump`
-
-### Find all monitors with No-Data
-
-`rake kennel:nodata TAG=team:foo`
-
-## Examples
-
-### Reusable monitors/dashes/etc
+### Reuse
 
 Add to `parts/<folder>`.
 
@@ -221,8 +244,30 @@ class Database < Kennel::Models::Project
   )
 end
 ```
+
+## Helpers
+
+### Listing un-muted alerts
+
+Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
+
+### Validating mentions work
+
+`rake kennel:validate_mentions` should run as part of CI
+
+### Grepping through all of datadog
+
+`TYPE=monitor rake kennel:dump`
+
+### Find all monitors with No-Data
+
+`rake kennel:nodata TAG=team:foo`
+
 <!-- NOT IN template/Readme.md -->
 
+
+## Development
+
 ### Integration testing
 
 ```Bash

data/lib/kennel.rb

@@ -29,7 +29,6 @@ require "kennel/models/project"
 require "kennel/models/team"
 
 module Kennel
-  MISSING_ID = 1
   class ValidationError < RuntimeError
   end
 

data/lib/kennel/importer.rb

@@ -40,7 +40,8 @@ module Kennel
           Kennel::Utils.parameterize(title)
         end
 
-      if resource == "monitor"
+      case resource
+      when "monitor"
         # flatten monitor options so they are all on the base
         data.merge!(data.delete(:options))
         data.merge!(data.delete(:thresholds) || {})
@@ -55,7 +56,7 @@ module Kennel
         end
 
         data[:type] = "query alert" if data[:type] == "metric alert"
-      elsif resource == "dashboard"
+      when "dashboard"
         widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
         widgets&.each { |widget| dry_up_query!(widget) }
       end

data/lib/kennel/models/dashboard.rb

@@ -13,16 +13,23 @@ module Kennel
       REQUEST_DEFAULTS = {
         style: { line_width: "normal", palette: "dog_classic", line_type: "solid" }
       }.freeze
-      TIMESERIES_DEFAULTS = { show_legend: false, legend_size: "0" }.freeze
+      WIDGET_DEFAULTS = {
+        "timeseries" => { show_legend: false, legend_size: "0" },
+        "note" => { background_color: "white", font_size: "14", show_tick: false, tick_edge: "left", tick_pos: "50%", text_align: "left" }
+      }.freeze
       SUPPORTED_DEFINITION_OPTIONS = [:events, :markers, :precision].freeze
 
+      DEFAULTS = {
+        template_variable_presets: nil
+      }.freeze
+
       settings :title, :description, :definitions, :widgets, :layout_type, :template_variable_presets
 
       defaults(
         description: -> { "" },
         definitions: -> { [] },
         widgets: -> { [] },
-        template_variable_presets: -> { [] },
+        template_variable_presets: -> { DEFAULTS.fetch(:template_variable_presets) },
         id: -> { nil }
       )
 
@@ -34,6 +41,8 @@ module Kennel
         def normalize(expected, actual)
           super
 
+          ignore_default(expected, actual, DEFAULTS)
+
           widgets_pairs(expected, actual).each do |pair|
             # conditional_formats ordering is randomly changed by datadog, compare a stable ordering
             pair.each do |widgets|
@@ -44,7 +53,7 @@ module Kennel
               end
             end
 
-            ignore_definition_defaults_for_type pair, "timeseries", TIMESERIES_DEFAULTS
+            ignore_widget_defaults pair
 
             ignore_request_defaults(*pair)
 
@@ -55,11 +64,12 @@ module Kennel
 
         private
 
-        def ignore_definition_defaults_for_type(pair, type, defaults)
+        def ignore_widget_defaults(pair)
           pair.map(&:size).max.times do |i|
-            if pair.all? { |w| w.dig(i, :definition, :type) == type }
-              ignore_defaults(pair[0], pair[1], defaults, nesting: :definition)
-            end
+            types = pair.map { |w| w.dig(i, :definition, :type) }.uniq
+            next unless types.size == 1
+            next unless defaults = WIDGET_DEFAULTS[types.first]
+            ignore_defaults(pair[0], pair[1], defaults, nesting: :definition)
           end
         end
 
@@ -122,7 +132,7 @@ module Kennel
         url[/\/dashboard\/([a-z\d-]+)/, 1]
       end
 
-      def resolve_linked_tracking_ids(id_map)
+      def resolve_linked_tracking_ids!(id_map, **args)
         widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
         widgets.each do |widget|
           next unless definition = widget[:definition]
@@ -130,16 +140,16 @@ module Kennel
           when "uptime"
             if ids = definition[:monitor_ids]
               definition[:monitor_ids] = ids.map do |id|
-                tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
+                tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
               end
             end
           when "alert_graph"
             if (id = definition[:alert_id]) && tracking_id?(id)
-              definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
+              definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
             end
           when "slo"
             if (id = definition[:slo_id]) && tracking_id?(id)
-              definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
+              definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
             end
           end
         end
@@ -172,7 +182,7 @@ module Kennel
 
         # Avoid diff from datadog presets sorting.
         presets = data[:template_variable_presets]
-        invalid! "template_variable_presets must be sorted by name" if presets != presets.sort_by { |p| p[:name] }
+        invalid! "template_variable_presets must be sorted by name" if presets && presets != presets.sort_by { |p| p[:name] }
       end
 
       def render_definitions

data/lib/kennel/models/monitor.rb

@@ -25,7 +25,7 @@ module Kennel
       settings(
         :query, :name, :message, :escalation_message, :critical, :type, :renotify_interval, :warning, :timeout_h, :evaluation_delay,
         :ok, :no_data_timeframe, :notify_no_data, :notify_audit, :tags, :critical_recovery, :warning_recovery, :require_full_window,
-        :threshold_windows, :new_host_delay
+        :threshold_windows, :new_host_delay, :groupby_simple_monitor
       )
 
       defaults(
@@ -44,7 +44,8 @@ module Kennel
         evaluation_delay: -> { MONITOR_OPTION_DEFAULTS.fetch(:evaluation_delay) },
         critical_recovery: -> { nil },
         warning_recovery: -> { nil },
-        threshold_windows: -> { nil }
+        threshold_windows: -> { nil },
+        groupby_simple_monitor: -> { nil }
       )
 
       def as_json
@@ -94,6 +95,11 @@ module Kennel
           end
         end
 
+        # option randomly pops up and cannot be removed
+        unless (group = groupby_simple_monitor).nil?
+          options[:groupby_simple_monitor] = group
+        end
+
         if windows = threshold_windows
           options[:threshold_windows] = windows
         end
@@ -103,11 +109,10 @@ module Kennel
         @as_json = data
       end
 
-      def resolve_linked_tracking_ids(id_map)
+      def resolve_linked_tracking_ids!(id_map, **args)
         if as_json[:type] == "composite"
           as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
-            # need force here since it validates the id exists
-            resolve_link($1, :monitor, id_map, force: true)
+            resolve_link($1, :monitor, id_map, **args)
           end
         end
       end
@@ -120,7 +125,7 @@ module Kennel
         Utils.path_to_url "/monitors##{id}/edit"
       end
 
-      # datadog uses both / and # as separator in it's links
+      # datadog uses / for show and # for edit as separator in it's links
       def self.parse_url(url)
         return unless id = url[/\/monitors[\/#](\d+)/, 1]
         Integer(id)
@@ -202,7 +207,7 @@ module Kennel
           # verify is_match uses available variables
           message = data.fetch(:message)
           used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
-          allowed = data.fetch(:query)[/by\s*[\({]([^\}\)]+)[}\)]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
+          allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
           unsupported = used - allowed
           if unsupported.any?
             invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"

data/lib/kennel/models/record.rb

@@ -60,23 +60,22 @@ module Kennel
         "#{project.kennel_id}:#{kennel_id}"
       end
 
-      def resolve_linked_tracking_ids(*)
+      def resolve_linked_tracking_ids!(*)
       end
 
       private
 
-      def resolve_link(id, type, id_map, force: false)
-        found = id_map[id]
-        return found if found && found != :new
-        api_resource = self.class.api_resource
-
-        if found == :new
+      def resolve_link(id, type, id_map, force:)
+        value = id_map[id]
+        if value == :new
           if force
-            invalid! "#{api_resource.capitalize} #{id} will be created in the current run and can only be used after that"
+            # TODO: remove the need for this by sorting monitors by missing resolutions
+            invalid! "#{id} needs to already exist, try again"
           else
-            Kennel.err.puts "#{api_resource.capitalize} #{id} will be created in the current run, the next run will link it properly"
-            Kennel::MISSING_ID
+            id # will be re-resolved by syncer after the linked object was created
           end
+        elsif value
+          value
         else
           invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
         end

data/lib/kennel/models/slo.rb

@@ -66,9 +66,9 @@ module Kennel
         url[/\/slo\?slo_id=([a-z\d]+)/, 1]
       end
 
-      def resolve_linked_tracking_ids(id_map)
+      def resolve_linked_tracking_ids!(id_map, **args)
         as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
-          id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
+          id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
         end
       end
 

data/lib/kennel/syncer.rb

@@ -38,12 +38,23 @@ module Kennel
     end
 
     def update
+      changed = (@create + @update).map { |_, e| e } unless @create.empty?
+
       @create.each do |_, e|
+        e.resolve_linked_tracking_ids!({}, force: true)
+
         reply = @api.create e.class.api_resource, e.as_json
-        Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(reply.fetch(:id))}"
+        id = reply.fetch(:id)
+
+        # resolve ids we could previously no resolve
+        changed.delete e
+        resolve_linked_tracking_ids! from: [reply], to: changed
+
+        Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
       end
 
       @update.each do |id, e|
+        e.resolve_linked_tracking_ids!({}, force: true)
         @api.update e.class.api_resource, id, e.as_json
         Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
       end
@@ -65,12 +76,10 @@ module Kennel
       @delete = []
 
       actual = Progress.progress("Downloading definitions") { download_definitions }
-
-      resolve_linked_tracking_ids actual
+      resolve_linked_tracking_ids! from: actual, to: @expected
+      filter_by_project! actual
 
       Progress.progress "Diffing" do
-        filter_by_project! actual
-
         items = actual.map do |a|
           e = matching_expected(a)
           if e && @expected.delete(e)
@@ -98,6 +107,7 @@ module Kennel
 
         ensure_all_ids_found
         @create = @expected.map { |e| [nil, e] }
+        @create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
       end
 
       @delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
@@ -178,7 +188,7 @@ module Kennel
     end
 
     # Do not add tracking-id when working with existing ids on a branch,
-    # so resource do not get deleted from merges to master.
+    # so resource do not get deleted fr:om merges to master.
     # Also make sure the diff still makes sense, by kicking out the now noop-update.
     #
     # Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
@@ -203,10 +213,10 @@ module Kennel
       end
     end
 
-    def resolve_linked_tracking_ids(actual)
-      map = actual.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
-      @expected.each { |e| map[e.tracking_id] ||= :new }
-      @expected.each { |e| e.resolve_linked_tracking_ids(map) }
+    def resolve_linked_tracking_ids!(from:, to:)
+      map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
+      to.each { |e| map[e.tracking_id] ||= :new }
+      to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
     end
 
     def filter_by_project!(definitions)

data/lib/kennel/tasks.rb

@@ -68,15 +68,16 @@ namespace :kennel do
     Kennel.update
   end
 
-  desc "update if this is a push to the default branch, otherwise plan"
-  task :travis do
-    on_default_branch = (ENV["TRAVIS_BRANCH"] == (ENV["DEFAULT_BRANCH"] || "master"))
-    is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false")
+  desc "update on push to the default branch, otherwise show plan"
+  task :ci do
+    branch = (ENV["TRAVIS_BRANCH"] || ENV["GITHUB_REF"]).to_s.sub(/^refs\/heads\//, "")
+    on_default_branch = (branch == (ENV["DEFAULT_BRANCH"] || "master"))
+    is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false" || ENV["GITHUB_EVENT_NAME"] == "push")
     task_name =
       if on_default_branch && is_push
         "kennel:update_datadog"
       else
-        "kennel:plan" # show plan in travis logs
+        "kennel:plan" # show plan in CI logs
       end
 
     Rake::Task[task_name].invoke

data/lib/kennel/version.rb

@@ -1,4 +1,4 @@
 # frozen_string_literal: true
 module Kennel
-  VERSION = "1.72.1"
+  VERSION = "1.75.0"
 end

data/template/Readme.md

@@ -1,16 +1,57 @@
-# Kennel
-
 ![](github/cage.jpg?raw=true)
 
-Manage datadog monitors/dashboards/slos as code
+Manage Datadog Monitors / Dashboards / Slos as code
 
- - Documented, reusable, and searchable
- - Changes are PR reviewed and auditable
+ - DRY, searchable, audited, documented
+ - Changes are PR reviewed and applied on merge
  - Updating shows diff before applying
- - Automated import of existing monitors/dashboards/slos
+ - Automated import of existing resources
+ - Resources are grouped into projects that belong to teams and inherit tags
+ - No copy-pasting of ids to create new resources
+ - Automated cleanup when removing code
+ - [Helpers](#helpers) for automating common tasks
+ 
+### Applying changes
 
 ![](github/screen.png?raw=true)
 
+### Example code
+
+```Ruby
+# teams/foo.rb
+module Teams
+  class Foo < Kennel::Models::Team
+    defaults(mention: -> { "@slack-my-team" })
+  end
+end
+
+# projects/bar.rb
+class Bar < Kennel::Models::Project
+  defaults(
+    team: -> { Teams::Foo.new }, # use mention and tags from the team
+    parts: -> {
+      [
+        Kennel::Models::Monitor.new(
+          self, # the current project
+          type: -> { "query alert" },
+          kennel_id: -> { "load-too-high" }, # pick a unique name
+          name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
+          message: -> {
+            <<~TEXT
+              This is bad!
+              #{super()} # inserts mention from team
+            TEXT
+          },
+          query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
+          critical: -> { 20 }
+        )
+      ]
+    }
+  )
+end
+```
+
+
 ## Structure
 
  - `projects/` monitors/dashboards/etc scoped by project
@@ -91,7 +132,7 @@ end
  - alternatively: `bundle exec rake generate` to only locally update the generated `json` files
  - review changes then `git commit`
  - make a PR ... get reviewed ... merge
- - datadog is updated by travis
+ - datadog is updated by CI
 
 ### Adding a new dashboard
  - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
  - figure out project name by converting the class name to snake-case
  - run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
 
-### Listing un-muted alerts
-
-Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
-
-### Validating mentions work
-
-`rake kennel:validate_mentions` should run as part of CI
-
-### Grepping through all of datadog
-
-`TYPE=monitor rake kennel:dump`
-
-### Find all monitors with No-Data
-
-`rake kennel:nodata TAG=team:foo`
-
-## Examples
-
-### Reusable monitors/dashes/etc
+### Reuse
 
 Add to `parts/<folder>`.
 
@@ -203,3 +226,22 @@ class Database < Kennel::Models::Project
   )
 end
 ```
+
+## Helpers
+
+### Listing un-muted alerts
+
+Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
+
+### Validating mentions work
+
+`rake kennel:validate_mentions` should run as part of CI
+
+### Grepping through all of datadog
+
+`TYPE=monitor rake kennel:dump`
+
+### Find all monitors with No-Data
+
+`rake kennel:nodata TAG=team:foo`
+

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kennel
 version: !ruby/object:Gem::Version
-  version: 1.72.1
+  version: 1.75.0
 platform: ruby
 authors:
 - Michael Grosser
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-06-25 00:00:00.000000000 Z
+date: 2020-07-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: faraday