kennel 2.14.0 → 2.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d289ac0af8158766eeced52cda258b67071189f889e6fa82d63cafe0b4cc7655
-  data.tar.gz: 5a231e9da146ca9e008a8b507e2cbd4e11913c6f53d19fbd64af5c93b5b1a0aa
+  metadata.gz: 3f5c11c126af9e586c2e7f48564760b8625248626204bdd76359001d78cd2038
+  data.tar.gz: 7ec6271b5eb235c8ee4b0bbaf3b9d59dc3b3f8796c5cf8288f6d753094e99a3c
 SHA512:
-  metadata.gz: a476cf37c12872c6cd4f1eeed55751840ffd5c6206dec0279d281f7862a47bd8e614fa14eefc0ffc32095a15f2d26d4a75b2298b60cc107bf09e1994562f0f15
-  data.tar.gz: 29dfd3a9b22848badfb52da141620fde5fc89224974cade1e0cdda957a69d9b82798e4a6a03e95339f1cf907e8c036939fa1b5289cf9b8292b83e9e73e68e03a
+  metadata.gz: 6778c767c1b28ad451f42e0a7f86557382e93627bfee517d7fcd857f90a674bd63294d67253704975c5653d4de616f731524eaf37bd11ff41822635dfcf78c8a
+  data.tar.gz: 626e57f6a9e8a8c8080defd2496d983fa93e0a9fcdf63e5ad87aad662c3f4aaca42ff7e5b0c6c9cb0d94fc66d43808fd10dee65add9d0ca3d1d60b76d2c801e9

data/Readme.md

@@ -206,6 +206,7 @@ end
   class MyProject < Kennel::Models::Project
     defaults(
       team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
+      # kennel_id: -> { "my_project" } # Custom kennel_id (default is snake_cased class name)
       parts: -> {
         [
           Kennel::Models::Monitor.new(
@@ -249,6 +250,7 @@ Remove the code that created the resource. The next update will delete it (see a
  - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to find a dashboard
  - run `URL='https://app.datadoghq.com/dashboard/bet-foo-bar' bundle exec rake kennel:import` and copy the output
  - find or create a project in `projects/`
+ - tags: only `team:` tags are submitted to datadog since nothing else is supported
  - add a dashboard to `parts: [` list, for example:
   ```Ruby
   class MyProject < Kennel::Models::Project
@@ -264,8 +266,9 @@ Remove the code that created the resource. The next update will delete it (see a
             template_variables: -> { ["environment"] }, # see https://docs.datadoghq.com/api/?lang=ruby#timeboards
             kennel_id: -> { "overview-dashboard" }, # make up a unique name
             layout_type: -> { "ordered" },
+            widgets: -> { "... raw widget definitions, most flexible ..." },
             definitions: -> {
-              [ # An array or arrays, each one is a graph in the dashboard, alternatively a hash for finer control
+              [ # each element is a graph in the dashboard, alternatively a hash for complete control just like in `widgets`
                 [
                   # title, viz, type, query, edit an existing graph and see the json definition
                   "Graph name", "timeseries", "area", "sum:mystats.foobar{$environment}"
@@ -274,7 +277,7 @@ Remove the code that created the resource. The next update will delete it (see a
                   # queries can be an Array as well, this will generate multiple requests
                   # for a single graph
                   "Graph name", "timeseries", "area", ["sum:mystats.foobar{$environment}", "sum:mystats.success{$environment}"],
-                  # add events too ...
+                  # add events too ... (also supports `:markers` and `:precision`)
                   events: [{q: "tags:foobar,deploy", tags_execution: "and"}]
                 ]
               ]
@@ -286,6 +289,55 @@ Remove the code that created the resource. The next update will delete it (see a
   end
  ```
 
+### Adding a new synthetic test
+ - go to [datadog synthetic tests UI](https://app.datadoghq.com/synthetics/tests) and click on _New_ to create a test
+ - see below
+
+### Updating an existing synthetic test
+ - go to [datadog synthetic tests UI](https://app.datadoghq.com/synthetics/tests) to find a test
+ - run `URL='https://app.datadoghq.com/synthetics/details/abc-def-ghi' bundle exec rake kennel:import` and copy the output
+ - find or create a project in `projects/`
+ - add a synthetic test to `parts: [` list, for example:
+  ```Ruby
+  class MyProject < Kennel::Models::Project
+    defaults(
+      team: -> { Teams::MyTeam.new },
+      parts: -> {
+        [
+          Kennel::Models::SyntheticTest.new(
+            self,
+            id: -> { "abc-def-ghi" }, # id from datadog url, not needed when creating a new test
+            kennel_id: -> { "my-api-test" },
+            name: -> { "My API Test" },
+            type: -> { "api" },
+            subtype: -> { "http" },
+            locations: -> { :all }, # use all locations, or specify: ["aws:us-east-1", "aws:eu-west-1"]
+            message: -> {
+              <<~TEXT
+                API check failed!
+                #{super()}
+              TEXT
+            },
+            options: -> {
+              {
+                tick_every: 60,
+                min_failure_duration: 0,
+                min_location_failed: 1
+              }
+            },
+            config: -> {
+              {
+                assertions: [{ type: "statusCode", operator: "is", target: 200 }],
+                request: { method: "GET", url: "https://example.com/health" }
+              }
+            }
+          )
+        ]
+      }
+    )
+  end
+  ```
+
 ### Updating existing resources with id
 Setting `id` makes kennel take over a manually created datadog resource.
 When manually creating to import, it is best to remove the `id` and delete the manually created resource.
@@ -327,10 +379,11 @@ module ProjectA
 - Use `TRACKING_ID=<project-kennel_id>:<resource-kennel_id>` for single resource:
 
   Use the project kennel_id and the resources kennel_id, for example `class ProjectA` and `FooAlert` would give `project_a:foo_alert`.
+  Alternatively use the path of the generated file `TRACKING_ID=generated/project_a/foo_alert.json`
 
 ### Skipping validations
 Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and
-to unblock use the `validate: -> { false }` option.
+to unblock use `ignored_errors: [:name_of_the_error]`.
 
 ### Linking resources with kennel_id
 Link resources with their kennel_id in the format `project kennel_id` + `:` + `resource kennel_id`,
@@ -339,7 +392,7 @@ so they can be created in a single update and can be re-created if any of them i
 
 |Resource|Type|Syntax|
 |---|---|---|
-|Dashboard|uptime|`monitor: {id: "foo:bar"}`|
+|Dashboard|uptime|`monitor_ids: ["foo:bar", "foo:baz"]`|
 |Dashboard|alert_graph|`alert_id: "foo:bar"`|
 |Dashboard|slo|`slo_id: "foo:bar"`|
 |Dashboard|timeseries|`queries: [{ data_source: "slo", slo_id: "foo:bar" }]`|
@@ -388,6 +441,9 @@ Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a
 ### Validating mentions work
 `rake kennel:validate_mentions` should run as part of CI
 
+Use `KNOWN=foo@bar.com,baz@bar.com` to exempt mentions that are not returned by the API.
+Use `KNOWN_RANDOM=@sns-foo,@sns-bar` to ignore for example SNS handles that are randomly invalid in the API.
+
 ### Grepping through all of datadog
 ```Bash
 rake kennel:dump > tmp/dump
@@ -405,6 +461,9 @@ https://foo.datadog.com/monitor/123
 ### Find all monitors with No-Data
 `rake kennel:nodata TAG=team:foo`
 
+- `FORMAT=json` to output as JSON with tracking IDs
+- `THRESHOLD_DAYS=N` to filter to monitors with N+ days in no-data
+
 ### Finding the tracking id of a resource
 
 When trying to link resources together, this avoids having to go through datadog UI.
@@ -421,6 +480,14 @@ rake kennel:tracking_id ID=123 RESOURCE=monitor
 - Setting `FORCE_GET_CACHE=true` will cache all get requests, which makes benchmarking improvements more reliable.
 - Setting `STORE=false` will make `rake plan` not update the files on disk and save a bit of time
 
+### Other Environment Variables
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `KENNEL_MARKER_TEXT` | Custom marker text to namespace multiple Kennel instances in the same Datadog account. Each instance will only manage resources with its marker. | `Managed by kennel` |
+| `KENNEL_API_CACHE_FILE` | Path to the API cache file for dashboard details. | `tmp/cache/details` |
+| `KENNEL_NO_GENERATE` | When set, skip generating files during `plan` or `update_datadog`. Useful when generated files are already up to date. | - |
+| `NO_IGNORED_ERRORS` | When set, show all validation errors including ones suppressed via `ignored_errors`. | - |
+
 ### Integration testing
 ```Bash
 rake play

data/lib/kennel/filter.rb

@@ -2,6 +2,7 @@
 
 module Kennel
   class Filter
+    ID_SEPARATOR = ":"
     attr_reader :project_filter
 
     def initialize
@@ -22,23 +23,18 @@ module Kennel
       !project_filter.nil?
     end
 
-    def matches_project_id?(project_id)
+    def filters_project_id?(project_id)
       !filtering? || project_filter.include?(project_id)
     end
 
-    def matches_tracking_id?(tracking_id)
+    def filters_tracking_id?(tracking_id)
       return true unless filtering?
       return tracking_id_filter.include?(tracking_id) if tracking_id_filter
 
-      project_id = tracking_id.split(":").first
+      project_id = tracking_id.split(ID_SEPARATOR, 2).first
       project_filter.include?(project_id)
     end
 
-    def tracking_id_for_path(tracking_id)
-      return tracking_id unless tracking_id.end_with?(".json")
-      tracking_id.sub("generated/", "").sub(".json", "").sub("/", ":")
-    end
-
     private
 
     attr_reader :tracking_id_filter
@@ -46,7 +42,7 @@ module Kennel
     # needs to be called after read_tracking_id_filter_from_env
     def read_project_filter_from_env
       project_names = ENV["PROJECT"]&.split(",")&.sort&.uniq
-      tracking_project_names = tracking_id_filter&.map { |id| id.split(":", 2).first }&.sort&.uniq
+      tracking_project_names = tracking_id_filter&.map { |id| id.split(ID_SEPARATOR, 2).first }&.sort&.uniq
       if project_names && tracking_project_names && project_names != tracking_project_names
         # avoid everything being filtered out
         raise "do not set a different PROJECT= when using TRACKING_ID="
@@ -57,8 +53,8 @@ module Kennel
     def read_tracking_id_filter_from_env
       return unless (tracking_id = ENV["TRACKING_ID"])
       tracking_id.split(",").map do |id|
-        # allow users to paste the generated/ path of an objects to update it without manually converting
-        tracking_id_for_path(id)
+        # allow using the generated/ path from `git diff` to update objects without manually converting
+        id.include?(ID_SEPARATOR) ? id : PartsSerializer.tracking_id_for_path(id)
       end.sort.uniq
     end
 

data/lib/kennel/models/team.rb

@@ -1,10 +1,12 @@
 # frozen_string_literal: true
 module Kennel
   module Models
-    class Team < Base
-      settings :mention, :tags, :renotify_interval, :kennel_id
+    class Team
+      include SettingsAsMethods
+
+      settings :mention, :tags, :renotify_interval
       defaults(
-        tags: -> { ["team:#{kennel_id.sub(/^teams_/, "").tr("_", "-")}"] },
+        tags: -> { ["team:#{StringUtils.snake_case(self.class.name).sub(/^teams_/, "").tr("_", "-")}"] },
         renotify_interval: -> { 0 }
       )
     end

data/lib/kennel/parts_serializer.rb

@@ -2,6 +2,9 @@
 
 module Kennel
   class PartsSerializer
+    FILE_EXTENSION = ".json"
+    FOLDER = "generated"
+
     def initialize(filter:)
       @filter = filter
     end
@@ -9,8 +12,15 @@ module Kennel
     def write(parts)
       Progress.progress "Storing" do
         existing = existing_files_and_folders
-        used = write_changed(parts)
-        FileUtils.rm_rf(existing - used)
+        used, changed = write_changed(parts)
+        FileUtils.rm_rf(existing - used) # cleanup abandoned
+        suggest_using_project_filter(changed)
+      end
+    end
+
+    class << self
+      def tracking_id_for_path(path)
+        path.sub("#{FOLDER}/", "").sub(FILE_EXTENSION, "").sub("/", ":")
       end
     end
 
@@ -20,28 +30,29 @@ module Kennel
 
     def write_changed(parts)
       used = []
+      changed = []
 
       Utils.parallel(parts, max: 2) do |part|
         path = path_for_tracking_id(part.tracking_id)
 
+        # match paths returned from existing_files_and_folders
         used << File.dirname(path) # we have 1 level of sub folders, so this is enough
         used << path
 
         content = part.as_json.merge(api_resource: part.class.api_resource)
-        write_file_if_necessary(path, content)
+        changed << path if write_file_if_necessary(path, content)
       end
-
-      used
+      [used, changed]
     end
 
     def existing_files_and_folders
-      paths = Dir["generated/**/*"]
+      paths = Dir["#{FOLDER}/**/*"] # we rely on this returning folders and files, see write_changed
 
       # when filtering we only need the files we are going to write
       if filter.filtering?
         paths.select! do |path|
-          tracking_id = filter.tracking_id_for_path(path)
-          filter.matches_tracking_id?(tracking_id)
+          tracking_id = self.class.tracking_id_for_path(path)
+          filter.filters_tracking_id?(tracking_id)
         end
       end
 
@@ -49,7 +60,7 @@ module Kennel
     end
 
     def path_for_tracking_id(tracking_id)
-      "generated/#{tracking_id.tr("/", ":").sub(":", "/")}.json"
+      "#{FOLDER}/#{tracking_id.tr("/", ":").sub(":", "/")}#{FILE_EXTENSION}"
     end
 
     def write_file_if_necessary(path, content)
@@ -58,13 +69,21 @@ module Kennel
 
       # 99% case
       begin
-        return if File.read(path) == content
-      rescue Errno::ENOENT
+        return false if File.read(path) == content
+      rescue Errno::ENOENT # file or even folder did not exist
         FileUtils.mkdir_p(File.dirname(path))
       end
 
       # slow 1% case
       File.write(path, content)
+      true
+    end
+
+    def suggest_using_project_filter(changed)
+      return if filter.filtering?
+      projects = changed.map { |path| path.split("/")[1] }.uniq
+      return if projects.size != 1
+      warn "Hint: Using PROJECT=#{projects[0]} is faster"
     end
   end
 end

data/lib/kennel/projects_provider.rb

@@ -12,7 +12,8 @@ module Kennel
     #   All requested projects. This is a slow operation when loading all projects.
     def projects
       load_requested
-      loaded_projects.map(&:new)
+      projects = loaded_projects.map(&:new)
+      @filter.filter_projects projects # in case we loaded more though dependencies
     end
 
     private

data/lib/kennel/syncer/resolver.rb

@@ -27,7 +27,7 @@ module Kennel
           # ignore when deleted from the codebase
           # (when running with filters we cannot see the other resources in the codebase)
           api_resource = a.fetch(:klass).api_resource
-          next if !id_map.get(api_resource, tracking_id) && filter.matches_tracking_id?(tracking_id)
+          next if !id_map.get(api_resource, tracking_id) && filter.filters_tracking_id?(tracking_id)
 
           id_map.set(api_resource, tracking_id, a.fetch(:id))
           if a.fetch(:klass).api_resource == "synthetics/tests"

data/lib/kennel/syncer.rb

@@ -179,7 +179,7 @@ module Kennel
 
       actual.select! do |a|
         tracking_id = a.fetch(:tracking_id)
-        tracking_id.nil? || filter.matches_tracking_id?(tracking_id)
+        tracking_id.nil? || filter.filters_tracking_id?(tracking_id)
       end
     end
   end

data/lib/kennel/tasks.rb

@@ -25,6 +25,7 @@ module Kennel
           source = ".env"
 
           # warn when users have things like DATADOG_TOKEN already set and it will not be loaded from .env
+          # (KENNEL_SILENCE_UPDATED_ENV is intentionally not documented - users see it when needed)
           unless ENV["KENNEL_SILENCE_UPDATED_ENV"]
             updated = Dotenv.parse(source).select { |k, v| ENV[k] && ENV[k] != v }
             warn "Environment variables #{updated.keys.join(", ")} need to be unset to be sourced from #{source}" if updated.any?

data/lib/kennel/version.rb

@@ -1,4 +1,4 @@
 # frozen_string_literal: true
 module Kennel
-  VERSION = "2.14.0"
+  VERSION = "2.16.0"
 end

data/lib/kennel.rb

@@ -69,7 +69,9 @@ module Kennel
 
     def generate
       parts = generated
-      PartsSerializer.new(filter: filter).write(parts) if ENV["STORE"] != "false" # quicker when debugging
+      if ENV["STORE"] != "false" # quicker when debugging
+        PartsSerializer.new(filter: filter).write(parts)
+      end
       parts
     end
 
@@ -109,8 +111,7 @@ module Kennel
     def generated(**kwargs)
       @generated ||= begin
         projects = Progress.progress "Loading projects", **kwargs do
-          projects = ProjectsProvider.new(filter: filter).projects
-          filter.filter_projects projects
+          ProjectsProvider.new(filter: filter).projects
         end
 
         parts = Progress.progress "Finding parts", **kwargs do

data/template/Readme.md

@@ -188,6 +188,7 @@ end
   class MyProject < Kennel::Models::Project
     defaults(
       team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
+      # kennel_id: -> { "my_project" } # Custom kennel_id (default is snake_cased class name)
       parts: -> {
         [
           Kennel::Models::Monitor.new(
@@ -231,6 +232,7 @@ Remove the code that created the resource. The next update will delete it (see a
  - go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to find a dashboard
  - run `URL='https://app.datadoghq.com/dashboard/bet-foo-bar' bundle exec rake kennel:import` and copy the output
  - find or create a project in `projects/`
+ - tags: only `team:` tags are submitted to datadog since nothing else is supported
  - add a dashboard to `parts: [` list, for example:
   ```Ruby
   class MyProject < Kennel::Models::Project
@@ -246,8 +248,9 @@ Remove the code that created the resource. The next update will delete it (see a
             template_variables: -> { ["environment"] }, # see https://docs.datadoghq.com/api/?lang=ruby#timeboards
             kennel_id: -> { "overview-dashboard" }, # make up a unique name
             layout_type: -> { "ordered" },
+            widgets: -> { "... raw widget definitions, most flexible ..." },
             definitions: -> {
-              [ # An array or arrays, each one is a graph in the dashboard, alternatively a hash for finer control
+              [ # each element is a graph in the dashboard, alternatively a hash for complete control just like in `widgets`
                 [
                   # title, viz, type, query, edit an existing graph and see the json definition
                   "Graph name", "timeseries", "area", "sum:mystats.foobar{$environment}"
@@ -256,7 +259,7 @@ Remove the code that created the resource. The next update will delete it (see a
                   # queries can be an Array as well, this will generate multiple requests
                   # for a single graph
                   "Graph name", "timeseries", "area", ["sum:mystats.foobar{$environment}", "sum:mystats.success{$environment}"],
-                  # add events too ...
+                  # add events too ... (also supports `:markers` and `:precision`)
                   events: [{q: "tags:foobar,deploy", tags_execution: "and"}]
                 ]
               ]
@@ -268,6 +271,55 @@ Remove the code that created the resource. The next update will delete it (see a
   end
  ```
 
+### Adding a new synthetic test
+ - go to [datadog synthetic tests UI](https://app.datadoghq.com/synthetics/tests) and click on _New_ to create a test
+ - see below
+
+### Updating an existing synthetic test
+ - go to [datadog synthetic tests UI](https://app.datadoghq.com/synthetics/tests) to find a test
+ - run `URL='https://app.datadoghq.com/synthetics/details/abc-def-ghi' bundle exec rake kennel:import` and copy the output
+ - find or create a project in `projects/`
+ - add a synthetic test to `parts: [` list, for example:
+  ```Ruby
+  class MyProject < Kennel::Models::Project
+    defaults(
+      team: -> { Teams::MyTeam.new },
+      parts: -> {
+        [
+          Kennel::Models::SyntheticTest.new(
+            self,
+            id: -> { "abc-def-ghi" }, # id from datadog url, not needed when creating a new test
+            kennel_id: -> { "my-api-test" },
+            name: -> { "My API Test" },
+            type: -> { "api" },
+            subtype: -> { "http" },
+            locations: -> { :all }, # use all locations, or specify: ["aws:us-east-1", "aws:eu-west-1"]
+            message: -> {
+              <<~TEXT
+                API check failed!
+                #{super()}
+              TEXT
+            },
+            options: -> {
+              {
+                tick_every: 60,
+                min_failure_duration: 0,
+                min_location_failed: 1
+              }
+            },
+            config: -> {
+              {
+                assertions: [{ type: "statusCode", operator: "is", target: 200 }],
+                request: { method: "GET", url: "https://example.com/health" }
+              }
+            }
+          )
+        ]
+      }
+    )
+  end
+  ```
+
 ### Updating existing resources with id
 Setting `id` makes kennel take over a manually created datadog resource.
 When manually creating to import, it is best to remove the `id` and delete the manually created resource.
@@ -309,10 +361,11 @@ module ProjectA
 - Use `TRACKING_ID=<project-kennel_id>:<resource-kennel_id>` for single resource:
 
   Use the project kennel_id and the resources kennel_id, for example `class ProjectA` and `FooAlert` would give `project_a:foo_alert`.
+  Alternatively use the path of the generated file `TRACKING_ID=generated/project_a/foo_alert.json`
 
 ### Skipping validations
 Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and
-to unblock use the `validate: -> { false }` option.
+to unblock use `ignored_errors: [:name_of_the_error]`.
 
 ### Linking resources with kennel_id
 Link resources with their kennel_id in the format `project kennel_id` + `:` + `resource kennel_id`,
@@ -321,7 +374,7 @@ so they can be created in a single update and can be re-created if any of them i
 
 |Resource|Type|Syntax|
 |---|---|---|
-|Dashboard|uptime|`monitor: {id: "foo:bar"}`|
+|Dashboard|uptime|`monitor_ids: ["foo:bar", "foo:baz"]`|
 |Dashboard|alert_graph|`alert_id: "foo:bar"`|
 |Dashboard|slo|`slo_id: "foo:bar"`|
 |Dashboard|timeseries|`queries: [{ data_source: "slo", slo_id: "foo:bar" }]`|
@@ -370,6 +423,9 @@ Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a
 ### Validating mentions work
 `rake kennel:validate_mentions` should run as part of CI
 
+Use `KNOWN=foo@bar.com,baz@bar.com` to exempt mentions that are not returned by the API.
+Use `KNOWN_RANDOM=@sns-foo,@sns-bar` to ignore for example SNS handles that are randomly invalid in the API.
+
 ### Grepping through all of datadog
 ```Bash
 rake kennel:dump > tmp/dump
@@ -387,6 +443,9 @@ https://foo.datadog.com/monitor/123
 ### Find all monitors with No-Data
 `rake kennel:nodata TAG=team:foo`
 
+- `FORMAT=json` to output as JSON with tracking IDs
+- `THRESHOLD_DAYS=N` to filter to monitors with N+ days in no-data
+
 ### Finding the tracking id of a resource
 
 When trying to link resources together, this avoids having to go through datadog UI.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: kennel
 version: !ruby/object:Gem::Version
-  version: 2.14.0
+  version: 2.16.0
 platform: ruby
 authors:
 - Michael Grosser