kennel 1.115.0 → 1.116.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 76e2802df7ab2a6583e3529643deb08c382367dea7f90a7253314959d29a5bec
-  data.tar.gz: 4249c23ca2892cc48ce72dbd9e24fc03f38883d6eaed9b509ef75c78cb31afeb
+  metadata.gz: 67342a80e2cd8dd1f0f2ad613cca326405bff85975ceb4253bc6180c856b03cc
+  data.tar.gz: 587fb9a0ca21bf084e71c496c411708cc132aa80f2f7ec3c34f56b5a3b1888b4
 SHA512:
-  metadata.gz: 8bd1740361932ba4e6dfe4d7aff4ad7cbcdc96796020428902fc6551b9768c6ccc434a279c214a638aa119e012267fd8dfe00e9107052c570789a1e563075505
-  data.tar.gz: 266aacb034b8933640c3e95a0319afd5fdc923c857d85a5b039e4e14a2bb47dc1096e8849a1d3060171fed2984d847934465ed845a1e45d27caa465c5ca9fcf4
+  metadata.gz: 89fb1183eb9ee4db1f0a138c342a79b9f6f5b1aab962090e8f5e83d0e1c7fcbd38486106cbfd506e7da33a747a0391b3a0944c214efeffbe3a5f1854cc0d0fc4
+  data.tar.gz: 4b5de65ad6957b45309e8e1158c0a23ac7fb95242a264d2f292f74d156481a23af5737d13236f7761664954c1f77471d6327a2febf30d5489361b0a4988545f2

data/Readme.md

@@ -289,23 +289,27 @@ Remove the code that created the resource. The next update will delete it (see a
  ```
 
 ### Updating existing resources with id
-
 Setting `id` makes kennel take over a manually created datadog resource.
 When manually creating to import, it is best to remove the `id` and delete the manually created resource.
 
 When an `id` is set and the original resource is deleted, kennel will fail to update,
 removing the `id` will cause kennel to create a new resource in datadog.
 
-### Organizing many projects
-Having many projects (and their sub-resources) can quickly get out of hand.
+### Organizing projects with many resources
+When project files get too long, this structure can keep things bite-sized.
 
-Use this class structure to keep things organized:
 ```Ruby
 # projects/project_a/base.rb
 module ProjectA
   class Base < Kennel::Models::Project
     defaults(
       kennel_id: -> { "project_a" },
+      parts: -> {
+        [
+          Monitors::FooAlert.new(self),
+          ...
+        ]
+      }
       ...
 
 # projects/project_a/monitors/foo_alert.rb
@@ -315,7 +319,16 @@ module ProjectA
       ...
 ```
 
-On the command line, use the projects `kennel_id` (and if none is set then snake_case of the class name) to refer to the project. For example here`PROJECT=project_a` but if it were `Foo::ProjectA` then `foo_project_a`.
+### Updating a single project or resource
+
+- Use `PROJECT=<kennel_id>` for single project:
+
+  Use the projects `kennel_id` (and if none is set then snake_case of the class name including modules)
+  to refer to the project. For example for `class ProjectA` use `PROJECT=project_a` but for `Foo::ProjectA` use `foo_project_a`.
+
+- Use `TRACKING_ID=<project-kennel_id>:<resource-kennel_id>` for single resource:
+
+  Use the project kennel_id and the resources kennel_id, for example `class ProjectA` and `FooAlert` would give `project_a:foo_alert`.
 
 ### Skipping validations
 Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and

data/lib/kennel/progress.rb

@@ -7,12 +7,12 @@ module Kennel
     def self.progress(name)
       Kennel.err.print "#{name} ... "
 
-      animation = "-\\|/"
-      count = 0
       stop = false
       result = nil
 
       spinner = Thread.new do
+        animation = "-\\|/"
+        count = 0
         loop do
           break if stop
           Kennel.err.print animation[count % animation.size]
@@ -30,7 +30,7 @@ module Kennel
 
       result
     ensure
-      stop = true
+      stop = true # make thread stop without killing it
     end
   end
 end

data/lib/kennel/syncer.rb

@@ -4,9 +4,10 @@ module Kennel
     DELETE_ORDER = ["dashboard", "slo", "monitor", "synthetics/tests"].freeze # dashboards references monitors + slos, slos reference monitors
     LINE_UP = "\e[1A\033[K" # go up and clear
 
-    def initialize(api, expected, project_filter: nil)
+    def initialize(api, expected, project_filter: nil, tracking_id_filter: nil)
       @api = api
       @project_filter = project_filter
+      @tracking_id_filter = tracking_id_filter
       @expected = Set.new expected # need set to speed up deletion
       calculate_diff
       validate_plan
@@ -105,7 +106,7 @@ module Kennel
 
       Progress.progress "Diffing" do
         populate_id_map @expected, actual
-        filter_actual_by_project! actual
+        filter_actual! actual
         resolve_linked_tracking_ids! @expected # resolve dependencies to avoid diff
 
         @expected.each(&:add_tracking_id) # avoid diff with actual
@@ -261,11 +262,12 @@ module Kennel
         next unless tracking_id = a.fetch(:tracking_id)
 
         # ignore when deleted from the codebase
-        # (when running with project filter we cannot see the other resources in the codebase)
+        # (when running with filters we cannot see the other resources in the codebase)
         api_resource = a.fetch(:klass).api_resource
         next if
           !@id_map.get(api_resource, tracking_id) &&
-          (!project_prefixes || tracking_id.start_with?(*project_prefixes))
+          (!project_prefixes || tracking_id.start_with?(*project_prefixes)) &&
+          (!@tracking_id_filter || @tracking_id_filter.include?(tracking_id))
 
         @id_map.set(api_resource, tracking_id, a.fetch(:id))
         if a[:klass].api_resource == "synthetics/tests"
@@ -278,12 +280,18 @@ module Kennel
       list.each { |e| e.resolve_linked_tracking_ids!(@id_map, force: force) }
     end
 
-    def filter_actual_by_project!(actual)
-      return unless @project_filter
-      project_prefixes = @project_filter&.map { |p| "#{p}:" }
-      actual.select! do |a|
-        tracking_id = a.fetch(:tracking_id)
-        !tracking_id || tracking_id.start_with?(*project_prefixes)
+    def filter_actual!(actual)
+      if @tracking_id_filter
+        actual.select! do |a|
+          tracking_id = a.fetch(:tracking_id)
+          !tracking_id || @tracking_id_filter.include?(tracking_id)
+        end
+      elsif @project_filter
+        project_prefixes = @project_filter.map { |p| "#{p}:" }
+        actual.select! do |a|
+          tracking_id = a.fetch(:tracking_id)
+          !tracking_id || tracking_id.start_with?(*project_prefixes)
+        end
       end
     end
   end

data/lib/kennel/utils.rb

@@ -113,7 +113,7 @@ module Kennel
               done[i] =
                 begin
                   yield item
-                rescue StandardError => e
+                rescue Exception => e # rubocop:disable Lint/RescueException
                   work.clear
                   e
                 end
@@ -121,7 +121,7 @@ module Kennel
           end
         end
         workers.each(&:join)
-        done.each { |d| raise d if d.is_a?(StandardError) }
+        done.each { |d| raise d if d.is_a?(Exception) }
       end
 
       def natural_order(name)

data/lib/kennel/version.rb

@@ -1,4 +1,4 @@
 # frozen_string_literal: true
 module Kennel
-  VERSION = "1.115.0"
+  VERSION = "1.116.2"
 end

data/lib/kennel.rb

@@ -65,13 +65,22 @@ module Kennel
 
     def store(parts)
       Progress.progress "Storing" do
-        old = Dir["generated/{#{(project_filter || ["**"]).join(",")}}/*"]
+        old = Dir[[
+          "generated",
+          if project_filter || tracking_id_filter
+            [
+              "{" + (project_filter || ["*"]).join(",") + "}",
+              "{" + (tracking_id_filter || ["*"]).join(",") + "}.json"
+            ]
+          else
+            "**"
+          end
+        ].join("/")]
         used = []
 
         Utils.parallel(parts, max: 2) do |part|
           path = "generated/#{part.tracking_id.tr("/", ":").sub(":", "/")}.json"
-          used << File.dirname(path) # only 1 level of sub folders, so this is safe
-          used << path
+          used.concat [File.dirname(path), path] # only 1 level of sub folders, so this is safe
           payload = part.as_json.merge(api_resource: part.class.api_resource)
           write_file_if_necessary(path, JSON.pretty_generate(payload) << "\n")
         end
@@ -94,7 +103,7 @@ module Kennel
     end
 
     def syncer
-      @syncer ||= Syncer.new(api, generated, project_filter: project_filter)
+      @syncer ||= Syncer.new(api, generated, project_filter: project_filter, tracking_id_filter: tracking_id_filter)
     end
 
     def api
@@ -105,22 +114,12 @@ module Kennel
       @generated ||= begin
         Progress.progress "Generating" do
           load_all
-          known = []
-          filter = project_filter
-
-          parts = Utils.parallel(Models::Project.recursive_subclasses) do |project_class|
-            project = project_class.new
-            kennel_id = project.kennel_id
-            if filter
-              known << kennel_id
-              next [] unless filter.include?(kennel_id)
-            end
-            project.validated_parts
-          end.flatten(1)
-
-          if filter && parts.empty?
-            raise "#{filter.join(", ")} does not match any projects, try any of these:\n#{known.uniq.sort.join("\n")}"
-          end
+
+          projects = Models::Project.recursive_subclasses.map(&:new)
+          filter_resources!(projects, :kennel_id, project_filter, "projects", "PROJECT")
+
+          parts = Utils.parallel(projects, &:validated_parts).flatten(1)
+          filter_resources!(parts, :tracking_id, tracking_id_filter, "resources", "TRACKING_ID")
 
           parts.group_by(&:tracking_id).each do |tracking_id, same|
             next if same.size == 1
@@ -139,7 +138,29 @@ module Kennel
     end
 
     def project_filter
-      ENV["PROJECT"]&.split(",")
+      projects = ENV["PROJECT"]&.split(",")
+      tracking_projects = tracking_id_filter&.map { |id| id.split(":", 2).first }
+      if projects && tracking_projects && projects != tracking_projects
+        raise "do not set PROJECT= when using TRACKING_ID="
+      end
+      projects || tracking_projects
+    end
+
+    def tracking_id_filter
+      (tracking_id = ENV["TRACKING_ID"]) && tracking_id.split(",")
+    end
+
+    def filter_resources!(resources, by, against, name, env)
+      return unless against
+
+      before = resources.dup
+      resources.select! { |p| against.include?(p.send(by)) }
+      return if resources.size == against.size
+
+      raise <<~MSG.rstrip
+        #{env}=#{against.join(",")} matched #{resources.size} #{name}, try any of these:
+        #{before.map(&by).sort.uniq.join("\n")}
+      MSG
     end
 
     def load_all
@@ -147,8 +168,9 @@ module Kennel
       Dir.exist?("teams") && loader.push_dir("teams", namespace: Teams)
       Dir.exist?("parts") && loader.push_dir("parts")
       loader.setup
+      loader.eager_load # TODO: this should not be needed but we see hanging CI processes when it's not added
 
-      # TODO: also do projects and update expected path too
+      # TODO: also auto-load projects and update expected path too
       ["projects"].each do |folder|
         Dir["#{folder}/**/*.rb"].sort.each { |f| require "./#{f}" }
       end

data/template/Readme.md

@@ -271,23 +271,27 @@ Remove the code that created the resource. The next update will delete it (see a
  ```
 
 ### Updating existing resources with id
-
 Setting `id` makes kennel take over a manually created datadog resource.
 When manually creating to import, it is best to remove the `id` and delete the manually created resource.
 
 When an `id` is set and the original resource is deleted, kennel will fail to update,
 removing the `id` will cause kennel to create a new resource in datadog.
 
-### Organizing many projects
-Having many projects (and their sub-resources) can quickly get out of hand.
+### Organizing projects with many resources
+When project files get too long, this structure can keep things bite-sized.
 
-Use this class structure to keep things organized:
 ```Ruby
 # projects/project_a/base.rb
 module ProjectA
   class Base < Kennel::Models::Project
     defaults(
       kennel_id: -> { "project_a" },
+      parts: -> {
+        [
+          Monitors::FooAlert.new(self),
+          ...
+        ]
+      }
       ...
 
 # projects/project_a/monitors/foo_alert.rb
@@ -297,7 +301,16 @@ module ProjectA
       ...
 ```
 
-On the command line, use the projects `kennel_id` (and if none is set then snake_case of the class name) to refer to the project. For example here`PROJECT=project_a` but if it were `Foo::ProjectA` then `foo_project_a`.
+### Updating a single project or resource
+
+- Use `PROJECT=<kennel_id>` for single project:
+
+  Use the projects `kennel_id` (and if none is set then snake_case of the class name including modules)
+  to refer to the project. For example for `class ProjectA` use `PROJECT=project_a` but for `Foo::ProjectA` use `foo_project_a`.
+
+- Use `TRACKING_ID=<project-kennel_id>:<resource-kennel_id>` for single resource:
+
+  Use the project kennel_id and the resources kennel_id, for example `class ProjectA` and `FooAlert` would give `project_a:foo_alert`.
 
 ### Skipping validations
 Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kennel
 version: !ruby/object:Gem::Version
-  version: 1.115.0
+  version: 1.116.2
 platform: ruby
 authors:
 - Michael Grosser
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-08-28 00:00:00.000000000 Z
+date: 2022-08-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: faraday