kennel 1.53.0 → 1.56.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Readme.md +25 -20
- data/lib/kennel.rb +3 -1
- data/lib/kennel/importer.rb +3 -1
- data/lib/kennel/models/base.rb +3 -155
- data/lib/kennel/models/dashboard.rb +2 -11
- data/lib/kennel/models/monitor.rb +3 -10
- data/lib/kennel/models/project.rb +2 -2
- data/lib/kennel/models/record.rb +98 -0
- data/lib/kennel/models/team.rb +1 -10
- data/lib/kennel/optional_validations.rb +1 -10
- data/lib/kennel/settings_as_methods.rb +87 -0
- data/lib/kennel/syncer.rb +1 -6
- data/lib/kennel/tasks.rb +30 -0
- data/lib/kennel/utils.rb +9 -0
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +25 -19
- metadata +4 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 491e174273f8af94504a4e81d9a2ff7da5820aafe287ea9cb1e78df93d07f0f0
|
4
|
+
data.tar.gz: bf918b4be464d408259e0f8204541a0619d1719670a9843a6e564cc4edc968d1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 6ff9000e02764618a7e08c7349f315737778ff7afadf8ca303b7dda8db5648d2bd966412a255e7987ebd2416f8bfd264c3c2979bf143ed0d1f6fdd1a759fc85f
|
7
|
+
data.tar.gz: daf144b7fc436cfb317c4cc298dbbd62d61d252b554261746ed00bcaf5279422bf1a4ab185d45c57974665bc38ebbbf61d0b8e28d8269225c011beb11a451d67
|
data/Readme.md
CHANGED
@@ -42,33 +42,38 @@ Keep datadog monitors/dashboards/etc in version control, avoid chaotic managemen
|
|
42
42
|
- `gem install bundler && bundle install`
|
43
43
|
- `cp .env.example .env`
|
44
44
|
- open [Datadog API Settings](https://app.datadoghq.com/account/settings#api)
|
45
|
-
- find or create your personal "Application Key" and add it to `.env` as `DATADOG_APP_KEY=` (will be on the last page if new)
|
46
45
|
- copy any `API Key` and add it to `.env` as `DATADOG_API_KEY`
|
46
|
+
- find or create (check last page) your personal "Application Key" and add it to `.env` as `DATADOG_APP_KEY=`
|
47
|
+
- change the `DATADOG_SUBDOMAIN=app` in `.env` to your companies subdomain if you have one
|
47
48
|
-->
|
48
49
|
|
49
50
|
### Adding a team
|
50
51
|
|
52
|
+
- `mention` is used for all team monitors via `super()`
|
53
|
+
- `renotify_interval` is used for all team monitors (defaults to `0` / off)
|
54
|
+
- `tags` is used for all team monitors/dashboards (defaults to `team:<team-name>`)
|
55
|
+
|
51
56
|
```Ruby
|
52
57
|
# teams/my_team.rb
|
53
58
|
module Teams
|
54
59
|
class MyTeam < Kennel::Models::Team
|
55
60
|
defaults(
|
56
|
-
|
57
|
-
email: -> { "my-team@example.com" }
|
61
|
+
mention: -> { "@slack-my-team" }
|
58
62
|
)
|
59
63
|
end
|
60
64
|
end
|
61
65
|
```
|
62
66
|
|
63
67
|
### Adding a new monitor
|
64
|
-
- use [datadog monitor UI](https://app.datadoghq.com/monitors#create
|
65
|
-
- get the `id` from the url
|
66
|
-
- `RESOURCE=monitor ID=12345 bundle exec rake kennel:import`
|
68
|
+
- use [datadog monitor UI](https://app.datadoghq.com/monitors#create) to create a monitor
|
67
69
|
- see below
|
68
70
|
|
69
71
|
### Updating an existing monitor
|
72
|
+
- use [datadog monitor UI](https://app.datadoghq.com/monitors/manage) to find a monitor
|
73
|
+
- get the `id` from the url
|
74
|
+
- run `RESOURCE=monitor ID=12345 bundle exec rake kennel:import` and copy the output
|
70
75
|
- find or create a project in `projects/`
|
71
|
-
- add
|
76
|
+
- add the monitor to `parts: [` list, for example:
|
72
77
|
```Ruby
|
73
78
|
# projects/my_project.rb
|
74
79
|
class MyProject < Kennel::Models::Project
|
@@ -83,7 +88,8 @@ end
|
|
83
88
|
kennel_id: -> { "load-too-high" }, # make up a unique name
|
84
89
|
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
85
90
|
message: -> {
|
86
|
-
# Explain what behavior to expect and how to fix the cause
|
91
|
+
# Explain what behavior to expect and how to fix the cause
|
92
|
+
# Use #{super()} to add team notifications.
|
87
93
|
<<~TEXT
|
88
94
|
Foobar will be slow and that could cause Barfoo to go down.
|
89
95
|
Add capacity or debug why it is suddenly slow.
|
@@ -98,21 +104,22 @@ end
|
|
98
104
|
)
|
99
105
|
end
|
100
106
|
```
|
101
|
-
- `bundle exec rake plan
|
102
|
-
- alternatively: `bundle exec rake generate` to only update the generated `json` files
|
107
|
+
- run `PROJECT=my_project bundle exec rake plan`, an Update to the existing monitor should be shown (not Create / Delete)
|
108
|
+
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
103
109
|
- review changes then `git commit`
|
104
110
|
- make a PR ... get reviewed ... merge
|
105
111
|
- datadog is updated by travis
|
106
112
|
|
107
113
|
### Adding a new dashboard
|
108
114
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
109
|
-
- get the `id` from the url
|
110
|
-
- `RESOURCE=dashboard ID=abc-def-ghi bundle exec rake kennel:import`
|
111
115
|
- see below
|
112
116
|
|
113
117
|
### Updating an existing dashboard
|
118
|
+
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to find a dashboard
|
119
|
+
- get the `id` from the url
|
120
|
+
- run `RESOURCE=dashboard ID=abc-def-ghi bundle exec rake kennel:import` and copy the output
|
114
121
|
- find or create a project in `projects/`
|
115
|
-
- add a dashboard to `parts: [` list
|
122
|
+
- add a dashboard to `parts: [` list, for example:
|
116
123
|
```Ruby
|
117
124
|
class MyProject < Kennel::Models::Project
|
118
125
|
defaults(
|
@@ -154,12 +161,6 @@ end
|
|
154
161
|
Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and
|
155
162
|
to unblock use the `validate: -> { false }` option.
|
156
163
|
|
157
|
-
### Monitor re-notification
|
158
|
-
|
159
|
-
Monitors inherit the re-notification setting from their `project.team`.
|
160
|
-
Set this to for example `renotify_interval: -> { 120 }` minutes,
|
161
|
-
to make alerts not get ignored by popping back up if they are still alerting.
|
162
|
-
|
163
164
|
### Linking with kennel_ids
|
164
165
|
|
165
166
|
To link to existing monitors via their kennel_id
|
@@ -178,6 +179,10 @@ To link to existing monitors via their kennel_id
|
|
178
179
|
|
179
180
|
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
180
181
|
|
182
|
+
### Validating mentions work
|
183
|
+
|
184
|
+
`rake kennel:validate_mentions` should run as part of CI
|
185
|
+
|
181
186
|
## Examples
|
182
187
|
|
183
188
|
### Reusable monitors/dashes/etc
|
@@ -202,7 +207,7 @@ Reuse it in multiple projects.
|
|
202
207
|
```Ruby
|
203
208
|
class Database < Kennel::Models::Project
|
204
209
|
defaults(
|
205
|
-
team: -> { Kennel::Models::Team.new(
|
210
|
+
team: -> { Kennel::Models::Team.new(mention: -> { '@slack-foo' }, kennel_id: -> { 'foo' }) },
|
206
211
|
parts: -> { [Monitors::LoadTooHigh.new(self, critical: -> { 13 })] }
|
207
212
|
)
|
208
213
|
end
|
data/lib/kennel.rb
CHANGED
@@ -10,14 +10,16 @@ require "kennel/syncer"
|
|
10
10
|
require "kennel/api"
|
11
11
|
require "kennel/github_reporter"
|
12
12
|
require "kennel/subclass_tracking"
|
13
|
+
require "kennel/settings_as_methods"
|
13
14
|
require "kennel/file_cache"
|
14
15
|
require "kennel/template_variables"
|
15
16
|
require "kennel/optional_validations"
|
16
17
|
require "kennel/unmuted_alerts"
|
17
18
|
|
18
19
|
require "kennel/models/base"
|
20
|
+
require "kennel/models/record"
|
19
21
|
|
20
|
-
#
|
22
|
+
# records
|
21
23
|
require "kennel/models/monitor"
|
22
24
|
require "kennel/models/dashboard"
|
23
25
|
|
data/lib/kennel/importer.rb
CHANGED
@@ -28,7 +28,7 @@ module Kennel
|
|
28
28
|
|
29
29
|
title_field = TITLES.detect { |f| data[f] }
|
30
30
|
title = data.fetch(title_field)
|
31
|
-
title.tr!(Kennel::Models::
|
31
|
+
title.tr!(Kennel::Models::Record::LOCK, "") # avoid double lock icon
|
32
32
|
|
33
33
|
# calculate or reuse kennel_id
|
34
34
|
# TODO: this is copy-pasted from syncer, need to find a nice way to reuse it
|
@@ -53,6 +53,8 @@ module Kennel
|
|
53
53
|
if query && critical
|
54
54
|
query.sub!(/([><=]) (#{Regexp.escape(critical.to_f.to_s)}|#{Regexp.escape(critical.to_i.to_s)})$/, "\\1 \#{critical}")
|
55
55
|
end
|
56
|
+
|
57
|
+
data[:type] = "query alert" if data[:type] == "metric alert"
|
56
58
|
elsif resource == "dashboard"
|
57
59
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
58
60
|
widgets&.each { |widget| dry_up_query!(widget) }
|
data/lib/kennel/models/base.rb
CHANGED
@@ -4,115 +4,10 @@ require "hashdiff"
|
|
4
4
|
module Kennel
|
5
5
|
module Models
|
6
6
|
class Base
|
7
|
-
|
8
|
-
|
9
|
-
:deleted, :matching_downtimes, :id, :created, :created_at, :creator, :org_id, :modified,
|
10
|
-
:overall_state_modified, :overall_state, :api_resource
|
11
|
-
].freeze
|
12
|
-
REQUEST_DEFAULTS = {
|
13
|
-
style: { width: "normal", palette: "dog_classic", type: "solid" },
|
14
|
-
conditional_formats: [],
|
15
|
-
aggregator: "avg"
|
16
|
-
}.freeze
|
17
|
-
OVERRIDABLE_METHODS = [:name, :kennel_id].freeze
|
7
|
+
extend SubclassTracking
|
8
|
+
include SettingsAsMethods
|
18
9
|
|
19
|
-
|
20
|
-
end
|
21
|
-
|
22
|
-
class << self
|
23
|
-
include SubclassTracking
|
24
|
-
|
25
|
-
def settings(*names)
|
26
|
-
duplicates = (@set & names)
|
27
|
-
if duplicates.any?
|
28
|
-
raise ArgumentError, "Settings #{duplicates.map(&:inspect).join(", ")} are already defined"
|
29
|
-
end
|
30
|
-
|
31
|
-
overrides = ((instance_methods - OVERRIDABLE_METHODS) & names)
|
32
|
-
if overrides.any?
|
33
|
-
raise ArgumentError, "Settings #{overrides.map(&:inspect).join(", ")} are already used as methods"
|
34
|
-
end
|
35
|
-
|
36
|
-
@set.concat names
|
37
|
-
names.each do |name|
|
38
|
-
next if method_defined?(name)
|
39
|
-
define_method name do
|
40
|
-
message = "Trying to call #{name} for #{self.class} but it was never set or passed as option"
|
41
|
-
raise_with_location ArgumentError, message
|
42
|
-
end
|
43
|
-
end
|
44
|
-
end
|
45
|
-
|
46
|
-
def defaults(options)
|
47
|
-
options.each do |name, block|
|
48
|
-
validate_setting_exists name
|
49
|
-
define_method name, &block
|
50
|
-
end
|
51
|
-
end
|
52
|
-
|
53
|
-
def inherited(child)
|
54
|
-
super
|
55
|
-
child.instance_variable_set(:@set, (@set || []).dup)
|
56
|
-
end
|
57
|
-
|
58
|
-
def validate_setting_exists(name)
|
59
|
-
return if !@set || @set.include?(name)
|
60
|
-
supported = @set.map(&:inspect)
|
61
|
-
raise ArgumentError, "Unsupported setting #{name.inspect}, supported settings are #{supported.join(", ")}"
|
62
|
-
end
|
63
|
-
|
64
|
-
private
|
65
|
-
|
66
|
-
def normalize(_expected, actual)
|
67
|
-
self::READONLY_ATTRIBUTES.each { |k| actual.delete k }
|
68
|
-
end
|
69
|
-
|
70
|
-
# discard styles/conditional_formats/aggregator if nothing would change when we applied (both are default or nil)
|
71
|
-
def ignore_request_defaults(expected, actual, level1, level2)
|
72
|
-
actual = actual[level1] || {}
|
73
|
-
expected = expected[level1] || {}
|
74
|
-
[expected.size.to_i, actual.size.to_i].max.times do |i|
|
75
|
-
a_r = actual.dig(i, level2, :requests) || []
|
76
|
-
e_r = expected.dig(i, level2, :requests) || []
|
77
|
-
ignore_defaults e_r, a_r, self::REQUEST_DEFAULTS
|
78
|
-
end
|
79
|
-
end
|
80
|
-
|
81
|
-
def ignore_defaults(expected, actual, defaults)
|
82
|
-
[expected&.size.to_i, actual&.size.to_i].max.times do |i|
|
83
|
-
e = expected[i] || {}
|
84
|
-
a = actual[i] || {}
|
85
|
-
ignore_default(e, a, defaults)
|
86
|
-
end
|
87
|
-
end
|
88
|
-
|
89
|
-
def ignore_default(expected, actual, defaults)
|
90
|
-
definitions = [actual, expected]
|
91
|
-
defaults.each do |key, default|
|
92
|
-
if definitions.all? { |r| !r.key?(key) || r[key] == default }
|
93
|
-
actual.delete(key)
|
94
|
-
expected.delete(key)
|
95
|
-
end
|
96
|
-
end
|
97
|
-
end
|
98
|
-
end
|
99
|
-
|
100
|
-
def initialize(options = {})
|
101
|
-
validate_options(options)
|
102
|
-
|
103
|
-
options.each do |name, block|
|
104
|
-
self.class.validate_setting_exists name
|
105
|
-
define_singleton_method name, &block
|
106
|
-
end
|
107
|
-
|
108
|
-
# need expand_path so it works wih rake and when run individually
|
109
|
-
pwd = /^#{Regexp.escape(Dir.pwd)}\//
|
110
|
-
@invocation_location = caller.detect do |l|
|
111
|
-
if found = File.expand_path(l).sub!(pwd, "")
|
112
|
-
break found
|
113
|
-
end
|
114
|
-
end
|
115
|
-
end
|
10
|
+
SETTING_OVERRIDABLE_METHODS = [:name, :kennel_id].freeze
|
116
11
|
|
117
12
|
def kennel_id
|
118
13
|
name = self.class.name
|
@@ -126,56 +21,9 @@ module Kennel
|
|
126
21
|
self.class.name
|
127
22
|
end
|
128
23
|
|
129
|
-
def diff(actual)
|
130
|
-
expected = as_json
|
131
|
-
expected.delete(:id)
|
132
|
-
|
133
|
-
self.class.send(:normalize, expected, actual)
|
134
|
-
|
135
|
-
HashDiff.diff(actual, expected, use_lcs: false)
|
136
|
-
end
|
137
|
-
|
138
|
-
def tracking_id
|
139
|
-
"#{project.kennel_id}:#{kennel_id}"
|
140
|
-
end
|
141
|
-
|
142
24
|
def to_json
|
143
25
|
raise NotImplementedError, "Use as_json"
|
144
26
|
end
|
145
|
-
|
146
|
-
def resolve_linked_tracking_ids(*)
|
147
|
-
end
|
148
|
-
|
149
|
-
private
|
150
|
-
|
151
|
-
def resolve_link(id, id_map, force:)
|
152
|
-
id_map[id] || begin
|
153
|
-
message = "Unable to find #{id} in existing monitors (they need to be created first to link them)"
|
154
|
-
force ? invalid!(message) : Kennel.err.puts(message)
|
155
|
-
end
|
156
|
-
end
|
157
|
-
|
158
|
-
# let users know which project/resource failed when something happens during diffing where the backtrace is hidden
|
159
|
-
def invalid!(message)
|
160
|
-
raise ValidationError, "#{tracking_id} #{message}"
|
161
|
-
end
|
162
|
-
|
163
|
-
def raise_with_location(error, message)
|
164
|
-
message = message.dup
|
165
|
-
message << " for project #{project.kennel_id}" if defined?(project)
|
166
|
-
message << " on #{@invocation_location}" if @invocation_location
|
167
|
-
raise error, message
|
168
|
-
end
|
169
|
-
|
170
|
-
def validate_options(options)
|
171
|
-
unless options.is_a?(Hash)
|
172
|
-
raise ArgumentError, "Expected #{self.class.name}.new options to be a Hash, got a #{options.class}"
|
173
|
-
end
|
174
|
-
options.each do |k, v|
|
175
|
-
next if v.class == Proc
|
176
|
-
raise ArgumentError, "Expected #{self.class.name}.new option :#{k} to be Proc, for example `#{k}: -> { 12 }`"
|
177
|
-
end
|
178
|
-
end
|
179
27
|
end
|
180
28
|
end
|
181
29
|
end
|
@@ -1,13 +1,13 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
module Kennel
|
3
3
|
module Models
|
4
|
-
class Dashboard <
|
4
|
+
class Dashboard < Record
|
5
5
|
include TemplateVariables
|
6
6
|
include OptionalValidations
|
7
7
|
|
8
8
|
API_LIST_INCOMPLETE = true
|
9
9
|
DASHBOARD_DEFAULTS = { template_variables: [] }.freeze
|
10
|
-
READONLY_ATTRIBUTES =
|
10
|
+
READONLY_ATTRIBUTES = superclass::READONLY_ATTRIBUTES + [
|
11
11
|
:author_handle, :author_name, :modified_at, :url, :is_read_only, :notify_list
|
12
12
|
]
|
13
13
|
REQUEST_DEFAULTS = {
|
@@ -32,8 +32,6 @@ module Kennel
|
|
32
32
|
def normalize(expected, actual)
|
33
33
|
super
|
34
34
|
|
35
|
-
ignore_default expected, actual, DASHBOARD_DEFAULTS
|
36
|
-
|
37
35
|
base_pairs(expected, actual).each do |pair|
|
38
36
|
# conditional_formats ordering is randomly changed by datadog, compare a stable ordering
|
39
37
|
pair.each do |b|
|
@@ -64,13 +62,6 @@ module Kennel
|
|
64
62
|
end
|
65
63
|
end
|
66
64
|
|
67
|
-
attr_reader :project
|
68
|
-
|
69
|
-
def initialize(project, *args)
|
70
|
-
@project = project
|
71
|
-
super(*args)
|
72
|
-
end
|
73
|
-
|
74
65
|
def as_json
|
75
66
|
return @json if @json
|
76
67
|
all_widgets = render_definitions + widgets
|
@@ -1,14 +1,14 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
module Kennel
|
3
3
|
module Models
|
4
|
-
class Monitor <
|
4
|
+
class Monitor < Record
|
5
5
|
include OptionalValidations
|
6
6
|
|
7
7
|
API_LIST_INCOMPLETE = false
|
8
8
|
RENOTIFY_INTERVALS = [0, 10, 20, 30, 40, 50, 60, 90, 120, 180, 240, 300, 360, 720, 1440].freeze # minutes
|
9
9
|
QUERY_INTERVALS = ["1m", "5m", "10m", "15m", "30m", "1h", "2h", "4h", "1d"].freeze
|
10
10
|
OPTIONAL_SERVICE_CHECK_THRESHOLDS = [:ok, :warning].freeze
|
11
|
-
READONLY_ATTRIBUTES =
|
11
|
+
READONLY_ATTRIBUTES = superclass::READONLY_ATTRIBUTES + [:multi]
|
12
12
|
|
13
13
|
# defaults that datadog uses when options are not sent, so safe to leave out if our values match their defaults
|
14
14
|
MONITOR_OPTION_DEFAULTS = {
|
@@ -27,7 +27,7 @@ module Kennel
|
|
27
27
|
)
|
28
28
|
|
29
29
|
defaults(
|
30
|
-
message: -> { "\n\n
|
30
|
+
message: -> { "\n\n#{project.mention}" },
|
31
31
|
escalation_message: -> { DEFAULT_ESCALATION_MESSAGE.first },
|
32
32
|
renotify_interval: -> { project.team.renotify_interval },
|
33
33
|
warning: -> { nil },
|
@@ -45,13 +45,6 @@ module Kennel
|
|
45
45
|
threshold_windows: -> { nil }
|
46
46
|
)
|
47
47
|
|
48
|
-
attr_reader :project
|
49
|
-
|
50
|
-
def initialize(project, *args)
|
51
|
-
@project = project
|
52
|
-
super(*args)
|
53
|
-
end
|
54
|
-
|
55
48
|
def as_json
|
56
49
|
return @as_json if @as_json
|
57
50
|
data = {
|
@@ -2,10 +2,10 @@
|
|
2
2
|
module Kennel
|
3
3
|
module Models
|
4
4
|
class Project < Base
|
5
|
-
settings :team, :parts, :tags, :
|
5
|
+
settings :team, :parts, :tags, :mention
|
6
6
|
defaults(
|
7
7
|
tags: -> { ["service:#{kennel_id}"] + team.tags },
|
8
|
-
|
8
|
+
mention: -> { team.mention }
|
9
9
|
)
|
10
10
|
|
11
11
|
def self.file_location
|
@@ -0,0 +1,98 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
module Kennel
|
3
|
+
module Models
|
4
|
+
class Record < Base
|
5
|
+
LOCK = "\u{1F512}"
|
6
|
+
READONLY_ATTRIBUTES = [
|
7
|
+
:deleted, :matching_downtimes, :id, :created, :created_at, :creator, :org_id, :modified,
|
8
|
+
:overall_state_modified, :overall_state, :api_resource
|
9
|
+
].freeze
|
10
|
+
REQUEST_DEFAULTS = {
|
11
|
+
style: { width: "normal", palette: "dog_classic", type: "solid" },
|
12
|
+
conditional_formats: [],
|
13
|
+
aggregator: "avg"
|
14
|
+
}.freeze
|
15
|
+
|
16
|
+
class ValidationError < RuntimeError
|
17
|
+
end
|
18
|
+
|
19
|
+
class << self
|
20
|
+
private
|
21
|
+
|
22
|
+
def normalize(_expected, actual)
|
23
|
+
self::READONLY_ATTRIBUTES.each { |k| actual.delete k }
|
24
|
+
end
|
25
|
+
|
26
|
+
# discard styles/conditional_formats/aggregator if nothing would change when we applied (both are default or nil)
|
27
|
+
def ignore_request_defaults(expected, actual, level1, level2)
|
28
|
+
actual = actual[level1] || {}
|
29
|
+
expected = expected[level1] || {}
|
30
|
+
[expected.size.to_i, actual.size.to_i].max.times do |i|
|
31
|
+
a_r = actual.dig(i, level2, :requests) || []
|
32
|
+
e_r = expected.dig(i, level2, :requests) || []
|
33
|
+
ignore_defaults e_r, a_r, self::REQUEST_DEFAULTS
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
def ignore_defaults(expected, actual, defaults)
|
38
|
+
[expected&.size.to_i, actual&.size.to_i].max.times do |i|
|
39
|
+
e = expected[i] || {}
|
40
|
+
a = actual[i] || {}
|
41
|
+
ignore_default(e, a, defaults)
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def ignore_default(expected, actual, defaults)
|
46
|
+
definitions = [actual, expected]
|
47
|
+
defaults.each do |key, default|
|
48
|
+
if definitions.all? { |r| !r.key?(key) || r[key] == default }
|
49
|
+
actual.delete(key)
|
50
|
+
expected.delete(key)
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
attr_reader :project
|
57
|
+
|
58
|
+
def initialize(project, *args)
|
59
|
+
@project = project
|
60
|
+
super(*args)
|
61
|
+
end
|
62
|
+
|
63
|
+
def diff(actual)
|
64
|
+
expected = as_json
|
65
|
+
expected.delete(:id)
|
66
|
+
|
67
|
+
self.class.send(:normalize, expected, actual)
|
68
|
+
|
69
|
+
HashDiff.diff(actual, expected, use_lcs: false)
|
70
|
+
end
|
71
|
+
|
72
|
+
def tracking_id
|
73
|
+
"#{project.kennel_id}:#{kennel_id}"
|
74
|
+
end
|
75
|
+
|
76
|
+
def resolve_linked_tracking_ids(*)
|
77
|
+
end
|
78
|
+
|
79
|
+
private
|
80
|
+
|
81
|
+
def resolve_link(id, id_map, force:)
|
82
|
+
id_map[id] || begin
|
83
|
+
message = "Unable to find #{id} in existing monitors (they need to be created first to link them)"
|
84
|
+
force ? invalid!(message) : Kennel.err.puts(message)
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
# let users know which project/resource failed when something happens during diffing where the backtrace is hidden
|
89
|
+
def invalid!(message)
|
90
|
+
raise ValidationError, "#{tracking_id} #{message}"
|
91
|
+
end
|
92
|
+
|
93
|
+
def raise_with_location(error, message)
|
94
|
+
super error, "#{message} for project #{project.kennel_id}"
|
95
|
+
end
|
96
|
+
end
|
97
|
+
end
|
98
|
+
end
|
data/lib/kennel/models/team.rb
CHANGED
@@ -2,20 +2,11 @@
|
|
2
2
|
module Kennel
|
3
3
|
module Models
|
4
4
|
class Team < Base
|
5
|
-
settings :
|
5
|
+
settings :mention, :tags, :renotify_interval, :kennel_id
|
6
6
|
defaults(
|
7
7
|
tags: -> { ["team:#{kennel_id.sub(/^teams_/, "")}"] },
|
8
8
|
renotify_interval: -> { 0 }
|
9
9
|
)
|
10
|
-
|
11
|
-
def initialize(*)
|
12
|
-
super
|
13
|
-
invalid! "remove leading # from slack" if slack.to_s.start_with?("#")
|
14
|
-
end
|
15
|
-
|
16
|
-
def tracking_id
|
17
|
-
kennel_id
|
18
|
-
end
|
19
10
|
end
|
20
11
|
end
|
21
12
|
end
|
@@ -6,19 +6,10 @@ module Kennel
|
|
6
6
|
base.defaults(validate: -> { true })
|
7
7
|
end
|
8
8
|
|
9
|
-
# https://stackoverflow.com/questions/20235206/ruby-get-all-keys-in-a-hash-including-sub-keys/53876255#53876255
|
10
|
-
def self.all_keys(items)
|
11
|
-
case items
|
12
|
-
when Hash then items.keys + items.values.flat_map { |v| all_keys(v) }
|
13
|
-
when Array then items.flat_map { |i| all_keys(i) }
|
14
|
-
else []
|
15
|
-
end
|
16
|
-
end
|
17
|
-
|
18
9
|
private
|
19
10
|
|
20
11
|
def validate_json(data)
|
21
|
-
bad =
|
12
|
+
bad = Kennel::Utils.all_keys(data).grep_v(Symbol)
|
22
13
|
return if bad.empty?
|
23
14
|
invalid!(
|
24
15
|
"Only use Symbols as hash keys to avoid permanent diffs when updating.\n" \
|
@@ -0,0 +1,87 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
module Kennel
|
3
|
+
module SettingsAsMethods
|
4
|
+
SETTING_OVERRIDABLE_METHODS = [].freeze
|
5
|
+
|
6
|
+
def self.included(base)
|
7
|
+
base.extend ClassMethods
|
8
|
+
base.instance_variable_set(:@settings, [])
|
9
|
+
end
|
10
|
+
|
11
|
+
module ClassMethods
|
12
|
+
def settings(*names)
|
13
|
+
duplicates = (@settings & names)
|
14
|
+
if duplicates.any?
|
15
|
+
raise ArgumentError, "Settings #{duplicates.map(&:inspect).join(", ")} are already defined"
|
16
|
+
end
|
17
|
+
|
18
|
+
overrides = ((instance_methods - self::SETTING_OVERRIDABLE_METHODS) & names)
|
19
|
+
if overrides.any?
|
20
|
+
raise ArgumentError, "Settings #{overrides.map(&:inspect).join(", ")} are already used as methods"
|
21
|
+
end
|
22
|
+
|
23
|
+
@settings.concat names
|
24
|
+
|
25
|
+
names.each do |name|
|
26
|
+
next if method_defined?(name)
|
27
|
+
define_method name do
|
28
|
+
message = "Trying to call #{name} for #{self.class} but it was never set or passed as option"
|
29
|
+
raise_with_location ArgumentError, message
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def defaults(options)
|
35
|
+
options.each do |name, block|
|
36
|
+
validate_setting_exist name
|
37
|
+
define_method name, &block
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
private
|
42
|
+
|
43
|
+
def validate_setting_exist(name)
|
44
|
+
return if !@settings || @settings.include?(name)
|
45
|
+
supported = @settings.map(&:inspect)
|
46
|
+
raise ArgumentError, "Unsupported setting #{name.inspect}, supported settings are #{supported.join(", ")}"
|
47
|
+
end
|
48
|
+
|
49
|
+
def inherited(child)
|
50
|
+
super
|
51
|
+
child.instance_variable_set(:@settings, (@settings || []).dup)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def initialize(options = {})
|
56
|
+
super()
|
57
|
+
|
58
|
+
unless options.is_a?(Hash)
|
59
|
+
raise ArgumentError, "Expected #{self.class.name}.new options to be a Hash, got a #{options.class}"
|
60
|
+
end
|
61
|
+
|
62
|
+
options.each do |k, v|
|
63
|
+
next if v.class == Proc
|
64
|
+
raise ArgumentError, "Expected #{self.class.name}.new option :#{k} to be Proc, for example `#{k}: -> { 12 }`"
|
65
|
+
end
|
66
|
+
|
67
|
+
options.each do |name, block|
|
68
|
+
self.class.send :validate_setting_exist, name
|
69
|
+
define_singleton_method name, &block
|
70
|
+
end
|
71
|
+
|
72
|
+
# need expand_path so it works wih rake and when run individually
|
73
|
+
pwd = /^#{Regexp.escape(Dir.pwd)}\//
|
74
|
+
@invocation_location = caller.detect do |l|
|
75
|
+
if found = File.expand_path(l).sub!(pwd, "")
|
76
|
+
break found
|
77
|
+
end
|
78
|
+
end
|
79
|
+
end
|
80
|
+
|
81
|
+
def raise_with_location(error, message)
|
82
|
+
message = message.dup
|
83
|
+
message << " on #{@invocation_location}" if @invocation_location
|
84
|
+
raise error, message
|
85
|
+
end
|
86
|
+
end
|
87
|
+
end
|
data/lib/kennel/syncer.rb
CHANGED
@@ -122,12 +122,7 @@ module Kennel
|
|
122
122
|
end
|
123
123
|
|
124
124
|
def download_definitions
|
125
|
-
|
126
|
-
next unless m.respond_to?(:api_resource)
|
127
|
-
m.api_resource
|
128
|
-
end
|
129
|
-
|
130
|
-
Utils.parallel(api_resources.compact.uniq) do |api_resource|
|
125
|
+
Utils.parallel(Models::Record.subclasses.map(&:api_resource)) do |api_resource|
|
131
126
|
results = @api.list(api_resource, with_downtimes: false) # lookup monitors without adding unnecessary downtime information
|
132
127
|
results = results[results.keys.first] if results.is_a?(Hash) # dashes/screens are nested in {dash: {}}
|
133
128
|
results.each { |c| c[:api_resource] = api_resource } # store api resource for later diffing
|
data/lib/kennel/tasks.rb
CHANGED
@@ -12,6 +12,36 @@ namespace :kennel do
|
|
12
12
|
abort "Error during diffing" unless $CHILD_STATUS.success?
|
13
13
|
end
|
14
14
|
|
15
|
+
# ideally do this on every run, but it's slow (~1.5s) and brittle (might not find all + might find false-positives)
|
16
|
+
# https://help.datadoghq.com/hc/en-us/requests/254114 for automatic validation
|
17
|
+
desc "Verify that all used monitor mentions are valid"
|
18
|
+
task validate_mentions: :environment do
|
19
|
+
known = Kennel.send(:api)
|
20
|
+
.send(:request, :get, "/monitor/notifications")
|
21
|
+
.fetch(:handles)
|
22
|
+
.values
|
23
|
+
.flatten(1)
|
24
|
+
.map { |v| v.fetch(:value) }
|
25
|
+
|
26
|
+
known += ENV["KNOWN"].to_s.split(",")
|
27
|
+
|
28
|
+
bad = []
|
29
|
+
Dir["generated/**/*.json"].each do |f|
|
30
|
+
next unless message = JSON.parse(File.read(f))["message"]
|
31
|
+
used = message.scan(/\s(@[^\s{,'"]+)/).flatten(1)
|
32
|
+
.grep(/^@.*@|^@.*-/) # ignore @here etc handles ... datadog uses @foo@bar.com for emails and @foo-bar for integrations
|
33
|
+
(used - known).each { |v| bad << [f, v] }
|
34
|
+
end
|
35
|
+
|
36
|
+
if bad.any?
|
37
|
+
subdomain = ENV["DATADOG_SUBDOMAIN"]
|
38
|
+
url = (subdomain ? "https://zendesk.datadoghq.com" : "") + "/account/settings"
|
39
|
+
puts "Invalid mentions found, either ignore them by adding to `KNOWN` env var or add them via #{url}"
|
40
|
+
bad.each { |f, v| puts "Invalid mention #{v} in monitor message of #{f}" }
|
41
|
+
abort
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
15
45
|
desc "generate local definitions"
|
16
46
|
task generate: :environment do
|
17
47
|
Kennel.generate
|
data/lib/kennel/utils.rb
CHANGED
@@ -133,6 +133,15 @@ module Kennel
|
|
133
133
|
Kennel.err.puts "Error #{e}, #{times} retries left"
|
134
134
|
retry
|
135
135
|
end
|
136
|
+
|
137
|
+
# https://stackoverflow.com/questions/20235206/ruby-get-all-keys-in-a-hash-including-sub-keys/53876255#53876255
|
138
|
+
def all_keys(items)
|
139
|
+
case items
|
140
|
+
when Hash then items.keys + items.values.flat_map { |v| all_keys(v) }
|
141
|
+
when Array then items.flat_map { |i| all_keys(i) }
|
142
|
+
else []
|
143
|
+
end
|
144
|
+
end
|
136
145
|
end
|
137
146
|
end
|
138
147
|
end
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -25,32 +25,37 @@ Keep datadog monitors/dashboards/etc in version control, avoid chaotic managemen
|
|
25
25
|
- `gem install bundler && bundle install`
|
26
26
|
- `cp .env.example .env`
|
27
27
|
- open [Datadog API Settings](https://app.datadoghq.com/account/settings#api)
|
28
|
-
- find or create your personal "Application Key" and add it to `.env` as `DATADOG_APP_KEY=` (will be on the last page if new)
|
29
28
|
- copy any `API Key` and add it to `.env` as `DATADOG_API_KEY`
|
29
|
+
- find or create (check last page) your personal "Application Key" and add it to `.env` as `DATADOG_APP_KEY=`
|
30
|
+
- change the `DATADOG_SUBDOMAIN=app` in `.env` to your companies subdomain if you have one
|
30
31
|
|
31
32
|
### Adding a team
|
32
33
|
|
34
|
+
- `mention` is used for all team monitors via `super()`
|
35
|
+
- `renotify_interval` is used for all team monitors (defaults to `0` / off)
|
36
|
+
- `tags` is used for all team monitors/dashboards (defaults to `team:<team-name>`)
|
37
|
+
|
33
38
|
```Ruby
|
34
39
|
# teams/my_team.rb
|
35
40
|
module Teams
|
36
41
|
class MyTeam < Kennel::Models::Team
|
37
42
|
defaults(
|
38
|
-
|
39
|
-
email: -> { "my-team@example.com" }
|
43
|
+
mention: -> { "@slack-my-team" }
|
40
44
|
)
|
41
45
|
end
|
42
46
|
end
|
43
47
|
```
|
44
48
|
|
45
49
|
### Adding a new monitor
|
46
|
-
- use [datadog monitor UI](https://app.datadoghq.com/monitors#create
|
47
|
-
- get the `id` from the url
|
48
|
-
- `RESOURCE=monitor ID=12345 bundle exec rake kennel:import`
|
50
|
+
- use [datadog monitor UI](https://app.datadoghq.com/monitors#create) to create a monitor
|
49
51
|
- see below
|
50
52
|
|
51
53
|
### Updating an existing monitor
|
54
|
+
- use [datadog monitor UI](https://app.datadoghq.com/monitors/manage) to find a monitor
|
55
|
+
- get the `id` from the url
|
56
|
+
- run `RESOURCE=monitor ID=12345 bundle exec rake kennel:import` and copy the output
|
52
57
|
- find or create a project in `projects/`
|
53
|
-
- add
|
58
|
+
- add the monitor to `parts: [` list, for example:
|
54
59
|
```Ruby
|
55
60
|
# projects/my_project.rb
|
56
61
|
class MyProject < Kennel::Models::Project
|
@@ -65,7 +70,8 @@ end
|
|
65
70
|
kennel_id: -> { "load-too-high" }, # make up a unique name
|
66
71
|
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
67
72
|
message: -> {
|
68
|
-
# Explain what behavior to expect and how to fix the cause
|
73
|
+
# Explain what behavior to expect and how to fix the cause
|
74
|
+
# Use #{super()} to add team notifications.
|
69
75
|
<<~TEXT
|
70
76
|
Foobar will be slow and that could cause Barfoo to go down.
|
71
77
|
Add capacity or debug why it is suddenly slow.
|
@@ -80,21 +86,22 @@ end
|
|
80
86
|
)
|
81
87
|
end
|
82
88
|
```
|
83
|
-
- `bundle exec rake plan
|
84
|
-
- alternatively: `bundle exec rake generate` to only update the generated `json` files
|
89
|
+
- run `PROJECT=my_project bundle exec rake plan`, an Update to the existing monitor should be shown (not Create / Delete)
|
90
|
+
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
85
91
|
- review changes then `git commit`
|
86
92
|
- make a PR ... get reviewed ... merge
|
87
93
|
- datadog is updated by travis
|
88
94
|
|
89
95
|
### Adding a new dashboard
|
90
96
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
91
|
-
- get the `id` from the url
|
92
|
-
- `RESOURCE=dashboard ID=abc-def-ghi bundle exec rake kennel:import`
|
93
97
|
- see below
|
94
98
|
|
95
99
|
### Updating an existing dashboard
|
100
|
+
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to find a dashboard
|
101
|
+
- get the `id` from the url
|
102
|
+
- run `RESOURCE=dashboard ID=abc-def-ghi bundle exec rake kennel:import` and copy the output
|
96
103
|
- find or create a project in `projects/`
|
97
|
-
- add a dashboard to `parts: [` list
|
104
|
+
- add a dashboard to `parts: [` list, for example:
|
98
105
|
```Ruby
|
99
106
|
class MyProject < Kennel::Models::Project
|
100
107
|
defaults(
|
@@ -136,11 +143,6 @@ end
|
|
136
143
|
Some validations might be too strict for your usecase or just wrong, please [open an issue](https://github.com/grosser/kennel/issues) and
|
137
144
|
to unblock use the `validate: -> { false }` option.
|
138
145
|
|
139
|
-
### Monitor re-notification
|
140
|
-
|
141
|
-
Monitors inherit the re-notification setting from their projects team.
|
142
|
-
By default this is `renotify_interval: -> { 120 }` minutes, which will make alerts not get ignored by popping back up.
|
143
|
-
|
144
146
|
### Linking with kennel_ids
|
145
147
|
|
146
148
|
To link to existing monitors via their kennel_id
|
@@ -159,6 +161,10 @@ To link to existing monitors via their kennel_id
|
|
159
161
|
|
160
162
|
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
161
163
|
|
164
|
+
### Validating mentions work
|
165
|
+
|
166
|
+
`rake kennel:validate_mentions` should run as part of CI
|
167
|
+
|
162
168
|
## Examples
|
163
169
|
|
164
170
|
### Reusable monitors/dashes/etc
|
@@ -183,7 +189,7 @@ Reuse it in multiple projects.
|
|
183
189
|
```Ruby
|
184
190
|
class Database < Kennel::Models::Project
|
185
191
|
defaults(
|
186
|
-
team: -> { Kennel::Models::Team.new(
|
192
|
+
team: -> { Kennel::Models::Team.new(mention: -> { '@slack-foo' }, kennel_id: -> { 'foo' }) },
|
187
193
|
parts: -> { [Monitors::LoadTooHigh.new(self, critical: -> { 13 })] }
|
188
194
|
)
|
189
195
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.56.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-
|
11
|
+
date: 2019-10-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|
@@ -68,9 +68,11 @@ files:
|
|
68
68
|
- lib/kennel/models/dashboard.rb
|
69
69
|
- lib/kennel/models/monitor.rb
|
70
70
|
- lib/kennel/models/project.rb
|
71
|
+
- lib/kennel/models/record.rb
|
71
72
|
- lib/kennel/models/team.rb
|
72
73
|
- lib/kennel/optional_validations.rb
|
73
74
|
- lib/kennel/progress.rb
|
75
|
+
- lib/kennel/settings_as_methods.rb
|
74
76
|
- lib/kennel/subclass_tracking.rb
|
75
77
|
- lib/kennel/syncer.rb
|
76
78
|
- lib/kennel/tasks.rb
|