kennel 1.73.0 → 1.76.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Readme.md +75 -29
- data/lib/kennel.rb +0 -1
- data/lib/kennel/importer.rb +12 -4
- data/lib/kennel/models/dashboard.rb +4 -4
- data/lib/kennel/models/monitor.rb +9 -8
- data/lib/kennel/models/record.rb +9 -10
- data/lib/kennel/models/slo.rb +2 -2
- data/lib/kennel/syncer.rb +20 -10
- data/lib/kennel/tasks.rb +17 -8
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +69 -26
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a109923efbaef4c2beb6ccf136ea593a43c06b92e45add81d7f07fcbd594779e
|
4
|
+
data.tar.gz: 44a1b83e6b2e708eb3f91d98c19e90aeb663f4a6cf0926f296faae2b2570064d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bfb64f23ff80eca8a8034a2bbb7c2cb08e867ec356879b85b487b093cea1109a5fca0da431355fdd597693a4da2b6a42231b99deb23398d562dfd74650448b64
|
7
|
+
data.tar.gz: e3034cd7215d04fd2020e6d12e27c1831f8e4167c3befaf24cd6725c81e9caafc71c7b14a8813d55fdd8439ae7ec4a4efa09a58a9c2d1c4616d302c77ed87a20
|
data/Readme.md
CHANGED
@@ -1,17 +1,58 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|

|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|

|
17
|
+
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
13
54
|
<!-- NOT IN template/Readme.md -->
|
14
|
-
##
|
55
|
+
## Installation
|
15
56
|
|
16
57
|
- create a new private `kennel` repo for your organization (do not fork this repo)
|
17
58
|
- use the template folder as starting point:
|
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
|
|
22
63
|
cd kennel && git add . && git commit -m 'initial'
|
23
64
|
```
|
24
65
|
- add a basic projects and teams so others can copy-paste to get started
|
25
|
-
- setup
|
26
|
-
- uncomment `.travis.yml` section for
|
66
|
+
- setup CI build for your repo (travis and Github Actions supported)
|
67
|
+
- uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)
|
27
68
|
- follow `Setup` in your repos Readme.md
|
28
69
|
<!-- NOT IN -->
|
29
70
|
|
@@ -109,7 +150,7 @@ end
|
|
109
150
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
110
151
|
- review changes then `git commit`
|
111
152
|
- make a PR ... get reviewed ... merge
|
112
|
-
- datadog is updated by
|
153
|
+
- datadog is updated by CI
|
113
154
|
|
114
155
|
### Adding a new dashboard
|
115
156
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
|
|
176
217
|
- figure out project name by converting the class name to snake-case
|
177
218
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
178
219
|
|
179
|
-
###
|
180
|
-
|
181
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
182
|
-
|
183
|
-
### Validating mentions work
|
184
|
-
|
185
|
-
`rake kennel:validate_mentions` should run as part of CI
|
186
|
-
|
187
|
-
### Grepping through all of datadog
|
188
|
-
|
189
|
-
`TYPE=monitor rake kennel:dump`
|
190
|
-
|
191
|
-
### Find all monitors with No-Data
|
192
|
-
|
193
|
-
`rake kennel:nodata TAG=team:foo`
|
194
|
-
|
195
|
-
## Examples
|
196
|
-
|
197
|
-
### Reusable monitors/dashes/etc
|
220
|
+
### Reuse
|
198
221
|
|
199
222
|
Add to `parts/<folder>`.
|
200
223
|
|
@@ -221,8 +244,31 @@ class Database < Kennel::Models::Project
|
|
221
244
|
)
|
222
245
|
end
|
223
246
|
```
|
247
|
+
|
248
|
+
## Helpers
|
249
|
+
|
250
|
+
### Listing un-muted alerts
|
251
|
+
|
252
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
253
|
+
|
254
|
+
### Validating mentions work
|
255
|
+
|
256
|
+
`rake kennel:validate_mentions` should run as part of CI
|
257
|
+
|
258
|
+
### Grepping through all of datadog
|
259
|
+
|
260
|
+
`rake kennel:dump`
|
261
|
+
focus on a single type: `TYPE=monitors`
|
262
|
+
|
263
|
+
### Find all monitors with No-Data
|
264
|
+
|
265
|
+
`rake kennel:nodata TAG=team:foo`
|
266
|
+
|
224
267
|
<!-- NOT IN template/Readme.md -->
|
225
268
|
|
269
|
+
|
270
|
+
## Development
|
271
|
+
|
226
272
|
### Integration testing
|
227
273
|
|
228
274
|
```Bash
|
data/lib/kennel.rb
CHANGED
data/lib/kennel/importer.rb
CHANGED
@@ -40,11 +40,17 @@ module Kennel
|
|
40
40
|
Kennel::Utils.parameterize(title)
|
41
41
|
end
|
42
42
|
|
43
|
-
|
44
|
-
|
43
|
+
case resource
|
44
|
+
when "monitor"
|
45
|
+
# flatten monitor options so they are all on the base which is how Monitor builds them
|
45
46
|
data.merge!(data.delete(:options))
|
46
47
|
data.merge!(data.delete(:thresholds) || {})
|
47
|
-
|
48
|
+
|
49
|
+
# clean up values that are the default
|
50
|
+
data.delete(:notify_no_data) if data[:notify_no_data] # Monitor uses true by default
|
51
|
+
data.delete(:notify_audit) unless data[:notify_audit] # Monitor uses false by default
|
52
|
+
|
53
|
+
# keep all values that are settable
|
48
54
|
data = data.slice(*model.instance_methods)
|
49
55
|
|
50
56
|
# make query use critical method if it matches
|
@@ -55,11 +61,13 @@ module Kennel
|
|
55
61
|
end
|
56
62
|
|
57
63
|
data[:type] = "query alert" if data[:type] == "metric alert"
|
58
|
-
|
64
|
+
when "dashboard"
|
59
65
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
60
66
|
widgets&.each { |widget| dry_up_query!(widget) }
|
61
67
|
end
|
62
68
|
|
69
|
+
data.delete(:tags) if data[:tags] == [] # do not create super + [] call
|
70
|
+
|
63
71
|
# simplify template_variables to array of string when possible
|
64
72
|
if vars = data[:template_variables]
|
65
73
|
vars.map! { |v| v[:default] == "*" && v[:prefix] == v[:name] ? v[:name] : v }
|
@@ -132,7 +132,7 @@ module Kennel
|
|
132
132
|
url[/\/dashboard\/([a-z\d-]+)/, 1]
|
133
133
|
end
|
134
134
|
|
135
|
-
def resolve_linked_tracking_ids(id_map)
|
135
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
136
136
|
widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
|
137
137
|
widgets.each do |widget|
|
138
138
|
next unless definition = widget[:definition]
|
@@ -140,16 +140,16 @@ module Kennel
|
|
140
140
|
when "uptime"
|
141
141
|
if ids = definition[:monitor_ids]
|
142
142
|
definition[:monitor_ids] = ids.map do |id|
|
143
|
-
tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
|
143
|
+
tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
|
144
144
|
end
|
145
145
|
end
|
146
146
|
when "alert_graph"
|
147
147
|
if (id = definition[:alert_id]) && tracking_id?(id)
|
148
|
-
definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
|
148
|
+
definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
|
149
149
|
end
|
150
150
|
when "slo"
|
151
151
|
if (id = definition[:slo_id]) && tracking_id?(id)
|
152
|
-
definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
|
152
|
+
definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
|
153
153
|
end
|
154
154
|
end
|
155
155
|
end
|
@@ -18,7 +18,9 @@ module Kennel
|
|
18
18
|
new_host_delay: 300,
|
19
19
|
timeout_h: 0,
|
20
20
|
renotify_interval: 0,
|
21
|
-
|
21
|
+
notify_audit: false,
|
22
|
+
no_data_timeframe: nil, # this works out ok since if notify_no_data is on, it would never be nil
|
23
|
+
groupby_simple_monitor: false
|
22
24
|
}.freeze
|
23
25
|
DEFAULT_ESCALATION_MESSAGE = ["", nil].freeze
|
24
26
|
|
@@ -35,9 +37,9 @@ module Kennel
|
|
35
37
|
warning: -> { nil },
|
36
38
|
ok: -> { nil },
|
37
39
|
id: -> { nil },
|
38
|
-
notify_no_data: -> { true },
|
40
|
+
notify_no_data: -> { true }, # datadog sets this to false by default, but true is the safer
|
39
41
|
no_data_timeframe: -> { 60 },
|
40
|
-
notify_audit: -> {
|
42
|
+
notify_audit: -> { MONITOR_OPTION_DEFAULTS.fetch(:notify_audit) },
|
41
43
|
new_host_delay: -> { MONITOR_OPTION_DEFAULTS.fetch(:new_host_delay) },
|
42
44
|
tags: -> { @project.tags },
|
43
45
|
timeout_h: -> { MONITOR_OPTION_DEFAULTS.fetch(:timeout_h) },
|
@@ -103,11 +105,10 @@ module Kennel
|
|
103
105
|
@as_json = data
|
104
106
|
end
|
105
107
|
|
106
|
-
def resolve_linked_tracking_ids(id_map)
|
108
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
107
109
|
if as_json[:type] == "composite"
|
108
110
|
as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
|
109
|
-
|
110
|
-
resolve_link($1, :monitor, id_map, force: true)
|
111
|
+
resolve_link($1, :monitor, id_map, **args)
|
111
112
|
end
|
112
113
|
end
|
113
114
|
end
|
@@ -120,7 +121,7 @@ module Kennel
|
|
120
121
|
Utils.path_to_url "/monitors##{id}/edit"
|
121
122
|
end
|
122
123
|
|
123
|
-
# datadog uses
|
124
|
+
# datadog uses / for show and # for edit as separator in it's links
|
124
125
|
def self.parse_url(url)
|
125
126
|
return unless id = url[/\/monitors[\/#](\d+)/, 1]
|
126
127
|
Integer(id)
|
@@ -202,7 +203,7 @@ module Kennel
|
|
202
203
|
# verify is_match uses available variables
|
203
204
|
message = data.fetch(:message)
|
204
205
|
used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
|
205
|
-
allowed = data.fetch(:query)[/by\s*[
|
206
|
+
allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
|
206
207
|
unsupported = used - allowed
|
207
208
|
if unsupported.any?
|
208
209
|
invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"
|
data/lib/kennel/models/record.rb
CHANGED
@@ -60,23 +60,22 @@ module Kennel
|
|
60
60
|
"#{project.kennel_id}:#{kennel_id}"
|
61
61
|
end
|
62
62
|
|
63
|
-
def resolve_linked_tracking_ids(*)
|
63
|
+
def resolve_linked_tracking_ids!(*)
|
64
64
|
end
|
65
65
|
|
66
66
|
private
|
67
67
|
|
68
|
-
def resolve_link(id, type, id_map, force:
|
69
|
-
|
70
|
-
|
71
|
-
api_resource = self.class.api_resource
|
72
|
-
|
73
|
-
if found == :new
|
68
|
+
def resolve_link(id, type, id_map, force:)
|
69
|
+
value = id_map[id]
|
70
|
+
if value == :new
|
74
71
|
if force
|
75
|
-
|
72
|
+
# TODO: remove the need for this by sorting monitors by missing resolutions
|
73
|
+
invalid! "#{id} needs to already exist, try again"
|
76
74
|
else
|
77
|
-
|
78
|
-
Kennel::MISSING_ID
|
75
|
+
id # will be re-resolved by syncer after the linked object was created
|
79
76
|
end
|
77
|
+
elsif value
|
78
|
+
value
|
80
79
|
else
|
81
80
|
invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
|
82
81
|
end
|
data/lib/kennel/models/slo.rb
CHANGED
@@ -66,9 +66,9 @@ module Kennel
|
|
66
66
|
url[/\/slo\?slo_id=([a-z\d]+)/, 1]
|
67
67
|
end
|
68
68
|
|
69
|
-
def resolve_linked_tracking_ids(id_map)
|
69
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
70
70
|
as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
|
71
|
-
id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
|
71
|
+
id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
data/lib/kennel/syncer.rb
CHANGED
@@ -38,12 +38,23 @@ module Kennel
|
|
38
38
|
end
|
39
39
|
|
40
40
|
def update
|
41
|
+
changed = (@create + @update).map { |_, e| e } unless @create.empty?
|
42
|
+
|
41
43
|
@create.each do |_, e|
|
44
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
45
|
+
|
42
46
|
reply = @api.create e.class.api_resource, e.as_json
|
43
|
-
|
47
|
+
id = reply.fetch(:id)
|
48
|
+
|
49
|
+
# resolve ids we could previously no resolve
|
50
|
+
changed.delete e
|
51
|
+
resolve_linked_tracking_ids! from: [reply], to: changed
|
52
|
+
|
53
|
+
Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
44
54
|
end
|
45
55
|
|
46
56
|
@update.each do |id, e|
|
57
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
47
58
|
@api.update e.class.api_resource, id, e.as_json
|
48
59
|
Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
49
60
|
end
|
@@ -65,12 +76,10 @@ module Kennel
|
|
65
76
|
@delete = []
|
66
77
|
|
67
78
|
actual = Progress.progress("Downloading definitions") { download_definitions }
|
68
|
-
|
69
|
-
|
79
|
+
resolve_linked_tracking_ids! from: actual, to: @expected
|
80
|
+
filter_by_project! actual
|
70
81
|
|
71
82
|
Progress.progress "Diffing" do
|
72
|
-
filter_by_project! actual
|
73
|
-
|
74
83
|
items = actual.map do |a|
|
75
84
|
e = matching_expected(a)
|
76
85
|
if e && @expected.delete(e)
|
@@ -98,6 +107,7 @@ module Kennel
|
|
98
107
|
|
99
108
|
ensure_all_ids_found
|
100
109
|
@create = @expected.map { |e| [nil, e] }
|
110
|
+
@create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
|
101
111
|
end
|
102
112
|
|
103
113
|
@delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
|
@@ -178,7 +188,7 @@ module Kennel
|
|
178
188
|
end
|
179
189
|
|
180
190
|
# Do not add tracking-id when working with existing ids on a branch,
|
181
|
-
# so resource do not get deleted
|
191
|
+
# so resource do not get deleted fr:om merges to master.
|
182
192
|
# Also make sure the diff still makes sense, by kicking out the now noop-update.
|
183
193
|
#
|
184
194
|
# Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
|
@@ -203,10 +213,10 @@ module Kennel
|
|
203
213
|
end
|
204
214
|
end
|
205
215
|
|
206
|
-
def resolve_linked_tracking_ids(
|
207
|
-
map =
|
208
|
-
|
209
|
-
|
216
|
+
def resolve_linked_tracking_ids!(from:, to:)
|
217
|
+
map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
|
218
|
+
to.each { |e| map[e.tracking_id] ||= :new }
|
219
|
+
to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
|
210
220
|
end
|
211
221
|
|
212
222
|
def filter_by_project!(definitions)
|
data/lib/kennel/tasks.rb
CHANGED
@@ -68,15 +68,16 @@ namespace :kennel do
|
|
68
68
|
Kennel.update
|
69
69
|
end
|
70
70
|
|
71
|
-
desc "update
|
72
|
-
task :
|
73
|
-
|
74
|
-
|
71
|
+
desc "update on push to the default branch, otherwise show plan"
|
72
|
+
task :ci do
|
73
|
+
branch = (ENV["TRAVIS_BRANCH"] || ENV["GITHUB_REF"]).to_s.sub(/^refs\/heads\//, "")
|
74
|
+
on_default_branch = (branch == (ENV["DEFAULT_BRANCH"] || "master"))
|
75
|
+
is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false" || ENV["GITHUB_EVENT_NAME"] == "push")
|
75
76
|
task_name =
|
76
77
|
if on_default_branch && is_push
|
77
78
|
"kennel:update_datadog"
|
78
79
|
else
|
79
|
-
"kennel:plan" # show plan in
|
80
|
+
"kennel:plan" # show plan in CI logs
|
80
81
|
end
|
81
82
|
|
82
83
|
Rake::Task[task_name].invoke
|
@@ -123,10 +124,18 @@ namespace :kennel do
|
|
123
124
|
Kennel.out.puts Kennel::Importer.new(Kennel.send(:api)).import(resource, id)
|
124
125
|
end
|
125
126
|
|
126
|
-
desc "Dump ALL of datadog config as raw json ... useful for grep/search TYPE=slo|monitor|dashboard"
|
127
|
+
desc "Dump ALL of datadog config as raw json ... useful for grep/search [TYPE=slo|monitor|dashboard]"
|
127
128
|
task dump: :environment do
|
128
|
-
|
129
|
-
|
129
|
+
resources =
|
130
|
+
if type = ENV["TYPE"]
|
131
|
+
[type]
|
132
|
+
else
|
133
|
+
Kennel::Models::Record.subclasses.map(&:api_resource)
|
134
|
+
end
|
135
|
+
resources.each do |resource|
|
136
|
+
Kennel.send(:api).list(resource).each do |r|
|
137
|
+
Kennel.out.puts JSON.pretty_generate(r)
|
138
|
+
end
|
130
139
|
end
|
131
140
|
end
|
132
141
|
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -1,16 +1,57 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|

|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|

|
13
17
|
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
|
14
55
|
## Structure
|
15
56
|
|
16
57
|
- `projects/` monitors/dashboards/etc scoped by project
|
@@ -91,7 +132,7 @@ end
|
|
91
132
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
92
133
|
- review changes then `git commit`
|
93
134
|
- make a PR ... get reviewed ... merge
|
94
|
-
- datadog is updated by
|
135
|
+
- datadog is updated by CI
|
95
136
|
|
96
137
|
### Adding a new dashboard
|
97
138
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
|
|
158
199
|
- figure out project name by converting the class name to snake-case
|
159
200
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
160
201
|
|
161
|
-
###
|
162
|
-
|
163
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
164
|
-
|
165
|
-
### Validating mentions work
|
166
|
-
|
167
|
-
`rake kennel:validate_mentions` should run as part of CI
|
168
|
-
|
169
|
-
### Grepping through all of datadog
|
170
|
-
|
171
|
-
`TYPE=monitor rake kennel:dump`
|
172
|
-
|
173
|
-
### Find all monitors with No-Data
|
174
|
-
|
175
|
-
`rake kennel:nodata TAG=team:foo`
|
176
|
-
|
177
|
-
## Examples
|
178
|
-
|
179
|
-
### Reusable monitors/dashes/etc
|
202
|
+
### Reuse
|
180
203
|
|
181
204
|
Add to `parts/<folder>`.
|
182
205
|
|
@@ -203,3 +226,23 @@ class Database < Kennel::Models::Project
|
|
203
226
|
)
|
204
227
|
end
|
205
228
|
```
|
229
|
+
|
230
|
+
## Helpers
|
231
|
+
|
232
|
+
### Listing un-muted alerts
|
233
|
+
|
234
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
235
|
+
|
236
|
+
### Validating mentions work
|
237
|
+
|
238
|
+
`rake kennel:validate_mentions` should run as part of CI
|
239
|
+
|
240
|
+
### Grepping through all of datadog
|
241
|
+
|
242
|
+
`rake kennel:dump`
|
243
|
+
focus on a single type: `TYPE=monitors`
|
244
|
+
|
245
|
+
### Find all monitors with No-Data
|
246
|
+
|
247
|
+
`rake kennel:nodata TAG=team:foo`
|
248
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.76.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-08-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|