kennel 1.73.0 → 1.76.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Readme.md +75 -29
- data/lib/kennel.rb +0 -1
- data/lib/kennel/importer.rb +12 -4
- data/lib/kennel/models/dashboard.rb +4 -4
- data/lib/kennel/models/monitor.rb +9 -8
- data/lib/kennel/models/record.rb +9 -10
- data/lib/kennel/models/slo.rb +2 -2
- data/lib/kennel/syncer.rb +20 -10
- data/lib/kennel/tasks.rb +17 -8
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +69 -26
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a109923efbaef4c2beb6ccf136ea593a43c06b92e45add81d7f07fcbd594779e
|
4
|
+
data.tar.gz: 44a1b83e6b2e708eb3f91d98c19e90aeb663f4a6cf0926f296faae2b2570064d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: bfb64f23ff80eca8a8034a2bbb7c2cb08e867ec356879b85b487b093cea1109a5fca0da431355fdd597693a4da2b6a42231b99deb23398d562dfd74650448b64
|
7
|
+
data.tar.gz: e3034cd7215d04fd2020e6d12e27c1831f8e4167c3befaf24cd6725c81e9caafc71c7b14a8813d55fdd8439ae7ec4a4efa09a58a9c2d1c4616d302c77ed87a20
|
data/Readme.md
CHANGED
@@ -1,17 +1,58 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](template/github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](template/github/screen.png?raw=true)
|
17
|
+
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
13
54
|
<!-- NOT IN template/Readme.md -->
|
14
|
-
##
|
55
|
+
## Installation
|
15
56
|
|
16
57
|
- create a new private `kennel` repo for your organization (do not fork this repo)
|
17
58
|
- use the template folder as starting point:
|
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
|
|
22
63
|
cd kennel && git add . && git commit -m 'initial'
|
23
64
|
```
|
24
65
|
- add a basic projects and teams so others can copy-paste to get started
|
25
|
-
- setup
|
26
|
-
- uncomment `.travis.yml` section for
|
66
|
+
- setup CI build for your repo (travis and Github Actions supported)
|
67
|
+
- uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)
|
27
68
|
- follow `Setup` in your repos Readme.md
|
28
69
|
<!-- NOT IN -->
|
29
70
|
|
@@ -109,7 +150,7 @@ end
|
|
109
150
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
110
151
|
- review changes then `git commit`
|
111
152
|
- make a PR ... get reviewed ... merge
|
112
|
-
- datadog is updated by
|
153
|
+
- datadog is updated by CI
|
113
154
|
|
114
155
|
### Adding a new dashboard
|
115
156
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
|
|
176
217
|
- figure out project name by converting the class name to snake-case
|
177
218
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
178
219
|
|
179
|
-
###
|
180
|
-
|
181
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
182
|
-
|
183
|
-
### Validating mentions work
|
184
|
-
|
185
|
-
`rake kennel:validate_mentions` should run as part of CI
|
186
|
-
|
187
|
-
### Grepping through all of datadog
|
188
|
-
|
189
|
-
`TYPE=monitor rake kennel:dump`
|
190
|
-
|
191
|
-
### Find all monitors with No-Data
|
192
|
-
|
193
|
-
`rake kennel:nodata TAG=team:foo`
|
194
|
-
|
195
|
-
## Examples
|
196
|
-
|
197
|
-
### Reusable monitors/dashes/etc
|
220
|
+
### Reuse
|
198
221
|
|
199
222
|
Add to `parts/<folder>`.
|
200
223
|
|
@@ -221,8 +244,31 @@ class Database < Kennel::Models::Project
|
|
221
244
|
)
|
222
245
|
end
|
223
246
|
```
|
247
|
+
|
248
|
+
## Helpers
|
249
|
+
|
250
|
+
### Listing un-muted alerts
|
251
|
+
|
252
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
253
|
+
|
254
|
+
### Validating mentions work
|
255
|
+
|
256
|
+
`rake kennel:validate_mentions` should run as part of CI
|
257
|
+
|
258
|
+
### Grepping through all of datadog
|
259
|
+
|
260
|
+
`rake kennel:dump`
|
261
|
+
focus on a single type: `TYPE=monitors`
|
262
|
+
|
263
|
+
### Find all monitors with No-Data
|
264
|
+
|
265
|
+
`rake kennel:nodata TAG=team:foo`
|
266
|
+
|
224
267
|
<!-- NOT IN template/Readme.md -->
|
225
268
|
|
269
|
+
|
270
|
+
## Development
|
271
|
+
|
226
272
|
### Integration testing
|
227
273
|
|
228
274
|
```Bash
|
data/lib/kennel.rb
CHANGED
data/lib/kennel/importer.rb
CHANGED
@@ -40,11 +40,17 @@ module Kennel
|
|
40
40
|
Kennel::Utils.parameterize(title)
|
41
41
|
end
|
42
42
|
|
43
|
-
|
44
|
-
|
43
|
+
case resource
|
44
|
+
when "monitor"
|
45
|
+
# flatten monitor options so they are all on the base which is how Monitor builds them
|
45
46
|
data.merge!(data.delete(:options))
|
46
47
|
data.merge!(data.delete(:thresholds) || {})
|
47
|
-
|
48
|
+
|
49
|
+
# clean up values that are the default
|
50
|
+
data.delete(:notify_no_data) if data[:notify_no_data] # Monitor uses true by default
|
51
|
+
data.delete(:notify_audit) unless data[:notify_audit] # Monitor uses false by default
|
52
|
+
|
53
|
+
# keep all values that are settable
|
48
54
|
data = data.slice(*model.instance_methods)
|
49
55
|
|
50
56
|
# make query use critical method if it matches
|
@@ -55,11 +61,13 @@ module Kennel
|
|
55
61
|
end
|
56
62
|
|
57
63
|
data[:type] = "query alert" if data[:type] == "metric alert"
|
58
|
-
|
64
|
+
when "dashboard"
|
59
65
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
60
66
|
widgets&.each { |widget| dry_up_query!(widget) }
|
61
67
|
end
|
62
68
|
|
69
|
+
data.delete(:tags) if data[:tags] == [] # do not create super + [] call
|
70
|
+
|
63
71
|
# simplify template_variables to array of string when possible
|
64
72
|
if vars = data[:template_variables]
|
65
73
|
vars.map! { |v| v[:default] == "*" && v[:prefix] == v[:name] ? v[:name] : v }
|
@@ -132,7 +132,7 @@ module Kennel
|
|
132
132
|
url[/\/dashboard\/([a-z\d-]+)/, 1]
|
133
133
|
end
|
134
134
|
|
135
|
-
def resolve_linked_tracking_ids(id_map)
|
135
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
136
136
|
widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
|
137
137
|
widgets.each do |widget|
|
138
138
|
next unless definition = widget[:definition]
|
@@ -140,16 +140,16 @@ module Kennel
|
|
140
140
|
when "uptime"
|
141
141
|
if ids = definition[:monitor_ids]
|
142
142
|
definition[:monitor_ids] = ids.map do |id|
|
143
|
-
tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
|
143
|
+
tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
|
144
144
|
end
|
145
145
|
end
|
146
146
|
when "alert_graph"
|
147
147
|
if (id = definition[:alert_id]) && tracking_id?(id)
|
148
|
-
definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
|
148
|
+
definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
|
149
149
|
end
|
150
150
|
when "slo"
|
151
151
|
if (id = definition[:slo_id]) && tracking_id?(id)
|
152
|
-
definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
|
152
|
+
definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
|
153
153
|
end
|
154
154
|
end
|
155
155
|
end
|
@@ -18,7 +18,9 @@ module Kennel
|
|
18
18
|
new_host_delay: 300,
|
19
19
|
timeout_h: 0,
|
20
20
|
renotify_interval: 0,
|
21
|
-
|
21
|
+
notify_audit: false,
|
22
|
+
no_data_timeframe: nil, # this works out ok since if notify_no_data is on, it would never be nil
|
23
|
+
groupby_simple_monitor: false
|
22
24
|
}.freeze
|
23
25
|
DEFAULT_ESCALATION_MESSAGE = ["", nil].freeze
|
24
26
|
|
@@ -35,9 +37,9 @@ module Kennel
|
|
35
37
|
warning: -> { nil },
|
36
38
|
ok: -> { nil },
|
37
39
|
id: -> { nil },
|
38
|
-
notify_no_data: -> { true },
|
40
|
+
notify_no_data: -> { true }, # datadog sets this to false by default, but true is the safer
|
39
41
|
no_data_timeframe: -> { 60 },
|
40
|
-
notify_audit: -> {
|
42
|
+
notify_audit: -> { MONITOR_OPTION_DEFAULTS.fetch(:notify_audit) },
|
41
43
|
new_host_delay: -> { MONITOR_OPTION_DEFAULTS.fetch(:new_host_delay) },
|
42
44
|
tags: -> { @project.tags },
|
43
45
|
timeout_h: -> { MONITOR_OPTION_DEFAULTS.fetch(:timeout_h) },
|
@@ -103,11 +105,10 @@ module Kennel
|
|
103
105
|
@as_json = data
|
104
106
|
end
|
105
107
|
|
106
|
-
def resolve_linked_tracking_ids(id_map)
|
108
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
107
109
|
if as_json[:type] == "composite"
|
108
110
|
as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
|
109
|
-
|
110
|
-
resolve_link($1, :monitor, id_map, force: true)
|
111
|
+
resolve_link($1, :monitor, id_map, **args)
|
111
112
|
end
|
112
113
|
end
|
113
114
|
end
|
@@ -120,7 +121,7 @@ module Kennel
|
|
120
121
|
Utils.path_to_url "/monitors##{id}/edit"
|
121
122
|
end
|
122
123
|
|
123
|
-
# datadog uses
|
124
|
+
# datadog uses / for show and # for edit as separator in it's links
|
124
125
|
def self.parse_url(url)
|
125
126
|
return unless id = url[/\/monitors[\/#](\d+)/, 1]
|
126
127
|
Integer(id)
|
@@ -202,7 +203,7 @@ module Kennel
|
|
202
203
|
# verify is_match uses available variables
|
203
204
|
message = data.fetch(:message)
|
204
205
|
used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
|
205
|
-
allowed = data.fetch(:query)[/by\s*[
|
206
|
+
allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
|
206
207
|
unsupported = used - allowed
|
207
208
|
if unsupported.any?
|
208
209
|
invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"
|
data/lib/kennel/models/record.rb
CHANGED
@@ -60,23 +60,22 @@ module Kennel
|
|
60
60
|
"#{project.kennel_id}:#{kennel_id}"
|
61
61
|
end
|
62
62
|
|
63
|
-
def resolve_linked_tracking_ids(*)
|
63
|
+
def resolve_linked_tracking_ids!(*)
|
64
64
|
end
|
65
65
|
|
66
66
|
private
|
67
67
|
|
68
|
-
def resolve_link(id, type, id_map, force:
|
69
|
-
|
70
|
-
|
71
|
-
api_resource = self.class.api_resource
|
72
|
-
|
73
|
-
if found == :new
|
68
|
+
def resolve_link(id, type, id_map, force:)
|
69
|
+
value = id_map[id]
|
70
|
+
if value == :new
|
74
71
|
if force
|
75
|
-
|
72
|
+
# TODO: remove the need for this by sorting monitors by missing resolutions
|
73
|
+
invalid! "#{id} needs to already exist, try again"
|
76
74
|
else
|
77
|
-
|
78
|
-
Kennel::MISSING_ID
|
75
|
+
id # will be re-resolved by syncer after the linked object was created
|
79
76
|
end
|
77
|
+
elsif value
|
78
|
+
value
|
80
79
|
else
|
81
80
|
invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
|
82
81
|
end
|
data/lib/kennel/models/slo.rb
CHANGED
@@ -66,9 +66,9 @@ module Kennel
|
|
66
66
|
url[/\/slo\?slo_id=([a-z\d]+)/, 1]
|
67
67
|
end
|
68
68
|
|
69
|
-
def resolve_linked_tracking_ids(id_map)
|
69
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
70
70
|
as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
|
71
|
-
id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
|
71
|
+
id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
data/lib/kennel/syncer.rb
CHANGED
@@ -38,12 +38,23 @@ module Kennel
|
|
38
38
|
end
|
39
39
|
|
40
40
|
def update
|
41
|
+
changed = (@create + @update).map { |_, e| e } unless @create.empty?
|
42
|
+
|
41
43
|
@create.each do |_, e|
|
44
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
45
|
+
|
42
46
|
reply = @api.create e.class.api_resource, e.as_json
|
43
|
-
|
47
|
+
id = reply.fetch(:id)
|
48
|
+
|
49
|
+
# resolve ids we could previously no resolve
|
50
|
+
changed.delete e
|
51
|
+
resolve_linked_tracking_ids! from: [reply], to: changed
|
52
|
+
|
53
|
+
Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
44
54
|
end
|
45
55
|
|
46
56
|
@update.each do |id, e|
|
57
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
47
58
|
@api.update e.class.api_resource, id, e.as_json
|
48
59
|
Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
49
60
|
end
|
@@ -65,12 +76,10 @@ module Kennel
|
|
65
76
|
@delete = []
|
66
77
|
|
67
78
|
actual = Progress.progress("Downloading definitions") { download_definitions }
|
68
|
-
|
69
|
-
|
79
|
+
resolve_linked_tracking_ids! from: actual, to: @expected
|
80
|
+
filter_by_project! actual
|
70
81
|
|
71
82
|
Progress.progress "Diffing" do
|
72
|
-
filter_by_project! actual
|
73
|
-
|
74
83
|
items = actual.map do |a|
|
75
84
|
e = matching_expected(a)
|
76
85
|
if e && @expected.delete(e)
|
@@ -98,6 +107,7 @@ module Kennel
|
|
98
107
|
|
99
108
|
ensure_all_ids_found
|
100
109
|
@create = @expected.map { |e| [nil, e] }
|
110
|
+
@create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
|
101
111
|
end
|
102
112
|
|
103
113
|
@delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
|
@@ -178,7 +188,7 @@ module Kennel
|
|
178
188
|
end
|
179
189
|
|
180
190
|
# Do not add tracking-id when working with existing ids on a branch,
|
181
|
-
# so resource do not get deleted
|
191
|
+
# so resource do not get deleted fr:om merges to master.
|
182
192
|
# Also make sure the diff still makes sense, by kicking out the now noop-update.
|
183
193
|
#
|
184
194
|
# Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
|
@@ -203,10 +213,10 @@ module Kennel
|
|
203
213
|
end
|
204
214
|
end
|
205
215
|
|
206
|
-
def resolve_linked_tracking_ids(
|
207
|
-
map =
|
208
|
-
|
209
|
-
|
216
|
+
def resolve_linked_tracking_ids!(from:, to:)
|
217
|
+
map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
|
218
|
+
to.each { |e| map[e.tracking_id] ||= :new }
|
219
|
+
to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
|
210
220
|
end
|
211
221
|
|
212
222
|
def filter_by_project!(definitions)
|
data/lib/kennel/tasks.rb
CHANGED
@@ -68,15 +68,16 @@ namespace :kennel do
|
|
68
68
|
Kennel.update
|
69
69
|
end
|
70
70
|
|
71
|
-
desc "update
|
72
|
-
task :
|
73
|
-
|
74
|
-
|
71
|
+
desc "update on push to the default branch, otherwise show plan"
|
72
|
+
task :ci do
|
73
|
+
branch = (ENV["TRAVIS_BRANCH"] || ENV["GITHUB_REF"]).to_s.sub(/^refs\/heads\//, "")
|
74
|
+
on_default_branch = (branch == (ENV["DEFAULT_BRANCH"] || "master"))
|
75
|
+
is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false" || ENV["GITHUB_EVENT_NAME"] == "push")
|
75
76
|
task_name =
|
76
77
|
if on_default_branch && is_push
|
77
78
|
"kennel:update_datadog"
|
78
79
|
else
|
79
|
-
"kennel:plan" # show plan in
|
80
|
+
"kennel:plan" # show plan in CI logs
|
80
81
|
end
|
81
82
|
|
82
83
|
Rake::Task[task_name].invoke
|
@@ -123,10 +124,18 @@ namespace :kennel do
|
|
123
124
|
Kennel.out.puts Kennel::Importer.new(Kennel.send(:api)).import(resource, id)
|
124
125
|
end
|
125
126
|
|
126
|
-
desc "Dump ALL of datadog config as raw json ... useful for grep/search TYPE=slo|monitor|dashboard"
|
127
|
+
desc "Dump ALL of datadog config as raw json ... useful for grep/search [TYPE=slo|monitor|dashboard]"
|
127
128
|
task dump: :environment do
|
128
|
-
|
129
|
-
|
129
|
+
resources =
|
130
|
+
if type = ENV["TYPE"]
|
131
|
+
[type]
|
132
|
+
else
|
133
|
+
Kennel::Models::Record.subclasses.map(&:api_resource)
|
134
|
+
end
|
135
|
+
resources.each do |resource|
|
136
|
+
Kennel.send(:api).list(resource).each do |r|
|
137
|
+
Kennel.out.puts JSON.pretty_generate(r)
|
138
|
+
end
|
130
139
|
end
|
131
140
|
end
|
132
141
|
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -1,16 +1,57 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](github/screen.png?raw=true)
|
13
17
|
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
|
14
55
|
## Structure
|
15
56
|
|
16
57
|
- `projects/` monitors/dashboards/etc scoped by project
|
@@ -91,7 +132,7 @@ end
|
|
91
132
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
92
133
|
- review changes then `git commit`
|
93
134
|
- make a PR ... get reviewed ... merge
|
94
|
-
- datadog is updated by
|
135
|
+
- datadog is updated by CI
|
95
136
|
|
96
137
|
### Adding a new dashboard
|
97
138
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
|
|
158
199
|
- figure out project name by converting the class name to snake-case
|
159
200
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
160
201
|
|
161
|
-
###
|
162
|
-
|
163
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
164
|
-
|
165
|
-
### Validating mentions work
|
166
|
-
|
167
|
-
`rake kennel:validate_mentions` should run as part of CI
|
168
|
-
|
169
|
-
### Grepping through all of datadog
|
170
|
-
|
171
|
-
`TYPE=monitor rake kennel:dump`
|
172
|
-
|
173
|
-
### Find all monitors with No-Data
|
174
|
-
|
175
|
-
`rake kennel:nodata TAG=team:foo`
|
176
|
-
|
177
|
-
## Examples
|
178
|
-
|
179
|
-
### Reusable monitors/dashes/etc
|
202
|
+
### Reuse
|
180
203
|
|
181
204
|
Add to `parts/<folder>`.
|
182
205
|
|
@@ -203,3 +226,23 @@ class Database < Kennel::Models::Project
|
|
203
226
|
)
|
204
227
|
end
|
205
228
|
```
|
229
|
+
|
230
|
+
## Helpers
|
231
|
+
|
232
|
+
### Listing un-muted alerts
|
233
|
+
|
234
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
235
|
+
|
236
|
+
### Validating mentions work
|
237
|
+
|
238
|
+
`rake kennel:validate_mentions` should run as part of CI
|
239
|
+
|
240
|
+
### Grepping through all of datadog
|
241
|
+
|
242
|
+
`rake kennel:dump`
|
243
|
+
focus on a single type: `TYPE=monitors`
|
244
|
+
|
245
|
+
### Find all monitors with No-Data
|
246
|
+
|
247
|
+
`rake kennel:nodata TAG=team:foo`
|
248
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.76.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-08-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|