kennel 1.72.2 → 1.75.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Readme.md +74 -29
- data/lib/kennel.rb +0 -1
- data/lib/kennel/importer.rb +3 -2
- data/lib/kennel/models/dashboard.rb +15 -11
- data/lib/kennel/models/monitor.rb +6 -6
- data/lib/kennel/models/record.rb +9 -10
- data/lib/kennel/models/slo.rb +2 -2
- data/lib/kennel/syncer.rb +20 -10
- data/lib/kennel/tasks.rb +6 -5
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +68 -26
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 23af2ecfb9019df555f7fe3a3954f4d992208927744093ff1f8886afd571e457
|
4
|
+
data.tar.gz: 6e2c902c47020e46b5fc0e647d8d63bbd7227657ce78303c673470be840ecbf9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c2dfa7e9f54c83d3369b48b91896cb7dc7cfe75bb94d672d7de07bc7a6b8ea3f63ebe38788dddf5de90ead3b73f977c42f87497d61e7b23def6ae385d0a42213
|
7
|
+
data.tar.gz: 9ebb18f6d9c54be18a783d8511e3128b9c59b5898f9f14a1be77c871dcdab837681bf04050ec97a1a2f675c84fda624469edaed0af9f5e0da11b60517ba93cea
|
data/Readme.md
CHANGED
@@ -1,17 +1,58 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](template/github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](template/github/screen.png?raw=true)
|
17
|
+
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
13
54
|
<!-- NOT IN template/Readme.md -->
|
14
|
-
##
|
55
|
+
## Installation
|
15
56
|
|
16
57
|
- create a new private `kennel` repo for your organization (do not fork this repo)
|
17
58
|
- use the template folder as starting point:
|
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
|
|
22
63
|
cd kennel && git add . && git commit -m 'initial'
|
23
64
|
```
|
24
65
|
- add a basic projects and teams so others can copy-paste to get started
|
25
|
-
- setup
|
26
|
-
- uncomment `.travis.yml` section for
|
66
|
+
- setup CI build for your repo (travis and Github Actions supported)
|
67
|
+
- uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)
|
27
68
|
- follow `Setup` in your repos Readme.md
|
28
69
|
<!-- NOT IN -->
|
29
70
|
|
@@ -109,7 +150,7 @@ end
|
|
109
150
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
110
151
|
- review changes then `git commit`
|
111
152
|
- make a PR ... get reviewed ... merge
|
112
|
-
- datadog is updated by
|
153
|
+
- datadog is updated by CI
|
113
154
|
|
114
155
|
### Adding a new dashboard
|
115
156
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
|
|
176
217
|
- figure out project name by converting the class name to snake-case
|
177
218
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
178
219
|
|
179
|
-
###
|
180
|
-
|
181
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
182
|
-
|
183
|
-
### Validating mentions work
|
184
|
-
|
185
|
-
`rake kennel:validate_mentions` should run as part of CI
|
186
|
-
|
187
|
-
### Grepping through all of datadog
|
188
|
-
|
189
|
-
`TYPE=monitor rake kennel:dump`
|
190
|
-
|
191
|
-
### Find all monitors with No-Data
|
192
|
-
|
193
|
-
`rake kennel:nodata TAG=team:foo`
|
194
|
-
|
195
|
-
## Examples
|
196
|
-
|
197
|
-
### Reusable monitors/dashes/etc
|
220
|
+
### Reuse
|
198
221
|
|
199
222
|
Add to `parts/<folder>`.
|
200
223
|
|
@@ -221,8 +244,30 @@ class Database < Kennel::Models::Project
|
|
221
244
|
)
|
222
245
|
end
|
223
246
|
```
|
247
|
+
|
248
|
+
## Helpers
|
249
|
+
|
250
|
+
### Listing un-muted alerts
|
251
|
+
|
252
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
253
|
+
|
254
|
+
### Validating mentions work
|
255
|
+
|
256
|
+
`rake kennel:validate_mentions` should run as part of CI
|
257
|
+
|
258
|
+
### Grepping through all of datadog
|
259
|
+
|
260
|
+
`TYPE=monitor rake kennel:dump`
|
261
|
+
|
262
|
+
### Find all monitors with No-Data
|
263
|
+
|
264
|
+
`rake kennel:nodata TAG=team:foo`
|
265
|
+
|
224
266
|
<!-- NOT IN template/Readme.md -->
|
225
267
|
|
268
|
+
|
269
|
+
## Development
|
270
|
+
|
226
271
|
### Integration testing
|
227
272
|
|
228
273
|
```Bash
|
data/lib/kennel.rb
CHANGED
data/lib/kennel/importer.rb
CHANGED
@@ -40,7 +40,8 @@ module Kennel
|
|
40
40
|
Kennel::Utils.parameterize(title)
|
41
41
|
end
|
42
42
|
|
43
|
-
|
43
|
+
case resource
|
44
|
+
when "monitor"
|
44
45
|
# flatten monitor options so they are all on the base
|
45
46
|
data.merge!(data.delete(:options))
|
46
47
|
data.merge!(data.delete(:thresholds) || {})
|
@@ -55,7 +56,7 @@ module Kennel
|
|
55
56
|
end
|
56
57
|
|
57
58
|
data[:type] = "query alert" if data[:type] == "metric alert"
|
58
|
-
|
59
|
+
when "dashboard"
|
59
60
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
60
61
|
widgets&.each { |widget| dry_up_query!(widget) }
|
61
62
|
end
|
@@ -13,7 +13,10 @@ module Kennel
|
|
13
13
|
REQUEST_DEFAULTS = {
|
14
14
|
style: { line_width: "normal", palette: "dog_classic", line_type: "solid" }
|
15
15
|
}.freeze
|
16
|
-
|
16
|
+
WIDGET_DEFAULTS = {
|
17
|
+
"timeseries" => { show_legend: false, legend_size: "0" },
|
18
|
+
"note" => { background_color: "white", font_size: "14", show_tick: false, tick_edge: "left", tick_pos: "50%", text_align: "left" }
|
19
|
+
}.freeze
|
17
20
|
SUPPORTED_DEFINITION_OPTIONS = [:events, :markers, :precision].freeze
|
18
21
|
|
19
22
|
DEFAULTS = {
|
@@ -26,7 +29,7 @@ module Kennel
|
|
26
29
|
description: -> { "" },
|
27
30
|
definitions: -> { [] },
|
28
31
|
widgets: -> { [] },
|
29
|
-
template_variable_presets: -> {
|
32
|
+
template_variable_presets: -> { DEFAULTS.fetch(:template_variable_presets) },
|
30
33
|
id: -> { nil }
|
31
34
|
)
|
32
35
|
|
@@ -50,7 +53,7 @@ module Kennel
|
|
50
53
|
end
|
51
54
|
end
|
52
55
|
|
53
|
-
|
56
|
+
ignore_widget_defaults pair
|
54
57
|
|
55
58
|
ignore_request_defaults(*pair)
|
56
59
|
|
@@ -61,11 +64,12 @@ module Kennel
|
|
61
64
|
|
62
65
|
private
|
63
66
|
|
64
|
-
def
|
67
|
+
def ignore_widget_defaults(pair)
|
65
68
|
pair.map(&:size).max.times do |i|
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
+
types = pair.map { |w| w.dig(i, :definition, :type) }.uniq
|
70
|
+
next unless types.size == 1
|
71
|
+
next unless defaults = WIDGET_DEFAULTS[types.first]
|
72
|
+
ignore_defaults(pair[0], pair[1], defaults, nesting: :definition)
|
69
73
|
end
|
70
74
|
end
|
71
75
|
|
@@ -128,7 +132,7 @@ module Kennel
|
|
128
132
|
url[/\/dashboard\/([a-z\d-]+)/, 1]
|
129
133
|
end
|
130
134
|
|
131
|
-
def resolve_linked_tracking_ids(id_map)
|
135
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
132
136
|
widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
|
133
137
|
widgets.each do |widget|
|
134
138
|
next unless definition = widget[:definition]
|
@@ -136,16 +140,16 @@ module Kennel
|
|
136
140
|
when "uptime"
|
137
141
|
if ids = definition[:monitor_ids]
|
138
142
|
definition[:monitor_ids] = ids.map do |id|
|
139
|
-
tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
|
143
|
+
tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
|
140
144
|
end
|
141
145
|
end
|
142
146
|
when "alert_graph"
|
143
147
|
if (id = definition[:alert_id]) && tracking_id?(id)
|
144
|
-
definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
|
148
|
+
definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
|
145
149
|
end
|
146
150
|
when "slo"
|
147
151
|
if (id = definition[:slo_id]) && tracking_id?(id)
|
148
|
-
definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
|
152
|
+
definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
|
149
153
|
end
|
150
154
|
end
|
151
155
|
end
|
@@ -18,7 +18,8 @@ module Kennel
|
|
18
18
|
new_host_delay: 300,
|
19
19
|
timeout_h: 0,
|
20
20
|
renotify_interval: 0,
|
21
|
-
no_data_timeframe: nil # this works out ok since if notify_no_data is on, it would never be nil
|
21
|
+
no_data_timeframe: nil, # this works out ok since if notify_no_data is on, it would never be nil
|
22
|
+
groupby_simple_monitor: false
|
22
23
|
}.freeze
|
23
24
|
DEFAULT_ESCALATION_MESSAGE = ["", nil].freeze
|
24
25
|
|
@@ -103,11 +104,10 @@ module Kennel
|
|
103
104
|
@as_json = data
|
104
105
|
end
|
105
106
|
|
106
|
-
def resolve_linked_tracking_ids(id_map)
|
107
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
107
108
|
if as_json[:type] == "composite"
|
108
109
|
as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
|
109
|
-
|
110
|
-
resolve_link($1, :monitor, id_map, force: true)
|
110
|
+
resolve_link($1, :monitor, id_map, **args)
|
111
111
|
end
|
112
112
|
end
|
113
113
|
end
|
@@ -120,7 +120,7 @@ module Kennel
|
|
120
120
|
Utils.path_to_url "/monitors##{id}/edit"
|
121
121
|
end
|
122
122
|
|
123
|
-
# datadog uses
|
123
|
+
# datadog uses / for show and # for edit as separator in it's links
|
124
124
|
def self.parse_url(url)
|
125
125
|
return unless id = url[/\/monitors[\/#](\d+)/, 1]
|
126
126
|
Integer(id)
|
@@ -202,7 +202,7 @@ module Kennel
|
|
202
202
|
# verify is_match uses available variables
|
203
203
|
message = data.fetch(:message)
|
204
204
|
used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
|
205
|
-
allowed = data.fetch(:query)[/by\s*[
|
205
|
+
allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
|
206
206
|
unsupported = used - allowed
|
207
207
|
if unsupported.any?
|
208
208
|
invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"
|
data/lib/kennel/models/record.rb
CHANGED
@@ -60,23 +60,22 @@ module Kennel
|
|
60
60
|
"#{project.kennel_id}:#{kennel_id}"
|
61
61
|
end
|
62
62
|
|
63
|
-
def resolve_linked_tracking_ids(*)
|
63
|
+
def resolve_linked_tracking_ids!(*)
|
64
64
|
end
|
65
65
|
|
66
66
|
private
|
67
67
|
|
68
|
-
def resolve_link(id, type, id_map, force:
|
69
|
-
|
70
|
-
|
71
|
-
api_resource = self.class.api_resource
|
72
|
-
|
73
|
-
if found == :new
|
68
|
+
def resolve_link(id, type, id_map, force:)
|
69
|
+
value = id_map[id]
|
70
|
+
if value == :new
|
74
71
|
if force
|
75
|
-
|
72
|
+
# TODO: remove the need for this by sorting monitors by missing resolutions
|
73
|
+
invalid! "#{id} needs to already exist, try again"
|
76
74
|
else
|
77
|
-
|
78
|
-
Kennel::MISSING_ID
|
75
|
+
id # will be re-resolved by syncer after the linked object was created
|
79
76
|
end
|
77
|
+
elsif value
|
78
|
+
value
|
80
79
|
else
|
81
80
|
invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
|
82
81
|
end
|
data/lib/kennel/models/slo.rb
CHANGED
@@ -66,9 +66,9 @@ module Kennel
|
|
66
66
|
url[/\/slo\?slo_id=([a-z\d]+)/, 1]
|
67
67
|
end
|
68
68
|
|
69
|
-
def resolve_linked_tracking_ids(id_map)
|
69
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
70
70
|
as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
|
71
|
-
id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
|
71
|
+
id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
data/lib/kennel/syncer.rb
CHANGED
@@ -38,12 +38,23 @@ module Kennel
|
|
38
38
|
end
|
39
39
|
|
40
40
|
def update
|
41
|
+
changed = (@create + @update).map { |_, e| e } unless @create.empty?
|
42
|
+
|
41
43
|
@create.each do |_, e|
|
44
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
45
|
+
|
42
46
|
reply = @api.create e.class.api_resource, e.as_json
|
43
|
-
|
47
|
+
id = reply.fetch(:id)
|
48
|
+
|
49
|
+
# resolve ids we could previously no resolve
|
50
|
+
changed.delete e
|
51
|
+
resolve_linked_tracking_ids! from: [reply], to: changed
|
52
|
+
|
53
|
+
Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
44
54
|
end
|
45
55
|
|
46
56
|
@update.each do |id, e|
|
57
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
47
58
|
@api.update e.class.api_resource, id, e.as_json
|
48
59
|
Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
49
60
|
end
|
@@ -65,12 +76,10 @@ module Kennel
|
|
65
76
|
@delete = []
|
66
77
|
|
67
78
|
actual = Progress.progress("Downloading definitions") { download_definitions }
|
68
|
-
|
69
|
-
|
79
|
+
resolve_linked_tracking_ids! from: actual, to: @expected
|
80
|
+
filter_by_project! actual
|
70
81
|
|
71
82
|
Progress.progress "Diffing" do
|
72
|
-
filter_by_project! actual
|
73
|
-
|
74
83
|
items = actual.map do |a|
|
75
84
|
e = matching_expected(a)
|
76
85
|
if e && @expected.delete(e)
|
@@ -98,6 +107,7 @@ module Kennel
|
|
98
107
|
|
99
108
|
ensure_all_ids_found
|
100
109
|
@create = @expected.map { |e| [nil, e] }
|
110
|
+
@create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
|
101
111
|
end
|
102
112
|
|
103
113
|
@delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
|
@@ -178,7 +188,7 @@ module Kennel
|
|
178
188
|
end
|
179
189
|
|
180
190
|
# Do not add tracking-id when working with existing ids on a branch,
|
181
|
-
# so resource do not get deleted
|
191
|
+
# so resource do not get deleted fr:om merges to master.
|
182
192
|
# Also make sure the diff still makes sense, by kicking out the now noop-update.
|
183
193
|
#
|
184
194
|
# Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
|
@@ -203,10 +213,10 @@ module Kennel
|
|
203
213
|
end
|
204
214
|
end
|
205
215
|
|
206
|
-
def resolve_linked_tracking_ids(
|
207
|
-
map =
|
208
|
-
|
209
|
-
|
216
|
+
def resolve_linked_tracking_ids!(from:, to:)
|
217
|
+
map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
|
218
|
+
to.each { |e| map[e.tracking_id] ||= :new }
|
219
|
+
to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
|
210
220
|
end
|
211
221
|
|
212
222
|
def filter_by_project!(definitions)
|
data/lib/kennel/tasks.rb
CHANGED
@@ -68,15 +68,16 @@ namespace :kennel do
|
|
68
68
|
Kennel.update
|
69
69
|
end
|
70
70
|
|
71
|
-
desc "update
|
72
|
-
task :
|
73
|
-
|
74
|
-
|
71
|
+
desc "update on push to the default branch, otherwise show plan"
|
72
|
+
task :ci do
|
73
|
+
branch = (ENV["TRAVIS_BRANCH"] || ENV["GITHUB_REF"]).to_s.sub(/^refs\/heads\//, "")
|
74
|
+
on_default_branch = (branch == (ENV["DEFAULT_BRANCH"] || "master"))
|
75
|
+
is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false" || ENV["GITHUB_EVENT_NAME"] == "push")
|
75
76
|
task_name =
|
76
77
|
if on_default_branch && is_push
|
77
78
|
"kennel:update_datadog"
|
78
79
|
else
|
79
|
-
"kennel:plan" # show plan in
|
80
|
+
"kennel:plan" # show plan in CI logs
|
80
81
|
end
|
81
82
|
|
82
83
|
Rake::Task[task_name].invoke
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -1,16 +1,57 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](github/screen.png?raw=true)
|
13
17
|
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
|
14
55
|
## Structure
|
15
56
|
|
16
57
|
- `projects/` monitors/dashboards/etc scoped by project
|
@@ -91,7 +132,7 @@ end
|
|
91
132
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
92
133
|
- review changes then `git commit`
|
93
134
|
- make a PR ... get reviewed ... merge
|
94
|
-
- datadog is updated by
|
135
|
+
- datadog is updated by CI
|
95
136
|
|
96
137
|
### Adding a new dashboard
|
97
138
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
|
|
158
199
|
- figure out project name by converting the class name to snake-case
|
159
200
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
160
201
|
|
161
|
-
###
|
162
|
-
|
163
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
164
|
-
|
165
|
-
### Validating mentions work
|
166
|
-
|
167
|
-
`rake kennel:validate_mentions` should run as part of CI
|
168
|
-
|
169
|
-
### Grepping through all of datadog
|
170
|
-
|
171
|
-
`TYPE=monitor rake kennel:dump`
|
172
|
-
|
173
|
-
### Find all monitors with No-Data
|
174
|
-
|
175
|
-
`rake kennel:nodata TAG=team:foo`
|
176
|
-
|
177
|
-
## Examples
|
178
|
-
|
179
|
-
### Reusable monitors/dashes/etc
|
202
|
+
### Reuse
|
180
203
|
|
181
204
|
Add to `parts/<folder>`.
|
182
205
|
|
@@ -203,3 +226,22 @@ class Database < Kennel::Models::Project
|
|
203
226
|
)
|
204
227
|
end
|
205
228
|
```
|
229
|
+
|
230
|
+
## Helpers
|
231
|
+
|
232
|
+
### Listing un-muted alerts
|
233
|
+
|
234
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
235
|
+
|
236
|
+
### Validating mentions work
|
237
|
+
|
238
|
+
`rake kennel:validate_mentions` should run as part of CI
|
239
|
+
|
240
|
+
### Grepping through all of datadog
|
241
|
+
|
242
|
+
`TYPE=monitor rake kennel:dump`
|
243
|
+
|
244
|
+
### Find all monitors with No-Data
|
245
|
+
|
246
|
+
`rake kennel:nodata TAG=team:foo`
|
247
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.75.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-07-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|