kennel 1.72.2 → 1.75.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Readme.md +74 -29
- data/lib/kennel.rb +0 -1
- data/lib/kennel/importer.rb +3 -2
- data/lib/kennel/models/dashboard.rb +15 -11
- data/lib/kennel/models/monitor.rb +6 -6
- data/lib/kennel/models/record.rb +9 -10
- data/lib/kennel/models/slo.rb +2 -2
- data/lib/kennel/syncer.rb +20 -10
- data/lib/kennel/tasks.rb +6 -5
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +68 -26
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 23af2ecfb9019df555f7fe3a3954f4d992208927744093ff1f8886afd571e457
|
4
|
+
data.tar.gz: 6e2c902c47020e46b5fc0e647d8d63bbd7227657ce78303c673470be840ecbf9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c2dfa7e9f54c83d3369b48b91896cb7dc7cfe75bb94d672d7de07bc7a6b8ea3f63ebe38788dddf5de90ead3b73f977c42f87497d61e7b23def6ae385d0a42213
|
7
|
+
data.tar.gz: 9ebb18f6d9c54be18a783d8511e3128b9c59b5898f9f14a1be77c871dcdab837681bf04050ec97a1a2f675c84fda624469edaed0af9f5e0da11b60517ba93cea
|
data/Readme.md
CHANGED
@@ -1,17 +1,58 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|

|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|

|
17
|
+
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
13
54
|
<!-- NOT IN template/Readme.md -->
|
14
|
-
##
|
55
|
+
## Installation
|
15
56
|
|
16
57
|
- create a new private `kennel` repo for your organization (do not fork this repo)
|
17
58
|
- use the template folder as starting point:
|
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
|
|
22
63
|
cd kennel && git add . && git commit -m 'initial'
|
23
64
|
```
|
24
65
|
- add a basic projects and teams so others can copy-paste to get started
|
25
|
-
- setup
|
26
|
-
- uncomment `.travis.yml` section for
|
66
|
+
- setup CI build for your repo (travis and Github Actions supported)
|
67
|
+
- uncomment `.travis.yml` section for datadog updates on merge (TODO: example setup for Github Actions)
|
27
68
|
- follow `Setup` in your repos Readme.md
|
28
69
|
<!-- NOT IN -->
|
29
70
|
|
@@ -109,7 +150,7 @@ end
|
|
109
150
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
110
151
|
- review changes then `git commit`
|
111
152
|
- make a PR ... get reviewed ... merge
|
112
|
-
- datadog is updated by
|
153
|
+
- datadog is updated by CI
|
113
154
|
|
114
155
|
### Adding a new dashboard
|
115
156
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
|
|
176
217
|
- figure out project name by converting the class name to snake-case
|
177
218
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
178
219
|
|
179
|
-
###
|
180
|
-
|
181
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
182
|
-
|
183
|
-
### Validating mentions work
|
184
|
-
|
185
|
-
`rake kennel:validate_mentions` should run as part of CI
|
186
|
-
|
187
|
-
### Grepping through all of datadog
|
188
|
-
|
189
|
-
`TYPE=monitor rake kennel:dump`
|
190
|
-
|
191
|
-
### Find all monitors with No-Data
|
192
|
-
|
193
|
-
`rake kennel:nodata TAG=team:foo`
|
194
|
-
|
195
|
-
## Examples
|
196
|
-
|
197
|
-
### Reusable monitors/dashes/etc
|
220
|
+
### Reuse
|
198
221
|
|
199
222
|
Add to `parts/<folder>`.
|
200
223
|
|
@@ -221,8 +244,30 @@ class Database < Kennel::Models::Project
|
|
221
244
|
)
|
222
245
|
end
|
223
246
|
```
|
247
|
+
|
248
|
+
## Helpers
|
249
|
+
|
250
|
+
### Listing un-muted alerts
|
251
|
+
|
252
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
253
|
+
|
254
|
+
### Validating mentions work
|
255
|
+
|
256
|
+
`rake kennel:validate_mentions` should run as part of CI
|
257
|
+
|
258
|
+
### Grepping through all of datadog
|
259
|
+
|
260
|
+
`TYPE=monitor rake kennel:dump`
|
261
|
+
|
262
|
+
### Find all monitors with No-Data
|
263
|
+
|
264
|
+
`rake kennel:nodata TAG=team:foo`
|
265
|
+
|
224
266
|
<!-- NOT IN template/Readme.md -->
|
225
267
|
|
268
|
+
|
269
|
+
## Development
|
270
|
+
|
226
271
|
### Integration testing
|
227
272
|
|
228
273
|
```Bash
|
data/lib/kennel.rb
CHANGED
data/lib/kennel/importer.rb
CHANGED
@@ -40,7 +40,8 @@ module Kennel
|
|
40
40
|
Kennel::Utils.parameterize(title)
|
41
41
|
end
|
42
42
|
|
43
|
-
|
43
|
+
case resource
|
44
|
+
when "monitor"
|
44
45
|
# flatten monitor options so they are all on the base
|
45
46
|
data.merge!(data.delete(:options))
|
46
47
|
data.merge!(data.delete(:thresholds) || {})
|
@@ -55,7 +56,7 @@ module Kennel
|
|
55
56
|
end
|
56
57
|
|
57
58
|
data[:type] = "query alert" if data[:type] == "metric alert"
|
58
|
-
|
59
|
+
when "dashboard"
|
59
60
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
60
61
|
widgets&.each { |widget| dry_up_query!(widget) }
|
61
62
|
end
|
@@ -13,7 +13,10 @@ module Kennel
|
|
13
13
|
REQUEST_DEFAULTS = {
|
14
14
|
style: { line_width: "normal", palette: "dog_classic", line_type: "solid" }
|
15
15
|
}.freeze
|
16
|
-
|
16
|
+
WIDGET_DEFAULTS = {
|
17
|
+
"timeseries" => { show_legend: false, legend_size: "0" },
|
18
|
+
"note" => { background_color: "white", font_size: "14", show_tick: false, tick_edge: "left", tick_pos: "50%", text_align: "left" }
|
19
|
+
}.freeze
|
17
20
|
SUPPORTED_DEFINITION_OPTIONS = [:events, :markers, :precision].freeze
|
18
21
|
|
19
22
|
DEFAULTS = {
|
@@ -26,7 +29,7 @@ module Kennel
|
|
26
29
|
description: -> { "" },
|
27
30
|
definitions: -> { [] },
|
28
31
|
widgets: -> { [] },
|
29
|
-
template_variable_presets: -> {
|
32
|
+
template_variable_presets: -> { DEFAULTS.fetch(:template_variable_presets) },
|
30
33
|
id: -> { nil }
|
31
34
|
)
|
32
35
|
|
@@ -50,7 +53,7 @@ module Kennel
|
|
50
53
|
end
|
51
54
|
end
|
52
55
|
|
53
|
-
|
56
|
+
ignore_widget_defaults pair
|
54
57
|
|
55
58
|
ignore_request_defaults(*pair)
|
56
59
|
|
@@ -61,11 +64,12 @@ module Kennel
|
|
61
64
|
|
62
65
|
private
|
63
66
|
|
64
|
-
def
|
67
|
+
def ignore_widget_defaults(pair)
|
65
68
|
pair.map(&:size).max.times do |i|
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
+
types = pair.map { |w| w.dig(i, :definition, :type) }.uniq
|
70
|
+
next unless types.size == 1
|
71
|
+
next unless defaults = WIDGET_DEFAULTS[types.first]
|
72
|
+
ignore_defaults(pair[0], pair[1], defaults, nesting: :definition)
|
69
73
|
end
|
70
74
|
end
|
71
75
|
|
@@ -128,7 +132,7 @@ module Kennel
|
|
128
132
|
url[/\/dashboard\/([a-z\d-]+)/, 1]
|
129
133
|
end
|
130
134
|
|
131
|
-
def resolve_linked_tracking_ids(id_map)
|
135
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
132
136
|
widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
|
133
137
|
widgets.each do |widget|
|
134
138
|
next unless definition = widget[:definition]
|
@@ -136,16 +140,16 @@ module Kennel
|
|
136
140
|
when "uptime"
|
137
141
|
if ids = definition[:monitor_ids]
|
138
142
|
definition[:monitor_ids] = ids.map do |id|
|
139
|
-
tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
|
143
|
+
tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
|
140
144
|
end
|
141
145
|
end
|
142
146
|
when "alert_graph"
|
143
147
|
if (id = definition[:alert_id]) && tracking_id?(id)
|
144
|
-
definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
|
148
|
+
definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
|
145
149
|
end
|
146
150
|
when "slo"
|
147
151
|
if (id = definition[:slo_id]) && tracking_id?(id)
|
148
|
-
definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
|
152
|
+
definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
|
149
153
|
end
|
150
154
|
end
|
151
155
|
end
|
@@ -18,7 +18,8 @@ module Kennel
|
|
18
18
|
new_host_delay: 300,
|
19
19
|
timeout_h: 0,
|
20
20
|
renotify_interval: 0,
|
21
|
-
no_data_timeframe: nil # this works out ok since if notify_no_data is on, it would never be nil
|
21
|
+
no_data_timeframe: nil, # this works out ok since if notify_no_data is on, it would never be nil
|
22
|
+
groupby_simple_monitor: false
|
22
23
|
}.freeze
|
23
24
|
DEFAULT_ESCALATION_MESSAGE = ["", nil].freeze
|
24
25
|
|
@@ -103,11 +104,10 @@ module Kennel
|
|
103
104
|
@as_json = data
|
104
105
|
end
|
105
106
|
|
106
|
-
def resolve_linked_tracking_ids(id_map)
|
107
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
107
108
|
if as_json[:type] == "composite"
|
108
109
|
as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
|
109
|
-
|
110
|
-
resolve_link($1, :monitor, id_map, force: true)
|
110
|
+
resolve_link($1, :monitor, id_map, **args)
|
111
111
|
end
|
112
112
|
end
|
113
113
|
end
|
@@ -120,7 +120,7 @@ module Kennel
|
|
120
120
|
Utils.path_to_url "/monitors##{id}/edit"
|
121
121
|
end
|
122
122
|
|
123
|
-
# datadog uses
|
123
|
+
# datadog uses / for show and # for edit as separator in it's links
|
124
124
|
def self.parse_url(url)
|
125
125
|
return unless id = url[/\/monitors[\/#](\d+)/, 1]
|
126
126
|
Integer(id)
|
@@ -202,7 +202,7 @@ module Kennel
|
|
202
202
|
# verify is_match uses available variables
|
203
203
|
message = data.fetch(:message)
|
204
204
|
used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
|
205
|
-
allowed = data.fetch(:query)[/by\s*[
|
205
|
+
allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
|
206
206
|
unsupported = used - allowed
|
207
207
|
if unsupported.any?
|
208
208
|
invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"
|
data/lib/kennel/models/record.rb
CHANGED
@@ -60,23 +60,22 @@ module Kennel
|
|
60
60
|
"#{project.kennel_id}:#{kennel_id}"
|
61
61
|
end
|
62
62
|
|
63
|
-
def resolve_linked_tracking_ids(*)
|
63
|
+
def resolve_linked_tracking_ids!(*)
|
64
64
|
end
|
65
65
|
|
66
66
|
private
|
67
67
|
|
68
|
-
def resolve_link(id, type, id_map, force:
|
69
|
-
|
70
|
-
|
71
|
-
api_resource = self.class.api_resource
|
72
|
-
|
73
|
-
if found == :new
|
68
|
+
def resolve_link(id, type, id_map, force:)
|
69
|
+
value = id_map[id]
|
70
|
+
if value == :new
|
74
71
|
if force
|
75
|
-
|
72
|
+
# TODO: remove the need for this by sorting monitors by missing resolutions
|
73
|
+
invalid! "#{id} needs to already exist, try again"
|
76
74
|
else
|
77
|
-
|
78
|
-
Kennel::MISSING_ID
|
75
|
+
id # will be re-resolved by syncer after the linked object was created
|
79
76
|
end
|
77
|
+
elsif value
|
78
|
+
value
|
80
79
|
else
|
81
80
|
invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
|
82
81
|
end
|
data/lib/kennel/models/slo.rb
CHANGED
@@ -66,9 +66,9 @@ module Kennel
|
|
66
66
|
url[/\/slo\?slo_id=([a-z\d]+)/, 1]
|
67
67
|
end
|
68
68
|
|
69
|
-
def resolve_linked_tracking_ids(id_map)
|
69
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
70
70
|
as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
|
71
|
-
id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
|
71
|
+
id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
data/lib/kennel/syncer.rb
CHANGED
@@ -38,12 +38,23 @@ module Kennel
|
|
38
38
|
end
|
39
39
|
|
40
40
|
def update
|
41
|
+
changed = (@create + @update).map { |_, e| e } unless @create.empty?
|
42
|
+
|
41
43
|
@create.each do |_, e|
|
44
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
45
|
+
|
42
46
|
reply = @api.create e.class.api_resource, e.as_json
|
43
|
-
|
47
|
+
id = reply.fetch(:id)
|
48
|
+
|
49
|
+
# resolve ids we could previously no resolve
|
50
|
+
changed.delete e
|
51
|
+
resolve_linked_tracking_ids! from: [reply], to: changed
|
52
|
+
|
53
|
+
Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
44
54
|
end
|
45
55
|
|
46
56
|
@update.each do |id, e|
|
57
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
47
58
|
@api.update e.class.api_resource, id, e.as_json
|
48
59
|
Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
49
60
|
end
|
@@ -65,12 +76,10 @@ module Kennel
|
|
65
76
|
@delete = []
|
66
77
|
|
67
78
|
actual = Progress.progress("Downloading definitions") { download_definitions }
|
68
|
-
|
69
|
-
|
79
|
+
resolve_linked_tracking_ids! from: actual, to: @expected
|
80
|
+
filter_by_project! actual
|
70
81
|
|
71
82
|
Progress.progress "Diffing" do
|
72
|
-
filter_by_project! actual
|
73
|
-
|
74
83
|
items = actual.map do |a|
|
75
84
|
e = matching_expected(a)
|
76
85
|
if e && @expected.delete(e)
|
@@ -98,6 +107,7 @@ module Kennel
|
|
98
107
|
|
99
108
|
ensure_all_ids_found
|
100
109
|
@create = @expected.map { |e| [nil, e] }
|
110
|
+
@create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
|
101
111
|
end
|
102
112
|
|
103
113
|
@delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
|
@@ -178,7 +188,7 @@ module Kennel
|
|
178
188
|
end
|
179
189
|
|
180
190
|
# Do not add tracking-id when working with existing ids on a branch,
|
181
|
-
# so resource do not get deleted
|
191
|
+
# so resource do not get deleted fr:om merges to master.
|
182
192
|
# Also make sure the diff still makes sense, by kicking out the now noop-update.
|
183
193
|
#
|
184
194
|
# Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
|
@@ -203,10 +213,10 @@ module Kennel
|
|
203
213
|
end
|
204
214
|
end
|
205
215
|
|
206
|
-
def resolve_linked_tracking_ids(
|
207
|
-
map =
|
208
|
-
|
209
|
-
|
216
|
+
def resolve_linked_tracking_ids!(from:, to:)
|
217
|
+
map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
|
218
|
+
to.each { |e| map[e.tracking_id] ||= :new }
|
219
|
+
to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
|
210
220
|
end
|
211
221
|
|
212
222
|
def filter_by_project!(definitions)
|
data/lib/kennel/tasks.rb
CHANGED
@@ -68,15 +68,16 @@ namespace :kennel do
|
|
68
68
|
Kennel.update
|
69
69
|
end
|
70
70
|
|
71
|
-
desc "update
|
72
|
-
task :
|
73
|
-
|
74
|
-
|
71
|
+
desc "update on push to the default branch, otherwise show plan"
|
72
|
+
task :ci do
|
73
|
+
branch = (ENV["TRAVIS_BRANCH"] || ENV["GITHUB_REF"]).to_s.sub(/^refs\/heads\//, "")
|
74
|
+
on_default_branch = (branch == (ENV["DEFAULT_BRANCH"] || "master"))
|
75
|
+
is_push = (ENV["TRAVIS_PULL_REQUEST"] == "false" || ENV["GITHUB_EVENT_NAME"] == "push")
|
75
76
|
task_name =
|
76
77
|
if on_default_branch && is_push
|
77
78
|
"kennel:update_datadog"
|
78
79
|
else
|
79
|
-
"kennel:plan" # show plan in
|
80
|
+
"kennel:plan" # show plan in CI logs
|
80
81
|
end
|
81
82
|
|
82
83
|
Rake::Task[task_name].invoke
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -1,16 +1,57 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|

|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|

|
13
17
|
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
|
14
55
|
## Structure
|
15
56
|
|
16
57
|
- `projects/` monitors/dashboards/etc scoped by project
|
@@ -91,7 +132,7 @@ end
|
|
91
132
|
- alternatively: `bundle exec rake generate` to only locally update the generated `json` files
|
92
133
|
- review changes then `git commit`
|
93
134
|
- make a PR ... get reviewed ... merge
|
94
|
-
- datadog is updated by
|
135
|
+
- datadog is updated by CI
|
95
136
|
|
96
137
|
### Adding a new dashboard
|
97
138
|
- go to [datadog dashboard UI](https://app.datadoghq.com/dashboard/lists) and click on _New Dashboard_ to create a dashboard
|
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
|
|
158
199
|
- figure out project name by converting the class name to snake-case
|
159
200
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
160
201
|
|
161
|
-
###
|
162
|
-
|
163
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
164
|
-
|
165
|
-
### Validating mentions work
|
166
|
-
|
167
|
-
`rake kennel:validate_mentions` should run as part of CI
|
168
|
-
|
169
|
-
### Grepping through all of datadog
|
170
|
-
|
171
|
-
`TYPE=monitor rake kennel:dump`
|
172
|
-
|
173
|
-
### Find all monitors with No-Data
|
174
|
-
|
175
|
-
`rake kennel:nodata TAG=team:foo`
|
176
|
-
|
177
|
-
## Examples
|
178
|
-
|
179
|
-
### Reusable monitors/dashes/etc
|
202
|
+
### Reuse
|
180
203
|
|
181
204
|
Add to `parts/<folder>`.
|
182
205
|
|
@@ -203,3 +226,22 @@ class Database < Kennel::Models::Project
|
|
203
226
|
)
|
204
227
|
end
|
205
228
|
```
|
229
|
+
|
230
|
+
## Helpers
|
231
|
+
|
232
|
+
### Listing un-muted alerts
|
233
|
+
|
234
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
235
|
+
|
236
|
+
### Validating mentions work
|
237
|
+
|
238
|
+
`rake kennel:validate_mentions` should run as part of CI
|
239
|
+
|
240
|
+
### Grepping through all of datadog
|
241
|
+
|
242
|
+
`TYPE=monitor rake kennel:dump`
|
243
|
+
|
244
|
+
### Find all monitors with No-Data
|
245
|
+
|
246
|
+
`rake kennel:nodata TAG=team:foo`
|
247
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.75.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-07-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|