kennel 1.72.0 → 1.74.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Readme.md +73 -28
- data/lib/kennel.rb +0 -1
- data/lib/kennel/importer.rb +3 -2
- data/lib/kennel/models/dashboard.rb +22 -12
- data/lib/kennel/models/monitor.rb +12 -7
- data/lib/kennel/models/record.rb +9 -10
- data/lib/kennel/models/slo.rb +2 -2
- data/lib/kennel/syncer.rb +20 -10
- data/lib/kennel/version.rb +1 -1
- data/template/Readme.md +67 -25
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ca8f1880a71be912c4d52c43b378781001c20eaa1797c923b46c7fab0134a92e
|
4
|
+
data.tar.gz: 5147a4185090e027300f8e74dd07df1b76d1d3a70b03e97986ca7043fa7a60ab
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 13cef2ac877cf523b13d34ccb969376af5ef451bca24e7484536785e8ebce86e713d2cd4572126cad0fdd62b7ee3d7d39c23563302d697570d18925ba1820e89
|
7
|
+
data.tar.gz: 9865461cb7c846ffabb03b5619ec7aa5833c4f7db1389efc834020b33f25246b4a6e39616df4ecd6cd0846c621a1d9e23d1672c243e43478ec4d747aeb28f36c
|
data/Readme.md
CHANGED
@@ -1,17 +1,58 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](template/github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](template/github/screen.png?raw=true)
|
17
|
+
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
13
54
|
<!-- NOT IN template/Readme.md -->
|
14
|
-
##
|
55
|
+
## Installation
|
15
56
|
|
16
57
|
- create a new private `kennel` repo for your organization (do not fork this repo)
|
17
58
|
- use the template folder as starting point:
|
@@ -22,8 +63,8 @@ Manage datadog monitors/dashboards/slos as code
|
|
22
63
|
cd kennel && git add . && git commit -m 'initial'
|
23
64
|
```
|
24
65
|
- add a basic projects and teams so others can copy-paste to get started
|
25
|
-
- setup travis build for your repo
|
26
|
-
- uncomment `.travis.yml` section for
|
66
|
+
- setup travis (CI) build for your repo
|
67
|
+
- uncomment `.travis.yml` section for datadog updates on merge
|
27
68
|
- follow `Setup` in your repos Readme.md
|
28
69
|
<!-- NOT IN -->
|
29
70
|
|
@@ -176,25 +217,7 @@ To link to existing monitors via their kennel_id
|
|
176
217
|
- figure out project name by converting the class name to snake-case
|
177
218
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
178
219
|
|
179
|
-
###
|
180
|
-
|
181
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
182
|
-
|
183
|
-
### Validating mentions work
|
184
|
-
|
185
|
-
`rake kennel:validate_mentions` should run as part of CI
|
186
|
-
|
187
|
-
### Grepping through all of datadog
|
188
|
-
|
189
|
-
`TYPE=monitor rake kennel:dump`
|
190
|
-
|
191
|
-
### Find all monitors with No-Data
|
192
|
-
|
193
|
-
`rake kennel:nodata TAG=team:foo`
|
194
|
-
|
195
|
-
## Examples
|
196
|
-
|
197
|
-
### Reusable monitors/dashes/etc
|
220
|
+
### Reuse
|
198
221
|
|
199
222
|
Add to `parts/<folder>`.
|
200
223
|
|
@@ -221,8 +244,30 @@ class Database < Kennel::Models::Project
|
|
221
244
|
)
|
222
245
|
end
|
223
246
|
```
|
247
|
+
|
248
|
+
## Helpers
|
249
|
+
|
250
|
+
### Listing un-muted alerts
|
251
|
+
|
252
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
253
|
+
|
254
|
+
### Validating mentions work
|
255
|
+
|
256
|
+
`rake kennel:validate_mentions` should run as part of CI
|
257
|
+
|
258
|
+
### Grepping through all of datadog
|
259
|
+
|
260
|
+
`TYPE=monitor rake kennel:dump`
|
261
|
+
|
262
|
+
### Find all monitors with No-Data
|
263
|
+
|
264
|
+
`rake kennel:nodata TAG=team:foo`
|
265
|
+
|
224
266
|
<!-- NOT IN template/Readme.md -->
|
225
267
|
|
268
|
+
|
269
|
+
## Development
|
270
|
+
|
226
271
|
### Integration testing
|
227
272
|
|
228
273
|
```Bash
|
data/lib/kennel.rb
CHANGED
data/lib/kennel/importer.rb
CHANGED
@@ -40,7 +40,8 @@ module Kennel
|
|
40
40
|
Kennel::Utils.parameterize(title)
|
41
41
|
end
|
42
42
|
|
43
|
-
|
43
|
+
case resource
|
44
|
+
when "monitor"
|
44
45
|
# flatten monitor options so they are all on the base
|
45
46
|
data.merge!(data.delete(:options))
|
46
47
|
data.merge!(data.delete(:thresholds) || {})
|
@@ -55,7 +56,7 @@ module Kennel
|
|
55
56
|
end
|
56
57
|
|
57
58
|
data[:type] = "query alert" if data[:type] == "metric alert"
|
58
|
-
|
59
|
+
when "dashboard"
|
59
60
|
widgets = data[:widgets]&.flat_map { |widget| widget.dig(:definition, :widgets) || [widget] }
|
60
61
|
widgets&.each { |widget| dry_up_query!(widget) }
|
61
62
|
end
|
@@ -13,16 +13,23 @@ module Kennel
|
|
13
13
|
REQUEST_DEFAULTS = {
|
14
14
|
style: { line_width: "normal", palette: "dog_classic", line_type: "solid" }
|
15
15
|
}.freeze
|
16
|
-
|
16
|
+
WIDGET_DEFAULTS = {
|
17
|
+
"timeseries" => { show_legend: false, legend_size: "0" },
|
18
|
+
"note" => { background_color: "white", font_size: "14", show_tick: false, tick_edge: "left", tick_pos: "50%", text_align: "left" }
|
19
|
+
}.freeze
|
17
20
|
SUPPORTED_DEFINITION_OPTIONS = [:events, :markers, :precision].freeze
|
18
21
|
|
22
|
+
DEFAULTS = {
|
23
|
+
template_variable_presets: nil
|
24
|
+
}.freeze
|
25
|
+
|
19
26
|
settings :title, :description, :definitions, :widgets, :layout_type, :template_variable_presets
|
20
27
|
|
21
28
|
defaults(
|
22
29
|
description: -> { "" },
|
23
30
|
definitions: -> { [] },
|
24
31
|
widgets: -> { [] },
|
25
|
-
template_variable_presets: -> {
|
32
|
+
template_variable_presets: -> { DEFAULTS.fetch(:template_variable_presets) },
|
26
33
|
id: -> { nil }
|
27
34
|
)
|
28
35
|
|
@@ -34,6 +41,8 @@ module Kennel
|
|
34
41
|
def normalize(expected, actual)
|
35
42
|
super
|
36
43
|
|
44
|
+
ignore_default(expected, actual, DEFAULTS)
|
45
|
+
|
37
46
|
widgets_pairs(expected, actual).each do |pair|
|
38
47
|
# conditional_formats ordering is randomly changed by datadog, compare a stable ordering
|
39
48
|
pair.each do |widgets|
|
@@ -44,7 +53,7 @@ module Kennel
|
|
44
53
|
end
|
45
54
|
end
|
46
55
|
|
47
|
-
|
56
|
+
ignore_widget_defaults pair
|
48
57
|
|
49
58
|
ignore_request_defaults(*pair)
|
50
59
|
|
@@ -55,11 +64,12 @@ module Kennel
|
|
55
64
|
|
56
65
|
private
|
57
66
|
|
58
|
-
def
|
67
|
+
def ignore_widget_defaults(pair)
|
59
68
|
pair.map(&:size).max.times do |i|
|
60
|
-
|
61
|
-
|
62
|
-
|
69
|
+
types = pair.map { |w| w.dig(i, :definition, :type) }.uniq
|
70
|
+
next unless types.size == 1
|
71
|
+
next unless defaults = WIDGET_DEFAULTS[types.first]
|
72
|
+
ignore_defaults(pair[0], pair[1], defaults, nesting: :definition)
|
63
73
|
end
|
64
74
|
end
|
65
75
|
|
@@ -122,7 +132,7 @@ module Kennel
|
|
122
132
|
url[/\/dashboard\/([a-z\d-]+)/, 1]
|
123
133
|
end
|
124
134
|
|
125
|
-
def resolve_linked_tracking_ids(id_map)
|
135
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
126
136
|
widgets = as_json[:widgets].flat_map { |w| [w, *w.dig(:definition, :widgets) || []] }
|
127
137
|
widgets.each do |widget|
|
128
138
|
next unless definition = widget[:definition]
|
@@ -130,16 +140,16 @@ module Kennel
|
|
130
140
|
when "uptime"
|
131
141
|
if ids = definition[:monitor_ids]
|
132
142
|
definition[:monitor_ids] = ids.map do |id|
|
133
|
-
tracking_id?(id) ? resolve_link(id, :monitor, id_map) : id
|
143
|
+
tracking_id?(id) ? resolve_link(id, :monitor, id_map, **args) : id
|
134
144
|
end
|
135
145
|
end
|
136
146
|
when "alert_graph"
|
137
147
|
if (id = definition[:alert_id]) && tracking_id?(id)
|
138
|
-
definition[:alert_id] = resolve_link(id, :monitor, id_map).to_s
|
148
|
+
definition[:alert_id] = resolve_link(id, :monitor, id_map, **args).to_s
|
139
149
|
end
|
140
150
|
when "slo"
|
141
151
|
if (id = definition[:slo_id]) && tracking_id?(id)
|
142
|
-
definition[:slo_id] = resolve_link(id, :slo, id_map).to_s
|
152
|
+
definition[:slo_id] = resolve_link(id, :slo, id_map, **args).to_s
|
143
153
|
end
|
144
154
|
end
|
145
155
|
end
|
@@ -172,7 +182,7 @@ module Kennel
|
|
172
182
|
|
173
183
|
# Avoid diff from datadog presets sorting.
|
174
184
|
presets = data[:template_variable_presets]
|
175
|
-
invalid! "template_variable_presets must be sorted by name" if presets != presets.sort_by { |p| p[:name] }
|
185
|
+
invalid! "template_variable_presets must be sorted by name" if presets && presets != presets.sort_by { |p| p[:name] }
|
176
186
|
end
|
177
187
|
|
178
188
|
def render_definitions
|
@@ -25,7 +25,7 @@ module Kennel
|
|
25
25
|
settings(
|
26
26
|
:query, :name, :message, :escalation_message, :critical, :type, :renotify_interval, :warning, :timeout_h, :evaluation_delay,
|
27
27
|
:ok, :no_data_timeframe, :notify_no_data, :notify_audit, :tags, :critical_recovery, :warning_recovery, :require_full_window,
|
28
|
-
:threshold_windows, :new_host_delay
|
28
|
+
:threshold_windows, :new_host_delay, :groupby_simple_monitor
|
29
29
|
)
|
30
30
|
|
31
31
|
defaults(
|
@@ -44,7 +44,8 @@ module Kennel
|
|
44
44
|
evaluation_delay: -> { MONITOR_OPTION_DEFAULTS.fetch(:evaluation_delay) },
|
45
45
|
critical_recovery: -> { nil },
|
46
46
|
warning_recovery: -> { nil },
|
47
|
-
threshold_windows: -> { nil }
|
47
|
+
threshold_windows: -> { nil },
|
48
|
+
groupby_simple_monitor: -> { nil }
|
48
49
|
)
|
49
50
|
|
50
51
|
def as_json
|
@@ -94,6 +95,11 @@ module Kennel
|
|
94
95
|
end
|
95
96
|
end
|
96
97
|
|
98
|
+
# option randomly pops up and cannot be removed
|
99
|
+
unless (group = groupby_simple_monitor).nil?
|
100
|
+
options[:groupby_simple_monitor] = group
|
101
|
+
end
|
102
|
+
|
97
103
|
if windows = threshold_windows
|
98
104
|
options[:threshold_windows] = windows
|
99
105
|
end
|
@@ -103,11 +109,10 @@ module Kennel
|
|
103
109
|
@as_json = data
|
104
110
|
end
|
105
111
|
|
106
|
-
def resolve_linked_tracking_ids(id_map)
|
112
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
107
113
|
if as_json[:type] == "composite"
|
108
114
|
as_json[:query] = as_json[:query].gsub(/%\{(.*?)\}/) do
|
109
|
-
|
110
|
-
resolve_link($1, :monitor, id_map, force: true)
|
115
|
+
resolve_link($1, :monitor, id_map, **args)
|
111
116
|
end
|
112
117
|
end
|
113
118
|
end
|
@@ -120,7 +125,7 @@ module Kennel
|
|
120
125
|
Utils.path_to_url "/monitors##{id}/edit"
|
121
126
|
end
|
122
127
|
|
123
|
-
# datadog uses
|
128
|
+
# datadog uses / for show and # for edit as separator in it's links
|
124
129
|
def self.parse_url(url)
|
125
130
|
return unless id = url[/\/monitors[\/#](\d+)/, 1]
|
126
131
|
Integer(id)
|
@@ -202,7 +207,7 @@ module Kennel
|
|
202
207
|
# verify is_match uses available variables
|
203
208
|
message = data.fetch(:message)
|
204
209
|
used = message.scan(/{{\s*#is_match\s*"([a-zA-Z\d_.-]+).name"/).flatten.uniq
|
205
|
-
allowed = data.fetch(:query)[/by\s*{([
|
210
|
+
allowed = data.fetch(:query)[/by\s*[({]([^})]+)[})]/, 1].to_s.gsub(/["']/, "").split(/\s*,\s*/)
|
206
211
|
unsupported = used - allowed
|
207
212
|
if unsupported.any?
|
208
213
|
invalid! "is_match used with #{unsupported}, but metric is only grouped by #{allowed}"
|
data/lib/kennel/models/record.rb
CHANGED
@@ -60,23 +60,22 @@ module Kennel
|
|
60
60
|
"#{project.kennel_id}:#{kennel_id}"
|
61
61
|
end
|
62
62
|
|
63
|
-
def resolve_linked_tracking_ids(*)
|
63
|
+
def resolve_linked_tracking_ids!(*)
|
64
64
|
end
|
65
65
|
|
66
66
|
private
|
67
67
|
|
68
|
-
def resolve_link(id, type, id_map, force:
|
69
|
-
|
70
|
-
|
71
|
-
api_resource = self.class.api_resource
|
72
|
-
|
73
|
-
if found == :new
|
68
|
+
def resolve_link(id, type, id_map, force:)
|
69
|
+
value = id_map[id]
|
70
|
+
if value == :new
|
74
71
|
if force
|
75
|
-
|
72
|
+
# TODO: remove the need for this by sorting monitors by missing resolutions
|
73
|
+
invalid! "#{id} needs to already exist, try again"
|
76
74
|
else
|
77
|
-
|
78
|
-
Kennel::MISSING_ID
|
75
|
+
id # will be re-resolved by syncer after the linked object was created
|
79
76
|
end
|
77
|
+
elsif value
|
78
|
+
value
|
80
79
|
else
|
81
80
|
invalid! "Unable to find #{type} #{id} (does not exist and is not being created by the current run)"
|
82
81
|
end
|
data/lib/kennel/models/slo.rb
CHANGED
@@ -66,9 +66,9 @@ module Kennel
|
|
66
66
|
url[/\/slo\?slo_id=([a-z\d]+)/, 1]
|
67
67
|
end
|
68
68
|
|
69
|
-
def resolve_linked_tracking_ids(id_map)
|
69
|
+
def resolve_linked_tracking_ids!(id_map, **args)
|
70
70
|
as_json[:monitor_ids] = as_json[:monitor_ids].map do |id|
|
71
|
-
id.is_a?(String) ? resolve_link(id, :monitor, id_map) : id
|
71
|
+
id.is_a?(String) ? resolve_link(id, :monitor, id_map, **args) : id
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
data/lib/kennel/syncer.rb
CHANGED
@@ -38,12 +38,23 @@ module Kennel
|
|
38
38
|
end
|
39
39
|
|
40
40
|
def update
|
41
|
+
changed = (@create + @update).map { |_, e| e } unless @create.empty?
|
42
|
+
|
41
43
|
@create.each do |_, e|
|
44
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
45
|
+
|
42
46
|
reply = @api.create e.class.api_resource, e.as_json
|
43
|
-
|
47
|
+
id = reply.fetch(:id)
|
48
|
+
|
49
|
+
# resolve ids we could previously no resolve
|
50
|
+
changed.delete e
|
51
|
+
resolve_linked_tracking_ids! from: [reply], to: changed
|
52
|
+
|
53
|
+
Kennel.out.puts "Created #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
44
54
|
end
|
45
55
|
|
46
56
|
@update.each do |id, e|
|
57
|
+
e.resolve_linked_tracking_ids!({}, force: true)
|
47
58
|
@api.update e.class.api_resource, id, e.as_json
|
48
59
|
Kennel.out.puts "Updated #{e.class.api_resource} #{tracking_id(e.as_json)} #{e.url(id)}"
|
49
60
|
end
|
@@ -65,12 +76,10 @@ module Kennel
|
|
65
76
|
@delete = []
|
66
77
|
|
67
78
|
actual = Progress.progress("Downloading definitions") { download_definitions }
|
68
|
-
|
69
|
-
|
79
|
+
resolve_linked_tracking_ids! from: actual, to: @expected
|
80
|
+
filter_by_project! actual
|
70
81
|
|
71
82
|
Progress.progress "Diffing" do
|
72
|
-
filter_by_project! actual
|
73
|
-
|
74
83
|
items = actual.map do |a|
|
75
84
|
e = matching_expected(a)
|
76
85
|
if e && @expected.delete(e)
|
@@ -98,6 +107,7 @@ module Kennel
|
|
98
107
|
|
99
108
|
ensure_all_ids_found
|
100
109
|
@create = @expected.map { |e| [nil, e] }
|
110
|
+
@create.sort_by! { |_, e| -DELETE_ORDER.index(e.class.api_resource) }
|
101
111
|
end
|
102
112
|
|
103
113
|
@delete.sort_by! { |_, _, a| DELETE_ORDER.index a.fetch(:api_resource) }
|
@@ -178,7 +188,7 @@ module Kennel
|
|
178
188
|
end
|
179
189
|
|
180
190
|
# Do not add tracking-id when working with existing ids on a branch,
|
181
|
-
# so resource do not get deleted
|
191
|
+
# so resource do not get deleted fr:om merges to master.
|
182
192
|
# Also make sure the diff still makes sense, by kicking out the now noop-update.
|
183
193
|
#
|
184
194
|
# Note: ideally we'd never add tracking in the first place, but at that point we do not know the diff yet
|
@@ -203,10 +213,10 @@ module Kennel
|
|
203
213
|
end
|
204
214
|
end
|
205
215
|
|
206
|
-
def resolve_linked_tracking_ids(
|
207
|
-
map =
|
208
|
-
|
209
|
-
|
216
|
+
def resolve_linked_tracking_ids!(from:, to:)
|
217
|
+
map = from.each_with_object({}) { |a, lookup| lookup[tracking_id(a)] = a.fetch(:id) }
|
218
|
+
to.each { |e| map[e.tracking_id] ||= :new }
|
219
|
+
to.each { |e| e.resolve_linked_tracking_ids!(map, force: false) }
|
210
220
|
end
|
211
221
|
|
212
222
|
def filter_by_project!(definitions)
|
data/lib/kennel/version.rb
CHANGED
data/template/Readme.md
CHANGED
@@ -1,16 +1,57 @@
|
|
1
|
-
# Kennel
|
2
|
-
|
3
1
|
![](github/cage.jpg?raw=true)
|
4
2
|
|
5
|
-
Manage
|
3
|
+
Manage Datadog Monitors / Dashboards / Slos as code
|
6
4
|
|
7
|
-
-
|
8
|
-
- Changes are PR reviewed and
|
5
|
+
- DRY, searchable, audited, documented
|
6
|
+
- Changes are PR reviewed and applied on merge
|
9
7
|
- Updating shows diff before applying
|
10
|
-
- Automated import of existing
|
8
|
+
- Automated import of existing resources
|
9
|
+
- Resources are grouped into projects that belong to teams and inherit tags
|
10
|
+
- No copy-pasting of ids to create new resources
|
11
|
+
- Automated cleanup when removing code
|
12
|
+
- [Helpers](#helpers) for automating common tasks
|
13
|
+
|
14
|
+
### Applying changes
|
11
15
|
|
12
16
|
![](github/screen.png?raw=true)
|
13
17
|
|
18
|
+
### Example code
|
19
|
+
|
20
|
+
```Ruby
|
21
|
+
# teams/foo.rb
|
22
|
+
module Teams
|
23
|
+
class Foo < Kennel::Models::Team
|
24
|
+
defaults(mention: -> { "@slack-my-team" })
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# projects/bar.rb
|
29
|
+
class Bar < Kennel::Models::Project
|
30
|
+
defaults(
|
31
|
+
team: -> { Teams::Foo.new }, # use mention and tags from the team
|
32
|
+
parts: -> {
|
33
|
+
[
|
34
|
+
Kennel::Models::Monitor.new(
|
35
|
+
self, # the current project
|
36
|
+
type: -> { "query alert" },
|
37
|
+
kennel_id: -> { "load-too-high" }, # pick a unique name
|
38
|
+
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
|
39
|
+
message: -> {
|
40
|
+
<<~TEXT
|
41
|
+
This is bad!
|
42
|
+
#{super()} # inserts mention from team
|
43
|
+
TEXT
|
44
|
+
},
|
45
|
+
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
|
46
|
+
critical: -> { 20 }
|
47
|
+
)
|
48
|
+
]
|
49
|
+
}
|
50
|
+
)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
|
14
55
|
## Structure
|
15
56
|
|
16
57
|
- `projects/` monitors/dashboards/etc scoped by project
|
@@ -158,25 +199,7 @@ To link to existing monitors via their kennel_id
|
|
158
199
|
- figure out project name by converting the class name to snake-case
|
159
200
|
- run `PROJECT=foo bundle exec rake kennel:update_datadog` to test changes for a single project
|
160
201
|
|
161
|
-
###
|
162
|
-
|
163
|
-
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
164
|
-
|
165
|
-
### Validating mentions work
|
166
|
-
|
167
|
-
`rake kennel:validate_mentions` should run as part of CI
|
168
|
-
|
169
|
-
### Grepping through all of datadog
|
170
|
-
|
171
|
-
`TYPE=monitor rake kennel:dump`
|
172
|
-
|
173
|
-
### Find all monitors with No-Data
|
174
|
-
|
175
|
-
`rake kennel:nodata TAG=team:foo`
|
176
|
-
|
177
|
-
## Examples
|
178
|
-
|
179
|
-
### Reusable monitors/dashes/etc
|
202
|
+
### Reuse
|
180
203
|
|
181
204
|
Add to `parts/<folder>`.
|
182
205
|
|
@@ -203,3 +226,22 @@ class Database < Kennel::Models::Project
|
|
203
226
|
)
|
204
227
|
end
|
205
228
|
```
|
229
|
+
|
230
|
+
## Helpers
|
231
|
+
|
232
|
+
### Listing un-muted alerts
|
233
|
+
|
234
|
+
Run `rake kennel:alerts TAG=service:my-service` to see all un-muted alerts for a given datadog monitor tag.
|
235
|
+
|
236
|
+
### Validating mentions work
|
237
|
+
|
238
|
+
`rake kennel:validate_mentions` should run as part of CI
|
239
|
+
|
240
|
+
### Grepping through all of datadog
|
241
|
+
|
242
|
+
`TYPE=monitor rake kennel:dump`
|
243
|
+
|
244
|
+
### Find all monitors with No-Data
|
245
|
+
|
246
|
+
`rake kennel:nodata TAG=team:foo`
|
247
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kennel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.74.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Grosser
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-07-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: faraday
|