cfn-guardian 0.14.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Gemfile.lock +1 -1
- data/README.md +1 -0
- data/docs/ecs_scheduled_tasks.md +99 -0
- data/docs/overview.md +2 -1
- data/lib/cfnguardian/compile.rb +1 -0
- data/lib/cfnguardian/models/alarm.rb +19 -0
- data/lib/cfnguardian/models/event_subscription.rb +24 -0
- data/lib/cfnguardian/resources/ecs_scheduled_task.rb +30 -0
- data/lib/cfnguardian/version.rb +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 97db5ada9a8fd070dbc9cbb851b3d0f1b0352db96014d9f29883c359a47b92a4
|
|
4
|
+
data.tar.gz: 8ce062a52661342ab8c78e0adccb357887234384e4ff2e1cd4307cb3066bd41b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b58d879093561d7f836e73b02aaceb6f3499209907ec0aaf4e66e272398ac53756b51d0820ef319cf89b7bfdeedd2fb44f57d7aee1ddcec904122af523a2ca64
|
|
7
|
+
data.tar.gz: 0e582fcd0bda145414d31888bde92e0b866a895c82e45841760085ee2ca9466281776750787c0d7c9973a96b227e69e200c79891d9246b4bd294f7b118155fab
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# ECS Scheduled Tasks
|
|
2
|
+
|
|
3
|
+
`ECSScheduledTask` monitors ECS tasks that run outside of an ECS service, for example tasks launched on a schedule by an EventBridge rule or EventBridge Scheduler. These tasks have no service to attach `ECSService` alarms to, so Guardian watches the `ECS Task State Change` events for the cluster instead and notifies when a task fails.
|
|
4
|
+
|
|
5
|
+
## Configuration
|
|
6
|
+
|
|
7
|
+
```yaml
|
|
8
|
+
Resources:
|
|
9
|
+
ECSScheduledTask:
|
|
10
|
+
# Id is the ECS cluster name. Monitors every standalone task in the cluster.
|
|
11
|
+
- Id: batch-jobs
|
|
12
|
+
# Optionally narrow to a single task definition family.
|
|
13
|
+
- Id: shared-cluster
|
|
14
|
+
TaskDefinitionFamily: report-export
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
| Key | Required | Description |
|
|
18
|
+
|---|---|---|
|
|
19
|
+
| `Id` | yes | ECS cluster name |
|
|
20
|
+
| `TaskDefinitionFamily` | no | Only match tasks from this task definition family, matched on the task definition ARN |
|
|
21
|
+
|
|
22
|
+
Every event subscription in this group, including custom ones, is scoped to:
|
|
23
|
+
|
|
24
|
+
- the cluster in `Id`
|
|
25
|
+
- tasks not started by an ECS service (the task's `group` doesn't start with `service:`), so service deployments don't trigger alerts
|
|
26
|
+
- the `TaskDefinitionFamily`, if set
|
|
27
|
+
|
|
28
|
+
If a scheduled task is launched with a custom `group` starting with `service:`, it won't be matched.
|
|
29
|
+
|
|
30
|
+
## Default Event Subscriptions
|
|
31
|
+
|
|
32
|
+
Both are sent to the `Events` topic by default.
|
|
33
|
+
|
|
34
|
+
| Name | Fires when |
|
|
35
|
+
|---|---|
|
|
36
|
+
| `TaskFailed` | A task stops and any of its containers has a non-zero exit code |
|
|
37
|
+
| `TaskFailedToStart` | A task stops with stop code `TaskFailedToStart`, e.g. the image can't be pulled or a secret can't be fetched |
|
|
38
|
+
|
|
39
|
+
A container with no exit code (null or missing, e.g. one that never started) doesn't match. EventBridge matches `exitCode` against every container in the task, so a sidecar (such as a log router) that exits non-zero will also trigger `TaskFailed`.
|
|
40
|
+
|
|
41
|
+
## Overriding Defaults
|
|
42
|
+
|
|
43
|
+
Event subscriptions are configured under the `EventSubscriptions` key. See [event subscriptions](event_subscriptions.md).
|
|
44
|
+
|
|
45
|
+
```yaml
|
|
46
|
+
EventSubscriptions:
|
|
47
|
+
ECSScheduledTask:
|
|
48
|
+
# page on failed runs instead of sending to the Events topic
|
|
49
|
+
TaskFailed:
|
|
50
|
+
Topic: Critical
|
|
51
|
+
# disable the failed-to-start notification
|
|
52
|
+
TaskFailedToStart: false
|
|
53
|
+
# custom subscriptions default to the ECS Task State Change detail type
|
|
54
|
+
# and are scoped to the cluster and family like the defaults
|
|
55
|
+
TaskStoppedBySpot:
|
|
56
|
+
Detail:
|
|
57
|
+
stopCode: [SpotInterruption]
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Alarms
|
|
61
|
+
|
|
62
|
+
There are no default alarms, as scheduled tasks only publish metrics while they run. If [Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html) is enabled on the cluster, alarms can be added under `Templates`. They use the `ECS/ContainerInsights` namespace with the `ClusterName` dimension, plus `TaskDefinitionFamily` if set, and treat missing data as not breaching.
|
|
63
|
+
|
|
64
|
+
Alarms are named `guardian-ECSScheduledTask-<Id>-<AlarmName>`, or `guardian-ECSScheduledTask-<Id>:<TaskDefinitionFamily>-<AlarmName>` when a family is set, so a cluster-wide entry and family entries on the same cluster each get their own alarm.
|
|
65
|
+
|
|
66
|
+
```yaml
|
|
67
|
+
Templates:
|
|
68
|
+
ECSScheduledTask:
|
|
69
|
+
MemoryUtilizedHigh:
|
|
70
|
+
MetricName: MemoryUtilized
|
|
71
|
+
Statistic: Maximum
|
|
72
|
+
Threshold: 1800 # MiB
|
|
73
|
+
AlarmAction: Warning
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Missed Runs
|
|
77
|
+
|
|
78
|
+
The event subscriptions only fire when a task runs and fails. They won't fire if the schedule never launches a task. If `RunTask` itself fails (for example on a bad network configuration), ECS creates no task and emits no events. That shows up in the launcher's own metrics instead: `FailedInvocations` in the `AWS/Events` namespace for an EventBridge rule, or `TargetErrorCount` and `InvocationDroppedCount` in the `AWS/Scheduler` namespace for EventBridge Scheduler (Scheduler doesn't publish `FailedInvocations`).
|
|
79
|
+
|
|
80
|
+
To catch a missed run, add a `LogGroup` metric filter that counts a line the task logs on success, and alarm when the count is below 1 over the schedule interval. A metric filter with no matches publishes no data, so the alarm must treat missing data as breaching. See [log group metric filters](custom_checks/log_group_metric_filters.md).
|
|
81
|
+
|
|
82
|
+
```yaml
|
|
83
|
+
Resources:
|
|
84
|
+
LogGroup:
|
|
85
|
+
- Id: batch-jobs
|
|
86
|
+
MetricFilters:
|
|
87
|
+
- MetricName: ReportExportSucceeded
|
|
88
|
+
Pattern: '"export complete"'
|
|
89
|
+
|
|
90
|
+
Templates:
|
|
91
|
+
LogGroup:
|
|
92
|
+
ReportExportSucceeded:
|
|
93
|
+
ComparisonOperator: LessThanThreshold
|
|
94
|
+
Threshold: 1
|
|
95
|
+
Period: 86400 # the schedule interval, here daily
|
|
96
|
+
EvaluationPeriods: 1
|
|
97
|
+
TreatMissingData: breaching
|
|
98
|
+
AlarmAction: Warning
|
|
99
|
+
```
|
data/docs/overview.md
CHANGED
|
@@ -24,4 +24,5 @@
|
|
|
24
24
|
10. [Dimension Variables](variables.md)
|
|
25
25
|
11. [Search Expression Alarms](search_expressions.md)
|
|
26
26
|
12. [Alarm Tags](alarm_tags.md)
|
|
27
|
-
13. [Anomaly Detection Alarms](anomaly_detection.md)
|
|
27
|
+
13. [Anomaly Detection Alarms](anomaly_detection.md)
|
|
28
|
+
14. [ECS Scheduled Tasks](ecs_scheduled_tasks.md)
|
data/lib/cfnguardian/compile.rb
CHANGED
|
@@ -20,6 +20,7 @@ require 'cfnguardian/resources/dynamodb_table'
|
|
|
20
20
|
require 'cfnguardian/resources/ec2_instance'
|
|
21
21
|
require 'cfnguardian/resources/ecs_cluster'
|
|
22
22
|
require 'cfnguardian/resources/ecs_service'
|
|
23
|
+
require 'cfnguardian/resources/ecs_scheduled_task'
|
|
23
24
|
require 'cfnguardian/resources/eks_container_insights'
|
|
24
25
|
require 'cfnguardian/resources/elastic_file_system'
|
|
25
26
|
require 'cfnguardian/resources/elasticache_replication_group'
|
|
@@ -271,6 +271,25 @@ module CfnGuardian
|
|
|
271
271
|
end
|
|
272
272
|
end
|
|
273
273
|
|
|
274
|
+
class ECSScheduledTaskAlarm < BaseAlarm
|
|
275
|
+
def initialize(resource)
|
|
276
|
+
super(resource)
|
|
277
|
+
@group = 'ECSScheduledTask'
|
|
278
|
+
@namespace = 'ECS/ContainerInsights'
|
|
279
|
+
@dimensions = { ClusterName: resource['Id'] }
|
|
280
|
+
if resource.has_key?('TaskDefinitionFamily')
|
|
281
|
+
@dimensions[:TaskDefinitionFamily] = resource['TaskDefinitionFamily']
|
|
282
|
+
# keep family alarms distinct from the cluster-wide and other family alarms on the same cluster.
|
|
283
|
+
# ECS cluster and family names can't contain a colon, so cluster `a` with family `b`
|
|
284
|
+
# can't share an alarm name with a cluster called `a-b`
|
|
285
|
+
@resource_hash = Digest::MD5.hexdigest [resource['Id'], resource['TaskDefinitionFamily']].to_json
|
|
286
|
+
@resource_name = "#{resource['Id']}:#{resource['TaskDefinitionFamily']}" if @resource_name.nil?
|
|
287
|
+
end
|
|
288
|
+
# scheduled tasks only publish metrics while they are running
|
|
289
|
+
@treat_missing_data = 'notBreaching'
|
|
290
|
+
end
|
|
291
|
+
end
|
|
292
|
+
|
|
274
293
|
class EKSContainerInsightsClusterAlarm < BaseAlarm
|
|
275
294
|
def initialize(resource)
|
|
276
295
|
super(resource)
|
|
@@ -95,6 +95,30 @@ module CfnGuardian
|
|
|
95
95
|
end
|
|
96
96
|
end
|
|
97
97
|
|
|
98
|
+
class ECSScheduledTaskEventSubscription < BaseEventSubscription
|
|
99
|
+
def initialize(resource)
|
|
100
|
+
super(resource)
|
|
101
|
+
@source = 'aws.ecs'
|
|
102
|
+
@detail_type = 'ECS Task State Change'
|
|
103
|
+
@hash = Digest::MD5.hexdigest [resource['Id'], resource['TaskDefinitionFamily']].to_json
|
|
104
|
+
@scope = {
|
|
105
|
+
'clusterArn' => ["arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/#{resource['Id']}"],
|
|
106
|
+
# tasks started by an ECS service have a group of service:<name>
|
|
107
|
+
'group' => [{ 'anything-but' => { 'prefix' => 'service:' } }]
|
|
108
|
+
}
|
|
109
|
+
if resource.has_key?('TaskDefinitionFamily')
|
|
110
|
+
# the trailing colon stops family `app` matching `app-v2`
|
|
111
|
+
@scope['taskDefinitionArn'] = [{ 'prefix' => "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/#{resource['TaskDefinitionFamily']}:" }]
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# Always scope to the resource's cluster and family, including custom and
|
|
116
|
+
# inherited subscriptions that set their own Detail.
|
|
117
|
+
def detail
|
|
118
|
+
return @detail.merge(@scope)
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
|
|
98
122
|
class AcmEventSubscription < BaseEventSubscription; end
|
|
99
123
|
class ApiGatewayEventSubscription < BaseEventSubscription; end
|
|
100
124
|
class ApplicationTargetGroupEventSubscription < BaseEventSubscription; end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
module CfnGuardian::Resource
|
|
2
|
+
# Monitors ECS tasks that run outside of an ECS service, such as tasks launched
|
|
3
|
+
# by an EventBridge rule or EventBridge Scheduler. These tasks have no service
|
|
4
|
+
# metrics, so failures are detected from the ECS Task State Change events.
|
|
5
|
+
# The cluster and task definition family scope is applied by
|
|
6
|
+
# ECSScheduledTaskEventSubscription, so custom subscriptions get it too.
|
|
7
|
+
class ECSScheduledTask < Base
|
|
8
|
+
|
|
9
|
+
def default_event_subscriptions()
|
|
10
|
+
event_subscription = CfnGuardian::Models::ECSScheduledTaskEventSubscription.new(@resource)
|
|
11
|
+
event_subscription.name = 'TaskFailed'
|
|
12
|
+
event_subscription.detail = {
|
|
13
|
+
'lastStatus' => ['STOPPED'],
|
|
14
|
+
# numeric matching skips containers with a null or missing exit code,
|
|
15
|
+
# which `anything-but: 0` would match
|
|
16
|
+
'containers' => { 'exitCode' => [{ 'numeric' => ['>', 0] }] }
|
|
17
|
+
}
|
|
18
|
+
@event_subscriptions.push(event_subscription)
|
|
19
|
+
|
|
20
|
+
event_subscription = CfnGuardian::Models::ECSScheduledTaskEventSubscription.new(@resource)
|
|
21
|
+
event_subscription.name = 'TaskFailedToStart'
|
|
22
|
+
event_subscription.detail = {
|
|
23
|
+
'lastStatus' => ['STOPPED'],
|
|
24
|
+
'stopCode' => ['TaskFailedToStart']
|
|
25
|
+
}
|
|
26
|
+
@event_subscriptions.push(event_subscription)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
end
|
|
30
|
+
end
|
data/lib/cfnguardian/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: cfn-guardian
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.15.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Guslington
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-11-05 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: thor
|
|
@@ -327,6 +327,7 @@ files:
|
|
|
327
327
|
- docs/custom_checks/tls.md
|
|
328
328
|
- docs/custom_checks/websocket.md
|
|
329
329
|
- docs/custom_metrics.md
|
|
330
|
+
- docs/ecs_scheduled_tasks.md
|
|
330
331
|
- docs/event_subscriptions.md
|
|
331
332
|
- docs/maintenance_mode.md
|
|
332
333
|
- docs/notifiers.md
|
|
@@ -369,6 +370,7 @@ files:
|
|
|
369
370
|
- lib/cfnguardian/resources/dynamodb_table.rb
|
|
370
371
|
- lib/cfnguardian/resources/ec2_instance.rb
|
|
371
372
|
- lib/cfnguardian/resources/ecs_cluster.rb
|
|
373
|
+
- lib/cfnguardian/resources/ecs_scheduled_task.rb
|
|
372
374
|
- lib/cfnguardian/resources/ecs_service.rb
|
|
373
375
|
- lib/cfnguardian/resources/eks_container_insights.rb
|
|
374
376
|
- lib/cfnguardian/resources/elastic_file_system.rb
|