@kensio/yulin 1.21.7 → 1.21.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -2
- package/dist/service/ses/command/authorize/sim-ses-authorizer.d.ts +9 -0
- package/dist/service/ses/command/authorize/sim-ses-authorizer.js +18 -2
- package/dist/service/ses/command/send/sim-ses-send-email.js +2 -1
- package/docs/README.md +7 -0
- package/docs/ai-skill/README.md +6 -1
- package/docs/sdk/README.md +15 -0
- package/docs/services/acm/README.md +24 -40
- package/docs/services/apigateway/README.md +52 -71
- package/docs/services/apigatewayv2/README.md +55 -74
- package/docs/services/athena/README.md +17 -26
- package/docs/services/backup/README.md +29 -39
- package/docs/services/bedrock/README.md +38 -52
- package/docs/services/cloudformation/README.md +43 -55
- package/docs/services/cloudfront/README.md +69 -95
- package/docs/services/cloudwatch/README.md +40 -54
- package/docs/services/cognito/README.md +30 -45
- package/docs/services/dynamodb/README.md +34 -51
- package/docs/services/ecr/README.md +36 -77
- package/docs/services/ecs/README.md +26 -46
- package/docs/services/elbv2/README.md +19 -29
- package/docs/services/eventbridge/README.md +14 -18
- package/docs/services/firehose/README.md +24 -32
- package/docs/services/glue/README.md +41 -78
- package/docs/services/iam/README.md +13 -15
- package/docs/services/kinesis/README.md +53 -93
- package/docs/services/kms/README.md +22 -32
- package/docs/services/lambda/README.md +53 -79
- package/docs/services/logs/README.md +41 -50
- package/docs/services/organizations/README.md +50 -85
- package/docs/services/personalize/README.md +28 -44
- package/docs/services/rekognition/README.md +26 -38
- package/docs/services/route53/README.md +17 -17
- package/docs/services/s3/README.md +47 -51
- package/docs/services/scheduler/README.md +41 -52
- package/docs/services/secretsmanager/README.md +27 -42
- package/docs/services/ses/README.md +24 -34
- package/docs/services/sns/README.md +18 -26
- package/docs/services/sqs/README.md +14 -14
- package/docs/services/ssm/README.md +13 -17
- package/docs/services/stepfunctions/README.md +18 -20
- package/docs/services/sts/README.md +32 -45
- package/docs/services/wafv2/README.md +12 -17
- package/docs/testing/README.md +228 -0
- package/docs/time/README.md +10 -0
- package/llms.txt +1 -0
- package/package.json +1 -1
|
@@ -1,31 +1,20 @@
|
|
|
1
1
|
# Simulated ECS
|
|
2
2
|
|
|
3
|
-
Yulin
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
Yulin simulates ECS clusters, task definitions, tasks and services. It runs JavaScript or TypeScript
|
|
4
|
+
handlers bound to container image URIs. Use `simAws.ecs()` directly or intercept an `ECSClient`.
|
|
5
|
+
ECS types are available from `@kensio/yulin/ecs`.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
## Binding code to a container image
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
Yulin uses an image URI as an identifier for an in-process handler. It does not pull or execute the
|
|
10
|
+
image. A container with a matching binding runs the handler. An unbound container is recorded as
|
|
11
|
+
unsimulated while the rest of the task continues.
|
|
10
12
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
in its own process.
|
|
13
|
+
A task definition may include application and support containers. Yulin stores and reports every
|
|
14
|
+
container, but only runs containers that have a binding.
|
|
14
15
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
way an image URI identifies a container image Lambda function. The rule that follows is that a
|
|
18
|
-
container matched to a handler runs that handler, and a container with no match is recorded as not
|
|
19
|
-
simulated while the rest of the task carries on.
|
|
20
|
-
|
|
21
|
-
A realistic task definition holds an application container, a log router and an observability agent.
|
|
22
|
-
Only the first of those is something Yulin could ever run, and all three are stored and reported
|
|
23
|
-
back exactly as declared.
|
|
24
|
-
|
|
25
|
-
The case this leaves out is a sidecar the application depends on, such as a Redis or a database in
|
|
26
|
-
the same task. Point its connection details, which are ordinary environment variables, at a real one
|
|
27
|
-
you run yourself. See [non-AWS dependencies](https://yulinsim.dev/non-aws-dependencies/) for how that fits
|
|
28
|
-
together.
|
|
16
|
+
Run required sidecars such as Redis separately and pass their connection details through environment
|
|
17
|
+
variables. See [non-AWS dependencies](https://yulinsim.dev/non-aws-dependencies/).
|
|
29
18
|
|
|
30
19
|
## Registering a task definition
|
|
31
20
|
|
|
@@ -76,9 +65,7 @@ console.log(described.taskDefinition?.containerDefinitions?.[0]?.image);
|
|
|
76
65
|
// "example.dkr.ecr.eu-west-2.amazonaws.com/checkout:1"
|
|
77
66
|
```
|
|
78
67
|
|
|
79
|
-
Container definitions
|
|
80
|
-
ever run it. That covers port mappings, environment variables and secrets. A `valueFrom` naming a
|
|
81
|
-
Secrets Manager secret is held as the identifier it is, and never resolved.
|
|
68
|
+
Container definitions retain their images, port mappings, environment variables and secrets.
|
|
82
69
|
|
|
83
70
|
A registration that declares something this simulation has no room for is refused outright, and
|
|
84
71
|
never trimmed. That way a declaration cannot go missing from the revision it made.
|
|
@@ -132,9 +119,8 @@ a full task definition ARN. The ARN of a revision is
|
|
|
132
119
|
|
|
133
120
|
## Deregistering a revision
|
|
134
121
|
|
|
135
|
-
`DeregisterTaskDefinition` marks
|
|
136
|
-
by `family:revision`
|
|
137
|
-
find out what it declared. What it stops being is the revision the family resolves to.
|
|
122
|
+
`DeregisterTaskDefinition` marks a revision `INACTIVE` without removing it. The revision remains
|
|
123
|
+
available by `family:revision` or ARN, but the family name no longer resolves to it.
|
|
138
124
|
|
|
139
125
|
```typescript sim-ecs-deregister-task-definition
|
|
140
126
|
/**
|
|
@@ -234,8 +220,8 @@ A family counts as inactive once every one of its revisions has been deregistere
|
|
|
234
220
|
|
|
235
221
|
## Clusters
|
|
236
222
|
|
|
237
|
-
A cluster
|
|
238
|
-
`
|
|
223
|
+
A cluster contains tasks and services. Use `CreateCluster`, `DescribeClusters`, `ListClusters` and
|
|
224
|
+
`DeleteCluster` to manage it.
|
|
239
225
|
|
|
240
226
|
```typescript sim-ecs-clusters
|
|
241
227
|
/**
|
|
@@ -290,13 +276,13 @@ A cluster is named either by its short name or by its full ARN, and the two are
|
|
|
290
276
|
ARN belonging to another account or region names a different cluster. `DescribeClusters` reports it
|
|
291
277
|
as a `MISSING` failure, and `DeleteCluster` refuses it.
|
|
292
278
|
|
|
293
|
-
|
|
294
|
-
|
|
279
|
+
Create a cluster before running a task. This includes the `default` cluster, which Yulin does not
|
|
280
|
+
create automatically.
|
|
295
281
|
|
|
296
282
|
## Running a task
|
|
297
283
|
|
|
298
|
-
`bindContainer`
|
|
299
|
-
handlers run in
|
|
284
|
+
`bindContainer` associates an image with a handler. `RunTask` starts the task and schedules bound
|
|
285
|
+
handlers to run in process.
|
|
300
286
|
|
|
301
287
|
```typescript sim-ecs-run-task
|
|
302
288
|
/**
|
|
@@ -376,9 +362,7 @@ The task stops with a `stopCode` of `TaskFailedToStart` saying that nothing ran.
|
|
|
376
362
|
|
|
377
363
|
### Binding by image repository
|
|
378
364
|
|
|
379
|
-
|
|
380
|
-
the container by hand is the wrong way round. A binding can name the repository instead, and the tag
|
|
381
|
-
is ignored on both sides.
|
|
365
|
+
Use a repository binding when image tags change between builds. Repository matching ignores the tag.
|
|
382
366
|
|
|
383
367
|
```typescript sim-ecs-bind-image-repository
|
|
384
368
|
/**
|
|
@@ -505,9 +489,8 @@ Read inside the handler to get the container's own.
|
|
|
505
489
|
|
|
506
490
|
## The task role
|
|
507
491
|
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
intercepted by `SimSdk` pick this up without the code under test knowing.
|
|
492
|
+
AWS calls made by a container use the task definition's `taskRoleArn`. An SDK client intercepted by
|
|
493
|
+
`SimSdk` picks up the role from the running handler.
|
|
511
494
|
|
|
512
495
|
```typescript sim-ecs-task-role
|
|
513
496
|
/**
|
|
@@ -600,10 +583,8 @@ alone, since there is no image to pull and no log driver to write to.
|
|
|
600
583
|
|
|
601
584
|
## Container secrets
|
|
602
585
|
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
the container's environment alongside its declared `environment`. A handler reads them through
|
|
606
|
-
`process.env` like anything else.
|
|
586
|
+
Yulin resolves container `secrets` from simulated Secrets Manager or SSM Parameter Store when the
|
|
587
|
+
task starts. The values appear in `process.env` alongside the declared environment variables.
|
|
607
588
|
|
|
608
589
|
They are read as the task definition's `executionRoleArn` rather than its `taskRoleArn`. That is the
|
|
609
590
|
split real ECS makes. The execution role is what the task agent pulls secrets with before a
|
|
@@ -765,8 +746,7 @@ is reported as it stands, keeping the reason it stopped for.
|
|
|
765
746
|
|
|
766
747
|
## Services
|
|
767
748
|
|
|
768
|
-
|
|
769
|
-
usually is. It is a named service in a cluster, running some number of tasks from a task definition.
|
|
749
|
+
An ECS service maintains a desired number of tasks from one task definition in a cluster.
|
|
770
750
|
|
|
771
751
|
`CreateService` creates one. Its tasks exist as soon as the request is answered and reach `RUNNING`
|
|
772
752
|
on the simulation's background work, as real ECS brings a new service up. The service reports the
|
|
@@ -1,24 +1,16 @@
|
|
|
1
1
|
# Simulated Elastic Load Balancing
|
|
2
2
|
|
|
3
|
-
Yulin
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
Only the application load balancer is simulated. A network or gateway load balancer routes below
|
|
16
|
-
HTTP, which nothing here speaks, and `Type: "network"` is refused outright.
|
|
17
|
-
|
|
18
|
-
No TLS is performed anywhere in this. An HTTPS listener holds a certificate and is checked against
|
|
19
|
-
simulated ACM, and everything travels in the clear. See
|
|
20
|
-
[HTTPS listeners and certificates](#https-listeners-and-certificates) for what that leaves a test
|
|
21
|
-
able to conclude.
|
|
3
|
+
Yulin simulates Application Load Balancers, target groups, listeners and listener rules for tests
|
|
4
|
+
and local development. Requests pass through the listener and rule configuration before reaching a
|
|
5
|
+
registered Lambda function or simulated ECS service. Simulated IAM authorizes every operation.
|
|
6
|
+
|
|
7
|
+
Each load balancer receives an AWS-shaped DNS name. A simulated
|
|
8
|
+
[Route 53](https://yulinsim.dev/services/route53/) record can point a local hostname at that name, so
|
|
9
|
+
an HTTP request follows the same routing path as the deployed application.
|
|
10
|
+
|
|
11
|
+
Network and Gateway Load Balancers are not simulated. HTTPS listeners validate their certificates
|
|
12
|
+
against simulated ACM, but Yulin does not perform TLS. Import ELBv2-specific types from
|
|
13
|
+
`@kensio/yulin/elbv2`.
|
|
22
14
|
|
|
23
15
|
## Creating a load balancer
|
|
24
16
|
|
|
@@ -73,9 +65,8 @@ why a load balancer cannot be named starting with `internal-`.
|
|
|
73
65
|
|
|
74
66
|
## Target groups hold functions or addresses
|
|
75
67
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
protocol and port at all.
|
|
68
|
+
`TargetType` determines what a target group can contain. It controls the number and shape of target
|
|
69
|
+
IDs, and whether the group requires a protocol and port.
|
|
79
70
|
|
|
80
71
|
```typescript sim-elbv2-lambda-target-group
|
|
81
72
|
/**
|
|
@@ -178,15 +169,14 @@ console.log(health.TargetHealthDescriptions?.length); // 1
|
|
|
178
169
|
console.log(health.TargetHealthDescriptions?.[0]?.Target.Port); // 8080
|
|
179
170
|
```
|
|
180
171
|
|
|
181
|
-
`TargetType: "instance"` is
|
|
182
|
-
|
|
183
|
-
target type at all is refused for the same reason, since real ELB defaults it to `instance`.
|
|
172
|
+
`TargetType: "instance"` is unsupported because Yulin does not simulate EC2 instances. Omitting
|
|
173
|
+
`TargetType` is also rejected because AWS would default it to `instance`.
|
|
184
174
|
|
|
185
175
|
## Listeners and the rules on them
|
|
186
176
|
|
|
187
|
-
A listener
|
|
188
|
-
|
|
189
|
-
|
|
177
|
+
A listener handles one port and provides the default action when no rule matches. When several
|
|
178
|
+
rules match, the lowest priority number wins. Two rules on the same listener cannot use the same
|
|
179
|
+
priority.
|
|
190
180
|
|
|
191
181
|
```typescript sim-elbv2-listener-rules
|
|
192
182
|
/**
|
|
@@ -1761,7 +1751,7 @@ console.log(created.LoadBalancers?.[0]?.DNSName);
|
|
|
1761
1751
|
// "shop-alb-0000000001.eu-west-2.elb.amazonaws.com"
|
|
1762
1752
|
```
|
|
1763
1753
|
|
|
1764
|
-
##
|
|
1754
|
+
## Supported operations
|
|
1765
1755
|
|
|
1766
1756
|
- `CreateLoadBalancer`, `DescribeLoadBalancers` and `DeleteLoadBalancer`, with a DNS name, ARN,
|
|
1767
1757
|
canonical hosted zone id, scheme and state on every load balancer.
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
# Simulated EventBridge
|
|
2
2
|
|
|
3
|
-
Yulin
|
|
4
|
-
|
|
3
|
+
Yulin simulates Amazon EventBridge event buses, rules, targets and `PutEvents` for tests and local
|
|
4
|
+
development. Rules can match events, run on a simulated schedule, and deliver to simulated Lambda,
|
|
5
|
+
SQS, SNS or ECS. Event buses and rules are stored in memory, and simulated IAM authorizes every
|
|
6
|
+
operation.
|
|
5
7
|
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
advances simulated time. [EventBridge Scheduler](https://yulinsim.dev/services/scheduler/) is a separate service with its own
|
|
9
|
-
docs. EventBridge-specific types are imported from the `@kensio/yulin/eventbridge` subpath.
|
|
8
|
+
[EventBridge Scheduler](https://yulinsim.dev/services/scheduler/) is simulated separately. Import
|
|
9
|
+
EventBridge-specific types from `@kensio/yulin/eventbridge`.
|
|
10
10
|
|
|
11
11
|
## Putting an event onto a bus
|
|
12
12
|
|
|
@@ -142,10 +142,8 @@ console.log(output.FailedEntryCount); // 0
|
|
|
142
142
|
console.log(output.Entries?.[0]?.EventId !== undefined); // true
|
|
143
143
|
```
|
|
144
144
|
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
looks exactly like a working call. That is worth knowing before it costs an afternoon, and the
|
|
148
|
-
simulation reproduces it faithfully.
|
|
145
|
+
AWS also accepts and drops an event sent to a bus that does not exist. The entry is not counted as a
|
|
146
|
+
failure, so check the bus name when `PutEvents` succeeds but no rule runs.
|
|
149
147
|
|
|
150
148
|
## Inspecting what a bus received
|
|
151
149
|
|
|
@@ -250,9 +248,8 @@ within the account. A rule ARN on the default bus is
|
|
|
250
248
|
`arn:aws:events:<region>:<account>:rule/<name>`, and a rule on a custom bus carries the bus as well,
|
|
251
249
|
as `rule/<bus>/<name>`.
|
|
252
250
|
|
|
253
|
-
`PutRule` creates
|
|
254
|
-
|
|
255
|
-
and a common surprise.
|
|
251
|
+
`PutRule` creates or updates a rule. An update replaces the stored rule instead of merging fields,
|
|
252
|
+
so omitting the description clears it.
|
|
256
253
|
|
|
257
254
|
`DisableRule` stops a rule matching, and `EnableRule` starts it again. A rule that was off picks up
|
|
258
255
|
from the next event and leaves what it missed behind. `DeleteRule` on a rule that was never there
|
|
@@ -285,10 +282,9 @@ field, the pattern matches when the two lists overlap, and a pattern naming one
|
|
|
285
282
|
event whose `resources` names several. `exists` is about the field and not its members, and a field
|
|
286
283
|
carrying an empty list still exists.
|
|
287
284
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
case-insensitive forms of `prefix` and `suffix`.
|
|
291
|
-
like a pattern that was simply too specific, and the rule would go unnoticed until the deployment.
|
|
285
|
+
`PutRule` rejects unsupported operators instead of storing a pattern that can never match. These
|
|
286
|
+
include `cidr`, `equals-ignore-case`, `wildcard`, `$or`, nested `anything-but`, and the
|
|
287
|
+
case-insensitive forms of `prefix` and `suffix`.
|
|
292
288
|
|
|
293
289
|
## Testing a pattern without a rule
|
|
294
290
|
|
|
@@ -1049,7 +1045,7 @@ events across accounts that way, but nothing here can reach another simulation's
|
|
|
1049
1045
|
treating a foreign ARN as local would let a test pass while the real call crossed a boundary it has
|
|
1050
1046
|
no permission for.
|
|
1051
1047
|
|
|
1052
|
-
##
|
|
1048
|
+
## Supported operations
|
|
1053
1049
|
|
|
1054
1050
|
- `CreateEventBus`, `DeleteEventBus`, `DescribeEventBus`, `ListEventBuses` and `PutEvents`.
|
|
1055
1051
|
- `PutRule`, `DeleteRule`, `DescribeRule`, `ListRules`, `EnableRule`, `DisableRule` and
|
|
@@ -1,18 +1,16 @@
|
|
|
1
1
|
# Simulated Kinesis Data Firehose
|
|
2
2
|
|
|
3
|
-
Yulin
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
event and assert on the Object it landed in, without an AWS account and without waiting five minutes
|
|
7
|
-
for a buffer to flush.
|
|
3
|
+
Yulin simulates Kinesis Data Firehose delivery streams in memory. A stream accepts records directly
|
|
4
|
+
or reads them from simulated Kinesis, buffers them, and writes each completed buffer to simulated S3.
|
|
5
|
+
Advance simulated time to flush interval-based buffers without waiting in real time.
|
|
8
6
|
|
|
9
7
|
Firehose specific types are imported from the `@kensio/yulin/firehose` subpath.
|
|
10
8
|
|
|
11
9
|
## Putting a record and finding the Object
|
|
12
10
|
|
|
13
|
-
`simAws.firehose()`
|
|
14
|
-
|
|
15
|
-
|
|
11
|
+
`simAws.firehose()` returns Firehose for the default account and region. Create the destination
|
|
12
|
+
bucket and delivery role before the delivery stream. Advance the clock past the buffering interval
|
|
13
|
+
to write the buffered records.
|
|
16
14
|
|
|
17
15
|
```typescript sim-firehose-put-and-deliver
|
|
18
16
|
/**
|
|
@@ -93,15 +91,13 @@ const { Contents } = await simAws
|
|
|
93
91
|
console.log(Contents?.[0]?.Key);
|
|
94
92
|
```
|
|
95
93
|
|
|
96
|
-
|
|
97
|
-
delivered. That delay is what a Firehose pipeline is built around.
|
|
94
|
+
`PutRecord` returns a record ID immediately. The bucket remains empty until the buffer is delivered.
|
|
98
95
|
|
|
99
96
|
## Buffering
|
|
100
97
|
|
|
101
|
-
A delivery stream
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
below does exactly that.
|
|
98
|
+
A delivery stream flushes when it reaches `SizeInMBs` or `IntervalInSeconds`, whichever comes first.
|
|
99
|
+
Firehose concatenates all records in the buffer into one S3 object. Add a newline to each record if
|
|
100
|
+
the object should contain separate lines.
|
|
105
101
|
|
|
106
102
|
`IntervalInSeconds` runs from the first record of a buffer. The default is 300 seconds and the
|
|
107
103
|
default size is 5 MB, as they are on real Firehose.
|
|
@@ -224,8 +220,8 @@ the background scheduler, the way real Firehose answers the producer before it w
|
|
|
224
220
|
|
|
225
221
|
## The Object key
|
|
226
222
|
|
|
227
|
-
|
|
228
|
-
|
|
223
|
+
Object keys contain the configured `Prefix`, UTC date path, delivery stream name, version, delivery
|
|
224
|
+
time, and a random suffix:
|
|
229
225
|
|
|
230
226
|
```
|
|
231
227
|
<Prefix>YYYY/MM/DD/HH/<delivery-stream-name>-<version>-YYYY-MM-DD-HH-MM-SS-<random>
|
|
@@ -234,15 +230,15 @@ delivery time and a random string:
|
|
|
234
230
|
A delivery stream with no `Prefix` gets the bare date path. The version is `1` and stays there,
|
|
235
231
|
since a delivery stream's configuration is fixed once it is created.
|
|
236
232
|
|
|
237
|
-
|
|
238
|
-
|
|
233
|
+
The date path and timestamp use simulated time. Set the clock before delivery to make the prefix
|
|
234
|
+
predictable.
|
|
239
235
|
|
|
240
236
|
## Reading from a Kinesis stream
|
|
241
237
|
|
|
242
|
-
A delivery stream can
|
|
238
|
+
A delivery stream can read records from simulated Kinesis. Create it with a
|
|
243
239
|
`DeliveryStreamType` of `KinesisStreamAsSource` and a `KinesisStreamSourceConfiguration` naming the
|
|
244
|
-
stream and
|
|
245
|
-
|
|
240
|
+
stream and its read role. Records added after the delivery stream is created are buffered and sent
|
|
241
|
+
to S3.
|
|
246
242
|
|
|
247
243
|
```typescript sim-firehose-kinesis-source
|
|
248
244
|
/**
|
|
@@ -382,11 +378,8 @@ Creating a delivery stream hands Firehose the destination `RoleARN`, and the sou
|
|
|
382
378
|
where it reads a Kinesis stream. `CreateDeliveryStream` authorizes `iam:PassRole` against each of
|
|
383
379
|
them. See [passing a Role to a service](https://yulinsim.dev/services/iam/#passing-a-role-to-a-service) in the IAM docs.
|
|
384
380
|
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
delivery. Real
|
|
388
|
-
Firehose answered that `PutRecord` minutes earlier, and what became of the buffer reaches the
|
|
389
|
-
producer through CloudWatch. The simulator keeps the failure for a test to read instead.
|
|
381
|
+
Delivery uses the delivery stream's `RoleARN`, not the identity that called `PutRecord`. The role
|
|
382
|
+
needs `s3:PutObject` on the destination. Failed writes appear in `getDeliveryFailures()`.
|
|
390
383
|
|
|
391
384
|
```typescript sim-firehose-permissions
|
|
392
385
|
/**
|
|
@@ -507,9 +500,8 @@ what a test checking the denial does. Anything else is also warned about on the
|
|
|
507
500
|
`KinesisStreamSourceConfiguration` and `Tags` are read. A `Ref` gives the delivery stream name and
|
|
508
501
|
`Fn::GetAtt` on `Arn` gives the ARN, the way real CloudFormation publishes them.
|
|
509
502
|
|
|
510
|
-
A CDK `DeliveryStream` with an `S3Bucket` destination synthesizes
|
|
511
|
-
|
|
512
|
-
synthesized template, and nothing here has to be written by hand.
|
|
503
|
+
A CDK `DeliveryStream` with an `S3Bucket` destination synthesizes this resource, its delivery role,
|
|
504
|
+
and the role policy. Deploy that synthesized template directly into simulated CloudFormation.
|
|
513
505
|
|
|
514
506
|
```typescript sim-firehose-cloudformation
|
|
515
507
|
/**
|
|
@@ -627,9 +619,9 @@ A `DeliveryStreamType` of `KinesisStreamAsSource` deploys as well. The
|
|
|
627
619
|
`KinesisStreamSourceConfiguration` names the stream by ARN and the Role to read it as, and a stack
|
|
628
620
|
declaring the stream beside the delivery stream archives what a producer puts on it.
|
|
629
621
|
|
|
630
|
-
A delivery stream
|
|
631
|
-
`stack.skippedResources`,
|
|
632
|
-
S3
|
|
622
|
+
A delivery stream with an unsupported source or destination is recorded in
|
|
623
|
+
`stack.skippedResources`, while the rest of the stack deploys. This includes destinations other than
|
|
624
|
+
S3 and unsupported source properties such as
|
|
633
625
|
`MSKSourceConfiguration` or `DatabaseSourceConfiguration`. The source property is what the skip is
|
|
634
626
|
decided on, because a template that leaves `DeliveryStreamType` out gets `DirectPut` by default.
|
|
635
627
|
|
|
@@ -1,26 +1,17 @@
|
|
|
1
1
|
# Simulated Glue
|
|
2
2
|
|
|
3
|
-
Yulin
|
|
4
|
-
|
|
5
|
-
`AWS::Glue::Database` and `AWS::Glue::Table`, and hands them back through `GetDatabase` and
|
|
6
|
-
`GetTable`. A test can assert that a stack declared the table definition it meant to, including the
|
|
7
|
-
Athena partition projection its parameters configure.
|
|
3
|
+
Yulin simulates Glue Data Catalog databases, tables and partitions. The catalog stores metadata. It
|
|
4
|
+
does not read the data in S3.
|
|
8
5
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
Simulated [Athena](https://yulinsim.dev/services/athena/ "Simulated Athena usage docs") reads this
|
|
13
|
-
catalog. A query naming a table no database here holds fails the way real Athena fails it, and a
|
|
14
|
-
table's partition projection is evaluated when a query runs against it. All four projection types
|
|
15
|
-
are covered, `enum`, `integer`, `date` and `injected`. A projection with a mistake in its parameters
|
|
16
|
-
fails the query that reads it.
|
|
6
|
+
Simulated [Athena](https://yulinsim.dev/services/athena/ "Simulated Athena usage docs") reads the
|
|
7
|
+
catalog and evaluates partition projection when it runs a query.
|
|
17
8
|
|
|
18
9
|
Glue-specific types are imported from the `@kensio/yulin/glue` subpath.
|
|
19
10
|
|
|
20
11
|
## Deploying a database and a table
|
|
21
12
|
|
|
22
|
-
|
|
23
|
-
|
|
13
|
+
Simulated CloudFormation deploys `AWS::Glue::Database` and `AWS::Glue::Table`. A table can name its
|
|
14
|
+
database through `Ref`.
|
|
24
15
|
|
|
25
16
|
```typescript sim-glue-cloudformation
|
|
26
17
|
/**
|
|
@@ -84,18 +75,15 @@ const { Table } = simAws
|
|
|
84
75
|
console.log(Table.Parameters["projection.enabled"]);
|
|
85
76
|
```
|
|
86
77
|
|
|
87
|
-
Partition projection
|
|
88
|
-
|
|
89
|
-
green while projecting none of them. Simulated Athena reads those same parameters when a query runs.
|
|
90
|
-
A broken projection fails that query.
|
|
78
|
+
Partition projection configuration is stored in `TableInput.Parameters`. Athena reads these
|
|
79
|
+
parameters when a query runs and rejects invalid projection configuration.
|
|
91
80
|
|
|
92
81
|
`Ref` answers with the database name and with the table name.
|
|
93
82
|
|
|
94
83
|
## Names are folded to lower case
|
|
95
84
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
reported and queried as `rainlytics`.
|
|
85
|
+
Database and table names are stored in lower case. A database created as `Rainlytics` is reported as
|
|
86
|
+
`rainlytics`.
|
|
99
87
|
|
|
100
88
|
```typescript sim-glue-name-folding
|
|
101
89
|
/**
|
|
@@ -123,36 +111,27 @@ const { Database } = glue.getDatabase(
|
|
|
123
111
|
console.log(Database.Name);
|
|
124
112
|
```
|
|
125
113
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
spelling it is asked for.
|
|
114
|
+
Names that differ only by case refer to the same database or table. Creating the second one raises
|
|
115
|
+
`AlreadyExistsException`.
|
|
129
116
|
|
|
130
|
-
Column
|
|
131
|
-
and simulated [Athena](https://yulinsim.dev/services/athena/ "Simulated Athena usage docs") folds a
|
|
132
|
-
column name only when it runs a query.
|
|
117
|
+
Column and partition key names keep their original case.
|
|
133
118
|
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
`analytics-stack-logtable-` and twelve more characters, where real CloudFormation ends the name in
|
|
137
|
-
twelve random ones. The name is trimmed to the 255 bytes the catalog allows, and [the CloudFormation docs](https://yulinsim.dev/services/cloudformation/#names-cloudformation-generates "Names CloudFormation generates")
|
|
138
|
-
cover how the stack name and the logical ID share what is left.
|
|
119
|
+
CloudFormation generates a lower-case name when the template omits one. See
|
|
120
|
+
[generated resource names](https://yulinsim.dev/services/cloudformation/#names-cloudformation-generates "Names CloudFormation generates").
|
|
139
121
|
|
|
140
122
|
## Reading the catalog back
|
|
141
123
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
authorization.
|
|
124
|
+
Use the SDK commands to read the catalog through IAM authorization. The `findDatabase`, `findTable`,
|
|
125
|
+
`allDatabases`, `tablesInDatabase`, `findPartition` and `partitionsInTable` accessors read the same
|
|
126
|
+
state directly.
|
|
146
127
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
storage descriptor's columns gives a table Athena refuses to query.
|
|
128
|
+
Storage descriptor columns and partition keys keep their declared order. Athena rejects a table
|
|
129
|
+
that repeats a partition key in the storage descriptor columns.
|
|
150
130
|
|
|
151
131
|
## Registering partitions
|
|
152
132
|
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
`PartitionKeys` declares them, and carries a storage descriptor saying where its own data sits.
|
|
133
|
+
Register partitions with the Glue partition commands. Values follow the order of the table's
|
|
134
|
+
`PartitionKeys`. Each partition may provide its own storage descriptor.
|
|
156
135
|
|
|
157
136
|
```typescript sim-glue-partitions
|
|
158
137
|
/**
|
|
@@ -219,24 +198,19 @@ console.log(Errors.length);
|
|
|
219
198
|
console.log(Partitions[1]?.StorageDescriptor?.Location);
|
|
220
199
|
```
|
|
221
200
|
|
|
222
|
-
`CreatePartition`
|
|
223
|
-
`
|
|
224
|
-
registration order. `DeletePartition` and `BatchDeletePartition` remove them.
|
|
201
|
+
`CreatePartition` and `BatchCreatePartition` register partitions. `GetPartition` reads one by its
|
|
202
|
+
values. `GetPartitions` returns them in registration order. The delete commands remove them.
|
|
225
203
|
|
|
226
|
-
A batch
|
|
227
|
-
|
|
228
|
-
job re-run over a week of days learns which days were already registered and registers the others.
|
|
204
|
+
A batch continues after an individual partition fails. Its `Errors` list contains the supplied
|
|
205
|
+
values and an `ErrorCode` for each failure.
|
|
229
206
|
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
Deleting a table removes the partitions registered against it, the way deleting a database removes
|
|
234
|
-
its tables.
|
|
207
|
+
The number of values must match the number of partition keys. A mismatch raises
|
|
208
|
+
`InvalidInputException`. Registering the same values twice raises `AlreadyExistsException`. Deleting
|
|
209
|
+
a table removes its partitions, and deleting a database removes its tables.
|
|
235
210
|
|
|
236
211
|
## Filtering partitions
|
|
237
212
|
|
|
238
|
-
`
|
|
239
|
-
none reads them all.
|
|
213
|
+
Pass `Expression` to `GetPartitions` to filter the result. Omitting it returns every partition.
|
|
240
214
|
|
|
241
215
|
```typescript sim-glue-partition-expressions
|
|
242
216
|
/**
|
|
@@ -295,28 +269,19 @@ const { Partitions } = glue.getPartitions(
|
|
|
295
269
|
console.log(JSON.stringify(Partitions.map((partition) => partition.Values)));
|
|
296
270
|
```
|
|
297
271
|
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
ends. `AND`, `OR` and `NOT` combine terms and brackets group them, with the precedence SQL gives
|
|
301
|
-
them (`NOT` binds tightest, then `AND`, then `OR`). `NOT` also sits in front of `LIKE`, `IN` and
|
|
302
|
-
`BETWEEN` to reverse that one term.
|
|
272
|
+
Expressions support `=`, `<>`, `!=`, `>`, `<`, `>=`, `<=`, `LIKE`, `IN` and `BETWEEN`. Combine terms
|
|
273
|
+
with `AND`, `OR`, `NOT` and parentheses. Operator precedence is `NOT`, then `AND`, then `OR`.
|
|
303
274
|
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
below it. Every other declared type compares as text, which is the order a date written the ISO way
|
|
307
|
-
already has. A key declared as a number is held to a numeric literal, and `hour = 'noon'` is refused
|
|
308
|
-
rather than matching nothing.
|
|
275
|
+
Numeric partition key types use numeric comparison. Other types use text comparison. A numeric key
|
|
276
|
+
requires a numeric literal.
|
|
309
277
|
|
|
310
|
-
`LIKE`
|
|
311
|
-
does. `%` stands for any run of characters and `_` for exactly one.
|
|
278
|
+
`LIKE` treats the value as text. `%` matches any sequence and `_` matches one character.
|
|
312
279
|
|
|
313
|
-
An
|
|
314
|
-
keys the table does have. So is one that cannot be read, and the message says where reading stopped.
|
|
280
|
+
An unknown partition key or invalid expression is rejected with a position-aware error.
|
|
315
281
|
|
|
316
282
|
## Intercepting a GlueClient
|
|
317
283
|
|
|
318
|
-
|
|
319
|
-
and sends its own Commands.
|
|
284
|
+
Intercept a `GlueClient` when application code constructs the client itself.
|
|
320
285
|
|
|
321
286
|
```typescript sim-glue-sdk-interception
|
|
322
287
|
/**
|
|
@@ -357,11 +322,9 @@ console.log(TableList?.[0]?.Name);
|
|
|
357
322
|
|
|
358
323
|
## Permissions
|
|
359
324
|
|
|
360
|
-
Every
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
table ARN is denied. Deleting a database needs permission on every table in it as well, since the
|
|
364
|
-
tables go with it.
|
|
325
|
+
Every command uses simulated IAM. Catalog resources form a hierarchy. A table operation needs
|
|
326
|
+
permission on the table, database and catalog. Deleting a database also needs permission on its
|
|
327
|
+
tables.
|
|
365
328
|
|
|
366
329
|
```typescript sim-glue-permissions
|
|
367
330
|
/**
|
|
@@ -441,7 +404,7 @@ const { Table } = glue.getTable(
|
|
|
441
404
|
console.log(Table.Name);
|
|
442
405
|
```
|
|
443
406
|
|
|
444
|
-
A policy
|
|
407
|
+
A policy that grants only the table ARN is insufficient.
|
|
445
408
|
|
|
446
409
|
## Available functionality
|
|
447
410
|
|