@kensio/yulin 1.20.15 → 1.20.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -0
- package/dist/sdk/module/sim-sdk-module-client-interceptor.d.ts +5 -0
- package/dist/sdk/module/sim-sdk-module-client-interceptor.js +14 -0
- package/dist/sdk/module/sim-sdk-static-client-factory.d.ts +26 -0
- package/dist/sdk/module/sim-sdk-static-client-factory.js +59 -0
- package/dist/sdk/send-patch.d.ts +8 -0
- package/dist/sdk/send-patch.js +11 -0
- package/dist/service/aws/caller/sim-aws-ambient-caller.d.ts +23 -0
- package/dist/service/aws/caller/sim-aws-ambient-caller.js +12 -0
- package/dist/service/aws/caller/sim-aws-caller-resolver.d.ts +39 -6
- package/dist/service/aws/caller/sim-aws-caller-resolver.js +32 -30
- package/dist/service/aws/caller/sim-aws-caller.d.ts +9 -0
- package/dist/service/aws/caller/sim-aws-resolved-caller.d.ts +19 -0
- package/dist/service/aws/caller/sim-aws-resolved-caller.js +26 -0
- package/dist/service/aws/factory/sim-aws-account-region-service-builder.d.ts +12 -0
- package/dist/service/aws/factory/sim-aws-account-region-service-builder.js +6 -0
- package/dist/service/aws/factory/sim-aws-account-service-cache.d.ts +12 -0
- package/dist/service/aws/factory/sim-aws-account-service-cache.js +9 -0
- package/dist/service/aws/factory/sim-aws-registered-service-builder.js +3 -1
- package/dist/service/aws/factory/sim-aws-service-factory.d.ts +5 -0
- package/dist/service/aws/factory/sim-aws-service-factory.js +14 -0
- package/dist/service/aws/sim-aws-properties.d.ts +15 -0
- package/dist/service/aws/sim-aws.js +2 -1
- package/dist/service/cloudformation/command/create-stack/create-stack.handler.d.ts +4 -0
- package/dist/service/cloudformation/command/create-stack/create-stack.handler.js +4 -1
- package/dist/service/cloudformation/deploy/sim-cfn-cdk-out-deployer.js +2 -1
- package/dist/service/cloudformation/deploy/sim-cfn-cdk-out-plan.d.ts +11 -2
- package/dist/service/cloudformation/deploy/sim-cfn-cdk-out-plan.js +2 -1
- package/dist/service/cloudformation/deploy/sim-cfn-cdk-out-stack-options.d.ts +6 -0
- package/dist/service/cloudformation/deploy/sim-cfn-template-deployer.d.ts +11 -0
- package/dist/service/cloudformation/deploy/sim-cfn-template-deployer.js +17 -15
- package/dist/service/cloudformation/deploy/sim-cfn-template-file-loader.d.ts +12 -0
- package/dist/service/cloudformation/deploy/sim-cfn-template-file-loader.js +2 -1
- package/dist/service/cloudformation/resource/cfn/cloudfront/sim-cloudfront-cache-policy-cfn.d.ts +23 -0
- package/dist/service/cloudformation/resource/cfn/cloudfront/sim-cloudfront-cache-policy-cfn.js +30 -0
- package/dist/service/cloudformation/resource/cfn/cloudfront/sim-cloudfront-cfn-value-adapter.js +8 -0
- package/dist/service/cloudformation/resource/name/sim-cfn-generated-resource-name.d.ts +29 -12
- package/dist/service/cloudformation/resource/name/sim-cfn-generated-resource-name.js +51 -20
- package/dist/service/cloudformation/stack/deploy/sim-cfn-resource-order.d.ts +19 -0
- package/dist/service/cloudformation/stack/deploy/sim-cfn-resource-order.js +6 -0
- package/dist/service/cloudformation/stack/deploy/sim-cfn-stack-resource-batch-creator.d.ts +7 -0
- package/dist/service/cloudformation/stack/deploy/sim-cfn-stack-resource-batch-creator.js +9 -2
- package/dist/service/cloudformation/stack/deploy/sim-cfn-stack-resource-creator.d.ts +2 -0
- package/dist/service/cloudformation/stack/deploy/sim-cfn-stack-resource-creator.js +2 -1
- package/dist/service/cloudformation/stack/sim-cfn-stack-resource-operations.d.ts +6 -0
- package/dist/service/cloudformation/stack/sim-cfn-stack-resource-operations.js +4 -1
- package/dist/service/cloudformation/stack/sim-cfn-stack.js +2 -1
- package/dist/service/cloudformation/stack/sim-cfn-stack.type.d.ts +6 -0
- package/dist/service/cloudfront/behaviour/sim-cloud-front-behavior.d.ts +5 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-cache-policy-registry.d.ts +37 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-cache-policy-registry.js +50 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-cache-policy.d.ts +25 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-cache-policy.js +21 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-managed-cache-policies.d.ts +28 -0
- package/dist/service/cloudfront/cache-policy/sim-cf-managed-cache-policies.js +48 -0
- package/dist/service/cloudfront/cff/function-code-input/cff-function-code-input.d.ts +8 -0
- package/dist/service/cloudfront/cff/function-code-input/cff-function-code-input.js +13 -0
- package/dist/service/cloudfront/cff/sim-cf-function-commands.d.ts +52 -0
- package/dist/service/cloudfront/cff/sim-cf-function-commands.js +51 -0
- package/dist/service/cloudfront/cff/sim-cff-configuration.d.ts +41 -0
- package/dist/service/cloudfront/cff/sim-cff-configuration.js +34 -0
- package/dist/service/cloudfront/cff/sim-cff-stage.d.ts +23 -0
- package/dist/service/cloudfront/cff/sim-cff-stage.js +30 -0
- package/dist/service/cloudfront/cff/sim-cloudfront-function.d.ts +6 -1
- package/dist/service/cloudfront/cff/sim-cloudfront-function.js +6 -0
- package/dist/service/cloudfront/cfn/cache-policy/sim-cfn-cf-cache-policy-config.d.ts +27 -0
- package/dist/service/cloudfront/cfn/cache-policy/sim-cfn-cf-cache-policy-config.js +42 -0
- package/dist/service/cloudfront/cfn/cache-policy/sim-cfn-cf-cache-policy-creator.d.ts +24 -0
- package/dist/service/cloudfront/cfn/cache-policy/sim-cfn-cf-cache-policy-creator.js +32 -0
- package/dist/service/cloudfront/cfn/distro/{sim-cfn-cf-distro-response-headers-policy.d.ts → sim-cfn-cf-distro-behavior-policies.d.ts} +8 -7
- package/dist/service/cloudfront/cfn/distro/{sim-cfn-cf-distro-response-headers-policy.js → sim-cfn-cf-distro-behavior-policies.js} +11 -37
- package/dist/service/cloudfront/cfn/distro/sim-cfn-cf-distro-creator.d.ts +6 -5
- package/dist/service/cloudfront/cfn/distro/sim-cfn-cf-distro-creator.js +8 -7
- package/dist/service/cloudfront/cfn/distro/sim-cfn-cf-distro-policy-drops.d.ts +23 -0
- package/dist/service/cloudfront/cfn/distro/sim-cfn-cf-distro-policy-drops.js +50 -0
- package/dist/service/cloudfront/cfn/sim-cfn-cloudfront-resource-factory.d.ts +1 -0
- package/dist/service/cloudfront/cfn/sim-cfn-cloudfront-resource-factory.js +10 -0
- package/dist/service/cloudfront/command/create-distribution/create-distribution.command.d.ts +1 -0
- package/dist/service/cloudfront/command/create-distribution/create-distribution.handler.d.ts +2 -0
- package/dist/service/cloudfront/command/create-function/create-function.handler.js +7 -1
- package/dist/service/cloudfront/command/function/sim-cf-describe-function.d.ts +22 -0
- package/dist/service/cloudfront/command/function/sim-cf-describe-function.js +32 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-access.d.ts +48 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-access.js +48 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-command.types.d.ts +97 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-command.types.js +10 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-lookup.d.ts +12 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-lookup.js +21 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-summary.d.ts +11 -0
- package/dist/service/cloudfront/command/function/sim-cf-function-summary.js +30 -0
- package/dist/service/cloudfront/command/function/sim-cf-get-function.d.ts +23 -0
- package/dist/service/cloudfront/command/function/sim-cf-get-function.js +33 -0
- package/dist/service/cloudfront/command/function/sim-cf-list-functions.d.ts +22 -0
- package/dist/service/cloudfront/command/function/sim-cf-list-functions.js +38 -0
- package/dist/service/cloudfront/command/update-distribution/update-distribution.handler.d.ts +2 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-cache-policy.d.ts +18 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-cache-policy.js +30 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-policies.d.ts +28 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-policies.js +41 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-properties.d.ts +2 -2
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-properties.js +6 -3
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-response-headers-policy.d.ts +0 -10
- package/dist/service/cloudfront/distribution/configurator/sim-cf-behavior-response-headers-policy.js +0 -19
- package/dist/service/cloudfront/distribution/configurator/sim-cf-distribution-configurator.factory.d.ts +2 -0
- package/dist/service/cloudfront/distribution/configurator/sim-cf-distribution-configurator.factory.js +4 -2
- package/dist/service/cloudfront/distribution/configurator/sim-cloud-front-behavior-configurator.d.ts +3 -3
- package/dist/service/cloudfront/distribution/configurator/sim-cloud-front-behavior-configurator.js +5 -5
- package/dist/service/cloudfront/distribution/configurator/sim-cloud-front-distribution-configurator.d.ts +3 -3
- package/dist/service/cloudfront/distribution/configurator/sim-cloud-front-distribution-configurator.js +4 -4
- package/dist/service/cloudfront/distribution/sim-cf-distribution-configuration-state.d.ts +3 -1
- package/dist/service/cloudfront/distribution/sim-cf-distribution-reconfigurer.d.ts +2 -0
- package/dist/service/cloudfront/error/sim-cloudfront.error.d.ts +21 -0
- package/dist/service/cloudfront/error/sim-cloudfront.error.js +25 -0
- package/dist/service/cloudfront/key-value-store/sim-cf-key-value-store-access.js +2 -10
- package/dist/service/cloudfront/key-value-store/sim-cf-key-value-store.js +2 -1
- package/dist/service/cloudfront/sdk/sim-cloudfront-sdk-command-router.js +12 -0
- package/dist/service/cloudfront/sim-cf-authorize.d.ts +16 -0
- package/dist/service/cloudfront/sim-cf-authorize.js +19 -0
- package/dist/service/cloudfront/sim-cf-etag.d.ts +7 -0
- package/dist/service/cloudfront/sim-cf-etag.js +10 -0
- package/dist/service/cloudfront/sim-cloudfront-commands.d.ts +8 -12
- package/dist/service/cloudfront/sim-cloudfront-commands.js +8 -17
- package/dist/service/cloudfront/sim-cloudfront-policies.d.ts +43 -0
- package/dist/service/cloudfront/sim-cloudfront-policies.js +53 -0
- package/dist/service/cloudfront/sim-cloudfront.d.ts +13 -17
- package/dist/service/cloudfront/sim-cloudfront.js +22 -25
- package/dist/service/ecr/authorize/sim-ecr-authorizer.d.ts +31 -0
- package/dist/service/ecr/authorize/sim-ecr-authorizer.js +38 -0
- package/dist/service/ecr/cfn/repository/sim-cfn-ecr-repository-creator.d.ts +11 -2
- package/dist/service/ecr/cfn/repository/sim-cfn-ecr-repository-creator.js +11 -2
- package/dist/service/ecr/cfn/sim-ecr-cfn-resource-factory.d.ts +4 -2
- package/dist/service/ecr/cfn/sim-ecr-cfn-resource-factory.js +8 -9
- package/dist/service/ecr/cfn/sim-ecr-cfn-unsupported-resource.d.ts +7 -0
- package/dist/service/ecr/cfn/sim-ecr-cfn-unsupported-resource.js +9 -0
- package/dist/service/ecr/sim-ecr.d.ts +7 -0
- package/dist/service/ecr/sim-ecr.js +16 -4
- package/dist/service/ecs/command/authorize/sim-ecs-authorizer.d.ts +12 -0
- package/dist/service/ecs/command/authorize/sim-ecs-authorizer.js +21 -0
- package/dist/service/ecs/command/register-task-definition/register-task-definition.handler.js +1 -0
- package/dist/service/ecs/sim-ecs-service-principal.d.ts +8 -0
- package/dist/service/ecs/sim-ecs-service-principal.js +8 -0
- package/dist/service/eventbridge/command/authorize/sim-event-bridge-authorizer.d.ts +12 -0
- package/dist/service/eventbridge/command/authorize/sim-event-bridge-authorizer.js +21 -0
- package/dist/service/eventbridge/command/rule/sim-event-bridge-rule-access.d.ts +13 -0
- package/dist/service/eventbridge/command/rule/sim-event-bridge-rule-access.js +9 -0
- package/dist/service/eventbridge/command/target/sim-event-bridge-put-targets.js +1 -0
- package/dist/service/firehose/command/authorize/sim-firehose-authorizer.d.ts +12 -0
- package/dist/service/firehose/command/authorize/sim-firehose-authorizer.js +20 -0
- package/dist/service/firehose/command/sim-firehose-delivery-stream-access.d.ts +4 -0
- package/dist/service/firehose/command/sim-firehose-delivery-stream-access.js +6 -0
- package/dist/service/firehose/command/stream/sim-firehose-create-delivery-stream.js +3 -1
- package/dist/service/firehose/sim-firehose-service-principal.d.ts +8 -0
- package/dist/service/firehose/sim-firehose-service-principal.js +8 -0
- package/dist/service/firehose/source/sim-firehose-source.d.ts +5 -0
- package/dist/service/firehose/source/sim-firehose-source.js +5 -0
- package/dist/service/glue/cfn/database/sim-cfn-glue-database-properties.js +2 -2
- package/dist/service/glue/cfn/sim-cfn-glue-generated-name.d.ts +10 -0
- package/dist/service/glue/cfn/sim-cfn-glue-generated-name.js +25 -0
- package/dist/service/glue/cfn/table/sim-cfn-glue-table-properties.js +2 -3
- package/dist/service/iam/authorize/allow/sim-iam-allow-requirement.d.ts +33 -2
- package/dist/service/iam/authorize/allow/sim-iam-allow-requirement.js +33 -2
- package/dist/service/iam/authorize/allow/sim-iam-allow-statements.d.ts +4 -3
- package/dist/service/iam/authorize/allow/sim-iam-allow-statements.js +9 -3
- package/dist/service/iam/authorize/context/sim-iam-auth-z-allow-requirement.d.ts +14 -5
- package/dist/service/iam/authorize/context/sim-iam-auth-z-allow-requirement.js +16 -12
- package/dist/service/iam/authorize/context/sim-iam-auth-z-caller-context-builder.d.ts +36 -7
- package/dist/service/iam/authorize/context/sim-iam-auth-z-caller-context-builder.js +22 -11
- package/dist/service/iam/authorize/context/sim-iam-auth-z-context-builder.d.ts +10 -71
- package/dist/service/iam/authorize/context/sim-iam-auth-z-context-builder.js +7 -2
- package/dist/service/iam/authorize/context/sim-iam-auth-z-input.d.ts +87 -0
- package/dist/service/iam/authorize/context/sim-iam-auth-z-input.js +1 -0
- package/dist/service/iam/authorize/match/sim-iam-policy-principal-matcher.d.ts +21 -3
- package/dist/service/iam/authorize/match/sim-iam-policy-principal-matcher.js +34 -10
- package/dist/service/iam/authorize/match/sim-iam-principal-match.d.ts +29 -8
- package/dist/service/iam/authorize/match/sim-iam-principal-match.js +38 -13
- package/dist/service/iam/authorize/pass-role/sim-iam-pass-role-authorizer.d.ts +74 -0
- package/dist/service/iam/authorize/pass-role/sim-iam-pass-role-authorizer.js +92 -0
- package/dist/service/iam/authorize/sim-iam-account-auth-z.d.ts +21 -5
- package/dist/service/iam/authorize/sim-iam-account-auth-z.js +7 -3
- package/dist/service/iam/authorize/sim-iam-authorizer.d.ts +5 -2
- package/dist/service/iam/authorize/sim-iam-authorizer.js +3 -1
- package/dist/service/iam/authorize/sim-iam-inter-service-auth-z.d.ts +1 -1
- package/dist/service/iam/authorize/sim-iam-wildcard.js +39 -4
- package/dist/service/iam/cfn/managed-policy/sim-cfn-iam-managed-policy-creator.js +2 -1
- package/dist/service/iam/cfn/name/sim-cfn-iam-generated-name.d.ts +16 -0
- package/dist/service/iam/cfn/name/sim-cfn-iam-generated-name.js +36 -0
- package/dist/service/iam/cfn/role/sim-cfn-iam-role-properties-parser.js +9 -8
- package/dist/service/iam/cfn/sim-cfn-iam-optional-string.d.ts +17 -0
- package/dist/service/iam/cfn/sim-cfn-iam-optional-string.js +17 -0
- package/dist/service/iam/cfn/user/sim-cfn-iam-user-properties-parser.js +4 -8
- package/dist/service/iam/command/policy/create-policy/create-policy-input-resolver.js +2 -0
- package/dist/service/iam/command/policy/put-role-policy/put-role-policy.handler.js +5 -0
- package/dist/service/iam/command/policy/put-user-policy/put-user-policy.handler.js +5 -0
- package/dist/service/iam/command/role/create-role/create-role.handler.js +4 -1
- package/dist/service/iam/error/sim-iam.error.d.ts +11 -0
- package/dist/service/iam/error/sim-iam.error.js +13 -0
- package/dist/service/iam/index.d.ts +1 -0
- package/dist/service/iam/policy/sim-iam-policy-document-of-size.d.ts +18 -0
- package/dist/service/iam/policy/sim-iam-policy-document-of-size.js +41 -0
- package/dist/service/iam/role/sim-iam-deploy-role.d.ts +56 -0
- package/dist/service/iam/role/sim-iam-deploy-role.js +50 -0
- package/dist/service/iam/role/sim-iam-role-with-policy.factory.d.ts +11 -0
- package/dist/service/iam/role/sim-iam-role-with-policy.factory.js +4 -2
- package/dist/service/iam/sim-iam-account-parts.d.ts +16 -0
- package/dist/service/iam/sim-iam-account-parts.js +2 -0
- package/dist/service/iam/sim-iam.d.ts +12 -2
- package/dist/service/iam/sim-iam.js +12 -0
- package/dist/service/iam/validate/size/sim-iam-policy-document-size.d.ts +57 -0
- package/dist/service/iam/validate/size/sim-iam-policy-document-size.js +71 -0
- package/dist/service/kms/command/authorize/sim-kms-authorizer.d.ts +3 -1
- package/dist/service/kms/command/authorize/sim-kms-authorizer.js +4 -1
- package/dist/service/kms/command/sim-kms-request-options.d.ts +11 -0
- package/dist/service/kms/key/sim-kms-key-policy.d.ts +5 -4
- package/dist/service/kms/key/sim-kms-key-policy.js +4 -3
- package/dist/service/lambda/cfn/function/sim-cfn-lambda-function-properties-parser.d.ts +2 -2
- package/dist/service/lambda/cfn/function/sim-cfn-lambda-function-properties-parser.js +4 -3
- package/dist/service/lambda/cfn/function/sim-cfn-lambda-generated-function-name.d.ts +10 -0
- package/dist/service/lambda/cfn/function/sim-cfn-lambda-generated-function-name.js +25 -0
- package/dist/service/lambda/command/authorize/sim-lambda-resource-policies.d.ts +1 -1
- package/dist/service/lambda/command/create-function/create-function-authorizer.d.ts +9 -2
- package/dist/service/lambda/command/create-function/create-function-authorizer.js +16 -2
- package/dist/service/lambda/command/create-function/create-function.handler.js +1 -1
- package/dist/service/lambda/command/update-function-configuration/update-function-configuration-authorizer.d.ts +7 -2
- package/dist/service/lambda/command/update-function-configuration/update-function-configuration-authorizer.js +14 -2
- package/dist/service/lambda/command/update-function-configuration/update-function-configuration.handler.js +1 -1
- package/dist/service/lambda/function/logging/sim-lambda-function-logging.d.ts +10 -1
- package/dist/service/lambda/function/logging/sim-lambda-function-logging.js +25 -1
- package/dist/service/lambda/function/logging/sim-lambda-invoke-error-log.d.ts +13 -0
- package/dist/service/lambda/function/logging/sim-lambda-invoke-error-log.js +18 -0
- package/dist/service/lambda/sim-lambda-service-principal.d.ts +7 -0
- package/dist/service/lambda/sim-lambda-service-principal.js +7 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-authorization.d.ts +46 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-authorization.js +49 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-creator.d.ts +5 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-creator.js +6 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-destination-creator.d.ts +5 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-destination-creator.js +6 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-source-creator.d.ts +5 -0
- package/dist/service/logs/cfn/delivery/sim-cfn-delivery-source-creator.js +6 -0
- package/dist/service/logs/cfn/group/sim-cfn-log-group-creator.d.ts +9 -2
- package/dist/service/logs/cfn/group/sim-cfn-log-group-creator.js +34 -5
- package/dist/service/logs/cfn/sim-logs-cfn-resource-deleter.js +1 -1
- package/dist/service/logs/cfn/sim-logs-cfn-resource-factory.d.ts +4 -0
- package/dist/service/logs/cfn/sim-logs-cfn-resource-factory.js +9 -4
- package/dist/service/logs/sim-logs-commands.d.ts +3 -0
- package/dist/service/logs/sim-logs-commands.js +4 -0
- package/dist/service/logs/sim-logs.js +6 -1
- package/dist/service/s3/cfn/bucket/sim-cfn-s3-bucket-creator.d.ts +2 -2
- package/dist/service/s3/cfn/bucket/sim-cfn-s3-bucket-creator.js +4 -3
- package/dist/service/s3/cfn/bucket/sim-cfn-s3-bucket-generated-name.d.ts +12 -0
- package/dist/service/s3/cfn/bucket/sim-cfn-s3-bucket-generated-name.js +22 -0
- package/dist/service/s3/command/authorize/sim-s3-bucket-resource-policies.d.ts +1 -1
- package/dist/service/scheduler/command/authorize/sim-scheduler-authorizer.d.ts +10 -1
- package/dist/service/scheduler/command/authorize/sim-scheduler-authorizer.js +19 -1
- package/dist/service/scheduler/command/schedule/sim-scheduler-create-schedule.js +1 -0
- package/dist/service/scheduler/command/schedule/sim-scheduler-schedule-access.d.ts +8 -0
- package/dist/service/scheduler/command/schedule/sim-scheduler-schedule-access.js +9 -0
- package/dist/service/scheduler/command/schedule/sim-scheduler-update-schedule.js +1 -0
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-generated-name.d.ts +15 -0
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-generated-name.js +22 -0
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-secret-properties.d.ts +4 -10
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-secret-properties.js +13 -31
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-secret-value.d.ts +17 -0
- package/dist/service/secretsmanager/cfn/secret/sim-cfn-secrets-manager-secret-value.js +30 -0
- package/dist/service/secretsmanager/secret/sim-secrets-manager-secret-name.d.ts +6 -0
- package/dist/service/secretsmanager/secret/sim-secrets-manager-secret-name.js +8 -3
- package/dist/service/sns/command/authorize/sim-sns-topic-resource-policies.d.ts +1 -1
- package/dist/service/sqs/command/authorize/sim-sqs-queue-resource-policies.d.ts +1 -1
- package/dist/service/ssm/cfn/parameter/sim-cfn-ssm-generated-parameter-name.d.ts +16 -0
- package/dist/service/ssm/cfn/parameter/sim-cfn-ssm-generated-parameter-name.js +23 -0
- package/dist/service/ssm/cfn/parameter/sim-cfn-ssm-parameter-properties.d.ts +2 -4
- package/dist/service/ssm/cfn/parameter/sim-cfn-ssm-parameter-properties.js +5 -5
- package/dist/service/ssm/parameter/sim-ssm-kms-crypto.d.ts +6 -4
- package/dist/service/ssm/parameter/sim-ssm-kms-crypto.js +4 -4
- package/dist/service/ssm/parameter/sim-ssm-parameter-arn.d.ts +17 -0
- package/dist/service/ssm/parameter/sim-ssm-parameter-arn.js +20 -0
- package/dist/service/ssm/parameter/sim-ssm-parameter-kms.d.ts +15 -9
- package/dist/service/ssm/parameter/sim-ssm-parameter-kms.js +20 -10
- package/dist/service/ssm/parameter/sim-ssm-parameter-name.js +3 -12
- package/dist/service/stepfunctions/command/authorize/sim-step-functions-authorizer.d.ts +41 -0
- package/dist/service/stepfunctions/command/authorize/sim-step-functions-authorizer.js +64 -0
- package/dist/service/stepfunctions/command/machine/sim-state-machine-create.d.ts +5 -2
- package/dist/service/stepfunctions/command/machine/sim-state-machine-create.js +6 -2
- package/dist/service/stepfunctions/command/machine/sim-state-machine-delete.d.ts +26 -0
- package/dist/service/stepfunctions/command/machine/sim-state-machine-delete.js +29 -0
- package/dist/service/stepfunctions/command/machine/sim-state-machine-lookup.d.ts +7 -0
- package/dist/service/stepfunctions/command/machine/sim-state-machine-lookup.js +12 -3
- package/dist/service/stepfunctions/command/machine/sim-state-machine-update.d.ts +19 -0
- package/dist/service/stepfunctions/command/machine/{sim-state-machine-writes.js → sim-state-machine-update.js} +3 -14
- package/dist/service/stepfunctions/sim-step-functions.d.ts +8 -2
- package/dist/service/stepfunctions/sim-step-functions.js +26 -9
- package/dist/service/sts/auth-z/assume-role-auth-z-coordinator.d.ts +9 -2
- package/dist/service/sts/auth-z/assume-role-auth-z-coordinator.js +2 -2
- package/dist/service/sts/auth-z/assume-role-source-account-auth-z.d.ts +10 -4
- package/dist/service/sts/auth-z/assume-role-source-account-auth-z.js +8 -7
- package/dist/service/sts/auth-z/assume-role-target-auth-z.d.ts +2 -2
- package/dist/service/sts/auth-z/assume-role-trust-policy-authorizer.d.ts +6 -2
- package/dist/service/sts/auth-z/assume-role-trust-policy-authorizer.js +14 -11
- package/dist/service/sts/command/assume-role/assume-role.handler.d.ts +10 -1
- package/dist/service/sts/command/assume-role/assume-role.handler.js +6 -2
- package/dist/service/sts/command/get-caller-identity/get-caller-identity.handler.d.ts +10 -1
- package/dist/service/sts/command/get-caller-identity/get-caller-identity.handler.js +5 -1
- package/dist/service/sts/service-role/sim-service-role.js +5 -1
- package/dist/service/sts/sim-sts.d.ts +18 -1
- package/dist/service/sts/sim-sts.js +8 -0
- package/dist/util/memo/bounded-memo.d.ts +36 -0
- package/dist/util/memo/bounded-memo.js +58 -0
- package/docs/README.md +54 -0
- package/docs/ai-skill/README.md +69 -0
- package/docs/cli/README.md +284 -0
- package/docs/factories/README.md +140 -0
- package/docs/lint/README.md +176 -0
- package/docs/non-aws-dependencies/README.md +261 -0
- package/docs/sdk/README.md +274 -0
- package/docs/serve/README.md +1126 -0
- package/docs/services/acm/README.md +823 -0
- package/docs/services/apigateway/README.md +1919 -0
- package/docs/services/apigatewayv2/README.md +2678 -0
- package/docs/services/athena/README.md +1044 -0
- package/docs/services/bedrock/README.md +350 -0
- package/docs/services/cloudformation/README.md +3625 -0
- package/docs/services/cloudfront/README.md +3098 -0
- package/docs/services/cloudwatch/README.md +422 -0
- package/docs/services/cognito/README.md +4924 -0
- package/docs/services/dynamodb/README.md +3785 -0
- package/docs/services/ecr/README.md +302 -0
- package/docs/services/ecs/README.md +2021 -0
- package/docs/services/elbv2/README.md +1901 -0
- package/docs/services/eventbridge/README.md +1118 -0
- package/docs/services/firehose/README.md +811 -0
- package/docs/services/glue/README.md +507 -0
- package/docs/services/iam/README.md +1578 -0
- package/docs/services/kinesis/README.md +455 -0
- package/docs/services/kms/README.md +664 -0
- package/docs/services/lambda/README.md +3719 -0
- package/docs/services/logs/README.md +882 -0
- package/docs/services/organizations/README.md +574 -0
- package/docs/services/personalize/README.md +835 -0
- package/docs/services/rekognition/README.md +1070 -0
- package/docs/services/route53/README.md +1783 -0
- package/docs/services/s3/README.md +3029 -0
- package/docs/services/scheduler/README.md +814 -0
- package/docs/services/secretsmanager/README.md +670 -0
- package/docs/services/ses/README.md +1013 -0
- package/docs/services/sns/README.md +1904 -0
- package/docs/services/sqs/README.md +1185 -0
- package/docs/services/ssm/README.md +1096 -0
- package/docs/services/stepfunctions/README.md +1341 -0
- package/docs/services/sts/README.md +245 -0
- package/docs/services/wafv2/README.md +1380 -0
- package/docs/terraform/README.md +248 -0
- package/docs/time/README.md +284 -0
- package/llms.txt +59 -0
- package/package.json +7 -4
- package/dist/service/stepfunctions/command/machine/sim-state-machine-writes.d.ts +0 -25
|
@@ -0,0 +1,3029 @@
|
|
|
1
|
+
# Simulated S3
|
|
2
|
+
|
|
3
|
+
Yulin includes a simulated S3 service for tests and local development.
|
|
4
|
+
|
|
5
|
+
Sim S3 can be used directly through `SimAws` or instantiated on its own as `SimS3` with isolated
|
|
6
|
+
state. Yulin can serve a simulated S3 service on localhost.
|
|
7
|
+
|
|
8
|
+
## Basic usage
|
|
9
|
+
|
|
10
|
+
Create a simulated AWS environment, get simulated S3, create a Bucket, and put an Object into it.
|
|
11
|
+
|
|
12
|
+
```typescript sim-s3-bucket
|
|
13
|
+
/**
|
|
14
|
+
* Creating a simulated S3 Bucket and putting an Object into it.
|
|
15
|
+
*/
|
|
16
|
+
|
|
17
|
+
import {
|
|
18
|
+
CreateBucketCommand,
|
|
19
|
+
GetObjectCommand,
|
|
20
|
+
PutObjectCommand,
|
|
21
|
+
} from "@aws-sdk/client-s3";
|
|
22
|
+
import { SimAws } from "@kensio/yulin";
|
|
23
|
+
|
|
24
|
+
const simAws = new SimAws();
|
|
25
|
+
const simS3 = simAws.s3();
|
|
26
|
+
|
|
27
|
+
await simS3.createBucket(
|
|
28
|
+
new CreateBucketCommand({
|
|
29
|
+
Bucket: "foo-bucket",
|
|
30
|
+
}),
|
|
31
|
+
);
|
|
32
|
+
|
|
33
|
+
await simS3.putObject(
|
|
34
|
+
new PutObjectCommand({
|
|
35
|
+
Bucket: "foo-bucket",
|
|
36
|
+
Key: "hello.txt",
|
|
37
|
+
Body: "Hello from simulated S3",
|
|
38
|
+
ContentType: "text/plain; charset=utf-8",
|
|
39
|
+
Metadata: {
|
|
40
|
+
source: "yulin",
|
|
41
|
+
},
|
|
42
|
+
}),
|
|
43
|
+
);
|
|
44
|
+
|
|
45
|
+
const objectOut = await simS3.getObject(
|
|
46
|
+
new GetObjectCommand({
|
|
47
|
+
Bucket: "foo-bucket",
|
|
48
|
+
Key: "hello.txt",
|
|
49
|
+
}),
|
|
50
|
+
);
|
|
51
|
+
|
|
52
|
+
console.log(objectOut.Body);
|
|
53
|
+
console.log(objectOut.Metadata?.["source"]);
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
`PutObjectCommand` currently accepts `string`, `Uint8Array`, or `undefined` for `Body`. An undefined
|
|
57
|
+
body is stored as an empty Object.
|
|
58
|
+
|
|
59
|
+
A read hands `ContentType` back in the field of the same name, and Bucket website responses are
|
|
60
|
+
served with it. It is one of several headers a write can say about an Object. See
|
|
61
|
+
[Object system metadata](#object-system-metadata).
|
|
62
|
+
|
|
63
|
+
## Accounts and Regions
|
|
64
|
+
|
|
65
|
+
Use `SimAws` scopes to simulate S3 in different AWS Accounts and Regions.
|
|
66
|
+
|
|
67
|
+
```typescript sim-s3-account-region-scoping
|
|
68
|
+
/**
|
|
69
|
+
* Simulated S3 Account and Region scoping.
|
|
70
|
+
*/
|
|
71
|
+
|
|
72
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
73
|
+
import { SimAws } from "@kensio/yulin";
|
|
74
|
+
|
|
75
|
+
const simAws = new SimAws();
|
|
76
|
+
|
|
77
|
+
const defaultS3 = simAws.s3();
|
|
78
|
+
const euWest2S3 = simAws.region("eu-west-2").s3();
|
|
79
|
+
const accountS3 = simAws.account("111111111111").s3();
|
|
80
|
+
const scopedS3 = simAws.account("222222222222").region("ap-east-1").s3();
|
|
81
|
+
|
|
82
|
+
await defaultS3.createBucket(
|
|
83
|
+
new CreateBucketCommand({
|
|
84
|
+
Bucket: "default-bucket",
|
|
85
|
+
}),
|
|
86
|
+
);
|
|
87
|
+
|
|
88
|
+
await euWest2S3.createBucket(
|
|
89
|
+
new CreateBucketCommand({
|
|
90
|
+
Bucket: "eu-west-2-bucket",
|
|
91
|
+
}),
|
|
92
|
+
);
|
|
93
|
+
|
|
94
|
+
await accountS3.createBucket(
|
|
95
|
+
new CreateBucketCommand({
|
|
96
|
+
Bucket: "account-bucket",
|
|
97
|
+
}),
|
|
98
|
+
);
|
|
99
|
+
|
|
100
|
+
await scopedS3.createBucket(
|
|
101
|
+
new CreateBucketCommand({
|
|
102
|
+
Bucket: "scoped-bucket",
|
|
103
|
+
}),
|
|
104
|
+
);
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Within one `SimAws` instance, Bucket names are globally registered across Accounts and Regions.
|
|
108
|
+
Creating a Bucket with a name already used in another simulated Region or Account throws an error.
|
|
109
|
+
|
|
110
|
+
Each `SimAws` instance has its own isolated state. Create a fresh one per test or share one across
|
|
111
|
+
all tests, as you prefer.
|
|
112
|
+
|
|
113
|
+
## Listing Buckets
|
|
114
|
+
|
|
115
|
+
Use `ListBucketsCommand` to inspect Buckets in the selected simulated S3 scope. Each Bucket reports the instant it was created, taken from [simulated time](https://yulinsim.dev/time/) rather than the host clock.
|
|
116
|
+
|
|
117
|
+
```typescript sim-s3-list-buckets
|
|
118
|
+
/**
|
|
119
|
+
* Listing Buckets in simulated S3.
|
|
120
|
+
*/
|
|
121
|
+
|
|
122
|
+
import { CreateBucketCommand, ListBucketsCommand } from "@aws-sdk/client-s3";
|
|
123
|
+
import { SimAws } from "@kensio/yulin";
|
|
124
|
+
|
|
125
|
+
const simAws = new SimAws();
|
|
126
|
+
const simS3 = simAws.s3();
|
|
127
|
+
|
|
128
|
+
await simS3.createBucket(
|
|
129
|
+
new CreateBucketCommand({
|
|
130
|
+
Bucket: "foo-bucket",
|
|
131
|
+
}),
|
|
132
|
+
);
|
|
133
|
+
|
|
134
|
+
const listBucketsOutput = await simS3.listBuckets(new ListBucketsCommand());
|
|
135
|
+
|
|
136
|
+
console.log(listBucketsOutput.Buckets?.map((bucket) => bucket.Name));
|
|
137
|
+
console.log(listBucketsOutput.Buckets?.[0]?.CreationDate);
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
## Asking whether something is there
|
|
141
|
+
|
|
142
|
+
`HeadObjectCommand` reports what a read would say about an Object without returning the Object, and `HeadBucketCommand` reports whether a Bucket is there and reachable. `HeadBucket` also reports the Region it was found in.
|
|
143
|
+
|
|
144
|
+
A HEAD response carries no body, so there is no document for an error code to travel in. Real S3 answers a `HeadObject` with `403` or `404`, and a `HeadBucket` with `400`, `403` or `404`, leaving the SDK to name the failure from the status alone. The simulator answers `404` for an absent Bucket and an absent Object alike, which an SDK client raises as `NotFound`, and `403` for a caller the permission is missing for. A read distinguishes `NoSuchBucket` from `NoSuchKey`, because a read has a body to say which.
|
|
145
|
+
|
|
146
|
+
`HeadObject` authorizes against `s3:GetObject` and `HeadBucket` against `s3:ListBucket`, as real S3 does, so knowing something is there needs the permission to read it.
|
|
147
|
+
|
|
148
|
+
## Listing Objects
|
|
149
|
+
|
|
150
|
+
Use `ListObjectsV2Command` to list the Objects in a Bucket. The simulator supports `Prefix`,
|
|
151
|
+
`Delimiter`, `MaxKeys`, `ContinuationToken` and `StartAfter`, and answers with `Contents`,
|
|
152
|
+
`CommonPrefixes`, `KeyCount`, `IsTruncated` and `NextContinuationToken`.
|
|
153
|
+
|
|
154
|
+
```typescript sim-s3-list-objects-v2
|
|
155
|
+
/**
|
|
156
|
+
* Listing Objects in a simulated S3 Bucket.
|
|
157
|
+
*/
|
|
158
|
+
|
|
159
|
+
import {
|
|
160
|
+
CreateBucketCommand,
|
|
161
|
+
ListObjectsV2Command,
|
|
162
|
+
PutObjectCommand,
|
|
163
|
+
} from "@aws-sdk/client-s3";
|
|
164
|
+
import { SimAws } from "@kensio/yulin";
|
|
165
|
+
|
|
166
|
+
const simAws = new SimAws();
|
|
167
|
+
const simS3 = simAws.s3();
|
|
168
|
+
|
|
169
|
+
await simS3.createBucket(
|
|
170
|
+
new CreateBucketCommand({
|
|
171
|
+
Bucket: "assets-bucket",
|
|
172
|
+
}),
|
|
173
|
+
);
|
|
174
|
+
|
|
175
|
+
for (const key of ["docs/index.html", "docs/guide.html", "images/logo.svg"]) {
|
|
176
|
+
await simS3.putObject(
|
|
177
|
+
new PutObjectCommand({
|
|
178
|
+
Bucket: "assets-bucket",
|
|
179
|
+
Key: key,
|
|
180
|
+
Body: "file contents",
|
|
181
|
+
}),
|
|
182
|
+
);
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
const listOutput = await simS3.listObjectsV2(
|
|
186
|
+
new ListObjectsV2Command({
|
|
187
|
+
Bucket: "assets-bucket",
|
|
188
|
+
Prefix: "docs/",
|
|
189
|
+
}),
|
|
190
|
+
);
|
|
191
|
+
|
|
192
|
+
console.log(listOutput.KeyCount);
|
|
193
|
+
|
|
194
|
+
const listedObjects = listOutput.Contents ?? [];
|
|
195
|
+
for (const object of listedObjects) {
|
|
196
|
+
console.log(object.Key, object.Size, object.ETag, object.LastModified);
|
|
197
|
+
}
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
Listings are sorted by key, and a page holds at most 1,000 keys, as in real S3. `MaxKeys` above that
|
|
201
|
+
is lowered to it, and the response reports the page size that was actually used. A `MaxKeys` of zero
|
|
202
|
+
returns no keys and completes the listing, and a negative one is refused with `InvalidArgument`.
|
|
203
|
+
|
|
204
|
+
A listing that found no keys has no `Contents` at all, and the example reaches for `Contents ?? []`
|
|
205
|
+
for that reason. `KeyCount` is the count either way.
|
|
206
|
+
|
|
207
|
+
### Walking a truncated listing
|
|
208
|
+
|
|
209
|
+
A truncated response carries `NextContinuationToken`, which the next request passes as
|
|
210
|
+
`ContinuationToken`. The token is opaque, as it is in real S3. Pass it back unchanged, read nothing
|
|
211
|
+
out of it, and simulated S3 refuses one it did not issue.
|
|
212
|
+
|
|
213
|
+
```typescript sim-s3-list-objects-v2-pagination
|
|
214
|
+
/**
|
|
215
|
+
* Walking a truncated Object listing in a simulated S3 Bucket.
|
|
216
|
+
*/
|
|
217
|
+
|
|
218
|
+
import {
|
|
219
|
+
CreateBucketCommand,
|
|
220
|
+
ListObjectsV2Command,
|
|
221
|
+
PutObjectCommand,
|
|
222
|
+
} from "@aws-sdk/client-s3";
|
|
223
|
+
import { SimAws } from "@kensio/yulin";
|
|
224
|
+
|
|
225
|
+
const simAws = new SimAws();
|
|
226
|
+
const simS3 = simAws.s3();
|
|
227
|
+
|
|
228
|
+
await simS3.createBucket(
|
|
229
|
+
new CreateBucketCommand({
|
|
230
|
+
Bucket: "paged-bucket",
|
|
231
|
+
}),
|
|
232
|
+
);
|
|
233
|
+
|
|
234
|
+
for (const key of ["a.txt", "b.txt", "c.txt"]) {
|
|
235
|
+
await simS3.putObject(
|
|
236
|
+
new PutObjectCommand({ Bucket: "paged-bucket", Key: key, Body: key }),
|
|
237
|
+
);
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
// Ask for a page of one, so the listing has to be continued.
|
|
241
|
+
let continuationToken: string | undefined;
|
|
242
|
+
const allKeys: string[] = [];
|
|
243
|
+
|
|
244
|
+
do {
|
|
245
|
+
const page = await simS3.listObjectsV2(
|
|
246
|
+
new ListObjectsV2Command({
|
|
247
|
+
Bucket: "paged-bucket",
|
|
248
|
+
MaxKeys: 1,
|
|
249
|
+
ContinuationToken: continuationToken,
|
|
250
|
+
}),
|
|
251
|
+
);
|
|
252
|
+
|
|
253
|
+
const pageObjects = page.Contents ?? [];
|
|
254
|
+
for (const object of pageObjects) {
|
|
255
|
+
allKeys.push(object.Key ?? "");
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
continuationToken = page.NextContinuationToken;
|
|
259
|
+
} while (continuationToken !== undefined);
|
|
260
|
+
|
|
261
|
+
console.log(allKeys);
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
Code that never names `MaxKeys` never continues a listing in a test small enough to be readable, and
|
|
265
|
+
its pagination goes unexercised. `configureMaxKeysPerPage` lowers the page size for a whole simulated
|
|
266
|
+
S3 instead. A Bucket of two Objects is then enough to make the caller walk a continuation:
|
|
267
|
+
|
|
268
|
+
```typescript sim-s3-list-page-size
|
|
269
|
+
/**
|
|
270
|
+
* Lowering the page size of a simulated S3 listing, so a caller that does not
|
|
271
|
+
* set MaxKeys still has to ask for a second page.
|
|
272
|
+
*/
|
|
273
|
+
|
|
274
|
+
import {
|
|
275
|
+
CreateBucketCommand,
|
|
276
|
+
ListObjectsV2Command,
|
|
277
|
+
PutObjectCommand,
|
|
278
|
+
} from "@aws-sdk/client-s3";
|
|
279
|
+
import { SimAws } from "@kensio/yulin";
|
|
280
|
+
|
|
281
|
+
const simAws = new SimAws();
|
|
282
|
+
const simS3 = simAws.s3();
|
|
283
|
+
simS3.configureMaxKeysPerPage(1);
|
|
284
|
+
|
|
285
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "small-pages" }));
|
|
286
|
+
|
|
287
|
+
for (const key of ["a.txt", "b.txt"]) {
|
|
288
|
+
await simS3.putObject(
|
|
289
|
+
new PutObjectCommand({ Bucket: "small-pages", Key: key, Body: key }),
|
|
290
|
+
);
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
const firstPage = await simS3.listObjectsV2(
|
|
294
|
+
new ListObjectsV2Command({ Bucket: "small-pages" }),
|
|
295
|
+
);
|
|
296
|
+
|
|
297
|
+
console.log(firstPage.IsTruncated, firstPage.KeyCount);
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
### Walking a Bucket as a folder tree
|
|
301
|
+
|
|
302
|
+
S3 stores keys flat and a `Delimiter` is what makes one look like a folder tree. Every key holding
|
|
303
|
+
the delimiter somewhere after the `Prefix` is rolled up into a common prefix, running from the start
|
|
304
|
+
of the key through the first delimiter. Those keys leave `Contents`, and the prefix appears once in
|
|
305
|
+
`CommonPrefixes` however many keys sit beneath it.
|
|
306
|
+
|
|
307
|
+
```typescript sim-s3-list-objects-v2-delimiter
|
|
308
|
+
/**
|
|
309
|
+
* Walking a simulated S3 Bucket one folder at a time.
|
|
310
|
+
*/
|
|
311
|
+
|
|
312
|
+
import {
|
|
313
|
+
CreateBucketCommand,
|
|
314
|
+
ListObjectsV2Command,
|
|
315
|
+
PutObjectCommand,
|
|
316
|
+
} from "@aws-sdk/client-s3";
|
|
317
|
+
import { SimAws } from "@kensio/yulin";
|
|
318
|
+
|
|
319
|
+
const simAws = new SimAws();
|
|
320
|
+
const simS3 = simAws.s3();
|
|
321
|
+
|
|
322
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "site-assets" }));
|
|
323
|
+
|
|
324
|
+
for (const key of ["img/logo.png", "img/icons/tick.png", "index.html"]) {
|
|
325
|
+
await simS3.putObject(
|
|
326
|
+
new PutObjectCommand({ Bucket: "site-assets", Key: key, Body: key }),
|
|
327
|
+
);
|
|
328
|
+
}
|
|
329
|
+
|
|
330
|
+
const top = await simS3.listObjectsV2(
|
|
331
|
+
new ListObjectsV2Command({ Bucket: "site-assets", Delimiter: "/" }),
|
|
332
|
+
);
|
|
333
|
+
|
|
334
|
+
// One folder, and the one key that sits beside it.
|
|
335
|
+
console.log(top.CommonPrefixes?.map((folder) => folder.Prefix)); // ["img/"]
|
|
336
|
+
console.log(top.Contents?.map((object) => object.Key)); // ["index.html"]
|
|
337
|
+
|
|
338
|
+
const folder = await simS3.listObjectsV2(
|
|
339
|
+
new ListObjectsV2Command({
|
|
340
|
+
Bucket: "site-assets",
|
|
341
|
+
Prefix: "img/",
|
|
342
|
+
Delimiter: "/",
|
|
343
|
+
}),
|
|
344
|
+
);
|
|
345
|
+
|
|
346
|
+
// A delimiter inside the Prefix is stepped over, so this lists what is
|
|
347
|
+
// directly in `img/` rather than rolling the whole Bucket back up.
|
|
348
|
+
console.log(folder.CommonPrefixes?.map((child) => child.Prefix)); // ["img/icons/"]
|
|
349
|
+
console.log(folder.Contents?.map((object) => object.Key)); // ["img/logo.png"]
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
A listing that rolled nothing up has no `CommonPrefixes` at all, the way one that found no keys has
|
|
353
|
+
no `Contents`. Reach for `CommonPrefixes ?? []`.
|
|
354
|
+
|
|
355
|
+
A common prefix counts against `MaxKeys` as a key does, and `KeyCount` counts the two together. Keys
|
|
356
|
+
and prefixes are ordered together, so a truncated page can end on either, and the continuation steps
|
|
357
|
+
over the whole rolled-up prefix rather than listing its keys again. `aws s3 ls s3://bucket/` against
|
|
358
|
+
[a simulation served on localhost](#serve-simulated-s3-on-localhost) prints these as `PRE` lines.
|
|
359
|
+
|
|
360
|
+
### The first version of the operation
|
|
361
|
+
|
|
362
|
+
`ListObjectsCommand` is also simulated, with the `Marker` and `NextMarker` shape it has in real S3.
|
|
363
|
+
It lists the same keys as `ListObjectsV2Command` and is bounded by the same page size.
|
|
364
|
+
|
|
365
|
+
```typescript sim-s3-list-objects
|
|
366
|
+
/**
|
|
367
|
+
* Listing Objects in a simulated S3 Bucket.
|
|
368
|
+
*/
|
|
369
|
+
|
|
370
|
+
import {
|
|
371
|
+
CreateBucketCommand,
|
|
372
|
+
ListObjectsCommand,
|
|
373
|
+
PutObjectCommand,
|
|
374
|
+
} from "@aws-sdk/client-s3";
|
|
375
|
+
import { SimAws } from "@kensio/yulin";
|
|
376
|
+
|
|
377
|
+
const simAws = new SimAws();
|
|
378
|
+
const simS3 = simAws.s3();
|
|
379
|
+
|
|
380
|
+
await simS3.createBucket(
|
|
381
|
+
new CreateBucketCommand({
|
|
382
|
+
Bucket: "assets-bucket",
|
|
383
|
+
}),
|
|
384
|
+
);
|
|
385
|
+
|
|
386
|
+
await simS3.putObject(
|
|
387
|
+
new PutObjectCommand({
|
|
388
|
+
Bucket: "assets-bucket",
|
|
389
|
+
Key: "images/logo.svg",
|
|
390
|
+
Body: "<svg></svg>",
|
|
391
|
+
ContentType: "image/svg+xml",
|
|
392
|
+
}),
|
|
393
|
+
);
|
|
394
|
+
|
|
395
|
+
await simS3.putObject(
|
|
396
|
+
new PutObjectCommand({
|
|
397
|
+
Bucket: "assets-bucket",
|
|
398
|
+
Key: "docs/index.html",
|
|
399
|
+
Body: "<h1>Docs</h1>",
|
|
400
|
+
ContentType: "text/html; charset=utf-8",
|
|
401
|
+
}),
|
|
402
|
+
);
|
|
403
|
+
|
|
404
|
+
const listObjectsOutput = await simS3.listObjects(
|
|
405
|
+
new ListObjectsCommand({
|
|
406
|
+
Bucket: "assets-bucket",
|
|
407
|
+
Prefix: "docs/",
|
|
408
|
+
MaxKeys: 10,
|
|
409
|
+
}),
|
|
410
|
+
);
|
|
411
|
+
|
|
412
|
+
const objectContentItems = listObjectsOutput.Contents ?? [];
|
|
413
|
+
for (const object of objectContentItems) {
|
|
414
|
+
console.log(object.Key, object.Size);
|
|
415
|
+
}
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
The marker is exclusive and lexicographic. A listing resumes after the key it names whether or not
|
|
419
|
+
the Bucket still holds it.
|
|
420
|
+
|
|
421
|
+
## Object ETags
|
|
422
|
+
|
|
423
|
+
Every Object has an ETag, the MD5 of its body in hex and quoted, as real S3 gives it for a
|
|
424
|
+
single-part upload. `PutObject`, `GetObject`, both list operations and the S3 REST endpoint all
|
|
425
|
+
report the same one. A tool can compare what a Bucket holds against a local file without reading the
|
|
426
|
+
Object back.
|
|
427
|
+
|
|
428
|
+
```typescript sim-s3-object-etag
|
|
429
|
+
/**
|
|
430
|
+
* Comparing a local file against a simulated S3 Object by content hash.
|
|
431
|
+
*/
|
|
432
|
+
|
|
433
|
+
import { createHash } from "node:crypto";
|
|
434
|
+
import {
|
|
435
|
+
CreateBucketCommand,
|
|
436
|
+
ListObjectsV2Command,
|
|
437
|
+
PutObjectCommand,
|
|
438
|
+
} from "@aws-sdk/client-s3";
|
|
439
|
+
import { SimAws } from "@kensio/yulin";
|
|
440
|
+
|
|
441
|
+
const simAws = new SimAws();
|
|
442
|
+
const simS3 = simAws.s3();
|
|
443
|
+
|
|
444
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "site-bucket" }));
|
|
445
|
+
|
|
446
|
+
const published = "<h1>Hello</h1>";
|
|
447
|
+
await simS3.putObject(
|
|
448
|
+
new PutObjectCommand({
|
|
449
|
+
Bucket: "site-bucket",
|
|
450
|
+
Key: "index.html",
|
|
451
|
+
Body: published,
|
|
452
|
+
}),
|
|
453
|
+
);
|
|
454
|
+
|
|
455
|
+
const listOutput = await simS3.listObjectsV2(
|
|
456
|
+
new ListObjectsV2Command({ Bucket: "site-bucket" }),
|
|
457
|
+
);
|
|
458
|
+
|
|
459
|
+
const localFile = Buffer.from(published);
|
|
460
|
+
const localETag = `"${createHash("md5").update(localFile).digest("hex")}"`;
|
|
461
|
+
|
|
462
|
+
const listedObjects = listOutput.Contents ?? [];
|
|
463
|
+
for (const object of listedObjects) {
|
|
464
|
+
// Nothing to upload: the Bucket already holds these bytes.
|
|
465
|
+
console.log(object.Key, object.ETag === localETag);
|
|
466
|
+
}
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
An event notification record carries the same value unquoted, in its `eTag` field, as real S3
|
|
470
|
+
reports it there.
|
|
471
|
+
|
|
472
|
+
An Object uploaded in parts gets a different form. See
|
|
473
|
+
[Uploading an Object in parts](#uploading-an-object-in-parts).
|
|
474
|
+
|
|
475
|
+
## Uploading an Object in parts
|
|
476
|
+
|
|
477
|
+
`aws s3 cp` switches to a multipart upload above eight megabytes, and `@aws-sdk/lib-storage` uploads
|
|
478
|
+
in parts whatever the size. Sim S3 answers the six operations that path is made of, over the SDK and
|
|
479
|
+
over a served endpoint alike.
|
|
480
|
+
|
|
481
|
+
```bash
|
|
482
|
+
aws s3 cp ./big.bin s3://widgets/big.bin # 12MB, multipart under the covers
|
|
483
|
+
aws s3 ls s3://widgets/ # reports the whole 12MB Object
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
An upload is started, the parts are sent under the id it issues, and completing it stores one
|
|
487
|
+
Object. The parts can be sent in any order.
|
|
488
|
+
|
|
489
|
+
```typescript sim-s3-multipart-upload
|
|
490
|
+
/**
|
|
491
|
+
* Uploading a simulated S3 Object in parts.
|
|
492
|
+
*/
|
|
493
|
+
|
|
494
|
+
import {
|
|
495
|
+
CompleteMultipartUploadCommand,
|
|
496
|
+
CreateBucketCommand,
|
|
497
|
+
CreateMultipartUploadCommand,
|
|
498
|
+
GetObjectCommand,
|
|
499
|
+
UploadPartCommand,
|
|
500
|
+
} from "@aws-sdk/client-s3";
|
|
501
|
+
import { SimAws } from "@kensio/yulin";
|
|
502
|
+
|
|
503
|
+
const simAws = new SimAws();
|
|
504
|
+
const simS3 = simAws.s3();
|
|
505
|
+
|
|
506
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "uploads-bucket" }));
|
|
507
|
+
|
|
508
|
+
const started = await simS3.createMultipartUpload(
|
|
509
|
+
new CreateMultipartUploadCommand({
|
|
510
|
+
Bucket: "uploads-bucket",
|
|
511
|
+
Key: "report.csv",
|
|
512
|
+
ContentType: "text/csv",
|
|
513
|
+
}),
|
|
514
|
+
);
|
|
515
|
+
|
|
516
|
+
const second = await simS3.uploadPart(
|
|
517
|
+
new UploadPartCommand({
|
|
518
|
+
Bucket: "uploads-bucket",
|
|
519
|
+
Key: "report.csv",
|
|
520
|
+
UploadId: started.UploadId,
|
|
521
|
+
PartNumber: 2,
|
|
522
|
+
Body: "2,two\n",
|
|
523
|
+
}),
|
|
524
|
+
);
|
|
525
|
+
|
|
526
|
+
const first = await simS3.uploadPart(
|
|
527
|
+
new UploadPartCommand({
|
|
528
|
+
Bucket: "uploads-bucket",
|
|
529
|
+
Key: "report.csv",
|
|
530
|
+
UploadId: started.UploadId,
|
|
531
|
+
PartNumber: 1,
|
|
532
|
+
Body: "id,name\n1,one\n",
|
|
533
|
+
}),
|
|
534
|
+
);
|
|
535
|
+
|
|
536
|
+
const completed = await simS3.completeMultipartUpload(
|
|
537
|
+
new CompleteMultipartUploadCommand({
|
|
538
|
+
Bucket: "uploads-bucket",
|
|
539
|
+
Key: "report.csv",
|
|
540
|
+
UploadId: started.UploadId,
|
|
541
|
+
MultipartUpload: {
|
|
542
|
+
Parts: [
|
|
543
|
+
{ PartNumber: 1, ETag: first.ETag },
|
|
544
|
+
{ PartNumber: 2, ETag: second.ETag },
|
|
545
|
+
],
|
|
546
|
+
},
|
|
547
|
+
}),
|
|
548
|
+
);
|
|
549
|
+
|
|
550
|
+
// The parts joined in part-number order, whichever order they arrived in.
|
|
551
|
+
console.log(completed.ETag);
|
|
552
|
+
|
|
553
|
+
const objectOut = await simS3.getObject(
|
|
554
|
+
new GetObjectCommand({ Bucket: "uploads-bucket", Key: "report.csv" }),
|
|
555
|
+
);
|
|
556
|
+
|
|
557
|
+
console.log(objectOut.Body);
|
|
558
|
+
```
|
|
559
|
+
|
|
560
|
+
The completed Object is an ordinary one. Every operation that reads an Object reads it, and the
|
|
561
|
+
system metadata the upload was started with (`ContentType` above) travels with it.
|
|
562
|
+
|
|
563
|
+
### The multipart ETag
|
|
564
|
+
|
|
565
|
+
Real S3 gives an Object uploaded in parts the ETag `<md5-of-the-part-md5s>-<partCount>`, and sim S3
|
|
566
|
+
gives it the same. A tool comparing content hashes checks for that `-N` suffix before trusting an
|
|
567
|
+
ETag. An Object assembled from parts therefore cannot report the MD5 of the joined bytes. The two
|
|
568
|
+
are different values.
|
|
569
|
+
|
|
570
|
+
`PutObject`, `GetObject`, `HeadObject` and both list operations all report the same one.
|
|
571
|
+
|
|
572
|
+
### Abandoning an upload
|
|
573
|
+
|
|
574
|
+
`AbortMultipartUploadCommand` discards the parts. Nothing was ever under the key, and the Bucket is
|
|
575
|
+
left as the upload found it. An unfinished upload puts no Object anywhere, and its parts are
|
|
576
|
+
invisible to a listing.
|
|
577
|
+
|
|
578
|
+
`ListMultipartUploadsCommand` reports what a Bucket has in flight, and `ListPartsCommand` reports the
|
|
579
|
+
parts stored against one upload. Both are how a cleanup finds an upload that stalled.
|
|
580
|
+
|
|
581
|
+
### Event notifications
|
|
582
|
+
|
|
583
|
+
A completed upload raises `s3:ObjectCreated:CompleteMultipartUpload`. A single-request upload raises
|
|
584
|
+
`s3:ObjectCreated:Put`, and real S3 keeps the two apart. `s3:ObjectCreated:*` covers both. See
|
|
585
|
+
[Event notifications](#event-notifications).
|
|
586
|
+
|
|
587
|
+
### Limitations
|
|
588
|
+
|
|
589
|
+
- `UploadPartCopy` is left out. It copies a byte range from another Object into an upload.
|
|
590
|
+
`CopyObject` copies a whole Object, and is simulated. See [Copying Objects](#copying-objects).
|
|
591
|
+
- Parts are held in memory, whatever storage the Bucket uses. A Bucket backed by a mounted directory
|
|
592
|
+
writes whole files and has nowhere to put half of one.
|
|
593
|
+
- Real S3 requires every part except the last to be at least five megabytes, and answers
|
|
594
|
+
`EntityTooSmall` for one that is not. Sim S3 takes a part of any size.
|
|
595
|
+
- A listing of uploads or of parts comes back on one page. `MaxUploads`, `MaxParts`, the markers that
|
|
596
|
+
page them, and `Delimiter` are all left out.
|
|
597
|
+
- No caller has to abort an upload. An `AbortIncompleteMultipartUpload` lifecycle rule abandons one
|
|
598
|
+
the clock has left unfinished for long enough, and takes its parts with it. See
|
|
599
|
+
[Lifecycle configuration](#lifecycle-configuration).
|
|
600
|
+
|
|
601
|
+
## Reading part of an Object
|
|
602
|
+
|
|
603
|
+
`GetObjectCommand` takes a `Range` and answers with the bytes it names. A client downloading a large
|
|
604
|
+
Object asks for its pieces at once and writes each response at the offset it asked for. `aws s3 cp`
|
|
605
|
+
downloads that way above eight megabytes.
|
|
606
|
+
|
|
607
|
+
```typescript sim-s3-ranged-read
|
|
608
|
+
/**
|
|
609
|
+
* Reading part of a simulated S3 Object.
|
|
610
|
+
*/
|
|
611
|
+
|
|
612
|
+
import {
|
|
613
|
+
CreateBucketCommand,
|
|
614
|
+
GetObjectCommand,
|
|
615
|
+
PutObjectCommand,
|
|
616
|
+
} from "@aws-sdk/client-s3";
|
|
617
|
+
import { SimAws } from "@kensio/yulin";
|
|
618
|
+
|
|
619
|
+
const simAws = new SimAws();
|
|
620
|
+
const simS3 = simAws.s3();
|
|
621
|
+
|
|
622
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "reports-bucket" }));
|
|
623
|
+
|
|
624
|
+
await simS3.putObject(
|
|
625
|
+
new PutObjectCommand({
|
|
626
|
+
Bucket: "reports-bucket",
|
|
627
|
+
Key: "quarter.csv",
|
|
628
|
+
Body: "region,revenue\neu-west-2,1200\n",
|
|
629
|
+
}),
|
|
630
|
+
);
|
|
631
|
+
|
|
632
|
+
const header = await simS3.getObject(
|
|
633
|
+
new GetObjectCommand({
|
|
634
|
+
Bucket: "reports-bucket",
|
|
635
|
+
Key: "quarter.csv",
|
|
636
|
+
Range: "bytes=0-13",
|
|
637
|
+
}),
|
|
638
|
+
);
|
|
639
|
+
|
|
640
|
+
// The first fourteen bytes, which are "region,revenue".
|
|
641
|
+
console.log(header.Body);
|
|
642
|
+
// 14
|
|
643
|
+
console.log(header.ContentLength);
|
|
644
|
+
// "bytes 0-13/30"
|
|
645
|
+
console.log(header.ContentRange);
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
`ContentLength` counts the bytes being sent, and `ContentRange` says which bytes of the Object they
|
|
649
|
+
are, in the `bytes <start>-<end>/<size>` form real S3 answers with. The `ETag` is the whole Object's.
|
|
650
|
+
A client reading an Object in pieces compares it across them to see whether the Object changed
|
|
651
|
+
underneath it.
|
|
652
|
+
|
|
653
|
+
Three forms are read:
|
|
654
|
+
|
|
655
|
+
- `bytes=0-499` takes the first five hundred bytes.
|
|
656
|
+
- `bytes=500-` takes everything from byte 500 to the end.
|
|
657
|
+
- `bytes=-500` takes the last five hundred bytes.
|
|
658
|
+
|
|
659
|
+
A range running past the end of the Object stops at the last byte, and a client that guessed the
|
|
660
|
+
size gets what there is. A range starting past the end raises `InvalidRange`, under the name and the
|
|
661
|
+
416 status real S3 gives it. A `Range` sim S3 cannot read (several ranges at once, or a unit other
|
|
662
|
+
than bytes) is ignored, and the whole Object comes back under a `200`.
|
|
663
|
+
|
|
664
|
+
Over a served endpoint, a ranged read answers `206 Partial Content` with a `content-range` header.
|
|
665
|
+
Both the S3 REST endpoint and an endpoint URL a client is pointed at answer the same way. See
|
|
666
|
+
[Serve simulated S3 on localhost](#serve-simulated-s3-on-localhost).
|
|
667
|
+
|
|
668
|
+
### Limitations
|
|
669
|
+
|
|
670
|
+
- `Range` on `HeadObject` is left out. A HEAD describes the whole Object however it is asked about,
|
|
671
|
+
over the SDK and over a served endpoint alike.
|
|
672
|
+
- `If-Range` is left out. A ranged read is answered without comparing the Object against the entity
|
|
673
|
+
tag or the date the client held.
|
|
674
|
+
- `PartNumber` is left out. A read names the bytes it wants, and the part they were uploaded in is
|
|
675
|
+
not something it can ask for.
|
|
676
|
+
|
|
677
|
+
## Copying Objects
|
|
678
|
+
|
|
679
|
+
`CopyObjectCommand` reads one Object and writes its bytes under another key, in the same Bucket or
|
|
680
|
+
in another one. A move and a rename are both a copy followed by a `DeleteObjectCommand`, and an
|
|
681
|
+
archive is a copy on its own.
|
|
682
|
+
|
|
683
|
+
`CopySource` names the source as `sourceBucket/sourceKey`, URL-encoded, and a leading slash on it is
|
|
684
|
+
accepted. Everything after the first slash is the key. A key with slashes of its own needs nothing
|
|
685
|
+
done to it.
|
|
686
|
+
|
|
687
|
+
```typescript sim-s3-copy-object
|
|
688
|
+
/**
|
|
689
|
+
* Copying an Object between simulated S3 Buckets.
|
|
690
|
+
*/
|
|
691
|
+
|
|
692
|
+
import {
|
|
693
|
+
CopyObjectCommand,
|
|
694
|
+
CreateBucketCommand,
|
|
695
|
+
DeleteObjectCommand,
|
|
696
|
+
PutObjectCommand,
|
|
697
|
+
} from "@aws-sdk/client-s3";
|
|
698
|
+
import { SimAws } from "@kensio/yulin";
|
|
699
|
+
|
|
700
|
+
const simAws = new SimAws();
|
|
701
|
+
const simS3 = simAws.s3();
|
|
702
|
+
|
|
703
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "inbox-bucket" }));
|
|
704
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "archive-bucket" }));
|
|
705
|
+
|
|
706
|
+
await simS3.putObject(
|
|
707
|
+
new PutObjectCommand({
|
|
708
|
+
Bucket: "inbox-bucket",
|
|
709
|
+
Key: "report.pdf",
|
|
710
|
+
Body: "quarterly figures",
|
|
711
|
+
ContentType: "application/pdf",
|
|
712
|
+
}),
|
|
713
|
+
);
|
|
714
|
+
|
|
715
|
+
const copy = await simS3.copyObject(
|
|
716
|
+
new CopyObjectCommand({
|
|
717
|
+
Bucket: "archive-bucket",
|
|
718
|
+
Key: "2026/report.pdf",
|
|
719
|
+
CopySource: "inbox-bucket/report.pdf",
|
|
720
|
+
}),
|
|
721
|
+
);
|
|
722
|
+
|
|
723
|
+
console.log(copy.CopyObjectResult?.ETag);
|
|
724
|
+
console.log(copy.CopyObjectResult?.LastModified);
|
|
725
|
+
|
|
726
|
+
// The copy carries the source's content type, because MetadataDirective
|
|
727
|
+
// defaults to COPY. Deleting the source turns the copy into a move.
|
|
728
|
+
await simS3.deleteObject(
|
|
729
|
+
new DeleteObjectCommand({ Bucket: "inbox-bucket", Key: "report.pdf" }),
|
|
730
|
+
);
|
|
731
|
+
```
|
|
732
|
+
|
|
733
|
+
A copy authorizes as two decisions. `s3:GetObject` on the source Object and `s3:PutObject` on the
|
|
734
|
+
destination Object, each against its own Bucket policy. A caller holding one and not the other gets
|
|
735
|
+
`AccessDenied`.
|
|
736
|
+
|
|
737
|
+
`MetadataDirective` says where the copy's metadata comes from. The default, `COPY`, carries the
|
|
738
|
+
source's content type, cache control and user metadata across. `REPLACE` takes all of it from the
|
|
739
|
+
request and leaves the source's behind. A copy of an Object onto itself under `REPLACE` is how an
|
|
740
|
+
Object's metadata gets corrected without uploading its bytes again.
|
|
741
|
+
|
|
742
|
+
The destination Bucket raises `s3:ObjectCreated:Copy`, and `s3:ObjectCreated:*` covers it. See
|
|
743
|
+
[Event notifications](#event-notifications).
|
|
744
|
+
|
|
745
|
+
Copying an Object onto itself without `REPLACE` is refused with `InvalidRequest`, as real S3 refuses
|
|
746
|
+
it. The copy would leave the Object exactly as it found it.
|
|
747
|
+
|
|
748
|
+
### Over a served endpoint
|
|
749
|
+
|
|
750
|
+
Real S3 states a copy as a `PUT` on the destination carrying an `x-amz-copy-source` header and an
|
|
751
|
+
empty body. The served endpoint reads that header and runs the operation an in-process caller
|
|
752
|
+
reaches. `aws s3 cp` and `aws s3 mv` between two served Buckets then behave as they do against real
|
|
753
|
+
S3, for a file under the CLI's eight-megabyte multipart threshold.
|
|
754
|
+
|
|
755
|
+
```bash
|
|
756
|
+
aws s3 cp ./report.pdf s3://inbox/report.pdf
|
|
757
|
+
aws s3 mv s3://inbox/report.pdf s3://archive/2026/report.pdf
|
|
758
|
+
aws s3 ls s3://archive/2026/
|
|
759
|
+
```
|
|
760
|
+
|
|
761
|
+
The source is decoded one key segment at a time, the way a key in the request path is, and
|
|
762
|
+
`x-amz-metadata-directive` carries `MetadataDirective`. A finished copy answers with the
|
|
763
|
+
`CopyObjectResult` document holding the ETag and the write time.
|
|
764
|
+
|
|
765
|
+
Real S3 answers a failed copy with `200` and an error document in the body (it has to start sending
|
|
766
|
+
the response while the bytes are still moving). Sim S3 copies in memory and answers with the status
|
|
767
|
+
the error maps to, and an SDK raises it as it raises any other S3 failure.
|
|
768
|
+
|
|
769
|
+
See [Serve simulated S3 on localhost](#serve-simulated-s3-on-localhost) for setting an endpoint up.
|
|
770
|
+
|
|
771
|
+
### Limitations
|
|
772
|
+
|
|
773
|
+
- `UploadPartCopy` is left out. An Object cannot be copied into a multipart upload. A served
|
|
774
|
+
endpoint refuses one with `NotImplemented` rather than storing an empty part. The `aws` CLI
|
|
775
|
+
switches to it above eight megabytes, and a move of a file that size is refused.
|
|
776
|
+
- Both Buckets have to belong to the same simulated S3. A copy across Accounts or Regions is left
|
|
777
|
+
out.
|
|
778
|
+
- A presigned copy is left out, and so is a copy reaching a Bucket through simulated CloudFront.
|
|
779
|
+
- `CopySourceIfMatch`, `CopySourceIfNoneMatch`, `CopySourceIfModifiedSince` and
|
|
780
|
+
`CopySourceIfUnmodifiedSince` are ignored. A conditional copy happens whatever the condition
|
|
781
|
+
says.
|
|
782
|
+
- `TaggingDirective`, `StorageClass`, `ACL` and the server-side encryption members are ignored. Sim
|
|
783
|
+
S3 models none of what they describe.
|
|
784
|
+
- A `versionId` in `CopySource` is refused with `NotImplemented`.
|
|
785
|
+
- A copy of an Object that was uploaded in parts gets a plain ETag rather than the multipart form.
|
|
786
|
+
Real S3 does the same for a copy under five gigabytes, because it rewrites the bytes as one
|
|
787
|
+
part.
|
|
788
|
+
|
|
789
|
+
## Deleting Objects
|
|
790
|
+
|
|
791
|
+
Use `DeleteObjectCommand` to remove one Object, and `DeleteObjectsCommand` to remove several in one
|
|
792
|
+
request. Both are authorized against `s3:DeleteObject` on the Object ARN. A caller allowed to read a
|
|
793
|
+
Bucket cannot empty it.
|
|
794
|
+
|
|
795
|
+
```typescript sim-s3-delete-object
|
|
796
|
+
/**
|
|
797
|
+
* Deleting Objects from a simulated S3 Bucket.
|
|
798
|
+
*/
|
|
799
|
+
|
|
800
|
+
import {
|
|
801
|
+
CreateBucketCommand,
|
|
802
|
+
DeleteObjectCommand,
|
|
803
|
+
DeleteObjectsCommand,
|
|
804
|
+
PutObjectCommand,
|
|
805
|
+
} from "@aws-sdk/client-s3";
|
|
806
|
+
import { SimAws } from "@kensio/yulin";
|
|
807
|
+
|
|
808
|
+
const simAws = new SimAws();
|
|
809
|
+
const simS3 = simAws.s3();
|
|
810
|
+
|
|
811
|
+
await simS3.createBucket(
|
|
812
|
+
new CreateBucketCommand({
|
|
813
|
+
Bucket: "uploads-bucket",
|
|
814
|
+
}),
|
|
815
|
+
);
|
|
816
|
+
|
|
817
|
+
for (const key of ["receipt.pdf", "invoice.pdf", "notes.txt"]) {
|
|
818
|
+
await simS3.putObject(
|
|
819
|
+
new PutObjectCommand({
|
|
820
|
+
Bucket: "uploads-bucket",
|
|
821
|
+
Key: key,
|
|
822
|
+
Body: "file contents",
|
|
823
|
+
}),
|
|
824
|
+
);
|
|
825
|
+
}
|
|
826
|
+
|
|
827
|
+
await simS3.deleteObject(
|
|
828
|
+
new DeleteObjectCommand({
|
|
829
|
+
Bucket: "uploads-bucket",
|
|
830
|
+
Key: "receipt.pdf",
|
|
831
|
+
}),
|
|
832
|
+
);
|
|
833
|
+
|
|
834
|
+
const batchOutput = await simS3.deleteObjects(
|
|
835
|
+
new DeleteObjectsCommand({
|
|
836
|
+
Bucket: "uploads-bucket",
|
|
837
|
+
Delete: {
|
|
838
|
+
Objects: [{ Key: "invoice.pdf" }, { Key: "notes.txt" }],
|
|
839
|
+
},
|
|
840
|
+
}),
|
|
841
|
+
);
|
|
842
|
+
|
|
843
|
+
const removedObjects = batchOutput.Deleted ?? [];
|
|
844
|
+
for (const removed of removedObjects) {
|
|
845
|
+
console.log(removed.Key);
|
|
846
|
+
}
|
|
847
|
+
|
|
848
|
+
const refusedObjects = batchOutput.Errors ?? [];
|
|
849
|
+
for (const refused of refusedObjects) {
|
|
850
|
+
console.log(refused.Key, refused.Code);
|
|
851
|
+
}
|
|
852
|
+
```
|
|
853
|
+
|
|
854
|
+
Deletion is idempotent, as it is in real S3. Deleting a key the Bucket never held succeeds, and
|
|
855
|
+
`DeleteObjects` reports it among the keys it deleted. Deleting from a Bucket that was never created
|
|
856
|
+
raises `NoSuchBucket`.
|
|
857
|
+
|
|
858
|
+
`DeleteObjects` authorizes each key on its own and carries on through the batch. A key the caller may
|
|
859
|
+
not delete appears in `Errors` with the code `AccessDenied`, while the rest are still removed and
|
|
860
|
+
reported in `Deleted`. Setting `Quiet: true` leaves `Deleted` out of the response, so only the
|
|
861
|
+
failures come back.
|
|
862
|
+
|
|
863
|
+
### Limitations
|
|
864
|
+
|
|
865
|
+
- Object versioning is left out. Deletion removes the Object rather than writing a delete marker,
|
|
866
|
+
and `VersionId` and `MFA` are both ignored on the request.
|
|
867
|
+
- A request naming no Objects, or more than the thousand S3 accepts, is refused with `MalformedXML`
|
|
868
|
+
before anything is deleted.
|
|
869
|
+
- A Bucket using filesystem-backed storage refuses deletion. See
|
|
870
|
+
[Filesystem-backed Bucket storage](#filesystem-backed-bucket-storage).
|
|
871
|
+
|
|
872
|
+
## Event notifications
|
|
873
|
+
|
|
874
|
+
A simulated S3 Bucket can notify a simulated Lambda function, a simulated SQS queue or a simulated
|
|
875
|
+
SNS topic when an Object is created or removed. The configuration is applied with
|
|
876
|
+
`PutBucketNotificationConfigurationCommand` and read back with
|
|
877
|
+
`GetBucketNotificationConfigurationCommand`.
|
|
878
|
+
|
|
879
|
+
The destination's own policy decides whether S3 may reach it. That is the function's resource
|
|
880
|
+
policy, the queue's `Policy` attribute, or the topic's. It is checked when the configuration is
|
|
881
|
+
applied, and again for every event, as real S3 does.
|
|
882
|
+
|
|
883
|
+
```typescript sim-s3-event-notifications
|
|
884
|
+
/**
|
|
885
|
+
* Notifying a simulated Lambda function when an Object is created.
|
|
886
|
+
*/
|
|
887
|
+
|
|
888
|
+
import {
|
|
889
|
+
AddPermissionCommand,
|
|
890
|
+
CreateFunctionCommand,
|
|
891
|
+
} from "@aws-sdk/client-lambda";
|
|
892
|
+
import {
|
|
893
|
+
CreateBucketCommand,
|
|
894
|
+
PutBucketNotificationConfigurationCommand,
|
|
895
|
+
PutObjectCommand,
|
|
896
|
+
} from "@aws-sdk/client-s3";
|
|
897
|
+
import { SimAws } from "@kensio/yulin";
|
|
898
|
+
import { makeLambdaZipFileInput } from "@kensio/yulin/lambda";
|
|
899
|
+
|
|
900
|
+
interface S3EventDocument {
|
|
901
|
+
Records: [{ eventName: string; s3: { object: { key: string } } }];
|
|
902
|
+
}
|
|
903
|
+
|
|
904
|
+
const simAws = new SimAws();
|
|
905
|
+
const thumbnailerArn = `arn:aws:lambda:${simAws.defaultRegionName}:${simAws.defaultAccountId}:function:thumbnailer`;
|
|
906
|
+
|
|
907
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "uploads" }));
|
|
908
|
+
|
|
909
|
+
await simAws.lambda().createFunction(
|
|
910
|
+
new CreateFunctionCommand({
|
|
911
|
+
FunctionName: "thumbnailer",
|
|
912
|
+
Role: `arn:aws:iam::${simAws.defaultAccountId}:role/ThumbnailerRole`,
|
|
913
|
+
Code: {
|
|
914
|
+
ZipFile: makeLambdaZipFileInput((event: S3EventDocument) => {
|
|
915
|
+
console.log(event.Records[0].eventName, event.Records[0].s3.object.key);
|
|
916
|
+
|
|
917
|
+
return "thumbnailed";
|
|
918
|
+
}),
|
|
919
|
+
},
|
|
920
|
+
}),
|
|
921
|
+
);
|
|
922
|
+
|
|
923
|
+
await simAws.lambda().addPermission(
|
|
924
|
+
new AddPermissionCommand({
|
|
925
|
+
FunctionName: "thumbnailer",
|
|
926
|
+
StatementId: "AllowS3",
|
|
927
|
+
Action: "lambda:InvokeFunction",
|
|
928
|
+
Principal: "s3.amazonaws.com",
|
|
929
|
+
SourceArn: "arn:aws:s3:::uploads",
|
|
930
|
+
SourceAccount: simAws.defaultAccountId,
|
|
931
|
+
}),
|
|
932
|
+
);
|
|
933
|
+
|
|
934
|
+
await simAws.s3().putBucketNotificationConfiguration(
|
|
935
|
+
new PutBucketNotificationConfigurationCommand({
|
|
936
|
+
Bucket: "uploads",
|
|
937
|
+
NotificationConfiguration: {
|
|
938
|
+
LambdaFunctionConfigurations: [
|
|
939
|
+
{
|
|
940
|
+
Id: "thumbnail-raw-uploads",
|
|
941
|
+
Events: ["s3:ObjectCreated:*"],
|
|
942
|
+
LambdaFunctionArn: thumbnailerArn,
|
|
943
|
+
Filter: { Key: { FilterRules: [{ Name: "prefix", Value: "raw/" }] } },
|
|
944
|
+
},
|
|
945
|
+
],
|
|
946
|
+
},
|
|
947
|
+
}),
|
|
948
|
+
);
|
|
949
|
+
|
|
950
|
+
await simAws.s3().putObject(
|
|
951
|
+
new PutObjectCommand({
|
|
952
|
+
Bucket: "uploads",
|
|
953
|
+
Key: "raw/cat.jpg",
|
|
954
|
+
Body: "cat picture",
|
|
955
|
+
}),
|
|
956
|
+
);
|
|
957
|
+
|
|
958
|
+
// Delivery happens in the background, so wait for the simulation to settle.
|
|
959
|
+
await simAws.backgroundTasksComplete();
|
|
960
|
+
```
|
|
961
|
+
|
|
962
|
+
The event types a configuration can name are `s3:ObjectCreated:*`, `s3:ObjectCreated:Put`,
|
|
963
|
+
`s3:ObjectRemoved:*` and `s3:ObjectRemoved:Delete`. Any other S3 event type is refused by name rather
|
|
964
|
+
than stored and never raised.
|
|
965
|
+
|
|
966
|
+
A configuration can filter on an object key prefix, a suffix, or both. Two configurations that share
|
|
967
|
+
an event type and whose filters could both match the same key are refused with `InvalidArgument`, as
|
|
968
|
+
real S3 refuses them. Overlapping prefixes are fine when the suffixes do not overlap, so one function
|
|
969
|
+
can take the `.jpg` files under a prefix while another takes the `.png` files under the same one.
|
|
970
|
+
The rule applies across the destination groups. A function and a queue that both want the same
|
|
971
|
+
event are refused as readily as two functions.
|
|
972
|
+
|
|
973
|
+
`PutBucketNotificationConfigurationCommand` replaces the whole configuration rather than adding to
|
|
974
|
+
it. `GetBucketNotificationConfigurationCommand` answers an empty configuration for a Bucket that has
|
|
975
|
+
none. Note that the response carries the destination groups at the top level, while the request nests
|
|
976
|
+
them under `NotificationConfiguration`:
|
|
977
|
+
|
|
978
|
+
```typescript
|
|
979
|
+
const read = await simAws
|
|
980
|
+
.s3()
|
|
981
|
+
.getBucketNotificationConfiguration(
|
|
982
|
+
new GetBucketNotificationConfigurationCommand({ Bucket: "uploads" }),
|
|
983
|
+
);
|
|
984
|
+
const configurations = read.LambdaFunctionConfigurations ?? [];
|
|
985
|
+
```
|
|
986
|
+
|
|
987
|
+
The two commands are authorized as `s3:PutBucketNotification` and `s3:GetBucketNotification`. Those
|
|
988
|
+
are the real IAM action names, and they do not match the API names.
|
|
989
|
+
|
|
990
|
+
### To a Lambda version or alias
|
|
991
|
+
|
|
992
|
+
A `LambdaFunctionArn` can carry a version number or an alias name on the end, and the events go to
|
|
993
|
+
the version that qualifier names. The permission it needs is one made on the same qualifier, which
|
|
994
|
+
`AddPermission` takes as a `Qualifier`:
|
|
995
|
+
|
|
996
|
+
```typescript sim-s3-notification-lambda-alias
|
|
997
|
+
/**
|
|
998
|
+
* Notifying a simulated Lambda alias, which runs the version it points at.
|
|
999
|
+
*/
|
|
1000
|
+
|
|
1001
|
+
import {
|
|
1002
|
+
AddPermissionCommand,
|
|
1003
|
+
CreateAliasCommand,
|
|
1004
|
+
CreateFunctionCommand,
|
|
1005
|
+
PublishVersionCommand,
|
|
1006
|
+
} from "@aws-sdk/client-lambda";
|
|
1007
|
+
import {
|
|
1008
|
+
CreateBucketCommand,
|
|
1009
|
+
PutBucketNotificationConfigurationCommand,
|
|
1010
|
+
PutObjectCommand,
|
|
1011
|
+
} from "@aws-sdk/client-s3";
|
|
1012
|
+
import { SimAws } from "@kensio/yulin";
|
|
1013
|
+
import { makeLambdaZipFileInput } from "@kensio/yulin/lambda";
|
|
1014
|
+
|
|
1015
|
+
const simAws = new SimAws();
|
|
1016
|
+
const lambda = simAws.lambda();
|
|
1017
|
+
const thumbnailerArn = `arn:aws:lambda:${simAws.defaultRegionName}:${simAws.defaultAccountId}:function:thumbnailer`;
|
|
1018
|
+
|
|
1019
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "uploads" }));
|
|
1020
|
+
|
|
1021
|
+
await lambda.createFunction(
|
|
1022
|
+
new CreateFunctionCommand({
|
|
1023
|
+
FunctionName: "thumbnailer",
|
|
1024
|
+
Role: `arn:aws:iam::${simAws.defaultAccountId}:role/ThumbnailerRole`,
|
|
1025
|
+
Code: {
|
|
1026
|
+
ZipFile: makeLambdaZipFileInput((_event, context) => {
|
|
1027
|
+
console.log(context.functionVersion); // "1", the version behind `live`
|
|
1028
|
+
|
|
1029
|
+
return "thumbnailed";
|
|
1030
|
+
}),
|
|
1031
|
+
},
|
|
1032
|
+
}),
|
|
1033
|
+
);
|
|
1034
|
+
|
|
1035
|
+
const published = await lambda.publishVersion(
|
|
1036
|
+
new PublishVersionCommand({ FunctionName: "thumbnailer" }),
|
|
1037
|
+
);
|
|
1038
|
+
|
|
1039
|
+
await lambda.createAlias(
|
|
1040
|
+
new CreateAliasCommand({
|
|
1041
|
+
FunctionName: "thumbnailer",
|
|
1042
|
+
Name: "live",
|
|
1043
|
+
FunctionVersion: published.Version,
|
|
1044
|
+
}),
|
|
1045
|
+
);
|
|
1046
|
+
|
|
1047
|
+
// The grant is made on the alias, which is the resource the notification names.
|
|
1048
|
+
await lambda.addPermission(
|
|
1049
|
+
new AddPermissionCommand({
|
|
1050
|
+
FunctionName: "thumbnailer",
|
|
1051
|
+
Qualifier: "live",
|
|
1052
|
+
StatementId: "AllowS3",
|
|
1053
|
+
Action: "lambda:InvokeFunction",
|
|
1054
|
+
Principal: "s3.amazonaws.com",
|
|
1055
|
+
SourceArn: "arn:aws:s3:::uploads",
|
|
1056
|
+
SourceAccount: simAws.defaultAccountId,
|
|
1057
|
+
}),
|
|
1058
|
+
);
|
|
1059
|
+
|
|
1060
|
+
await simAws.s3().putBucketNotificationConfiguration(
|
|
1061
|
+
new PutBucketNotificationConfigurationCommand({
|
|
1062
|
+
Bucket: "uploads",
|
|
1063
|
+
NotificationConfiguration: {
|
|
1064
|
+
LambdaFunctionConfigurations: [
|
|
1065
|
+
{
|
|
1066
|
+
Id: "thumbnail-uploads",
|
|
1067
|
+
Events: ["s3:ObjectCreated:*"],
|
|
1068
|
+
LambdaFunctionArn: `${thumbnailerArn}:live`,
|
|
1069
|
+
},
|
|
1070
|
+
],
|
|
1071
|
+
},
|
|
1072
|
+
}),
|
|
1073
|
+
);
|
|
1074
|
+
|
|
1075
|
+
await simAws
|
|
1076
|
+
.s3()
|
|
1077
|
+
.putObject(
|
|
1078
|
+
new PutObjectCommand({ Bucket: "uploads", Key: "cat.jpg", Body: "cat" }),
|
|
1079
|
+
);
|
|
1080
|
+
await simAws.backgroundTasksComplete();
|
|
1081
|
+
```
|
|
1082
|
+
|
|
1083
|
+
`UpdateAlias` moves what the notification reaches, and the configuration stays as it is. A qualifier
|
|
1084
|
+
naming no version and no alias is refused where the configuration is applied, the way a missing
|
|
1085
|
+
function is.
|
|
1086
|
+
|
|
1087
|
+
### To an SQS queue
|
|
1088
|
+
|
|
1089
|
+
A `QueueConfigurations` entry names a queue by ARN. The whole `Records` document arrives as one
|
|
1090
|
+
message body, and a consumer parses `record.body` to get at the event. Put a Lambda event source
|
|
1091
|
+
mapping on the queue and the chain runs end to end after one `backgroundTasksComplete()`.
|
|
1092
|
+
|
|
1093
|
+
The queue's `Policy` attribute has to allow `sqs:SendMessage` for the `s3.amazonaws.com` service
|
|
1094
|
+
principal. S3 supplies `aws:SourceArn` and `aws:SourceAccount`. The `ArnLike` condition CDK's
|
|
1095
|
+
`SqsDestination` writes and the `StringEquals aws:SourceAccount` guard AWS documents are both
|
|
1096
|
+
satisfied.
|
|
1097
|
+
|
|
1098
|
+
```typescript sim-s3-sqs-notification
|
|
1099
|
+
/**
|
|
1100
|
+
* An Object event reaching a Lambda function through an SQS queue.
|
|
1101
|
+
*/
|
|
1102
|
+
|
|
1103
|
+
import { CreateRoleCommand, PutRolePolicyCommand } from "@aws-sdk/client-iam";
|
|
1104
|
+
import {
|
|
1105
|
+
CreateEventSourceMappingCommand,
|
|
1106
|
+
CreateFunctionCommand,
|
|
1107
|
+
} from "@aws-sdk/client-lambda";
|
|
1108
|
+
import {
|
|
1109
|
+
CreateBucketCommand,
|
|
1110
|
+
PutBucketNotificationConfigurationCommand,
|
|
1111
|
+
PutObjectCommand,
|
|
1112
|
+
} from "@aws-sdk/client-s3";
|
|
1113
|
+
import {
|
|
1114
|
+
CreateQueueCommand,
|
|
1115
|
+
SetQueueAttributesCommand,
|
|
1116
|
+
} from "@aws-sdk/client-sqs";
|
|
1117
|
+
import { SimAws } from "@kensio/yulin";
|
|
1118
|
+
import {
|
|
1119
|
+
makeLambdaZipFileInput,
|
|
1120
|
+
type SimLambdaSqsEvent,
|
|
1121
|
+
} from "@kensio/yulin/lambda";
|
|
1122
|
+
|
|
1123
|
+
interface S3EventDocument {
|
|
1124
|
+
Records: [{ eventName: string; s3: { object: { key: string } } }];
|
|
1125
|
+
}
|
|
1126
|
+
|
|
1127
|
+
const simAws = new SimAws();
|
|
1128
|
+
const queueArn = `arn:aws:sqs:${simAws.defaultRegionName}:${simAws.defaultAccountId}:uploads`;
|
|
1129
|
+
|
|
1130
|
+
const { QueueUrl } = await simAws
|
|
1131
|
+
.sqs()
|
|
1132
|
+
.createQueue(new CreateQueueCommand({ QueueName: "uploads" }));
|
|
1133
|
+
|
|
1134
|
+
// The queue policy is the whole of what admits S3, which owns no identity
|
|
1135
|
+
// policies anywhere.
|
|
1136
|
+
await simAws.sqs().setQueueAttributes(
|
|
1137
|
+
new SetQueueAttributesCommand({
|
|
1138
|
+
QueueUrl,
|
|
1139
|
+
Attributes: {
|
|
1140
|
+
Policy: JSON.stringify({
|
|
1141
|
+
Version: "2012-10-17",
|
|
1142
|
+
Statement: {
|
|
1143
|
+
Effect: "Allow",
|
|
1144
|
+
Principal: { Service: "s3.amazonaws.com" },
|
|
1145
|
+
Action: "sqs:SendMessage",
|
|
1146
|
+
Resource: queueArn,
|
|
1147
|
+
Condition: { ArnLike: { "aws:SourceArn": "arn:aws:s3:::uploads" } },
|
|
1148
|
+
},
|
|
1149
|
+
}),
|
|
1150
|
+
},
|
|
1151
|
+
}),
|
|
1152
|
+
);
|
|
1153
|
+
|
|
1154
|
+
const role = await simAws.iam().createRole(
|
|
1155
|
+
new CreateRoleCommand({
|
|
1156
|
+
RoleName: "UploadConsumerRole",
|
|
1157
|
+
AssumeRolePolicyDocument: JSON.stringify({
|
|
1158
|
+
Version: "2012-10-17",
|
|
1159
|
+
Statement: {
|
|
1160
|
+
Effect: "Allow",
|
|
1161
|
+
Principal: { Service: "lambda.amazonaws.com" },
|
|
1162
|
+
Action: "sts:AssumeRole",
|
|
1163
|
+
},
|
|
1164
|
+
}),
|
|
1165
|
+
}),
|
|
1166
|
+
);
|
|
1167
|
+
|
|
1168
|
+
await simAws.iam().putRolePolicy(
|
|
1169
|
+
new PutRolePolicyCommand({
|
|
1170
|
+
RoleName: "UploadConsumerRole",
|
|
1171
|
+
PolicyName: "ConsumeUploads",
|
|
1172
|
+
PolicyDocument: JSON.stringify({
|
|
1173
|
+
Version: "2012-10-17",
|
|
1174
|
+
Statement: {
|
|
1175
|
+
Effect: "Allow",
|
|
1176
|
+
Action: [
|
|
1177
|
+
"sqs:ReceiveMessage",
|
|
1178
|
+
"sqs:DeleteMessage",
|
|
1179
|
+
"sqs:GetQueueAttributes",
|
|
1180
|
+
],
|
|
1181
|
+
Resource: queueArn,
|
|
1182
|
+
},
|
|
1183
|
+
}),
|
|
1184
|
+
}),
|
|
1185
|
+
);
|
|
1186
|
+
|
|
1187
|
+
await simAws.lambda().createFunction(
|
|
1188
|
+
new CreateFunctionCommand({
|
|
1189
|
+
FunctionName: "upload-consumer",
|
|
1190
|
+
Role: role.Role.Arn,
|
|
1191
|
+
Code: {
|
|
1192
|
+
ZipFile: makeLambdaZipFileInput((event: SimLambdaSqsEvent) => {
|
|
1193
|
+
for (const record of event.Records) {
|
|
1194
|
+
// The S3 event document is the SQS message body, so it is parsed
|
|
1195
|
+
// out of the record rather than being the event itself.
|
|
1196
|
+
const document = JSON.parse(record.body) as S3EventDocument;
|
|
1197
|
+
|
|
1198
|
+
console.log(document.Records[0].s3.object.key); // "raw/cat.jpg"
|
|
1199
|
+
}
|
|
1200
|
+
}),
|
|
1201
|
+
},
|
|
1202
|
+
}),
|
|
1203
|
+
);
|
|
1204
|
+
|
|
1205
|
+
await simAws.lambda().createEventSourceMapping(
|
|
1206
|
+
new CreateEventSourceMappingCommand({
|
|
1207
|
+
EventSourceArn: queueArn,
|
|
1208
|
+
FunctionName: "upload-consumer",
|
|
1209
|
+
}),
|
|
1210
|
+
);
|
|
1211
|
+
|
|
1212
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "uploads" }));
|
|
1213
|
+
|
|
1214
|
+
await simAws.s3().putBucketNotificationConfiguration(
|
|
1215
|
+
new PutBucketNotificationConfigurationCommand({
|
|
1216
|
+
Bucket: "uploads",
|
|
1217
|
+
NotificationConfiguration: {
|
|
1218
|
+
QueueConfigurations: [
|
|
1219
|
+
{
|
|
1220
|
+
Id: "raw-uploads",
|
|
1221
|
+
Events: ["s3:ObjectCreated:*"],
|
|
1222
|
+
QueueArn: queueArn,
|
|
1223
|
+
},
|
|
1224
|
+
],
|
|
1225
|
+
},
|
|
1226
|
+
}),
|
|
1227
|
+
);
|
|
1228
|
+
|
|
1229
|
+
await simAws.s3().putObject(
|
|
1230
|
+
new PutObjectCommand({
|
|
1231
|
+
Bucket: "uploads",
|
|
1232
|
+
Key: "raw/cat.jpg",
|
|
1233
|
+
Body: "cat picture",
|
|
1234
|
+
}),
|
|
1235
|
+
);
|
|
1236
|
+
|
|
1237
|
+
// One wait covers the delivery to the queue and the poll that follows it.
|
|
1238
|
+
await simAws.backgroundTasksComplete();
|
|
1239
|
+
```
|
|
1240
|
+
|
|
1241
|
+
The queue has to be in the Bucket's Region, as real S3 requires. It can be in another Account, since
|
|
1242
|
+
its own policy and its own Account's IAM are what admit the Bucket. A FIFO queue is refused by name.
|
|
1243
|
+
|
|
1244
|
+
### To an SNS topic
|
|
1245
|
+
|
|
1246
|
+
A `TopicConfigurations` entry names a topic by ARN. The whole `Records` document is published as the
|
|
1247
|
+
SNS `Message`, with a `Subject` of `Amazon S3 Notification`, as real S3 publishes it. A queue
|
|
1248
|
+
subscribed to the topic therefore has two envelopes to reach through. Parse the message body for the
|
|
1249
|
+
SNS envelope, then parse its `Message` for the S3 event.
|
|
1250
|
+
|
|
1251
|
+
The topic's `Policy` attribute has to allow `sns:Publish` for the `s3.amazonaws.com` service
|
|
1252
|
+
principal. S3 supplies `aws:SourceArn` and `aws:SourceAccount`. The `ArnLike` condition CDK's
|
|
1253
|
+
`SnsDestination` writes and the `StringEquals aws:SourceAccount` guard AWS documents are both
|
|
1254
|
+
satisfied.
|
|
1255
|
+
|
|
1256
|
+
```typescript sim-s3-sns-notification
|
|
1257
|
+
/**
|
|
1258
|
+
* An Object event reaching a queue through an SNS topic.
|
|
1259
|
+
*/
|
|
1260
|
+
|
|
1261
|
+
import {
|
|
1262
|
+
CreateBucketCommand,
|
|
1263
|
+
PutBucketNotificationConfigurationCommand,
|
|
1264
|
+
PutObjectCommand,
|
|
1265
|
+
} from "@aws-sdk/client-s3";
|
|
1266
|
+
import {
|
|
1267
|
+
CreateTopicCommand,
|
|
1268
|
+
SetTopicAttributesCommand,
|
|
1269
|
+
SubscribeCommand,
|
|
1270
|
+
} from "@aws-sdk/client-sns";
|
|
1271
|
+
import {
|
|
1272
|
+
CreateQueueCommand,
|
|
1273
|
+
ReceiveMessageCommand,
|
|
1274
|
+
SetQueueAttributesCommand,
|
|
1275
|
+
} from "@aws-sdk/client-sqs";
|
|
1276
|
+
import { SimAws } from "@kensio/yulin";
|
|
1277
|
+
|
|
1278
|
+
interface SnsEnvelope {
|
|
1279
|
+
Subject: string;
|
|
1280
|
+
Message: string;
|
|
1281
|
+
}
|
|
1282
|
+
|
|
1283
|
+
interface S3EventDocument {
|
|
1284
|
+
Records: [{ eventName: string; s3: { object: { key: string } } }];
|
|
1285
|
+
}
|
|
1286
|
+
|
|
1287
|
+
const simAws = new SimAws();
|
|
1288
|
+
const { defaultRegionName: region, defaultAccountId: account } = simAws;
|
|
1289
|
+
const bucketArn = "arn:aws:s3:::uploads";
|
|
1290
|
+
const topicArn = `arn:aws:sns:${region}:${account}:uploads`;
|
|
1291
|
+
const queueArn = `arn:aws:sqs:${region}:${account}:uploads-queue`;
|
|
1292
|
+
|
|
1293
|
+
const { TopicArn } = await simAws
|
|
1294
|
+
.sns()
|
|
1295
|
+
.createTopic(new CreateTopicCommand({ Name: "uploads" }));
|
|
1296
|
+
|
|
1297
|
+
// The topic policy is the whole decision, because S3 owns no identity
|
|
1298
|
+
// policies. S3 supplies aws:SourceArn, so the grant names one Bucket.
|
|
1299
|
+
await simAws.sns().setTopicAttributes(
|
|
1300
|
+
new SetTopicAttributesCommand({
|
|
1301
|
+
TopicArn,
|
|
1302
|
+
AttributeName: "Policy",
|
|
1303
|
+
AttributeValue: JSON.stringify({
|
|
1304
|
+
Version: "2012-10-17",
|
|
1305
|
+
Statement: [
|
|
1306
|
+
{
|
|
1307
|
+
Effect: "Allow",
|
|
1308
|
+
Principal: { Service: "s3.amazonaws.com" },
|
|
1309
|
+
Action: "sns:Publish",
|
|
1310
|
+
Resource: topicArn,
|
|
1311
|
+
Condition: { ArnLike: { "aws:SourceArn": bucketArn } },
|
|
1312
|
+
},
|
|
1313
|
+
],
|
|
1314
|
+
}),
|
|
1315
|
+
}),
|
|
1316
|
+
);
|
|
1317
|
+
|
|
1318
|
+
const { QueueUrl } = await simAws
|
|
1319
|
+
.sqs()
|
|
1320
|
+
.createQueue(new CreateQueueCommand({ QueueName: "uploads-queue" }));
|
|
1321
|
+
|
|
1322
|
+
await simAws.sqs().setQueueAttributes(
|
|
1323
|
+
new SetQueueAttributesCommand({
|
|
1324
|
+
QueueUrl,
|
|
1325
|
+
Attributes: {
|
|
1326
|
+
Policy: JSON.stringify({
|
|
1327
|
+
Version: "2012-10-17",
|
|
1328
|
+
Statement: [
|
|
1329
|
+
{
|
|
1330
|
+
Effect: "Allow",
|
|
1331
|
+
Principal: { Service: "sns.amazonaws.com" },
|
|
1332
|
+
Action: "sqs:SendMessage",
|
|
1333
|
+
Resource: queueArn,
|
|
1334
|
+
Condition: { ArnLike: { "aws:SourceArn": topicArn } },
|
|
1335
|
+
},
|
|
1336
|
+
],
|
|
1337
|
+
}),
|
|
1338
|
+
},
|
|
1339
|
+
}),
|
|
1340
|
+
);
|
|
1341
|
+
|
|
1342
|
+
await simAws
|
|
1343
|
+
.sns()
|
|
1344
|
+
.subscribe(
|
|
1345
|
+
new SubscribeCommand({ TopicArn, Protocol: "sqs", Endpoint: queueArn }),
|
|
1346
|
+
);
|
|
1347
|
+
|
|
1348
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "uploads" }));
|
|
1349
|
+
|
|
1350
|
+
await simAws.s3().putBucketNotificationConfiguration(
|
|
1351
|
+
new PutBucketNotificationConfigurationCommand({
|
|
1352
|
+
Bucket: "uploads",
|
|
1353
|
+
NotificationConfiguration: {
|
|
1354
|
+
TopicConfigurations: [
|
|
1355
|
+
{
|
|
1356
|
+
Id: "raw-uploads",
|
|
1357
|
+
Events: ["s3:ObjectCreated:*"],
|
|
1358
|
+
TopicArn,
|
|
1359
|
+
Filter: { Key: { FilterRules: [{ Name: "prefix", Value: "raw/" }] } },
|
|
1360
|
+
},
|
|
1361
|
+
],
|
|
1362
|
+
},
|
|
1363
|
+
}),
|
|
1364
|
+
);
|
|
1365
|
+
|
|
1366
|
+
await simAws.s3().putObject(
|
|
1367
|
+
new PutObjectCommand({
|
|
1368
|
+
Bucket: "uploads",
|
|
1369
|
+
Key: "raw/cat.jpg",
|
|
1370
|
+
Body: "cat picture",
|
|
1371
|
+
}),
|
|
1372
|
+
);
|
|
1373
|
+
|
|
1374
|
+
// One wait covers the publish to the topic and the delivery to the queue.
|
|
1375
|
+
await simAws.backgroundTasksComplete();
|
|
1376
|
+
|
|
1377
|
+
const received = await simAws
|
|
1378
|
+
.sqs()
|
|
1379
|
+
.receiveMessage(new ReceiveMessageCommand({ QueueUrl }));
|
|
1380
|
+
|
|
1381
|
+
// Two envelopes to reach through: the SNS envelope, then the S3 event.
|
|
1382
|
+
const envelope = JSON.parse(received.Messages?.[0]?.Body ?? "") as SnsEnvelope;
|
|
1383
|
+
|
|
1384
|
+
console.log(envelope.Subject); // "Amazon S3 Notification"
|
|
1385
|
+
|
|
1386
|
+
const event = JSON.parse(envelope.Message) as S3EventDocument;
|
|
1387
|
+
|
|
1388
|
+
console.log(event.Records[0].s3.object.key); // "raw/cat.jpg"
|
|
1389
|
+
```
|
|
1390
|
+
|
|
1391
|
+
The topic has to be in the Bucket's Region, as real S3 requires. It can be in another Account, since
|
|
1392
|
+
its own policy and its own Account's IAM are what admit the Bucket. A FIFO topic is refused by name.
|
|
1393
|
+
|
|
1394
|
+
The publish goes through the ordinary `Publish` path, and the topic's own subscriptions take it from
|
|
1395
|
+
there. That means a topic destination reaches everything the topic reaches, and a subscribed queue is
|
|
1396
|
+
two hops from the Object that was written. One `backgroundTasksComplete()` covers both.
|
|
1397
|
+
|
|
1398
|
+
### From a CloudFormation template
|
|
1399
|
+
|
|
1400
|
+
The `NotificationConfiguration` property of `AWS::S3::Bucket` deploys through the same
|
|
1401
|
+
`PutBucketNotificationConfiguration` path, and a template and an SDK caller get identical validation.
|
|
1402
|
+
CloudFormation names the same configuration differently in several places. It writes
|
|
1403
|
+
`LambdaConfigurations` where the SDK writes `LambdaFunctionConfigurations`, a single `Event` string
|
|
1404
|
+
where the SDK takes an `Events` list, `Function` for `LambdaFunctionArn`, `Queue` for `QueueArn`,
|
|
1405
|
+
`Topic` for `TopicArn`, and `Filter.S3Key.Rules` for `Filter.Key.FilterRules`. `QueueConfigurations`
|
|
1406
|
+
and `TopicConfigurations` are the names both spell the same way. Yulin reads the CloudFormation names
|
|
1407
|
+
and refuses the others, so a template using the SDK spelling fails the stack. An unfiltered
|
|
1408
|
+
configuration would deploy otherwise.
|
|
1409
|
+
|
|
1410
|
+
```typescript sim-s3-cfn-event-notification
|
|
1411
|
+
/**
|
|
1412
|
+
* Configuring Bucket event notifications from a CloudFormation template.
|
|
1413
|
+
*/
|
|
1414
|
+
|
|
1415
|
+
import { PutObjectCommand } from "@aws-sdk/client-s3";
|
|
1416
|
+
import { SimAws } from "@kensio/yulin";
|
|
1417
|
+
|
|
1418
|
+
const simAws = new SimAws();
|
|
1419
|
+
|
|
1420
|
+
const stack = await simAws.cloudFormation().deployTemplate({
|
|
1421
|
+
stackName: "uploads-stack",
|
|
1422
|
+
template: {
|
|
1423
|
+
Resources: {
|
|
1424
|
+
Thumbnailer: {
|
|
1425
|
+
Type: "AWS::Lambda::Function",
|
|
1426
|
+
Properties: {
|
|
1427
|
+
FunctionName: "thumbnailer",
|
|
1428
|
+
Role: { "Fn::GetAtt": ["ThumbnailerRole", "Arn"] },
|
|
1429
|
+
Handler: "index.handler",
|
|
1430
|
+
Runtime: "nodejs20.x",
|
|
1431
|
+
Code: { ZipFile: "exports.handler = async () => 'thumbnailed';" },
|
|
1432
|
+
},
|
|
1433
|
+
},
|
|
1434
|
+
ThumbnailerRole: {
|
|
1435
|
+
Type: "AWS::IAM::Role",
|
|
1436
|
+
Properties: {
|
|
1437
|
+
RoleName: "thumbnailer-role",
|
|
1438
|
+
AssumeRolePolicyDocument: {
|
|
1439
|
+
Version: "2012-10-17",
|
|
1440
|
+
Statement: [
|
|
1441
|
+
{
|
|
1442
|
+
Effect: "Allow",
|
|
1443
|
+
Principal: { Service: "lambda.amazonaws.com" },
|
|
1444
|
+
Action: "sts:AssumeRole",
|
|
1445
|
+
},
|
|
1446
|
+
],
|
|
1447
|
+
},
|
|
1448
|
+
},
|
|
1449
|
+
},
|
|
1450
|
+
ThumbnailerPermission: {
|
|
1451
|
+
Type: "AWS::Lambda::Permission",
|
|
1452
|
+
Properties: {
|
|
1453
|
+
Action: "lambda:InvokeFunction",
|
|
1454
|
+
FunctionName: { "Fn::GetAtt": ["Thumbnailer", "Arn"] },
|
|
1455
|
+
Principal: "s3.amazonaws.com",
|
|
1456
|
+
SourceAccount: { Ref: "AWS::AccountId" },
|
|
1457
|
+
SourceArn: "arn:aws:s3:::uploads",
|
|
1458
|
+
},
|
|
1459
|
+
},
|
|
1460
|
+
UploadsBucket: {
|
|
1461
|
+
Type: "AWS::S3::Bucket",
|
|
1462
|
+
DependsOn: ["ThumbnailerPermission"],
|
|
1463
|
+
Properties: {
|
|
1464
|
+
BucketName: "uploads",
|
|
1465
|
+
NotificationConfiguration: {
|
|
1466
|
+
LambdaConfigurations: [
|
|
1467
|
+
{
|
|
1468
|
+
Event: "s3:ObjectCreated:*",
|
|
1469
|
+
Function: { "Fn::GetAtt": ["Thumbnailer", "Arn"] },
|
|
1470
|
+
Filter: {
|
|
1471
|
+
S3Key: { Rules: [{ Name: "prefix", Value: "raw/" }] },
|
|
1472
|
+
},
|
|
1473
|
+
},
|
|
1474
|
+
],
|
|
1475
|
+
},
|
|
1476
|
+
},
|
|
1477
|
+
},
|
|
1478
|
+
},
|
|
1479
|
+
},
|
|
1480
|
+
});
|
|
1481
|
+
await stack.waitForDeployComplete();
|
|
1482
|
+
|
|
1483
|
+
await simAws.s3().putObject(
|
|
1484
|
+
new PutObjectCommand({
|
|
1485
|
+
Bucket: "uploads",
|
|
1486
|
+
Key: "raw/cat.jpg",
|
|
1487
|
+
Body: "cat picture",
|
|
1488
|
+
}),
|
|
1489
|
+
);
|
|
1490
|
+
|
|
1491
|
+
// Delivery happens in the background, so wait for the simulation to settle.
|
|
1492
|
+
await simAws.backgroundTasksComplete();
|
|
1493
|
+
```
|
|
1494
|
+
|
|
1495
|
+
Two things in that template are there because real CloudFormation needs them, and simulated
|
|
1496
|
+
CloudFormation needs them for the same reasons. The Bucket names itself, where CloudFormation would
|
|
1497
|
+
otherwise name it, and the permission names the Bucket by ARN literal, where `Fn::GetAtt` would
|
|
1498
|
+
otherwise give it. Written the other way round, the Bucket needs the function's ARN and the
|
|
1499
|
+
permission needs the Bucket's, a circular dependency. The `DependsOn` then puts the permission in
|
|
1500
|
+
place before S3 validates the destination the notification names.
|
|
1501
|
+
|
|
1502
|
+
S3 generates the configuration id, because CloudFormation has no property for stating one. Read it
|
|
1503
|
+
back with `GetBucketNotificationConfigurationCommand` if a test needs it.
|
|
1504
|
+
|
|
1505
|
+
### From a CDK app
|
|
1506
|
+
|
|
1507
|
+
`bucket.addEventNotification(...)` deploys through simulated CloudFormation. CDK writes a
|
|
1508
|
+
`Custom::S3BucketNotifications` resource for it rather than the `AWS::S3::Bucket`
|
|
1509
|
+
`NotificationConfiguration` property. The resource carries the same request
|
|
1510
|
+
`PutBucketNotificationConfigurationCommand` takes, alongside the `AWS::Lambda::Permission` that lets
|
|
1511
|
+
S3 invoke the function. Yulin applies that request through the same command path an SDK caller
|
|
1512
|
+
reaches, and a configuration is validated the same way whichever it arrives by.
|
|
1513
|
+
|
|
1514
|
+
`SqsDestination` and `SnsDestination` write their entry into the same resource, alongside the
|
|
1515
|
+
`AWS::SQS::QueuePolicy` or `AWS::SNS::TopicPolicy` that grants S3 access. Both of those deploy, as
|
|
1516
|
+
does the `AWS::SNS::Topic` beside them. A stack whose Bucket notifies a topic needs nothing set up
|
|
1517
|
+
by hand.
|
|
1518
|
+
|
|
1519
|
+
Deploy into an Account and Region matching the ones the CDK app synthesized for. The `SourceAccount`
|
|
1520
|
+
on the permission CDK writes beside the notification is a synth-time literal. A stack deployed into
|
|
1521
|
+
another Account leaves S3 unable to validate the destination, and the stack fails.
|
|
1522
|
+
|
|
1523
|
+
```typescript sim-s3-cdk-event-notification
|
|
1524
|
+
/**
|
|
1525
|
+
* Deploying a CDK Bucket event notification into simulated AWS.
|
|
1526
|
+
*/
|
|
1527
|
+
|
|
1528
|
+
import { PutObjectCommand } from "@aws-sdk/client-s3";
|
|
1529
|
+
import { SimAws } from "@kensio/yulin";
|
|
1530
|
+
|
|
1531
|
+
const simAws = new SimAws();
|
|
1532
|
+
|
|
1533
|
+
// The Account and Region the CDK app synthesized for.
|
|
1534
|
+
const scope = simAws.account("111111111111").region("eu-west-2");
|
|
1535
|
+
|
|
1536
|
+
await scope
|
|
1537
|
+
.cloudFormation()
|
|
1538
|
+
.deployTemplateFile("cdk.out/TestStack.template.json");
|
|
1539
|
+
|
|
1540
|
+
await scope.s3().putObject(
|
|
1541
|
+
new PutObjectCommand({
|
|
1542
|
+
Bucket: "uploads",
|
|
1543
|
+
Key: "raw/cat.jpg",
|
|
1544
|
+
Body: "cat picture",
|
|
1545
|
+
}),
|
|
1546
|
+
);
|
|
1547
|
+
|
|
1548
|
+
// Delivery happens in the background, so wait for the simulation to settle.
|
|
1549
|
+
await simAws.backgroundTasksComplete();
|
|
1550
|
+
```
|
|
1551
|
+
|
|
1552
|
+
CDK's own provider function for this resource is written in Python, so simulated CloudFormation skips
|
|
1553
|
+
it on its runtime and Yulin does the work the function would have done. The `ServiceToken` naming it
|
|
1554
|
+
is read and ignored.
|
|
1555
|
+
|
|
1556
|
+
A resource carrying `Managed: false` is refused, and the stack fails. CDK writes it for a Bucket the
|
|
1557
|
+
app imported rather than declared. It asks S3 to merge the configuration with the configurations
|
|
1558
|
+
already on the Bucket, where simulated S3 only replaces, so applying it as written would drop
|
|
1559
|
+
configurations that survive on real AWS. Declare the Bucket in the same stack to
|
|
1560
|
+
get a managed notification configuration.
|
|
1561
|
+
|
|
1562
|
+
### What arrives at the destination
|
|
1563
|
+
|
|
1564
|
+
A function is invoked with the `Records` document real S3 sends. A queue gets the same document as
|
|
1565
|
+
one message body, and a topic gets it as the published `Message`. One event produces one record.
|
|
1566
|
+
|
|
1567
|
+
Creation records carry the Object's `size` and its `eTag`, the MD5 of the bytes as it is for an
|
|
1568
|
+
Object real S3 stored in one part. Removal records leave both out, because the Object they describe
|
|
1569
|
+
is gone. Both carry a `sequencer`, which orders the events for one object key. The object key is
|
|
1570
|
+
form-URL-encoded, so `red flower.jpg` arrives as `red+flower.jpg`.
|
|
1571
|
+
|
|
1572
|
+
`eventTime` comes from the simulation's clock, and a frozen clock produces a fixed timestamp.
|
|
1573
|
+
|
|
1574
|
+
The document is typed as `SimS3Event`, with `SimS3EventRecord` for one record, and a handler can be
|
|
1575
|
+
written against it. Assigning it to the `aws-lambda` typings package's `S3Event` fails,
|
|
1576
|
+
deliberately. That package declares `Records` mutable and requires `s3.object.size` and `eTag`, which a
|
|
1577
|
+
removal record leaves out. A handler typed against `S3Event` still receives this
|
|
1578
|
+
document at runtime, and typing it as `SimS3Event` is what describes what actually arrives.
|
|
1579
|
+
|
|
1580
|
+
### Making an event notification without a Bucket
|
|
1581
|
+
|
|
1582
|
+
A test of the handler on its own, with no Bucket and no configuration, still has to pass it a whole
|
|
1583
|
+
event. `s3NotificationEventFactory` makes one, and `s3NotificationEventRecordFactory` makes the
|
|
1584
|
+
records in it:
|
|
1585
|
+
|
|
1586
|
+
```typescript sim-s3-notification-event-factory
|
|
1587
|
+
/**
|
|
1588
|
+
* Making an S3 event notification to call a handler with.
|
|
1589
|
+
*/
|
|
1590
|
+
|
|
1591
|
+
import { VariantFactory } from "@kensio/part-factory";
|
|
1592
|
+
|
|
1593
|
+
import { s3NotificationEventFactory, type SimS3Event } from "@kensio/yulin/s3";
|
|
1594
|
+
|
|
1595
|
+
function thumbnailKeys(event: SimS3Event): readonly string[] {
|
|
1596
|
+
return event.Records.filter((record) =>
|
|
1597
|
+
record.eventName.startsWith("ObjectCreated"),
|
|
1598
|
+
).map((record) => `${record.s3.bucket.name}/${record.s3.object.key}`);
|
|
1599
|
+
}
|
|
1600
|
+
|
|
1601
|
+
const uploaded = s3NotificationEventFactory.make({
|
|
1602
|
+
Records: [
|
|
1603
|
+
{ s3: { bucket: { name: "uploads" }, object: { key: "cat.jpg" } } },
|
|
1604
|
+
],
|
|
1605
|
+
});
|
|
1606
|
+
|
|
1607
|
+
// [ 'uploads/cat.jpg' ]
|
|
1608
|
+
console.log(thumbnailKeys(uploaded));
|
|
1609
|
+
|
|
1610
|
+
// A removal is a variation worth naming, since it reports no Object detail.
|
|
1611
|
+
const objectRemovedFactory = new VariantFactory(s3NotificationEventFactory, {
|
|
1612
|
+
Records: [{ eventName: "ObjectRemoved:Delete" }],
|
|
1613
|
+
});
|
|
1614
|
+
|
|
1615
|
+
// []
|
|
1616
|
+
console.log(thumbnailKeys(objectRemovedFactory.make()));
|
|
1617
|
+
```
|
|
1618
|
+
|
|
1619
|
+
The default is the single record one Object event produces, all real S3 delivers to a function at
|
|
1620
|
+
once. What a record says in more than one place is computed from the rest. The Bucket ARN is the ARN
|
|
1621
|
+
of the Bucket named, and a removal carries no `size` and no `eTag` where a creation carries both.
|
|
1622
|
+
The key is carried as a record carries it, form-URL-encoded, and a key with a space in it goes in as
|
|
1623
|
+
`red+flower.jpg`.
|
|
1624
|
+
|
|
1625
|
+
The [event factories page](https://yulinsim.dev/factories/ "Test factories for AWS event shapes usage docs")
|
|
1626
|
+
covers what these have in common with the factories for the other event shapes.
|
|
1627
|
+
|
|
1628
|
+
### When delivery fails
|
|
1629
|
+
|
|
1630
|
+
Real S3 tells the caller who wrote the Object nothing about a delivery, and the simulator says as
|
|
1631
|
+
little. A handler that throws leaves the `PutObject` successful and `backgroundTasksComplete()`
|
|
1632
|
+
resolved. The outcome is still readable:
|
|
1633
|
+
|
|
1634
|
+
```typescript
|
|
1635
|
+
for (const failure of simAws.s3().getNotificationDeliveryFailures()) {
|
|
1636
|
+
console.log(failure.destinationArn, failure.reason, failure.wasRefused);
|
|
1637
|
+
}
|
|
1638
|
+
```
|
|
1639
|
+
|
|
1640
|
+
A handler that threw is also warned about on the console, once per destination and cause. A
|
|
1641
|
+
destination that refused the event, because its resource policy no longer admits the Bucket, is
|
|
1642
|
+
recorded without a warning.
|
|
1643
|
+
|
|
1644
|
+
A handler that writes back into the Bucket that triggered it notifies itself forever, and in process
|
|
1645
|
+
there is nothing to slow it down. Filter the configuration by prefix or suffix, so the handler's own
|
|
1646
|
+
writes fall outside it. Without that, the simulation stops after a thousand deliveries and
|
|
1647
|
+
`backgroundTasksComplete()` raises an error naming the Bucket.
|
|
1648
|
+
|
|
1649
|
+
### Limitations
|
|
1650
|
+
|
|
1651
|
+
- A Lambda function, an SQS queue and an SNS topic are the destinations. EventBridge is refused by
|
|
1652
|
+
name.
|
|
1653
|
+
- A destination goes where the group it was declared in says, and its ARN has no say. A queue ARN
|
|
1654
|
+
under `LambdaFunctionConfigurations` is refused for failing to be a function ARN, and never
|
|
1655
|
+
delivered to as a queue.
|
|
1656
|
+
- Four event types are raised: `s3:ObjectCreated:Put`, `s3:ObjectCreated:Copy`,
|
|
1657
|
+
`s3:ObjectCreated:CompleteMultipartUpload` and `s3:ObjectRemoved:Delete`. `Post`,
|
|
1658
|
+
`DeleteMarkerCreated`, the `ObjectRestore:*`, `Replication:*`, `LifecycleExpiration:*` and
|
|
1659
|
+
`ObjectTagging:*` families, `LifecycleTransition`, `IntelligentTiering`, `ObjectAcl:Put` and
|
|
1660
|
+
`ReducedRedundancyLostObject` are refused by name. `s3:ObjectCreated:*` expands to the three
|
|
1661
|
+
creations and `s3:ObjectRemoved:*` to the one removal.
|
|
1662
|
+
- `userIdentity.principalId` carries the caller's ARN rather than the `AIDA...` unique id real S3
|
|
1663
|
+
puts there. Simulated IAM has no unique-id namespace to draw one from, and an ARN is what a test
|
|
1664
|
+
would assert on. `requestParameters.sourceIPAddress` is the loopback address, because the request
|
|
1665
|
+
was made in this process, and the `responseElements` request ids are generated per event and match
|
|
1666
|
+
nothing.
|
|
1667
|
+
- `eventVersion` is the version the S3 event message structure page documents now. AWS increments the
|
|
1668
|
+
minor version whenever it adds a field, so compare the major for equality and leave the whole
|
|
1669
|
+
string alone.
|
|
1670
|
+
- `versionId` is absent from every record, as it is on real S3 for a Bucket without versioning.
|
|
1671
|
+
Versioning is left out.
|
|
1672
|
+
- A notification cannot be configured on a standalone `SimS3`. It has no other simulated services to
|
|
1673
|
+
notify, and no shared background scheduler for `backgroundTasksComplete()` to drain. Reach
|
|
1674
|
+
simulated S3 through `SimAws` instead.
|
|
1675
|
+
- An `EventBridgeConfiguration` in an `AWS::S3::Bucket` `NotificationConfiguration` is refused by
|
|
1676
|
+
name, as it is for an SDK caller.
|
|
1677
|
+
- `Managed: false` on a `Custom::S3BucketNotifications` resource is refused outright, and an
|
|
1678
|
+
EventBridge destination in one is refused by name as it is for an SDK caller.
|
|
1679
|
+
- A FIFO queue destination is refused by name, as real S3 refuses one. Simulated SQS has no FIFO
|
|
1680
|
+
queues either, and simulated SNS has no FIFO topics, so a FIFO topic destination is refused the
|
|
1681
|
+
same way.
|
|
1682
|
+
- The KMS key policy statement CDK's `SqsDestination` writes for an encrypted queue is ignored.
|
|
1683
|
+
Queue encryption is left out.
|
|
1684
|
+
- A CDK `BucketDeployment` and `mountBucketFilesystem(...)` both replace the whole storage backend
|
|
1685
|
+
rather than putting Objects, and neither raises an event. Real CDK `BucketDeployment` fires one
|
|
1686
|
+
`ObjectCreated:Put` per file.
|
|
1687
|
+
- A topic destination publishes with no message attributes, since real S3 publishes none. The only
|
|
1688
|
+
thing on the message besides the event document is the `Amazon S3 Notification` subject.
|
|
1689
|
+
- `s3:TestEvent` is left out. Real S3 puts one on a queue or topic when a configuration naming it is
|
|
1690
|
+
applied, carrying a flat `{Service, Event, Time, Bucket, RequestId, HostId}` document with no
|
|
1691
|
+
`Records` in it. Sending it here would make the simplest test two messages long and hand a
|
|
1692
|
+
consumer a body it cannot parse as an event. What the message exists to prove, that S3 may reach
|
|
1693
|
+
the destination, is simulated directly by the destination check.
|
|
1694
|
+
|
|
1695
|
+
## Buckets from CloudFormation
|
|
1696
|
+
|
|
1697
|
+
An `AWS::S3::Bucket` resource carries five properties simulated S3 acts on. Those are `BucketName`,
|
|
1698
|
+
`LifecycleConfiguration`, `NotificationConfiguration`, `PublicAccessBlockConfiguration` and
|
|
1699
|
+
`WebsiteConfiguration`. See [Lifecycle configuration](#lifecycle-configuration) for the parts of a
|
|
1700
|
+
rule that are read.
|
|
1701
|
+
|
|
1702
|
+
A Bucket with no `BucketName` is named from the stack name, the logical ID and a tail derived from
|
|
1703
|
+
both, lower cased as a bucket name has to be. A `SiteBucket` in `orders-stack` becomes
|
|
1704
|
+
`orders-stack-sitebucket-` and twelve more characters, where real CloudFormation ends the name in
|
|
1705
|
+
twelve random ones. The name is trimmed to the 63 characters a bucket name allows, and [the CloudFormation docs](https://yulinsim.dev/services/cloudformation/#names-cloudformation-generates "Names CloudFormation generates")
|
|
1706
|
+
cover how the stack name and the logical ID share what is left.
|
|
1707
|
+
|
|
1708
|
+
Any other property is left out and recorded in
|
|
1709
|
+
[`stack.ignoredProperties`](https://yulinsim.dev/services/cloudformation/#properties-a-resource-was-created-without),
|
|
1710
|
+
and the Bucket is created and the stack carries on. That matters because a Bucket deployed without
|
|
1711
|
+
the versioning, replication or CORS configuration its template asked for looks configured and
|
|
1712
|
+
behaves as though it were bare, and the failure that causes turns up somewhere else entirely. The
|
|
1713
|
+
record is where a test checks which of those it is standing on. A property name `AWS::S3::Bucket`
|
|
1714
|
+
never had is recorded the same way, and a typo leaves the stack standing.
|
|
1715
|
+
|
|
1716
|
+
One of the five given in the wrong shape still fails the stack, and so does a `BucketName` that is
|
|
1717
|
+
something other than a string. There is no Bucket to create under a name nothing else in the
|
|
1718
|
+
template refers to.
|
|
1719
|
+
|
|
1720
|
+
`BucketEncryption` and `Tags` are read, ignored and left out of the record, because nothing this
|
|
1721
|
+
simulator models can tell the difference. There is no simulated KMS, Object bytes are stored as they
|
|
1722
|
+
arrive, and no simulated service reads a Bucket tag. CDK puts both on almost every Bucket it synthesizes, and
|
|
1723
|
+
listing a difference no test could observe would only bury the ones that matter.
|
|
1724
|
+
|
|
1725
|
+
## Bucket policies
|
|
1726
|
+
|
|
1727
|
+
A Bucket policy is a resource policy stored on the Bucket. Sim IAM evaluates it alongside the
|
|
1728
|
+
caller's identity policies whenever an Object command is authorized. A policy can grant access to a
|
|
1729
|
+
principal that holds no identity policy at all, including an anonymous caller.
|
|
1730
|
+
|
|
1731
|
+
Apply one with `PutBucketPolicyCommand`, read it back with `GetBucketPolicyCommand`, and remove it
|
|
1732
|
+
with `DeleteBucketPolicyCommand`. Each is authorized in its own right, against `s3:PutBucketPolicy`,
|
|
1733
|
+
`s3:GetBucketPolicy` and `s3:DeleteBucketPolicy`.
|
|
1734
|
+
|
|
1735
|
+
In a CloudFormation template, a Bucket policy is a separate `AWS::S3::BucketPolicy` resource rather
|
|
1736
|
+
than a property of `AWS::S3::Bucket`. CDK synthesizes one for `bucket.grantRead(...)`,
|
|
1737
|
+
`grantPut(...)` and `addToResourcePolicy(...)`, and a template reaches it whether or not the app
|
|
1738
|
+
mentions a Bucket policy itself. Sim CloudFormation attaches it through the same `PutBucketPolicy`
|
|
1739
|
+
path an SDK call takes, and the document is validated and enforced identically either way.
|
|
1740
|
+
|
|
1741
|
+
```typescript sim-s3-bucket-policy
|
|
1742
|
+
/**
|
|
1743
|
+
* Granting access to a simulated S3 Bucket with a Bucket policy.
|
|
1744
|
+
*/
|
|
1745
|
+
|
|
1746
|
+
import { CreateRoleCommand } from "@aws-sdk/client-iam";
|
|
1747
|
+
import {
|
|
1748
|
+
GetBucketPolicyCommand,
|
|
1749
|
+
GetObjectCommand,
|
|
1750
|
+
PutObjectCommand,
|
|
1751
|
+
} from "@aws-sdk/client-s3";
|
|
1752
|
+
import { SimAws } from "@kensio/yulin";
|
|
1753
|
+
|
|
1754
|
+
const simAws = new SimAws();
|
|
1755
|
+
const simIam = simAws.iam();
|
|
1756
|
+
const simS3 = simAws.s3();
|
|
1757
|
+
|
|
1758
|
+
// The principal the Bucket policy will name. It gets no identity policy, so
|
|
1759
|
+
// the Bucket policy is the whole of its access.
|
|
1760
|
+
const roleOut = await simIam.createRole(
|
|
1761
|
+
new CreateRoleCommand({
|
|
1762
|
+
RoleName: "ReportReader",
|
|
1763
|
+
AssumeRolePolicyDocument: JSON.stringify({
|
|
1764
|
+
Version: "2012-10-17",
|
|
1765
|
+
Statement: {
|
|
1766
|
+
Effect: "Allow",
|
|
1767
|
+
Principal: { Service: "lambda.amazonaws.com" },
|
|
1768
|
+
Action: "sts:AssumeRole",
|
|
1769
|
+
},
|
|
1770
|
+
}),
|
|
1771
|
+
}),
|
|
1772
|
+
);
|
|
1773
|
+
|
|
1774
|
+
await simAws.cloudFormation().deployTemplate({
|
|
1775
|
+
stackName: "reports-stack",
|
|
1776
|
+
template: {
|
|
1777
|
+
Resources: {
|
|
1778
|
+
ReportsBucket: {
|
|
1779
|
+
Type: "AWS::S3::Bucket",
|
|
1780
|
+
Properties: { BucketName: "reports" },
|
|
1781
|
+
},
|
|
1782
|
+
ReportsBucketPolicy: {
|
|
1783
|
+
Type: "AWS::S3::BucketPolicy",
|
|
1784
|
+
Properties: {
|
|
1785
|
+
Bucket: { Ref: "ReportsBucket" },
|
|
1786
|
+
PolicyDocument: {
|
|
1787
|
+
Version: "2012-10-17",
|
|
1788
|
+
Statement: [
|
|
1789
|
+
{
|
|
1790
|
+
Effect: "Allow",
|
|
1791
|
+
Principal: { AWS: roleOut.Role.Arn },
|
|
1792
|
+
Action: "s3:GetObject",
|
|
1793
|
+
Resource: "arn:aws:s3:::reports/*",
|
|
1794
|
+
},
|
|
1795
|
+
],
|
|
1796
|
+
},
|
|
1797
|
+
},
|
|
1798
|
+
},
|
|
1799
|
+
},
|
|
1800
|
+
},
|
|
1801
|
+
});
|
|
1802
|
+
|
|
1803
|
+
await simS3.putObject(
|
|
1804
|
+
new PutObjectCommand({
|
|
1805
|
+
Bucket: "reports",
|
|
1806
|
+
Key: "q3/report.txt",
|
|
1807
|
+
Body: "quarterly numbers",
|
|
1808
|
+
}),
|
|
1809
|
+
);
|
|
1810
|
+
|
|
1811
|
+
// The deployed policy authorizes the read.
|
|
1812
|
+
const objectOut = await simS3.getObject(
|
|
1813
|
+
new GetObjectCommand({ Bucket: "reports", Key: "q3/report.txt" }),
|
|
1814
|
+
{ caller: { kind: "arn", arn: roleOut.Role.Arn } },
|
|
1815
|
+
);
|
|
1816
|
+
|
|
1817
|
+
console.log(objectOut.Metadata);
|
|
1818
|
+
|
|
1819
|
+
// The same document comes back out as a JSON string.
|
|
1820
|
+
const policyOut = await simS3.getBucketPolicy(
|
|
1821
|
+
new GetBucketPolicyCommand({ Bucket: "reports" }),
|
|
1822
|
+
);
|
|
1823
|
+
|
|
1824
|
+
console.log(policyOut.Policy);
|
|
1825
|
+
```
|
|
1826
|
+
|
|
1827
|
+
`GetBucketPolicyCommand` throws `NoSuchBucketPolicy` when the Bucket exists but has no policy, as
|
|
1828
|
+
real S3 separates that from a missing Bucket. `DeleteBucketPolicyCommand` succeeds either way,
|
|
1829
|
+
matching S3's idempotent behaviour.
|
|
1830
|
+
|
|
1831
|
+
A Bucket policy granting `Principal: "*"` is refused by default. See
|
|
1832
|
+
[Block Public Access](#block-public-access) below.
|
|
1833
|
+
|
|
1834
|
+
### Where a request came from
|
|
1835
|
+
|
|
1836
|
+
A request can say what it is being made for, and a simulated service supplies that when it reaches a
|
|
1837
|
+
Bucket on a resource's behalf. `sourceArn` and `sourceAccount` go alongside the caller and
|
|
1838
|
+
reach IAM as the `aws:SourceArn` and `aws:SourceAccount` condition keys:
|
|
1839
|
+
|
|
1840
|
+
```typescript
|
|
1841
|
+
await simS3.getObject(
|
|
1842
|
+
new GetObjectCommand({ Bucket: "site", Key: "index.html" }),
|
|
1843
|
+
{
|
|
1844
|
+
caller: { kind: "service", service: "cloudfront.amazonaws.com" },
|
|
1845
|
+
sourceArn: "arn:aws:cloudfront::111111111111:distribution/E1EXAMPLE",
|
|
1846
|
+
},
|
|
1847
|
+
);
|
|
1848
|
+
```
|
|
1849
|
+
|
|
1850
|
+
That is the condition a Bucket policy granting a service principal usually carries, since a service
|
|
1851
|
+
principal is shared by every resource of that service. A request carrying no such value leaves the
|
|
1852
|
+
key out entirely, and a statement conditioned on it fails to match. Condition key names are matched
|
|
1853
|
+
case insensitively, so CDK's `AWS:SourceArn` spelling matches the same key.
|
|
1854
|
+
|
|
1855
|
+
Sim CloudFront supplies both when a Distribution's S3 Origin has an origin access control, and that
|
|
1856
|
+
is [how it serves a private Bucket](https://yulinsim.dev/services/cloudfront/#origin-access-controls).
|
|
1857
|
+
|
|
1858
|
+
## Block Public Access
|
|
1859
|
+
|
|
1860
|
+
Real S3 turns on all four Block Public Access settings for every new Bucket, and `BlockPublicPolicy`
|
|
1861
|
+
makes `PutBucketPolicy` reject a policy that allows public access. Sim S3 does the same. A Bucket
|
|
1862
|
+
starts closed, and a public Bucket policy is refused with `AccessDenied` until the Bucket opts out:
|
|
1863
|
+
|
|
1864
|
+
```typescript
|
|
1865
|
+
await simS3.putPublicAccessBlock(
|
|
1866
|
+
new PutPublicAccessBlockCommand({
|
|
1867
|
+
Bucket: "site",
|
|
1868
|
+
PublicAccessBlockConfiguration: { BlockPublicPolicy: false },
|
|
1869
|
+
}),
|
|
1870
|
+
);
|
|
1871
|
+
```
|
|
1872
|
+
|
|
1873
|
+
The configuration you supply replaces the previous one wholesale, and a setting you leave out of it
|
|
1874
|
+
is off. That matches CDK. `BlockPublicAccess.BLOCK_ACLS` names only the two ACL settings, and pairing
|
|
1875
|
+
it with `publicReadAccess: true` is the usual way to build a public website Bucket.
|
|
1876
|
+
|
|
1877
|
+
`GetPublicAccessBlockCommand` reads the settings back, and `DeletePublicAccessBlockCommand` removes
|
|
1878
|
+
them, which returns the Bucket to fully blocked. In a CloudFormation
|
|
1879
|
+
template the settings are the `PublicAccessBlockConfiguration` property of `AWS::S3::Bucket`, and a
|
|
1880
|
+
Stack whose `AWS::S3::BucketPolicy` is public without that opt-out fails to deploy, exactly as the
|
|
1881
|
+
real deployment would.
|
|
1882
|
+
|
|
1883
|
+
The settings govern what may be written rather than what is already stored, so turning
|
|
1884
|
+
`BlockPublicPolicy` back on afterwards leaves an existing public policy in place.
|
|
1885
|
+
|
|
1886
|
+
### What counts as public
|
|
1887
|
+
|
|
1888
|
+
A statement is public when it allows a wildcard `Principal` without pinning the caller down. A
|
|
1889
|
+
`Condition` fixing `aws:SourceAccount`, `aws:SourceArn`, `aws:PrincipalOrgID`, `aws:SourceVpc`,
|
|
1890
|
+
`aws:SourceVpce`, `aws:SourceOwner`, `aws:userid`, `s3:DataAccessPointArn` or
|
|
1891
|
+
`s3:DataAccessPointAccount` to a value with no wildcard in it makes the statement non-public, as it
|
|
1892
|
+
does in real S3. A `Service` principal is never a wildcard, and a `Deny` statement is never public.
|
|
1893
|
+
|
|
1894
|
+
### Limitations
|
|
1895
|
+
|
|
1896
|
+
Only `BlockPublicPolicy` changes behaviour. The other three settings are stored and reported, and go
|
|
1897
|
+
no further. `BlockPublicAcls` and `IgnorePublicAcls` govern ACLs, which this simulator leaves out.
|
|
1898
|
+
`RestrictPublicBuckets` changes how an existing public policy is evaluated for cross-account callers
|
|
1899
|
+
rather than rejecting a write, and that evaluation is absent so far.
|
|
1900
|
+
|
|
1901
|
+
Anything the simulator cannot classify confidently counts as public and is refused, which makes it
|
|
1902
|
+
stricter than real S3 in places. A `NotPrincipal` statement, a statement with no `Principal`, and a
|
|
1903
|
+
`Condition` on `aws:SourceIp` all count as public here. Real S3 accepts a sufficiently narrow
|
|
1904
|
+
`aws:SourceIp` CIDR range as non-public, where the simulator judges no range breadth at all.
|
|
1905
|
+
|
|
1906
|
+
Account-level and organisation-level Block Public Access, access points, and `GetBucketPolicyStatus`
|
|
1907
|
+
are left out.
|
|
1908
|
+
|
|
1909
|
+
The static website endpoint authorizes a request that names a principal as that principal, where a
|
|
1910
|
+
real S3 website endpoint supports only publicly readable content and authenticates nothing. The
|
|
1911
|
+
simulator is looser here. A website reachable in a test as a named principal can be unreachable in
|
|
1912
|
+
the same way against real S3.
|
|
1913
|
+
|
|
1914
|
+
Bucket ACLs and Object ownership settings are left out, and stay that way by choice. Object Ownership
|
|
1915
|
+
defaults to Bucket owner enforced on new Buckets, which disables ACLs, and AWS recommends keeping
|
|
1916
|
+
them disabled in favour of policies.
|
|
1917
|
+
|
|
1918
|
+
## Lifecycle configuration
|
|
1919
|
+
|
|
1920
|
+
Sim S3 stores a Bucket's lifecycle rules and acts on them. An `Expiration` rule removes the Objects
|
|
1921
|
+
it selects once simulated time passes the boundary, and an `AbortIncompleteMultipartUpload` rule
|
|
1922
|
+
discards uploads that were started and left unfinished.
|
|
1923
|
+
|
|
1924
|
+
Retention is otherwise the one property of a log or a backup Bucket a test cannot demonstrate.
|
|
1925
|
+
Reading the rules back off a deployed Bucket says the rules arrived. Putting an Object, moving the
|
|
1926
|
+
clock and finding the Object gone says the Bucket keeps what it was configured to keep.
|
|
1927
|
+
|
|
1928
|
+
```typescript sim-s3-lifecycle-expiry
|
|
1929
|
+
/**
|
|
1930
|
+
* Expiring simulated S3 Objects against a lifecycle rule.
|
|
1931
|
+
*/
|
|
1932
|
+
|
|
1933
|
+
import {
|
|
1934
|
+
CreateBucketCommand,
|
|
1935
|
+
ListObjectsV2Command,
|
|
1936
|
+
PutBucketLifecycleConfigurationCommand,
|
|
1937
|
+
PutObjectCommand,
|
|
1938
|
+
} from "@aws-sdk/client-s3";
|
|
1939
|
+
import { SimAws } from "@kensio/yulin";
|
|
1940
|
+
|
|
1941
|
+
const simAws = new SimAws();
|
|
1942
|
+
const simS3 = simAws.region("eu-west-2").s3();
|
|
1943
|
+
|
|
1944
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "logs" }));
|
|
1945
|
+
await simS3.putBucketLifecycleConfiguration(
|
|
1946
|
+
new PutBucketLifecycleConfigurationCommand({
|
|
1947
|
+
Bucket: "logs",
|
|
1948
|
+
LifecycleConfiguration: {
|
|
1949
|
+
Rules: [
|
|
1950
|
+
{
|
|
1951
|
+
ID: "expire-raw-logs",
|
|
1952
|
+
Status: "Enabled",
|
|
1953
|
+
Filter: { Prefix: "raw/" },
|
|
1954
|
+
Expiration: { Days: 365 },
|
|
1955
|
+
},
|
|
1956
|
+
],
|
|
1957
|
+
},
|
|
1958
|
+
}),
|
|
1959
|
+
);
|
|
1960
|
+
|
|
1961
|
+
await simS3.putObject(
|
|
1962
|
+
new PutObjectCommand({
|
|
1963
|
+
Bucket: "logs",
|
|
1964
|
+
Key: "raw/2026-08-24.gz",
|
|
1965
|
+
Body: "one raw log line",
|
|
1966
|
+
}),
|
|
1967
|
+
);
|
|
1968
|
+
|
|
1969
|
+
await simAws.clock().advanceBy({ days: 366 });
|
|
1970
|
+
|
|
1971
|
+
const listing = await simS3.listObjectsV2(
|
|
1972
|
+
new ListObjectsV2Command({ Bucket: "logs", Prefix: "raw/" }),
|
|
1973
|
+
);
|
|
1974
|
+
|
|
1975
|
+
// The rule expired the Object, so the listing is empty.
|
|
1976
|
+
console.log(listing.Contents ?? []);
|
|
1977
|
+
```
|
|
1978
|
+
|
|
1979
|
+
An Object goes the moment the clock reaches the boundary. Real S3 removes an expired Object some
|
|
1980
|
+
time after it and bills up to the removal, which a test would have to wait out. Expiring on the
|
|
1981
|
+
boundary is the answer a test can assert against.
|
|
1982
|
+
|
|
1983
|
+
Expiry happens when the Bucket is read. What `ListObjectsV2`, `GetObject` and `HeadObject` find is
|
|
1984
|
+
what the rules leave at that instant, and a Bucket carrying no rules costs one comparison. Moving
|
|
1985
|
+
the clock backwards afterwards leaves an expired Object gone, because the rule deleted it on the way
|
|
1986
|
+
past.
|
|
1987
|
+
|
|
1988
|
+
### What a rule selects
|
|
1989
|
+
|
|
1990
|
+
A rule selects Objects by its `Filter`, or by the older top-level `Prefix`. A rule with no scope at
|
|
1991
|
+
all covers every key in the Bucket. `Filter.Prefix`, `Filter.And.Prefix`, `ObjectSizeGreaterThan` and
|
|
1992
|
+
`ObjectSizeLessThan` are all read. A `Disabled` rule is stored and skipped.
|
|
1993
|
+
|
|
1994
|
+
A multipart upload is selected by its key alone. Half an upload has no size. A rule narrowed by an
|
|
1995
|
+
object size bound abandons no upload.
|
|
1996
|
+
|
|
1997
|
+
Sim S3 holds no Object tags. A rule narrowed by `Filter.Tag`, `Filter.And.Tags` or a template's
|
|
1998
|
+
`TagFilters` selects no Object, and expires none.
|
|
1999
|
+
|
|
2000
|
+
### Reading and replacing the rules
|
|
2001
|
+
|
|
2002
|
+
```typescript sim-s3-lifecycle-configuration
|
|
2003
|
+
import {
|
|
2004
|
+
GetBucketLifecycleConfigurationCommand,
|
|
2005
|
+
PutBucketLifecycleConfigurationCommand,
|
|
2006
|
+
} from "@aws-sdk/client-s3";
|
|
2007
|
+
import { SimAws } from "@kensio/yulin";
|
|
2008
|
+
|
|
2009
|
+
const simAws = new SimAws();
|
|
2010
|
+
const simS3 = simAws.s3();
|
|
2011
|
+
|
|
2012
|
+
await simAws.cloudFormation().deployTemplate({
|
|
2013
|
+
stackName: "logs-stack",
|
|
2014
|
+
template: {
|
|
2015
|
+
Resources: {
|
|
2016
|
+
LogBucket: {
|
|
2017
|
+
Type: "AWS::S3::Bucket",
|
|
2018
|
+
Properties: {
|
|
2019
|
+
BucketName: "logs",
|
|
2020
|
+
LifecycleConfiguration: {
|
|
2021
|
+
Rules: [
|
|
2022
|
+
{
|
|
2023
|
+
Id: "expire-raw-logs",
|
|
2024
|
+
Status: "Enabled",
|
|
2025
|
+
Prefix: "raw/",
|
|
2026
|
+
ExpirationInDays: 365,
|
|
2027
|
+
},
|
|
2028
|
+
],
|
|
2029
|
+
},
|
|
2030
|
+
},
|
|
2031
|
+
},
|
|
2032
|
+
},
|
|
2033
|
+
},
|
|
2034
|
+
});
|
|
2035
|
+
|
|
2036
|
+
// The template's rule reads back off the deployed Bucket, in the shape the SDK
|
|
2037
|
+
// states one in.
|
|
2038
|
+
const deployed = await simS3.getBucketLifecycleConfiguration(
|
|
2039
|
+
new GetBucketLifecycleConfigurationCommand({ Bucket: "logs" }),
|
|
2040
|
+
);
|
|
2041
|
+
|
|
2042
|
+
console.log(deployed.Rules);
|
|
2043
|
+
|
|
2044
|
+
// A put replaces the whole configuration, so a rule it leaves out is gone.
|
|
2045
|
+
await simS3.putBucketLifecycleConfiguration(
|
|
2046
|
+
new PutBucketLifecycleConfigurationCommand({
|
|
2047
|
+
Bucket: "logs",
|
|
2048
|
+
LifecycleConfiguration: {
|
|
2049
|
+
Rules: [
|
|
2050
|
+
{
|
|
2051
|
+
ID: "abort-incomplete-uploads",
|
|
2052
|
+
Status: "Enabled",
|
|
2053
|
+
Filter: { Prefix: "" },
|
|
2054
|
+
AbortIncompleteMultipartUpload: { DaysAfterInitiation: 7 },
|
|
2055
|
+
},
|
|
2056
|
+
],
|
|
2057
|
+
},
|
|
2058
|
+
}),
|
|
2059
|
+
);
|
|
2060
|
+
```
|
|
2061
|
+
|
|
2062
|
+
`DeleteBucketLifecycleCommand` removes the configuration, and is idempotent as in real S3. A Bucket
|
|
2063
|
+
carrying no rules answers `GetBucketLifecycleConfigurationCommand` with
|
|
2064
|
+
`NoSuchLifecycleConfiguration` rather than an empty list, which is how real S3 separates a Bucket
|
|
2065
|
+
nobody configured from one configured to do nothing.
|
|
2066
|
+
|
|
2067
|
+
A configuration stating no rules at all is refused with `MalformedXML`. So is a rule whose `Status`
|
|
2068
|
+
is anything but `Enabled` or `Disabled`, and a rule stating no action to take, meaning none of
|
|
2069
|
+
`Expiration`, `Transitions`, `NoncurrentVersionExpiration`, `NoncurrentVersionTransitions` or
|
|
2070
|
+
`AbortIncompleteMultipartUpload`. An empty list of transitions counts as no action. Real S3 refuses
|
|
2071
|
+
all three, and a rule stored here that real S3 would have rejected reads back looking configured.
|
|
2072
|
+
|
|
2073
|
+
### From a CloudFormation template
|
|
2074
|
+
|
|
2075
|
+
CloudFormation spells some rule fields differently from the request. `Id` becomes `ID`,
|
|
2076
|
+
`ExpirationInDays` and `ExpirationDate` are gathered under `Expiration`, and a transition's
|
|
2077
|
+
`TransitionInDays` becomes `Days`. The singular `Transition` a template may state alongside
|
|
2078
|
+
`Transitions` joins the list. Everything else, `Status`, `Prefix`, `AbortIncompleteMultipartUpload`,
|
|
2079
|
+
`TagFilters` and the object size bounds among them, is carried across as the template stated it.
|
|
2080
|
+
|
|
2081
|
+
`LifecycleConfiguration` is one of the properties simulated S3 acts on. It stays out of
|
|
2082
|
+
[`stack.ignoredProperties`](https://yulinsim.dev/services/cloudformation/#properties-a-resource-was-created-without).
|
|
2083
|
+
The two actions it enforces are `Expiration`, whether the template flattened it onto the rule or
|
|
2084
|
+
not, and `AbortIncompleteMultipartUpload`. A `Transitions` rule is stored and read back and goes no
|
|
2085
|
+
further. Which Objects an enforced action reaches is decided by the fields listed under
|
|
2086
|
+
[What a rule selects](#what-a-rule-selects).
|
|
2087
|
+
|
|
2088
|
+
### Limitations
|
|
2089
|
+
|
|
2090
|
+
No Object moves between storage classes. Storage classes are left out of the simulator entirely. A
|
|
2091
|
+
`Transitions` rule is stored and read back and goes no further.
|
|
2092
|
+
|
|
2093
|
+
Real S3 raises `s3:LifecycleExpiration:Delete` when a rule removes an Object. That event family is
|
|
2094
|
+
among the ones sim S3 leaves out. An expiry here is silent.
|
|
2095
|
+
|
|
2096
|
+
`NoncurrentVersionExpiration`, `NoncurrentVersionTransitions` and `ExpiredObjectDeleteMarker` are
|
|
2097
|
+
stored and unread, because Object versions are left out.
|
|
2098
|
+
|
|
2099
|
+
A Bucket mounted on a filesystem directory refuses the deletion an expiry asks for, the way it
|
|
2100
|
+
refuses `DeleteObject`, and answers `NotImplemented`. Removing a real file off the mounted directory
|
|
2101
|
+
is worse than reporting that the rule cannot run. Use the default in-memory storage to test
|
|
2102
|
+
retention.
|
|
2103
|
+
|
|
2104
|
+
`GetBucketLifecycleConfiguration` and its siblings are reachable through the SDK and not over the
|
|
2105
|
+
served S3 REST endpoint.
|
|
2106
|
+
|
|
2107
|
+
## Static website hosting
|
|
2108
|
+
|
|
2109
|
+
Configure Bucket website hosting with `PutBucketWebsiteCommand`.
|
|
2110
|
+
|
|
2111
|
+
Website hosting settles which Object answers a request. Who may read it is a separate question. A
|
|
2112
|
+
browser asking for a page is anonymous, and anonymous holds nothing unless a Bucket policy grants
|
|
2113
|
+
it. A site with no Bucket policy answers `403` to every ordinary visitor, as it does on real S3. See
|
|
2114
|
+
[Block Public Access](#block-public-access) for the two commands a public site needs. The localhost
|
|
2115
|
+
serving example below shows them in place. The examples in this section configure hosting without
|
|
2116
|
+
serving it, and leave that out.
|
|
2117
|
+
|
|
2118
|
+
A request that does name a principal, through a signature or the `x-sim-aws-caller` header, is
|
|
2119
|
+
authorized as that principal, and an identity policy granting `s3:GetObject` reaches the website
|
|
2120
|
+
endpoint too. Real S3 has no such thing. Its website endpoint supports only publicly readable
|
|
2121
|
+
content and never authenticates a request. This is a deliberate simulator affordance, in keeping
|
|
2122
|
+
with the other simulated services that serve HTTP, and it means a website test driven as a named
|
|
2123
|
+
principal proves less than one driven as a browser would be.
|
|
2124
|
+
|
|
2125
|
+
```typescript sim-s3-static-website-hosting
|
|
2126
|
+
/**
|
|
2127
|
+
* Simulated S3 static website hosting.
|
|
2128
|
+
*/
|
|
2129
|
+
|
|
2130
|
+
import {
|
|
2131
|
+
CreateBucketCommand,
|
|
2132
|
+
PutBucketWebsiteCommand,
|
|
2133
|
+
PutObjectCommand,
|
|
2134
|
+
} from "@aws-sdk/client-s3";
|
|
2135
|
+
import { SimAws } from "@kensio/yulin";
|
|
2136
|
+
|
|
2137
|
+
const simAws = new SimAws();
|
|
2138
|
+
const simS3 = simAws.region("eu-west-2").s3();
|
|
2139
|
+
|
|
2140
|
+
await simS3.createBucket(
|
|
2141
|
+
new CreateBucketCommand({
|
|
2142
|
+
Bucket: "foo-site",
|
|
2143
|
+
}),
|
|
2144
|
+
);
|
|
2145
|
+
|
|
2146
|
+
await simS3.putObject(
|
|
2147
|
+
new PutObjectCommand({
|
|
2148
|
+
Bucket: "foo-site",
|
|
2149
|
+
Key: "index.html",
|
|
2150
|
+
Body: "<h1>Hello from simulated S3</h1>",
|
|
2151
|
+
ContentType: "text/html; charset=utf-8",
|
|
2152
|
+
}),
|
|
2153
|
+
);
|
|
2154
|
+
|
|
2155
|
+
await simS3.putObject(
|
|
2156
|
+
new PutObjectCommand({
|
|
2157
|
+
Bucket: "foo-site",
|
|
2158
|
+
Key: "docs/index.html",
|
|
2159
|
+
Body: "<h1>Docs</h1>",
|
|
2160
|
+
ContentType: "text/html; charset=utf-8",
|
|
2161
|
+
}),
|
|
2162
|
+
);
|
|
2163
|
+
|
|
2164
|
+
await simS3.putBucketWebsite(
|
|
2165
|
+
new PutBucketWebsiteCommand({
|
|
2166
|
+
Bucket: "foo-site",
|
|
2167
|
+
WebsiteConfiguration: {
|
|
2168
|
+
IndexDocument: {
|
|
2169
|
+
Suffix: "index.html",
|
|
2170
|
+
},
|
|
2171
|
+
},
|
|
2172
|
+
}),
|
|
2173
|
+
);
|
|
2174
|
+
|
|
2175
|
+
console.log(simS3.getBucketWebsiteUrl("foo-site").toString());
|
|
2176
|
+
```
|
|
2177
|
+
|
|
2178
|
+
With an index document configured:
|
|
2179
|
+
|
|
2180
|
+
- `/` resolves to `index.html`
|
|
2181
|
+
- `/docs/` resolves to `docs/index.html`
|
|
2182
|
+
- `/docs` redirects to `/docs/` when `docs/index.html` exists
|
|
2183
|
+
|
|
2184
|
+
Static website hosting must be enabled before the sim Bucket can be served over HTTP. The localhost
|
|
2185
|
+
server returns `403` until it is.
|
|
2186
|
+
|
|
2187
|
+
## Serve simulated S3 on localhost
|
|
2188
|
+
|
|
2189
|
+
Use `serveSimAws` when you want application code to make real HTTP requests to the simulated S3, or
|
|
2190
|
+
to access the simulated services via your browser or commandline with curl.
|
|
2191
|
+
|
|
2192
|
+
```typescript sim-s3-serve-localhost
|
|
2193
|
+
/**
|
|
2194
|
+
* Serving simulated S3 on localhost.
|
|
2195
|
+
*/
|
|
2196
|
+
|
|
2197
|
+
import {
|
|
2198
|
+
CreateBucketCommand,
|
|
2199
|
+
PutBucketPolicyCommand,
|
|
2200
|
+
PutBucketWebsiteCommand,
|
|
2201
|
+
PutObjectCommand,
|
|
2202
|
+
PutPublicAccessBlockCommand,
|
|
2203
|
+
} from "@aws-sdk/client-s3";
|
|
2204
|
+
import { SimAws } from "@kensio/yulin";
|
|
2205
|
+
import { serveSimAws } from "@kensio/yulin/serve";
|
|
2206
|
+
|
|
2207
|
+
const simAws = new SimAws();
|
|
2208
|
+
const srv = await serveSimAws({ simAws });
|
|
2209
|
+
|
|
2210
|
+
try {
|
|
2211
|
+
const simS3 = simAws.region("eu-west-2").s3();
|
|
2212
|
+
|
|
2213
|
+
await simS3.createBucket(
|
|
2214
|
+
new CreateBucketCommand({
|
|
2215
|
+
Bucket: "foo-site",
|
|
2216
|
+
}),
|
|
2217
|
+
);
|
|
2218
|
+
|
|
2219
|
+
await simS3.putObject(
|
|
2220
|
+
new PutObjectCommand({
|
|
2221
|
+
Bucket: "foo-site",
|
|
2222
|
+
Key: "index.html",
|
|
2223
|
+
Body: "<h1>Hello from localhost S3</h1>",
|
|
2224
|
+
ContentType: "text/html; charset=utf-8",
|
|
2225
|
+
}),
|
|
2226
|
+
);
|
|
2227
|
+
|
|
2228
|
+
await simS3.putBucketWebsite(
|
|
2229
|
+
new PutBucketWebsiteCommand({
|
|
2230
|
+
Bucket: "foo-site",
|
|
2231
|
+
WebsiteConfiguration: {
|
|
2232
|
+
IndexDocument: {
|
|
2233
|
+
Suffix: "index.html",
|
|
2234
|
+
},
|
|
2235
|
+
},
|
|
2236
|
+
}),
|
|
2237
|
+
);
|
|
2238
|
+
|
|
2239
|
+
// A website endpoint serves only what the Bucket policy makes readable, and
|
|
2240
|
+
// a public policy needs the Block Public Access opt-out first.
|
|
2241
|
+
await simS3.putPublicAccessBlock(
|
|
2242
|
+
new PutPublicAccessBlockCommand({
|
|
2243
|
+
Bucket: "foo-site",
|
|
2244
|
+
PublicAccessBlockConfiguration: {
|
|
2245
|
+
BlockPublicAcls: true,
|
|
2246
|
+
IgnorePublicAcls: true,
|
|
2247
|
+
},
|
|
2248
|
+
}),
|
|
2249
|
+
);
|
|
2250
|
+
await simS3.putBucketPolicy(
|
|
2251
|
+
new PutBucketPolicyCommand({
|
|
2252
|
+
Bucket: "foo-site",
|
|
2253
|
+
Policy: JSON.stringify({
|
|
2254
|
+
Version: "2012-10-17",
|
|
2255
|
+
Statement: {
|
|
2256
|
+
Effect: "Allow",
|
|
2257
|
+
Principal: "*",
|
|
2258
|
+
Action: "s3:GetObject",
|
|
2259
|
+
Resource: "arn:aws:s3:::foo-site/*",
|
|
2260
|
+
},
|
|
2261
|
+
}),
|
|
2262
|
+
}),
|
|
2263
|
+
);
|
|
2264
|
+
|
|
2265
|
+
const websiteUrl = simS3.getBucketWebsiteUrl("foo-site");
|
|
2266
|
+
const localWebsiteUrl = srv.localUrl(websiteUrl);
|
|
2267
|
+
|
|
2268
|
+
const response = await fetch(localWebsiteUrl);
|
|
2269
|
+
|
|
2270
|
+
console.log(response.status);
|
|
2271
|
+
console.log(response.headers.get("content-type"));
|
|
2272
|
+
console.log(await response.text());
|
|
2273
|
+
} finally {
|
|
2274
|
+
await srv.close();
|
|
2275
|
+
}
|
|
2276
|
+
```
|
|
2277
|
+
|
|
2278
|
+
The `getBucketWebsiteUrl(...)` method returns the simulated S3 website URL for the Bucket. The
|
|
2279
|
+
`localUrl(...)` method on the localhost server adapts that URL so the request is sent to the local
|
|
2280
|
+
server while preserving the simulated S3 website hostname.
|
|
2281
|
+
|
|
2282
|
+
## Presigned URLs
|
|
2283
|
+
|
|
2284
|
+
Sim S3 serves a REST API endpoint alongside the website endpoint, and it accepts presigned URLs
|
|
2285
|
+
built by the real AWS presigner, `getSignedUrl` from `@aws-sdk/s3-request-presigner`. Nothing about
|
|
2286
|
+
the signing is simulated. An `S3Client` is pointed at the simulated endpoint and signs as it would
|
|
2287
|
+
against real S3, and sim IAM verifies the signature it produced.
|
|
2288
|
+
|
|
2289
|
+
Presigning is entirely client-side, and this works whether or not the URL is ever fetched over a
|
|
2290
|
+
real socket. Install the presigner alongside the SDK:
|
|
2291
|
+
|
|
2292
|
+
```bash
|
|
2293
|
+
npm install --save-dev @aws-sdk/s3-request-presigner
|
|
2294
|
+
```
|
|
2295
|
+
|
|
2296
|
+
`simS3.getServiceUrl()` gives the endpoint to configure the client with. Sim S3 also has
|
|
2297
|
+
`getBucketUrl(...)` for the virtual-hosted endpoint of one Bucket, though a client adds the Bucket
|
|
2298
|
+
to the service endpoint for itself.
|
|
2299
|
+
|
|
2300
|
+
A client pointed at an endpoint URL presigns too, the `http://localhost:<port>` form that
|
|
2301
|
+
`--endpoint-url` and `AWS_ENDPOINT_URL` take. Such a URL names no service in its hostname and is
|
|
2302
|
+
routed by the credential scope it carries, so sign it with `forcePathStyle` and the Bucket goes in
|
|
2303
|
+
the path. See [S3 over the endpoint](https://yulinsim.dev/serve/#s3-over-the-endpoint).
|
|
2304
|
+
|
|
2305
|
+
```typescript sim-s3-presigned-url
|
|
2306
|
+
/**
|
|
2307
|
+
* Downloading a simulated S3 Object through a presigned URL.
|
|
2308
|
+
*/
|
|
2309
|
+
|
|
2310
|
+
import {
|
|
2311
|
+
CreateAccessKeyCommand,
|
|
2312
|
+
CreateUserCommand,
|
|
2313
|
+
PutUserPolicyCommand,
|
|
2314
|
+
} from "@aws-sdk/client-iam";
|
|
2315
|
+
import {
|
|
2316
|
+
CreateBucketCommand,
|
|
2317
|
+
GetObjectCommand,
|
|
2318
|
+
PutObjectCommand,
|
|
2319
|
+
S3Client,
|
|
2320
|
+
} from "@aws-sdk/client-s3";
|
|
2321
|
+
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
|
|
2322
|
+
import { SimAws } from "@kensio/yulin";
|
|
2323
|
+
import { serveSimAws } from "@kensio/yulin/serve";
|
|
2324
|
+
|
|
2325
|
+
const simAws = new SimAws();
|
|
2326
|
+
const srv = await serveSimAws({ simAws });
|
|
2327
|
+
|
|
2328
|
+
try {
|
|
2329
|
+
const simS3 = simAws.region("eu-west-2").s3();
|
|
2330
|
+
const simIam = simAws.iam();
|
|
2331
|
+
|
|
2332
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "reports" }));
|
|
2333
|
+
await simS3.putObject(
|
|
2334
|
+
new PutObjectCommand({
|
|
2335
|
+
Bucket: "reports",
|
|
2336
|
+
Key: "q3/report.txt",
|
|
2337
|
+
Body: "quarterly numbers",
|
|
2338
|
+
ContentType: "text/plain",
|
|
2339
|
+
}),
|
|
2340
|
+
);
|
|
2341
|
+
|
|
2342
|
+
// Whoever presigns the URL needs permission for what it will be used for.
|
|
2343
|
+
await simIam.createUser(new CreateUserCommand({ UserName: "Publisher" }));
|
|
2344
|
+
await simIam.putUserPolicy(
|
|
2345
|
+
new PutUserPolicyCommand({
|
|
2346
|
+
UserName: "Publisher",
|
|
2347
|
+
PolicyName: "ReadReports",
|
|
2348
|
+
PolicyDocument: JSON.stringify({
|
|
2349
|
+
Version: "2012-10-17",
|
|
2350
|
+
Statement: {
|
|
2351
|
+
Effect: "Allow",
|
|
2352
|
+
Action: "s3:GetObject",
|
|
2353
|
+
Resource: "arn:aws:s3:::reports/*",
|
|
2354
|
+
},
|
|
2355
|
+
}),
|
|
2356
|
+
}),
|
|
2357
|
+
);
|
|
2358
|
+
const accessKey = await simIam.createAccessKey(
|
|
2359
|
+
new CreateAccessKeyCommand({ UserName: "Publisher" }),
|
|
2360
|
+
);
|
|
2361
|
+
|
|
2362
|
+
// The endpoint includes the port the local server took, because a presigned
|
|
2363
|
+
// URL signs its own host and cannot be redirected elsewhere afterwards.
|
|
2364
|
+
const s3Client = new S3Client({
|
|
2365
|
+
region: "eu-west-2",
|
|
2366
|
+
endpoint: srv.localUrl(simS3.getServiceUrl()).toString(),
|
|
2367
|
+
credentials: {
|
|
2368
|
+
accessKeyId: accessKey.AccessKey.AccessKeyId,
|
|
2369
|
+
secretAccessKey: accessKey.AccessKey.SecretAccessKey,
|
|
2370
|
+
},
|
|
2371
|
+
});
|
|
2372
|
+
|
|
2373
|
+
const url = await getSignedUrl(
|
|
2374
|
+
s3Client,
|
|
2375
|
+
new GetObjectCommand({ Bucket: "reports", Key: "q3/report.txt" }),
|
|
2376
|
+
{ expiresIn: 900 },
|
|
2377
|
+
);
|
|
2378
|
+
|
|
2379
|
+
const response = await fetch(url);
|
|
2380
|
+
|
|
2381
|
+
console.log(response.status);
|
|
2382
|
+
console.log(await response.text());
|
|
2383
|
+
} finally {
|
|
2384
|
+
await srv.close();
|
|
2385
|
+
}
|
|
2386
|
+
```
|
|
2387
|
+
|
|
2388
|
+
A presigned URL grants exactly what the principal who signed it holds. Sim IAM resolves that
|
|
2389
|
+
principal from the signature and authorizes `s3:GetObject` as them. A user without permission cannot
|
|
2390
|
+
presign around it. Temporary credentials from an STS `AssumeRoleCommand` work the same way,
|
|
2391
|
+
carrying their session token in the URL.
|
|
2392
|
+
|
|
2393
|
+
A request to the REST endpoint presenting no signature and naming no principal in the
|
|
2394
|
+
`x-sim-aws-caller` header is anonymous, and anonymous holds nothing unless a Bucket policy says
|
|
2395
|
+
otherwise. That header is always enabled and wins over a signature, and a request driven by hand can
|
|
2396
|
+
be any principal without signing anything, exactly as it can against the other simulated services
|
|
2397
|
+
that serve HTTP. See
|
|
2398
|
+
[the sim IAM docs](https://yulinsim.dev/services/iam/#what-the-simulator-reports-back) for the whole boundary.
|
|
2399
|
+
|
|
2400
|
+
### Expiry in simulated time
|
|
2401
|
+
|
|
2402
|
+
`X-Amz-Expires` is judged against Yulin's simulated clock. A frozen clock keeps a URL usable however
|
|
2403
|
+
long a test spends, and advancing past the window expires it with the `AccessDenied` and
|
|
2404
|
+
`Request has expired` real S3 answers with:
|
|
2405
|
+
|
|
2406
|
+
```typescript
|
|
2407
|
+
simAws.clock().freeze();
|
|
2408
|
+
const url = await getSignedUrl(s3Client, command, { expiresIn: 900 });
|
|
2409
|
+
|
|
2410
|
+
await simAws.clock().advanceBy({ minutes: 20 });
|
|
2411
|
+
const response = await fetch(url); // 403
|
|
2412
|
+
```
|
|
2413
|
+
|
|
2414
|
+
### Uploads and checksums
|
|
2415
|
+
|
|
2416
|
+
Presigned `PutObjectCommand` URLs work in the same way, with one thing to watch. The AWS SDK computes
|
|
2417
|
+
a checksum when it presigns, before there is a body to hash, and hoists it into the signed
|
|
2418
|
+
URL. Uploading anything else through that URL then fails against real S3, and fails here too, with
|
|
2419
|
+
`XAmzContentChecksumMismatch`. Build the client with
|
|
2420
|
+
`requestChecksumCalculation: "WHEN_REQUIRED"` to presign upload URLs that accept a body:
|
|
2421
|
+
|
|
2422
|
+
```typescript
|
|
2423
|
+
const s3Client = new S3Client({
|
|
2424
|
+
region: "eu-west-2",
|
|
2425
|
+
endpoint: srv.localUrl(simS3.getServiceUrl()).toString(),
|
|
2426
|
+
requestChecksumCalculation: "WHEN_REQUIRED",
|
|
2427
|
+
credentials,
|
|
2428
|
+
});
|
|
2429
|
+
```
|
|
2430
|
+
|
|
2431
|
+
### Limitations
|
|
2432
|
+
|
|
2433
|
+
- `GET`, `HEAD`, `PUT` and `DELETE` of an Object are served over a Bucket's own REST endpoint, which
|
|
2434
|
+
is what a presigned URL addresses. Bucket operations and multipart uploads there are refused with
|
|
2435
|
+
`501`. `DeleteObjects` is a `POST` to the Bucket, so it is available through the SDK and
|
|
2436
|
+
unavailable over a presigned URL. The shared endpoint `serveSimAws` binds serves all of them. See
|
|
2437
|
+
[Serve simulated S3 on localhost](#serve-simulated-s3-on-localhost).
|
|
2438
|
+
- `createPresignedPost` and SigV4A presigning are left out.
|
|
2439
|
+
- Checksums are verified for CRC32, SHA1 and SHA256. An upload stating a CRC32C or CRC64NVME checksum
|
|
2440
|
+
is refused, and never stored unchecked.
|
|
2441
|
+
- Responses carry the Object's `ETag` and `Last-Modified`, and no conditional request is honoured.
|
|
2442
|
+
`If-None-Match` and `If-Modified-Since` are ignored, and the Object is served in full.
|
|
2443
|
+
|
|
2444
|
+
## Error documents
|
|
2445
|
+
|
|
2446
|
+
Configure an error document to return custom content with a `404` response when an Object is
|
|
2447
|
+
missing.
|
|
2448
|
+
|
|
2449
|
+
```typescript sim-s3-error-document
|
|
2450
|
+
/**
|
|
2451
|
+
* Simulated S3 error documents.
|
|
2452
|
+
*/
|
|
2453
|
+
|
|
2454
|
+
import {
|
|
2455
|
+
CreateBucketCommand,
|
|
2456
|
+
PutBucketWebsiteCommand,
|
|
2457
|
+
PutObjectCommand,
|
|
2458
|
+
} from "@aws-sdk/client-s3";
|
|
2459
|
+
import { SimAws } from "@kensio/yulin";
|
|
2460
|
+
|
|
2461
|
+
const simAws = new SimAws();
|
|
2462
|
+
const simS3 = simAws.s3();
|
|
2463
|
+
|
|
2464
|
+
await simS3.createBucket(
|
|
2465
|
+
new CreateBucketCommand({
|
|
2466
|
+
Bucket: "error-site",
|
|
2467
|
+
}),
|
|
2468
|
+
);
|
|
2469
|
+
|
|
2470
|
+
await simS3.putObject(
|
|
2471
|
+
new PutObjectCommand({
|
|
2472
|
+
Bucket: "error-site",
|
|
2473
|
+
Key: "error.html",
|
|
2474
|
+
Body: "<h1>Not found</h1>",
|
|
2475
|
+
ContentType: "text/html; charset=utf-8",
|
|
2476
|
+
}),
|
|
2477
|
+
);
|
|
2478
|
+
|
|
2479
|
+
await simS3.putBucketWebsite(
|
|
2480
|
+
new PutBucketWebsiteCommand({
|
|
2481
|
+
Bucket: "error-site",
|
|
2482
|
+
WebsiteConfiguration: {
|
|
2483
|
+
IndexDocument: {
|
|
2484
|
+
Suffix: "index.html",
|
|
2485
|
+
},
|
|
2486
|
+
ErrorDocument: {
|
|
2487
|
+
Key: "error.html",
|
|
2488
|
+
},
|
|
2489
|
+
},
|
|
2490
|
+
}),
|
|
2491
|
+
);
|
|
2492
|
+
```
|
|
2493
|
+
|
|
2494
|
+
## Website redirects
|
|
2495
|
+
|
|
2496
|
+
Sim S3 supports common S3 website redirect configuration.
|
|
2497
|
+
|
|
2498
|
+
Redirect all requests to another host:
|
|
2499
|
+
|
|
2500
|
+
```typescript sim-s3-website-redirect
|
|
2501
|
+
/**
|
|
2502
|
+
* Simulated S3 website redirects.
|
|
2503
|
+
*/
|
|
2504
|
+
|
|
2505
|
+
import {
|
|
2506
|
+
CreateBucketCommand,
|
|
2507
|
+
PutBucketWebsiteCommand,
|
|
2508
|
+
} from "@aws-sdk/client-s3";
|
|
2509
|
+
import { SimAws } from "@kensio/yulin";
|
|
2510
|
+
|
|
2511
|
+
const simAws = new SimAws();
|
|
2512
|
+
const simS3 = simAws.s3();
|
|
2513
|
+
|
|
2514
|
+
await simS3.createBucket(
|
|
2515
|
+
new CreateBucketCommand({
|
|
2516
|
+
Bucket: "redirect-site",
|
|
2517
|
+
}),
|
|
2518
|
+
);
|
|
2519
|
+
|
|
2520
|
+
await simS3.putBucketWebsite(
|
|
2521
|
+
new PutBucketWebsiteCommand({
|
|
2522
|
+
Bucket: "redirect-site",
|
|
2523
|
+
WebsiteConfiguration: {
|
|
2524
|
+
RedirectAllRequestsTo: {
|
|
2525
|
+
HostName: "example.test",
|
|
2526
|
+
Protocol: "https",
|
|
2527
|
+
},
|
|
2528
|
+
},
|
|
2529
|
+
}),
|
|
2530
|
+
);
|
|
2531
|
+
```
|
|
2532
|
+
|
|
2533
|
+
Add routing rules for conditional redirects:
|
|
2534
|
+
|
|
2535
|
+
```typescript sim-s3-conditional-redirect
|
|
2536
|
+
/**
|
|
2537
|
+
* Conditional redirects in simulated S3.
|
|
2538
|
+
*/
|
|
2539
|
+
|
|
2540
|
+
import {
|
|
2541
|
+
CreateBucketCommand,
|
|
2542
|
+
PutBucketWebsiteCommand,
|
|
2543
|
+
} from "@aws-sdk/client-s3";
|
|
2544
|
+
import { SimAws } from "@kensio/yulin";
|
|
2545
|
+
|
|
2546
|
+
const simAws = new SimAws();
|
|
2547
|
+
const simS3 = simAws.s3();
|
|
2548
|
+
|
|
2549
|
+
await simS3.createBucket(
|
|
2550
|
+
new CreateBucketCommand({
|
|
2551
|
+
Bucket: "docs-site",
|
|
2552
|
+
}),
|
|
2553
|
+
);
|
|
2554
|
+
|
|
2555
|
+
await simS3.putBucketWebsite(
|
|
2556
|
+
new PutBucketWebsiteCommand({
|
|
2557
|
+
Bucket: "docs-site",
|
|
2558
|
+
WebsiteConfiguration: {
|
|
2559
|
+
IndexDocument: {
|
|
2560
|
+
Suffix: "index.html",
|
|
2561
|
+
},
|
|
2562
|
+
RoutingRules: [
|
|
2563
|
+
{
|
|
2564
|
+
Condition: {
|
|
2565
|
+
KeyPrefixEquals: "old/",
|
|
2566
|
+
},
|
|
2567
|
+
Redirect: {
|
|
2568
|
+
ReplaceKeyPrefixWith: "new/",
|
|
2569
|
+
},
|
|
2570
|
+
},
|
|
2571
|
+
{
|
|
2572
|
+
Condition: {
|
|
2573
|
+
HttpErrorCodeReturnedEquals: "404",
|
|
2574
|
+
},
|
|
2575
|
+
Redirect: {
|
|
2576
|
+
HttpRedirectCode: "302",
|
|
2577
|
+
ReplaceKeyWith: "not-found.html",
|
|
2578
|
+
},
|
|
2579
|
+
},
|
|
2580
|
+
],
|
|
2581
|
+
},
|
|
2582
|
+
}),
|
|
2583
|
+
);
|
|
2584
|
+
```
|
|
2585
|
+
|
|
2586
|
+
The first matching routing rule is used. A rule can match by `KeyPrefixEquals`, by
|
|
2587
|
+
`HttpErrorCodeReturnedEquals`, by both, or by no condition at all. Redirects support configured host, protocol,
|
|
2588
|
+
replacement key, replacement key prefix, and redirect status code.
|
|
2589
|
+
|
|
2590
|
+
## Filesystem-backed Bucket storage
|
|
2591
|
+
|
|
2592
|
+
By default, simulated S3 stores Objects in memory. For local development, you can mount a Bucket to a
|
|
2593
|
+
filesystem directory. This is handy for serving a static website on the local filesystem through
|
|
2594
|
+
simulated S3.
|
|
2595
|
+
|
|
2596
|
+
```typescript sim-s3-filesystem-storage
|
|
2597
|
+
/**
|
|
2598
|
+
* Local filesystem storage for simulated S3 Buckets.
|
|
2599
|
+
*/
|
|
2600
|
+
|
|
2601
|
+
import path from "node:path";
|
|
2602
|
+
|
|
2603
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
2604
|
+
import { SimAws } from "@kensio/yulin";
|
|
2605
|
+
|
|
2606
|
+
const simAws = new SimAws();
|
|
2607
|
+
const simS3 = simAws.s3();
|
|
2608
|
+
|
|
2609
|
+
await simS3.createBucket(
|
|
2610
|
+
new CreateBucketCommand({
|
|
2611
|
+
Bucket: "public-assets",
|
|
2612
|
+
}),
|
|
2613
|
+
);
|
|
2614
|
+
|
|
2615
|
+
simS3.mountBucketFilesystem(
|
|
2616
|
+
"public-assets",
|
|
2617
|
+
path.join(process.cwd(), "public"),
|
|
2618
|
+
);
|
|
2619
|
+
```
|
|
2620
|
+
|
|
2621
|
+
After mounting, Object reads and writes for that Bucket use the filesystem directory.
|
|
2622
|
+
|
|
2623
|
+
### Reloading the browser when the directory changes
|
|
2624
|
+
|
|
2625
|
+
The Bucket is reading the files, and a rebuild copies nothing into it. All that is left is telling
|
|
2626
|
+
the browser. Give the mount somewhere to reload and it watches the directory for you:
|
|
2627
|
+
|
|
2628
|
+
```typescript sim-s3-mount-reload
|
|
2629
|
+
/**
|
|
2630
|
+
* Reloading the browser when a build writes into a mounted directory.
|
|
2631
|
+
*/
|
|
2632
|
+
|
|
2633
|
+
import path from "node:path";
|
|
2634
|
+
|
|
2635
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
2636
|
+
import { SimAws } from "@kensio/yulin";
|
|
2637
|
+
import { serveSimAws } from "@kensio/yulin/serve";
|
|
2638
|
+
|
|
2639
|
+
const simAws = new SimAws();
|
|
2640
|
+
const srv = await serveSimAws({ simAws, port: 8787, liveReload: true });
|
|
2641
|
+
|
|
2642
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "site" }));
|
|
2643
|
+
|
|
2644
|
+
simAws.s3().mountBucketFilesystem("site", path.join(process.cwd(), "public"), {
|
|
2645
|
+
reload: srv,
|
|
2646
|
+
});
|
|
2647
|
+
```
|
|
2648
|
+
|
|
2649
|
+
A build writing a whole tree of files is one reload, and never one per file. The writes are held
|
|
2650
|
+
until they stop arriving. `settleMs` is how long that wait is, in milliseconds, for a generator that
|
|
2651
|
+
pauses part way through a build:
|
|
2652
|
+
|
|
2653
|
+
```typescript sim-s3-mount-reload-settle
|
|
2654
|
+
/**
|
|
2655
|
+
* Waiting longer for a slow build to finish writing.
|
|
2656
|
+
*/
|
|
2657
|
+
|
|
2658
|
+
import path from "node:path";
|
|
2659
|
+
|
|
2660
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
2661
|
+
import { SimAws } from "@kensio/yulin";
|
|
2662
|
+
import { serveSimAws } from "@kensio/yulin/serve";
|
|
2663
|
+
|
|
2664
|
+
const simAws = new SimAws();
|
|
2665
|
+
const srv = await serveSimAws({ simAws, liveReload: true });
|
|
2666
|
+
|
|
2667
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "site" }));
|
|
2668
|
+
|
|
2669
|
+
simAws.s3().mountBucketFilesystem("site", path.join(process.cwd(), "dist"), {
|
|
2670
|
+
reload: srv,
|
|
2671
|
+
settleMs: 500,
|
|
2672
|
+
});
|
|
2673
|
+
```
|
|
2674
|
+
|
|
2675
|
+
Anything with a `reload()` method will do, and a test can watch a mount without serving anything.
|
|
2676
|
+
|
|
2677
|
+
The watch is recursive, and holds an open filesystem handle that keeps the process alive. A dev
|
|
2678
|
+
process wants exactly that. Anything with an end, such as a test, calls
|
|
2679
|
+
`simAws.s3().stopWatchingMountedDirectories()` when it is done.
|
|
2680
|
+
`simAws.s3().watchedMountedDirectories()` says which directories are being watched.
|
|
2681
|
+
[`simAws.close()`](https://yulinsim.dev/serve/#stopping-and-restarting) is the one that names no service
|
|
2682
|
+
and no scope. It lets go of the mounted directory watches along with everything else the environment
|
|
2683
|
+
is holding, and a served environment gets that from `srv.close()`.
|
|
2684
|
+
|
|
2685
|
+
Under [`yulin watch`](https://yulinsim.dev/serve/#restarting-on-a-file-change), a mount that reloads for
|
|
2686
|
+
itself is left alone by the supervisor. A rebuild reloads the page rather than restarting the
|
|
2687
|
+
process and taking every simulated Bucket, Table and Stack with it. A mount without a reload target
|
|
2688
|
+
is still reported to the supervisor as a directory to watch, and a change in it restarts the
|
|
2689
|
+
process.
|
|
2690
|
+
|
|
2691
|
+
Filesystem storage is somewhat restrictive to make it slightly safer:
|
|
2692
|
+
|
|
2693
|
+
- The directory path must be absolute
|
|
2694
|
+
- The directory must not be the filesystem root
|
|
2695
|
+
- The directory must not be the user's home directory
|
|
2696
|
+
- The path must not contain `..`
|
|
2697
|
+
- Object keys must not be absolute paths or contain `..`
|
|
2698
|
+
- Only files whose extension is on a cautious list are served (see below)
|
|
2699
|
+
- Symlinks are ignored when listing Objects
|
|
2700
|
+
- Deletion is refused, and never unlinks a real file
|
|
2701
|
+
|
|
2702
|
+
`DeleteObject` against a filesystem-backed Bucket raises `NotImplemented`, and `DeleteObjects`
|
|
2703
|
+
reports the same code for every key. This is stricter than real S3, deliberately. The directory a
|
|
2704
|
+
Bucket is mounted on is an ordinary directory of yours, and removing files from it because a test
|
|
2705
|
+
called `DeleteObject` would be a poor default. Leave a Bucket on the default in-memory storage
|
|
2706
|
+
when a test needs deletion to work.
|
|
2707
|
+
|
|
2708
|
+
When reading files from filesystem-backed storage, Yulin infers common `content-type` metadata from
|
|
2709
|
+
file extensions such as `.html`, `.css`, `.js`, `.json`, `.png`, `.svg`, `.txt`, `.csv`, `.pdf`,
|
|
2710
|
+
`.xml`, and common font and image formats. A served file whose extension falls outside that set gets
|
|
2711
|
+
`binary/octet-stream`, as S3 reports for an Object whose type it was never told. That only comes up
|
|
2712
|
+
for an extension a mount named itself, below. No other file is served at all, with or without a
|
|
2713
|
+
type.
|
|
2714
|
+
|
|
2715
|
+
### Serving a file extension of your own
|
|
2716
|
+
|
|
2717
|
+
A mounted Bucket only serves files whose extension is on a cautious list (the web's own types, and
|
|
2718
|
+
nothing else) so that pointing a Bucket at a directory cannot be talked into reading whatever else
|
|
2719
|
+
happens to be in it. A file with any other extension goes unserved, and a `GetObject` for it comes
|
|
2720
|
+
back as though the file were absent. That is the right default and the wrong answer for a site with
|
|
2721
|
+
a data file of its own. A mount can name the extensions it needs:
|
|
2722
|
+
|
|
2723
|
+
```typescript sim-s3-mount-file-extensions
|
|
2724
|
+
/**
|
|
2725
|
+
* Serving a data file whose extension is not one of the web's own.
|
|
2726
|
+
*/
|
|
2727
|
+
|
|
2728
|
+
import path from "node:path";
|
|
2729
|
+
|
|
2730
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
2731
|
+
import { SimAws } from "@kensio/yulin";
|
|
2732
|
+
|
|
2733
|
+
const simAws = new SimAws();
|
|
2734
|
+
|
|
2735
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "site" }));
|
|
2736
|
+
|
|
2737
|
+
simAws.s3().mountBucketFilesystem("site", path.join(process.cwd(), "public"), {
|
|
2738
|
+
// A pinyin dictionary ships a binary frequency table beside its text files.
|
|
2739
|
+
additionalFileExtensions: [".freq"],
|
|
2740
|
+
});
|
|
2741
|
+
```
|
|
2742
|
+
|
|
2743
|
+
These are added to the list rather than replacing it, so naming one cannot cost you `.html`, and a
|
|
2744
|
+
leading dot is optional. Everything not named is still refused.
|
|
2745
|
+
|
|
2746
|
+
### Metadata a file cannot carry
|
|
2747
|
+
|
|
2748
|
+
A stored Object holds what S3 was told when it was written. A file holds its bytes and its name, so
|
|
2749
|
+
a mounted Bucket has only the extension to go on, and reports a `content-type` and no more.
|
|
2750
|
+
Anything a deployment would have set is either inherited from the deployment, below, or declared on
|
|
2751
|
+
the mount, for the Objects under a key prefix.
|
|
2752
|
+
|
|
2753
|
+
`ContentEncoding` is the one a site can be broken without. A directory of brotli files served with
|
|
2754
|
+
no `content-encoding` is bytes no browser can decode:
|
|
2755
|
+
|
|
2756
|
+
```typescript sim-s3-mount-system-metadata
|
|
2757
|
+
/**
|
|
2758
|
+
* Declaring the encoding of a compressed mirror in a mounted directory.
|
|
2759
|
+
*/
|
|
2760
|
+
|
|
2761
|
+
import path from "node:path";
|
|
2762
|
+
|
|
2763
|
+
import { CreateBucketCommand } from "@aws-sdk/client-s3";
|
|
2764
|
+
import { SimAws } from "@kensio/yulin";
|
|
2765
|
+
|
|
2766
|
+
const simAws = new SimAws();
|
|
2767
|
+
|
|
2768
|
+
await simAws.s3().createBucket(new CreateBucketCommand({ Bucket: "site" }));
|
|
2769
|
+
|
|
2770
|
+
simAws.s3().mountBucketFilesystem("site", path.join(process.cwd(), "public"), {
|
|
2771
|
+
// The mirrored copies keep their own names, so `br/js/app.js` is still typed
|
|
2772
|
+
// `text/javascript` from its extension. Nothing about the file says it is
|
|
2773
|
+
// compressed, which is what this declares.
|
|
2774
|
+
systemMetadata: [{ keyPrefix: "br/", metadata: { ContentEncoding: "br" } }],
|
|
2775
|
+
});
|
|
2776
|
+
```
|
|
2777
|
+
|
|
2778
|
+
The fields are the ones a [`PutObjectCommand`](#object-system-metadata) sets, and every value is a
|
|
2779
|
+
string, including `Expires`. Every declaration whose prefix the key starts with applies, in the
|
|
2780
|
+
order they were given, and a later one wins where two name the same header. An empty prefix is every
|
|
2781
|
+
Object in the Bucket. A declared `ContentType` replaces the one guessed from the extension.
|
|
2782
|
+
|
|
2783
|
+
### Inheriting what the deployment set
|
|
2784
|
+
|
|
2785
|
+
A mount rarely has to declare any of that, because something in the same simulated account has
|
|
2786
|
+
already said it. A CDK [`BucketDeployment`](https://yulinsim.dev/services/cloudformation/#cdk-s3-bucketdeployment)
|
|
2787
|
+
sets these headers through its own `SystemMetadata`, and says so on the destination Bucket as well as
|
|
2788
|
+
setting them on the Objects it copies. Mounting a directory over that Bucket replaces the Objects and
|
|
2789
|
+
inherits what the Bucket was told about them. The files on disk are then served as the deployed ones
|
|
2790
|
+
were:
|
|
2791
|
+
|
|
2792
|
+
```typescript sim-s3-mount-deployed-system-metadata
|
|
2793
|
+
/**
|
|
2794
|
+
* Serving a rebuilt directory as the deployment that filled the Bucket did.
|
|
2795
|
+
*/
|
|
2796
|
+
|
|
2797
|
+
import path from "node:path";
|
|
2798
|
+
|
|
2799
|
+
import { SimAws } from "@kensio/yulin";
|
|
2800
|
+
|
|
2801
|
+
const simAws = new SimAws();
|
|
2802
|
+
|
|
2803
|
+
// The Stack publishes the site. Its BucketDeployments say what they set, such
|
|
2804
|
+
// as `content-encoding: br` for the compressed mirror under `br/`.
|
|
2805
|
+
await simAws
|
|
2806
|
+
.cloudFormation()
|
|
2807
|
+
.deployTemplateFile("cdk.out/SiteStack.template.json");
|
|
2808
|
+
|
|
2809
|
+
// The Bucket then serves the generator's output as it is rebuilt. Nothing about
|
|
2810
|
+
// those files says how they were compressed, and nothing here has to either.
|
|
2811
|
+
simAws
|
|
2812
|
+
.s3()
|
|
2813
|
+
.mountBucketFilesystem("site-bucket", path.join(process.cwd(), "public"));
|
|
2814
|
+
```
|
|
2815
|
+
|
|
2816
|
+
The order is free. A directory can be mounted into a Bucket before the Stack describing it is
|
|
2817
|
+
deployed, and the mount answers with whatever the Bucket has been told by the time an Object is read.
|
|
2818
|
+
|
|
2819
|
+
What a deployment published is what it is sure of, and a file it copied is described exactly. A file
|
|
2820
|
+
a later build wrote is described by the rule the deployment would have published it under (its
|
|
2821
|
+
destination key prefix and its filters) as long as only one deployment claims it. Where two
|
|
2822
|
+
deployments into one Bucket could both have published a file that neither did, nothing is inherited
|
|
2823
|
+
for it. Serving a page as another deployment's brotli breaks it, where serving the file as it is on
|
|
2824
|
+
disk leaves it readable. Declare those on the mount.
|
|
2825
|
+
|
|
2826
|
+
Anything declared on the mount goes over the top of all of it, one header at a time. That is how a
|
|
2827
|
+
mount answers differently on purpose. A deployed site caching its assets for a year is the usual
|
|
2828
|
+
reason:
|
|
2829
|
+
|
|
2830
|
+
```typescript sim-s3-mount-override-system-metadata
|
|
2831
|
+
/**
|
|
2832
|
+
* Keeping a deployment's encoding while dropping its caching locally.
|
|
2833
|
+
*/
|
|
2834
|
+
|
|
2835
|
+
import path from "node:path";
|
|
2836
|
+
|
|
2837
|
+
import { SimAws } from "@kensio/yulin";
|
|
2838
|
+
|
|
2839
|
+
const simAws = new SimAws();
|
|
2840
|
+
|
|
2841
|
+
await simAws
|
|
2842
|
+
.cloudFormation()
|
|
2843
|
+
.deployTemplateFile("cdk.out/SiteStack.template.json");
|
|
2844
|
+
|
|
2845
|
+
simAws
|
|
2846
|
+
.s3()
|
|
2847
|
+
.mountBucketFilesystem("site-bucket", path.join(process.cwd(), "public"), {
|
|
2848
|
+
// `content-encoding` is still the deployment's, because this says nothing
|
|
2849
|
+
// about it. A year of caching is not what a rebuild wants reaching the
|
|
2850
|
+
// browser, so that one is answered here instead.
|
|
2851
|
+
systemMetadata: [{ keyPrefix: "", metadata: { CacheControl: "no-store" } }],
|
|
2852
|
+
});
|
|
2853
|
+
```
|
|
2854
|
+
|
|
2855
|
+
Pages served with [live reload](https://yulinsim.dev/serve/) are already sent `no-store`, and an HTML
|
|
2856
|
+
document is never what a stale cache is holding on to. Assets a build rewrites in place are, and a
|
|
2857
|
+
declaration like this one is what they need.
|
|
2858
|
+
|
|
2859
|
+
## Object system metadata
|
|
2860
|
+
|
|
2861
|
+
S3 keeps a handful of headers about an Object when it is written and hands them back on every read.
|
|
2862
|
+
Sim S3 stores and returns `cache-control`, `content-disposition`, `content-encoding`,
|
|
2863
|
+
`content-language`, `content-type` and `expires`, alongside a `content-length` describing the body
|
|
2864
|
+
being served.
|
|
2865
|
+
|
|
2866
|
+
`GetObjectCommand` and `HeadObjectCommand` answer with these in fields of their own (`ContentType`,
|
|
2867
|
+
`CacheControl` and the rest), the way real S3 does. `Metadata` carries the user-defined metadata a
|
|
2868
|
+
write attached, and nothing else.
|
|
2869
|
+
|
|
2870
|
+
Every path that serves an Object goes through the same mapping. The REST endpoint, the
|
|
2871
|
+
[website endpoint](#static-website-hosting) and a CloudFront S3 Origin all report the same headers
|
|
2872
|
+
for it. `content-encoding` is the one that matters most. Bytes served without it are bytes no client
|
|
2873
|
+
can decode, and an Object stored as brotli is only usable if the header comes back with it.
|
|
2874
|
+
|
|
2875
|
+
`PutObjectCommand` sets them, one request field per header.
|
|
2876
|
+
|
|
2877
|
+
```typescript sim-s3-object-system-metadata
|
|
2878
|
+
/**
|
|
2879
|
+
* Writing an Object with the system metadata S3 returns on a read.
|
|
2880
|
+
*/
|
|
2881
|
+
|
|
2882
|
+
import {
|
|
2883
|
+
CreateBucketCommand,
|
|
2884
|
+
GetObjectCommand,
|
|
2885
|
+
PutObjectCommand,
|
|
2886
|
+
} from "@aws-sdk/client-s3";
|
|
2887
|
+
import { SimAws } from "@kensio/yulin";
|
|
2888
|
+
|
|
2889
|
+
const simS3 = new SimAws().s3();
|
|
2890
|
+
|
|
2891
|
+
await simS3.createBucket(new CreateBucketCommand({ Bucket: "site" }));
|
|
2892
|
+
|
|
2893
|
+
await simS3.putObject(
|
|
2894
|
+
new PutObjectCommand({
|
|
2895
|
+
Bucket: "site",
|
|
2896
|
+
Key: "app.js",
|
|
2897
|
+
Body: "compressed bytes",
|
|
2898
|
+
CacheControl: "public, max-age=31536000, immutable",
|
|
2899
|
+
ContentDisposition: 'inline; filename="app.js"',
|
|
2900
|
+
ContentEncoding: "br",
|
|
2901
|
+
ContentLanguage: "en-GB",
|
|
2902
|
+
ContentType: "text/javascript",
|
|
2903
|
+
Expires: new Date("2027-01-02T03:04:05Z"),
|
|
2904
|
+
}),
|
|
2905
|
+
);
|
|
2906
|
+
|
|
2907
|
+
const objectOut = await simS3.getObject(
|
|
2908
|
+
new GetObjectCommand({ Bucket: "site", Key: "app.js" }),
|
|
2909
|
+
);
|
|
2910
|
+
|
|
2911
|
+
// Each header comes back in the field a read describes an Object with.
|
|
2912
|
+
console.log(objectOut.ContentEncoding); // br
|
|
2913
|
+
console.log(objectOut.ExpiresString); // Sat, 02 Jan 2027 03:04:05 GMT
|
|
2914
|
+
```
|
|
2915
|
+
|
|
2916
|
+
A header the write says nothing about is left unset, and never stored empty, so a read leaves it
|
|
2917
|
+
out. Content type is the exception. S3 gives an Object one whether the write named it or not, and a
|
|
2918
|
+
read of an Object written without one reports `binary/octet-stream`.
|
|
2919
|
+
|
|
2920
|
+
`Expires` is the one field that takes something other than a string. The SDK takes a `Date` on the
|
|
2921
|
+
way in. A read hands back the stored HTTP date as `ExpiresString`, alongside the same value parsed
|
|
2922
|
+
into a `Date` as `Expires`.
|
|
2923
|
+
|
|
2924
|
+
A CDK BucketDeployment's `SystemMetadata` sets the same headers on every Object it copies. See
|
|
2925
|
+
[CDK S3 BucketDeployment](https://yulinsim.dev/services/cloudformation/#cdk-s3-bucketdeployment). A
|
|
2926
|
+
[mounted directory](#metadata-a-file-cannot-carry) declares them for the Objects under a key prefix,
|
|
2927
|
+
since a file on disk carries none of them.
|
|
2928
|
+
|
|
2929
|
+
## Standalone SimS3
|
|
2930
|
+
|
|
2931
|
+
If you only need S3 alone, you can instantiate `SimS3` directly.
|
|
2932
|
+
|
|
2933
|
+
```typescript sim-s3-standalone
|
|
2934
|
+
/**
|
|
2935
|
+
* Standalone simulated S3 instance.
|
|
2936
|
+
*/
|
|
2937
|
+
|
|
2938
|
+
import { CreateBucketCommand, PutObjectCommand } from "@aws-sdk/client-s3";
|
|
2939
|
+
import { SimS3 } from "@kensio/yulin/s3";
|
|
2940
|
+
|
|
2941
|
+
const simS3 = new SimS3();
|
|
2942
|
+
|
|
2943
|
+
await simS3.createBucket(
|
|
2944
|
+
new CreateBucketCommand({
|
|
2945
|
+
Bucket: "standalone-bucket",
|
|
2946
|
+
}),
|
|
2947
|
+
);
|
|
2948
|
+
|
|
2949
|
+
await simS3.putObject(
|
|
2950
|
+
new PutObjectCommand({
|
|
2951
|
+
Bucket: "standalone-bucket",
|
|
2952
|
+
Key: "hello.txt",
|
|
2953
|
+
Body: "Hello from standalone SimS3",
|
|
2954
|
+
}),
|
|
2955
|
+
);
|
|
2956
|
+
```
|
|
2957
|
+
|
|
2958
|
+
A standalone `SimS3` instance has its own isolated state, with no wider `SimAws` environment behind
|
|
2959
|
+
it.
|
|
2960
|
+
|
|
2961
|
+
## Available functionality
|
|
2962
|
+
|
|
2963
|
+
Sim S3 currently supports:
|
|
2964
|
+
|
|
2965
|
+
- `CreateBucketCommand` and `ListBucketsCommand`
|
|
2966
|
+
- `HeadObjectCommand` and `HeadBucketCommand`, describing an Object or a Bucket without reading it
|
|
2967
|
+
- `PutObjectCommand`, `GetObjectCommand`, `ListObjectsV2Command` and `ListObjectsCommand`, with an
|
|
2968
|
+
ETag and a last-modified time on every Object
|
|
2969
|
+
- `Delimiter` on a listing, rolling keys up into `CommonPrefixes` so a Bucket can be walked as a
|
|
2970
|
+
folder tree, over the SDK and over a served endpoint
|
|
2971
|
+
- `CreateMultipartUploadCommand`, `UploadPartCommand`, `CompleteMultipartUploadCommand`,
|
|
2972
|
+
`AbortMultipartUploadCommand`, `ListMultipartUploadsCommand` and `ListPartsCommand`, so `aws s3 cp`
|
|
2973
|
+
and `@aws-sdk/lib-storage` can upload a file of real size
|
|
2974
|
+
- `Range` on `GetObjectCommand`, answering with the bytes asked for and `206 Partial Content` over a
|
|
2975
|
+
served endpoint, so `aws s3 cp` downloads a file of real size unchanged
|
|
2976
|
+
- `CopyObjectCommand`, authorized as a read of the source and a write of the destination, with a
|
|
2977
|
+
`MetadataDirective` deciding which metadata the copy carries, over the SDK and over a served
|
|
2978
|
+
endpoint, letting `aws s3 cp` and `aws s3 mv` move an Object between two served Buckets
|
|
2979
|
+
- `DeleteObjectCommand` and `DeleteObjectsCommand`, authorized per Object by sim IAM
|
|
2980
|
+
- `PutBucketNotificationConfigurationCommand` and `GetBucketNotificationConfigurationCommand`, with
|
|
2981
|
+
Object events delivered to a simulated Lambda function, a simulated SQS queue or a simulated SNS
|
|
2982
|
+
topic
|
|
2983
|
+
- `PutBucketWebsiteCommand`, for static website hosting
|
|
2984
|
+
- `PutBucketLifecycleConfigurationCommand`, `GetBucketLifecycleConfigurationCommand` and
|
|
2985
|
+
`DeleteBucketLifecycleCommand`, storing a Bucket's lifecycle rules and handing them back without
|
|
2986
|
+
expiring or transitioning any Object against them
|
|
2987
|
+
- `PutBucketPolicyCommand`, `GetBucketPolicyCommand` and `DeleteBucketPolicyCommand`, evaluated by
|
|
2988
|
+
sim IAM alongside identity policies
|
|
2989
|
+
- The `AWS::S3::Bucket` and `AWS::S3::BucketPolicy` CloudFormation resources
|
|
2990
|
+
- Block Public Access, on by default as in real S3, refusing a public Bucket policy unless the Bucket
|
|
2991
|
+
opts out with `PutPublicAccessBlockCommand` or `PublicAccessBlockConfiguration`
|
|
2992
|
+
- Serving static website requests on localhost with `serveSimAws`
|
|
2993
|
+
- Serving Object `GET`, `HEAD`, `PUT` and `DELETE` over the S3 REST endpoint, authorized by sim IAM,
|
|
2994
|
+
and the `?uploads` and `?uploadId` sub-resources a multipart upload is made of
|
|
2995
|
+
- Presigned URLs built by the real `@aws-sdk/s3-request-presigner`, with expiry in simulated time
|
|
2996
|
+
- Object system metadata set by a `PutObjectCommand` and returned on a read, over every endpoint
|
|
2997
|
+
that serves an Object
|
|
2998
|
+
- Bucket website index documents, error documents, trailing-slash redirects, redirect-all
|
|
2999
|
+
configuration, and routing-rule redirects
|
|
3000
|
+
- Bucket-global uniqueness within a `SimAws` instance across simulated Accounts and Regions
|
|
3001
|
+
- In-memory Object storage by default
|
|
3002
|
+
- Optional filesystem-backed Bucket storage with `mountBucketFilesystem(...)`, watching the mounted
|
|
3003
|
+
directory and reloading connected browsers when it is rebuilt, and reporting the system metadata a
|
|
3004
|
+
CDK `BucketDeployment` into the same Bucket published, alongside anything the mount declares for a
|
|
3005
|
+
key prefix itself
|
|
3006
|
+
|
|
3007
|
+
The simulator aims at useful behaviour for tests and local development, short of full S3 feature
|
|
3008
|
+
parity. Unsupported S3 options may be ignored or may throw errors depending on whether the simulator
|
|
3009
|
+
needs them to model the requested behaviour.
|
|
3010
|
+
|
|
3011
|
+
## Limitations
|
|
3012
|
+
|
|
3013
|
+
These apply across the page. The sections above each list what is specific to them.
|
|
3014
|
+
|
|
3015
|
+
- Object versioning is left out. There are no version ids, no delete markers and no `VersionId` on
|
|
3016
|
+
any request or response.
|
|
3017
|
+
- A listing reports `StorageClass` as `STANDARD` for every Object. Storage classes themselves are
|
|
3018
|
+
left out, and every Object is in that one.
|
|
3019
|
+
- `EncodingType` is ignored on a listing, and keys come back unencoded.
|
|
3020
|
+
- Object tags, ACLs, replication and server-side encryption are left out. A lifecycle rule expires
|
|
3021
|
+
Objects and abandons uploads, and transitions nothing between storage classes. See
|
|
3022
|
+
[Lifecycle configuration](#lifecycle-configuration).
|
|
3023
|
+
- A Bucket using filesystem-backed storage cannot delete Objects, and raises no event
|
|
3024
|
+
notifications, because it swaps the whole storage backend in place of putting Objects.
|
|
3025
|
+
- An upload over the S3 REST endpoint keeps its `content-type` and no other system metadata, leaving
|
|
3026
|
+
a presigned `PUT` unable to set the rest. A `PutObjectCommand` through the SDK keeps all of them.
|
|
3027
|
+
- A presigned `GetObject` ignores the `response-content-type`, `response-cache-control` and other
|
|
3028
|
+
`response-*` parameters that override a response header in real S3. An Object is served with the
|
|
3029
|
+
system metadata it was written with.
|