@63klabs/cache-data 1.2.5 → 1.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +28 -1
- package/README.md +52 -1230
- package/SECURITY.md +1 -1
- package/docs/00-example-implementation/README.md +55 -0
- package/docs/00-example-implementation/example-buildspec.yml +15 -0
- package/docs/00-example-implementation/example-config.js +0 -0
- package/docs/00-example-implementation/example-handler.js +0 -0
- package/docs/00-example-implementation/example-template-lambda-function.yml +119 -0
- package/docs/00-example-implementation/example-template-parameters.yml +66 -0
- package/docs/00-example-implementation/example-template-s3-and-dynamodb-cache-store.yml +77 -0
- package/docs/00-example-implementation/generate-put-ssm.py +209 -0
- package/docs/00-example-implementation/template-configuration.json +14 -0
- package/docs/00-quick-start-implementation/README.md +615 -0
- package/docs/01-advanced-implementation-for-web-service/README.md +13 -0
- package/docs/README.md +9 -0
- package/docs/features/README.md +5 -0
- package/docs/features/cache/README.md +3 -0
- package/docs/features/endpoint/README.md +3 -0
- package/docs/features/tools/README.md +341 -0
- package/docs/lambda-optimization/README.md +178 -0
- package/package.json +4 -6
- package/src/lib/dao-cache.js +82 -28
package/SECURITY.md
CHANGED
|
@@ -2,4 +2,4 @@
|
|
|
2
2
|
|
|
3
3
|
## Reporting a Vulnerability
|
|
4
4
|
|
|
5
|
-
Report all vulnerabilities under the [Security menu in the Cache-Data GitHub repository](https://github.com/63klabs/
|
|
5
|
+
Report all vulnerabilities under the [Security menu in the Cache-Data GitHub repository](https://github.com/63klabs/cache-data/security/advisories).
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Implementation Examples
|
|
2
|
+
|
|
3
|
+
> Note: To implement Cache Data, a DynamoDb and S3 bucket needs to be created to store the cache. See the [Cache-Data DynamoDb and S3 CloudFormation Resource Templates](#cache-data-dynamodb-and-s3-cloudformation-resource-templates) section below.
|
|
4
|
+
|
|
5
|
+
A full implementation example and tutorial is available through the [Atlantis Tutorials repository](https://github.com/63klabs/atlantis-tutorials) as [Tutorial #2](https://github.com/63Klabs/atlantis-tutorials/blob/main/tutorials/02-s3-dynamo-api-gateway-lambda-cache-data-node/README.md). (Atlantis is a collection of templates and deployment scripts to assist in starting and automating serverless deployments using AWS SAM and CloudFormation.)
|
|
6
|
+
|
|
7
|
+
## CloudFormation Template Examples
|
|
8
|
+
|
|
9
|
+
### Cache-Data DynamoDb and S3 CloudFormation Resource Example Templates
|
|
10
|
+
|
|
11
|
+
Use the [S3 and DynamoDb example template](./example-template-s3-and-dynamodb-cache-store.yml) for creating your DynamoDb and S3 cache storage locations.
|
|
12
|
+
|
|
13
|
+
You can create a centrally-shared DynamoDb table and S3 location for use among all your instances (each instance has its own encryption key to prevent cache-sharing/exposure).
|
|
14
|
+
|
|
15
|
+
You can also add the `CacheDataDynamoDbTable` and `CacheDataS3Bucket` to each application template individually. However, this will quickly increase the number of S3 buckets and DynamoDb tables to keep track of.
|
|
16
|
+
|
|
17
|
+
### Lambda Function Resource Template Example
|
|
18
|
+
|
|
19
|
+
Use the [Lambda Example Template](./example-template-lambda-function.yml) for properties, environment variables, and execution permissions to use in your Lambda function.
|
|
20
|
+
|
|
21
|
+
The Cache Data packages utilizes environment variables for cache, logging, and other configuration settings.
|
|
22
|
+
|
|
23
|
+
Conditionals or parameters can be utilized to provide deployment context switching instead of hard-coding values.
|
|
24
|
+
|
|
25
|
+
In order to utilize X-Ray tracing, Lambda Insights, and to write and read data from the DynamoDb and S3 data stores, IAM permissions must be provided in a Lambda execution role.
|
|
26
|
+
|
|
27
|
+
### Cache-Data Parameter Examples
|
|
28
|
+
|
|
29
|
+
If you want to pass cache-data settings as CloudFormation template parameters during deployments, see [Cache-Data Template Parameter Examples](./example-template-parameters.yml).
|
|
30
|
+
|
|
31
|
+
Even if not using the parameters, the parameter example also provides descriptions, examples, and value requirements to further understand how to use the settings.
|
|
32
|
+
|
|
33
|
+
## CodeBuild Script Examples
|
|
34
|
+
|
|
35
|
+
Cache-Data requires a secret key to encrypt cached data in S3 and DynamoDb.
|
|
36
|
+
|
|
37
|
+
This key is stored in SSM Parameter store and can be generated using [the generate-put-ssm.py build script](./generate-put-ssm.py)
|
|
38
|
+
|
|
39
|
+
The script can be used to store additional parameters at build time. Review comments in the script for usage information.
|
|
40
|
+
|
|
41
|
+
The script will also look for a [template-configuration.json](./template-configuration.json) file in the script or parent directory and apply supplied tags to the parameter. If placeholders are used in tag values (for example `$VARIBLE_NAME$`) it is replaced with the corresponding environment variable.
|
|
42
|
+
|
|
43
|
+
Template configuration files are often used alongside buildspec.yml files for supplying `Parameters` and `Tags` to SAM deployments.
|
|
44
|
+
|
|
45
|
+
To implement the generate-put-ssm.py script during your CodeBuild process, you can refer to the [example buildspec.yml](./example-buildspec.yml).
|
|
46
|
+
|
|
47
|
+
## Code Examples
|
|
48
|
+
|
|
49
|
+
### Configuration
|
|
50
|
+
|
|
51
|
+
[Configuration Example Code](./example-config.js)
|
|
52
|
+
|
|
53
|
+
### Lambda Handler
|
|
54
|
+
|
|
55
|
+
[Lambda Handler Example Code](./example-handler.js)
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
version: 0.2
|
|
2
|
+
|
|
3
|
+
phases:
|
|
4
|
+
|
|
5
|
+
pre_build:
|
|
6
|
+
commands:
|
|
7
|
+
|
|
8
|
+
# Generate a random 256 bit key (hexidecimal) in SSM for cache data:
|
|
9
|
+
- generate-put-ssm.py /webservice/app-name/CacheData_SecureDataKey --generate 256
|
|
10
|
+
|
|
11
|
+
# Generate a parameter with a preset value:
|
|
12
|
+
- generate-put-ssm.py /webservice/app-name/HelloWorld --value "some-value"
|
|
13
|
+
|
|
14
|
+
# Generate a parameter with value of 'BLANK' to fill in later:
|
|
15
|
+
- generate-put-ssm.py /webservice/app-name/WeatherServiceApiKey
|
|
File without changes
|
|
File without changes
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# template.yml
|
|
2
|
+
# This is a template for a Lambda function that uses the 63klabs/cache-data package
|
|
3
|
+
|
|
4
|
+
Resources:
|
|
5
|
+
|
|
6
|
+
# API Gateway
|
|
7
|
+
|
|
8
|
+
WebApi:
|
|
9
|
+
Type: AWS::Serverless::Api
|
|
10
|
+
Properties:
|
|
11
|
+
PropagateTags: True
|
|
12
|
+
TracingEnabled: True
|
|
13
|
+
OpenApiVersion: 3.0.0
|
|
14
|
+
|
|
15
|
+
# Lambda Function
|
|
16
|
+
|
|
17
|
+
AppFunction:
|
|
18
|
+
Type: AWS::Serverless::Function
|
|
19
|
+
Properties:
|
|
20
|
+
# ...
|
|
21
|
+
Runtime: nodejs22.x
|
|
22
|
+
MemorySize: 1028
|
|
23
|
+
Role: !GetAtt LambdaExecutionRole.Arn
|
|
24
|
+
|
|
25
|
+
# Lambda Insights and X-Ray
|
|
26
|
+
Tracing: "Active" # X-Ray
|
|
27
|
+
|
|
28
|
+
Layers:
|
|
29
|
+
# Lambda Insights and Param Secrets - Account and Version are Mapped in as they vary by region and architecture
|
|
30
|
+
- !Sub "arn:aws:lambda:${AWS::Region}:${AWSInsightsAccount}:layer:LambdaInsightsExtension:${Version}"
|
|
31
|
+
- !Sub "arn:aws:lambda:${AWS::Region}:${AWSParamAccount}:layer:AWS-Parameters-and-Secrets-Lambda-Extension:${Version}"
|
|
32
|
+
|
|
33
|
+
Environment:
|
|
34
|
+
Variables:
|
|
35
|
+
|
|
36
|
+
DEPLOY_ENVIRONMENT: "DEV" # PROD, TEST, DEV - a different category of environment
|
|
37
|
+
LOG_LEVEL: 5 # 0 for prod, 2-5 for non-prod
|
|
38
|
+
|
|
39
|
+
# Cache-Data settings - Parameters may be used instead of hard-coded values - see docs/00-example-implementation/example-template-parameters.yml
|
|
40
|
+
CACHE_DATA_DYNAMO_DB_TABLE: your-dynamodb-table-name # The DynamoDb table name to use for caching data
|
|
41
|
+
CACHE_DATA_S3_BUCKET: your-s3-bucket-name # The S3 bucket name to use for caching data
|
|
42
|
+
CACHE_DATA_SECURE_DATA_ALGORITHM: "aes-256-cbc" # The cryptographic algorithm to use for storing sensitive cached data in S3 and DynamoDb
|
|
43
|
+
CACHE_DATA_ID_HASH_ALGORITHM: "RSA-SHA256" # The hash algorithm to use for generating the URI ID to identify cached requests
|
|
44
|
+
CACHE_DATA_DYNAMO_DB_MAX_CACHE_SIZE_KB: 10 # The cut-off in KB that large objects should be stored in S3 instead of DynamoDb
|
|
45
|
+
CACHE_DATA_PURGE_EXPIRED_CACHE_ENTRIES_AFTER_X_HRS: 24 # The number of hours expired cached data should be kept before purging
|
|
46
|
+
CACHE_DATA_ERROR_EXP_IN_SECONDS: 300 # How long should errors be cached? This prevents retrying a service that is currently in error too often
|
|
47
|
+
CACHE_DATA_TIME_ZONE_FOR_INTERVAL: "Etc/UTC" # Cache-Data may expire using an interval such as every four, six, twelve, ... hours on the hour starting at midnight. What timezone holds the midnight to calculate from?
|
|
48
|
+
CACHE_DATA_AWS_X_RAY_ON: true # Enable X-Ray tracing for Cache-Data
|
|
49
|
+
CACHE_DATA_USE_TOOLS_HASH_METHOD: true # Use the tools.hash method for generating the URI ID to identify cached requests (default is false)
|
|
50
|
+
|
|
51
|
+
# -- LambdaFunction Execution Role --
|
|
52
|
+
|
|
53
|
+
LambdaExecutionRole:
|
|
54
|
+
Type: AWS::IAM::Role
|
|
55
|
+
Properties:
|
|
56
|
+
RoleName: !Sub "${LAMBDA_EXECUTION_ROLE_NAME}-ExecutionRole"
|
|
57
|
+
Description: "IAM Role that allows the Lambda permission to execute and access resources"
|
|
58
|
+
Path: /
|
|
59
|
+
|
|
60
|
+
AssumeRolePolicyDocument:
|
|
61
|
+
Statement:
|
|
62
|
+
- Effect: Allow
|
|
63
|
+
Principal:
|
|
64
|
+
Service: [lambda.amazonaws.com]
|
|
65
|
+
Action: sts:AssumeRole
|
|
66
|
+
|
|
67
|
+
# These are for application monitoring via LambdaInsights and X-Ray
|
|
68
|
+
ManagedPolicyArns:
|
|
69
|
+
- 'arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy'
|
|
70
|
+
- 'arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess'
|
|
71
|
+
|
|
72
|
+
# These are the resources your Lambda function needs access to
|
|
73
|
+
# Logs, SSM Parameters, DynamoDb, S3, etc.
|
|
74
|
+
# Define specific actions such as get/put (read/write)
|
|
75
|
+
Policies:
|
|
76
|
+
- PolicyName: LambdaResourceAccessPolicies
|
|
77
|
+
PolicyDocument:
|
|
78
|
+
Statement:
|
|
79
|
+
|
|
80
|
+
- Sid: LambdaAccessToWriteLogs
|
|
81
|
+
Action:
|
|
82
|
+
- logs:CreateLogGroup
|
|
83
|
+
- logs:CreateLogStream
|
|
84
|
+
- logs:PutLogEvents
|
|
85
|
+
Effect: Allow
|
|
86
|
+
Resource: !GetAtt AppLogGroup.Arn
|
|
87
|
+
|
|
88
|
+
# cache-data Parameter Read Access (from: https://www.npmjs.com/package/@63klabs/cache-data)
|
|
89
|
+
- Sid: LambdaAccessToSSMParameters
|
|
90
|
+
Action:
|
|
91
|
+
- ssm:DescribeParameters
|
|
92
|
+
- ssm:GetParameters
|
|
93
|
+
- ssm:GetParameter
|
|
94
|
+
- ssm:GetParametersByPath
|
|
95
|
+
Effect: Allow
|
|
96
|
+
Resource:
|
|
97
|
+
- !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter${ParameterPath}*"
|
|
98
|
+
|
|
99
|
+
# cache-data S3 bucket (from: https://www.npmjs.com/package/@63klabs/cache-data)
|
|
100
|
+
- Sid: LambdaAccessToS3BucketCacheData
|
|
101
|
+
Action:
|
|
102
|
+
- s3:PutObject
|
|
103
|
+
- s3:GetObject
|
|
104
|
+
- s3:GetObjectVersion
|
|
105
|
+
Effect: Allow
|
|
106
|
+
Resource: !Join [ '', [ !GetAtt CacheDataS3Bucket.Arn, '/cache/*' ] ]
|
|
107
|
+
|
|
108
|
+
# cache-data DynamoDb table (from: https://www.npmjs.com/package/@63klabs/cache-data)
|
|
109
|
+
- Sid: LambdaAccessToDynamoDBTableCacheData
|
|
110
|
+
Action:
|
|
111
|
+
- dynamodb:GetItem
|
|
112
|
+
- dynamodb:Scan
|
|
113
|
+
- dynamodb:Query
|
|
114
|
+
- dynamodb:BatchGetItem
|
|
115
|
+
- dynamodb:PutItem
|
|
116
|
+
- dynamodb:UpdateItem
|
|
117
|
+
- dynamodb:BatchWriteItem
|
|
118
|
+
Effect: Allow
|
|
119
|
+
Resource: !GetAtt CacheDataDynamoDbTable.Arn
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# The following parameters can be used to customize the Cache-Data implementation on a per-deployment basis.
|
|
2
|
+
# They can then be used to set the Lambda environment variables in the template.
|
|
3
|
+
# However, you can just as easily set these directly in the Lambda environment variables
|
|
4
|
+
|
|
5
|
+
Parameters:
|
|
6
|
+
|
|
7
|
+
# ...
|
|
8
|
+
|
|
9
|
+
# Cache-Data Parameters
|
|
10
|
+
|
|
11
|
+
CacheDataDbMaxCacheSizeInKB:
|
|
12
|
+
Type: Number
|
|
13
|
+
Description: "DynamoDb does better when storing smaller pieces of data. Choose the cut-off in KB that large objects should be stored in S3 instead (10)"
|
|
14
|
+
Default: 10
|
|
15
|
+
MinValue: 10
|
|
16
|
+
MaxValue: 200
|
|
17
|
+
ConstraintDescription: "Numeric value between 10 and 200 (inclusive)"
|
|
18
|
+
|
|
19
|
+
CacheDataCryptIdHashAlgorithm:
|
|
20
|
+
Type: String
|
|
21
|
+
Description: "Hash algorithm used for generating the URI ID to identify cached requests. This is for generating IDs, not crypto."
|
|
22
|
+
Default: "RSA-SHA256"
|
|
23
|
+
AllowedValues: ["RSA-SHA256", "RSA-SHA3-224", "RSA-SHA3-256", "RSA-SHA3-384", "RSA-SHA3-512"]
|
|
24
|
+
ConstraintDescription: "Use possible hashes available from Node.js in the RSA- category (RSA-SHA256 to RSA-SM3)"
|
|
25
|
+
|
|
26
|
+
CacheDataCryptSecureDataAlg:
|
|
27
|
+
Type: String
|
|
28
|
+
Description: "Cryptographic algorithm to use for storing sensitive cached data in S3 and DynamoDb"
|
|
29
|
+
Default: "aes-256-cbc"
|
|
30
|
+
AllowedValues: ["aes-256-cbc", "aes-256-cfb", "aes-256-cfb1", "aes-256-cfb8", "aes-256-ofb"]
|
|
31
|
+
ConstraintDescription: "Use possible cipher algorithms available (crypto.getCiphers()) from Node.js in the aes-256-xxx category"
|
|
32
|
+
|
|
33
|
+
CacheDataErrorExpirationInSeconds:
|
|
34
|
+
Type: Number
|
|
35
|
+
Description: "How long should errors be cached? This prevents retrying a service that is currenlty in error too often (300 is recommended)"
|
|
36
|
+
Default: 300
|
|
37
|
+
MinValue: 1
|
|
38
|
+
ConstraintDescription: "Choose a value of 1 or greater"
|
|
39
|
+
|
|
40
|
+
CacheDataPurgeExpiredCacheEntriesInHours:
|
|
41
|
+
Type: Number
|
|
42
|
+
Description: "The number of hours expired cached data should be kept before purging. Expired cache data may be used if the source returns an error."
|
|
43
|
+
Default: 24
|
|
44
|
+
MinValue: 1
|
|
45
|
+
ConstraintDescription: "Choose a value of 1 or greater"
|
|
46
|
+
|
|
47
|
+
CacheDataPurgeAgeOfCachedBucketObjInDays:
|
|
48
|
+
Type: Number
|
|
49
|
+
Description: "Similar to CacheData_PurgeExpiredCacheEntriesInHours, but for the S3 Bucket. S3 calculates from time object is created/last modified (not accessed). This should be longer than your longest cache expiration set in custom/policies. Keeping objects in S3 for too long increases storage costs. (30 is recommended)"
|
|
50
|
+
Default: 15
|
|
51
|
+
MinValue: 3
|
|
52
|
+
ConstraintDescription: "Choose a value of 3 days or greater. This should be slightly longer than the longest cache expiration expected"
|
|
53
|
+
|
|
54
|
+
CacheDataTimeZoneForInterval:
|
|
55
|
+
Type: String
|
|
56
|
+
Description: "Cache-Data may expire using an interval such as every four, six, twelve, ... hours on the hour starting at midnight. What timezone holds the midnight to calculate from?"
|
|
57
|
+
Default: "Etc/UTC"
|
|
58
|
+
AllowedValues: ["Etc/UTC", "America/Puerto_Rico", "America/New_York", "America/Indianapolis", "America/Chicago", "America/Denver", "America/Phoenix", "America/Los_Angeles", "America/Anchorage", "Pacific/Honolulu"] # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
|
|
59
|
+
ConstraintDescription: "Common examples for United States of America. Accepted values can be changed in the template for your region."
|
|
60
|
+
|
|
61
|
+
CacheDataAWSXRayOn:
|
|
62
|
+
Type: String
|
|
63
|
+
Description: "Turn on AWS XRay tracing for Cache-Data"
|
|
64
|
+
Default: "false"
|
|
65
|
+
AllowedValues: ["true", "false"]
|
|
66
|
+
ConstraintDescription: "Accepted values are true or false"
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
Resources:
|
|
2
|
+
|
|
3
|
+
# ... all your other resources
|
|
4
|
+
# ... make sure your Lambda function has between 512MB and 1024MB allocated (256MB minimum)
|
|
5
|
+
# ... also make sure you added environment variables to your Lambda function
|
|
6
|
+
# ... and make sure your Lambda Execution Role grants access to your DynamoDb and S3 buckets
|
|
7
|
+
|
|
8
|
+
# ---------------------------------------------------------------------------
|
|
9
|
+
# Cache-Data
|
|
10
|
+
# From: https://www.npmjs.com/package/@63klabs/cache-data
|
|
11
|
+
# Your Lambda function will need access via the Execution Role
|
|
12
|
+
|
|
13
|
+
# -- Cache-Data DynamoDb Table --
|
|
14
|
+
|
|
15
|
+
CacheDataDynamoDbTable:
|
|
16
|
+
Type: AWS::DynamoDB::Table
|
|
17
|
+
Description: Table to store Cache-Data.
|
|
18
|
+
Properties:
|
|
19
|
+
TableName: !Sub '${YOUR-DYNAMODB-TABLE}-CacheData'
|
|
20
|
+
AttributeDefinitions:
|
|
21
|
+
- AttributeName: "id_hash"
|
|
22
|
+
AttributeType: "S"
|
|
23
|
+
KeySchema:
|
|
24
|
+
- AttributeName: "id_hash"
|
|
25
|
+
KeyType: "HASH"
|
|
26
|
+
TimeToLiveSpecification:
|
|
27
|
+
AttributeName: "purge_ts"
|
|
28
|
+
Enabled: true
|
|
29
|
+
BillingMode: "PAY_PER_REQUEST"
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
# -- Cache-Data S3 Bucket --
|
|
33
|
+
|
|
34
|
+
CacheDataS3Bucket:
|
|
35
|
+
Type: AWS::S3::Bucket
|
|
36
|
+
Description: S3 Bucket to store Cache-Data too big for DynamoDb. Cache-Data stores objects in /cache directory. The application may store additional data outside of the cache directory.
|
|
37
|
+
Properties:
|
|
38
|
+
BucketName: !Sub "${YOUR-BUCKET-NAME}-cachedata"
|
|
39
|
+
PublicAccessBlockConfiguration:
|
|
40
|
+
BlockPublicAcls: true
|
|
41
|
+
BlockPublicPolicy: true
|
|
42
|
+
IgnorePublicAcls: true
|
|
43
|
+
RestrictPublicBuckets: true
|
|
44
|
+
BucketEncryption:
|
|
45
|
+
ServerSideEncryptionConfiguration:
|
|
46
|
+
- ServerSideEncryptionByDefault:
|
|
47
|
+
SSEAlgorithm: AES256
|
|
48
|
+
LifecycleConfiguration:
|
|
49
|
+
Rules:
|
|
50
|
+
- Id: "ExpireObjects"
|
|
51
|
+
AbortIncompleteMultipartUpload:
|
|
52
|
+
DaysAfterInitiation: 1
|
|
53
|
+
ExpirationInDays: !Ref CacheDataPurgeAgeOfCachedBucketObjInDays
|
|
54
|
+
Prefix: "cache" # this will limit this policy to YOURBUCKETNAME/cache/*
|
|
55
|
+
NoncurrentVersionExpirationInDays: !Ref CacheDataPurgeAgeOfCachedBucketObjInDays
|
|
56
|
+
Status: "Enabled" # Enable only if you are going to use this LifecycleConfiguration
|
|
57
|
+
|
|
58
|
+
# -- Cache-Data S3 Bucket Policy --
|
|
59
|
+
|
|
60
|
+
CacheDataS3BucketPolicy:
|
|
61
|
+
Type: AWS::S3::BucketPolicy
|
|
62
|
+
Properties:
|
|
63
|
+
Bucket: !Ref CacheDataS3Bucket
|
|
64
|
+
PolicyDocument:
|
|
65
|
+
Version: "2012-10-17"
|
|
66
|
+
Id: SecurityPolicy
|
|
67
|
+
Statement:
|
|
68
|
+
- Sid: "DenyNonSecureTransportAccess"
|
|
69
|
+
Effect: Deny
|
|
70
|
+
Principal: "*"
|
|
71
|
+
Action: "s3:*"
|
|
72
|
+
Resource:
|
|
73
|
+
- !GetAtt CacheDataS3Bucket.Arn
|
|
74
|
+
- !Join [ '', [ !GetAtt CacheDataS3Bucket.Arn, '/*' ] ]
|
|
75
|
+
Condition:
|
|
76
|
+
Bool:
|
|
77
|
+
"aws:SecureTransport": false
|
|
@@ -0,0 +1,209 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
# From npm @63klabs/cache-data
|
|
4
|
+
# This script is used to create a secure SSM Parameter in AWS Parameter Store.
|
|
5
|
+
# Cache-Data requires a secure key to be stored in SSM Parameter Store.
|
|
6
|
+
# This script can be used to generate random keys or use provided values.
|
|
7
|
+
# Include this script in your CI/CD pipeline to automate the process of creating secure parameters.
|
|
8
|
+
# It reads tags from a template-configuration.json file, which is common for SAM deployments,
|
|
9
|
+
# and applies them to the parameter.
|
|
10
|
+
|
|
11
|
+
import sys
|
|
12
|
+
import secrets
|
|
13
|
+
import boto3
|
|
14
|
+
from botocore.exceptions import ClientError
|
|
15
|
+
import json
|
|
16
|
+
import os
|
|
17
|
+
import argparse
|
|
18
|
+
import re
|
|
19
|
+
|
|
20
|
+
def usage():
|
|
21
|
+
print(f"""Creates a secure SSM Parameter in AWS Parameter Store.
|
|
22
|
+
This script can generate a random key of specified bit length or use a provided value.
|
|
23
|
+
It also reads tags from a template-configuration.json file (common for SAM deployments)
|
|
24
|
+
and applies them to the parameter.
|
|
25
|
+
|
|
26
|
+
NOTE:
|
|
27
|
+
It is designed to be used in a CI/CD pipeline, such as AWS CodePipeline.
|
|
28
|
+
It requires the AWS SDK for Python (boto3) and the botocore library.
|
|
29
|
+
Ensure CodeBuild has the necessary permissions to create and manage SSM parameters.
|
|
30
|
+
It can be run manually from a local CLI with a specific AWS profile if needed.
|
|
31
|
+
There is a --dryrun option to check if the parameter exists (and see values and tags) without creating it for testing purposes.
|
|
32
|
+
IT WILL NOT OVERWRITE AN EXISTING PARAMETER! DELETE IT FIRST IF YOU WANT TO REPLACE IT,
|
|
33
|
+
or, update using the AWS CLI, or manually through the console.
|
|
34
|
+
|
|
35
|
+
Usage: {sys.argv[0]} <PARAM_NAME> [--generate BITS | --value VALUE] [--dryrun] [--profile PROFILE]
|
|
36
|
+
PARAM_NAME
|
|
37
|
+
The name of the key parameter
|
|
38
|
+
For example, '/webservices/myapp/CacheData_SecureDataKey'
|
|
39
|
+
--generate BITS
|
|
40
|
+
Generate a random key with specified number of bits (mutually exclusive with --value)
|
|
41
|
+
--value VALUE
|
|
42
|
+
Use the provided value directly (mutually exclusive with --generate)
|
|
43
|
+
--dryrun
|
|
44
|
+
Check if parameter exists but don't create it
|
|
45
|
+
--profile PROFILE
|
|
46
|
+
AWS profile to use for the request""", file=sys.stderr)
|
|
47
|
+
|
|
48
|
+
def generate_key(key_len):
|
|
49
|
+
"""Generate a random hex key of specified bit length"""
|
|
50
|
+
return secrets.token_hex(key_len // 8) # key_len bits ÷ 8 bits/byte = bytes needed
|
|
51
|
+
|
|
52
|
+
def put_parameter(ssm_client, param_full_name, value, tags, dryrun=False):
|
|
53
|
+
"""Store a parameter in SSM Parameter Store"""
|
|
54
|
+
print(f"Checking for SSM Parameter: {param_full_name} ...")
|
|
55
|
+
|
|
56
|
+
try:
|
|
57
|
+
ssm_client.get_parameter(Name=param_full_name)
|
|
58
|
+
print("Parameter already exists. Skipping.")
|
|
59
|
+
except ClientError as e:
|
|
60
|
+
if e.response['Error']['Code'] == 'ParameterNotFound':
|
|
61
|
+
print("...parameter does not exist...")
|
|
62
|
+
if dryrun:
|
|
63
|
+
print(f"[DRYRUN] Would store parameter: {param_full_name}")
|
|
64
|
+
else:
|
|
65
|
+
print(f"Storing parameter: {param_full_name} ...")
|
|
66
|
+
ssm_client.put_parameter(
|
|
67
|
+
Name=param_full_name,
|
|
68
|
+
Value=value,
|
|
69
|
+
Type='SecureString',
|
|
70
|
+
Tags=tags,
|
|
71
|
+
Overwrite=False
|
|
72
|
+
)
|
|
73
|
+
else:
|
|
74
|
+
raise
|
|
75
|
+
|
|
76
|
+
def get_tags():
|
|
77
|
+
|
|
78
|
+
tags = []
|
|
79
|
+
|
|
80
|
+
# read in template-configuration.json from current directory or parent directory
|
|
81
|
+
config_file = None
|
|
82
|
+
if os.path.exists('template-configuration.json'):
|
|
83
|
+
config_file = 'template-configuration.json'
|
|
84
|
+
elif os.path.exists('../template-configuration.json'):
|
|
85
|
+
config_file = '../template-configuration.json'
|
|
86
|
+
|
|
87
|
+
try:
|
|
88
|
+
if config_file:
|
|
89
|
+
with open(config_file, 'r') as f:
|
|
90
|
+
config = json.load(f)
|
|
91
|
+
|
|
92
|
+
# Find the Tags property in config
|
|
93
|
+
# {
|
|
94
|
+
# "Tags": {
|
|
95
|
+
# "Provisioner": "CloudFormation",
|
|
96
|
+
# "Environment": "$DEPLOY_ENVIRONMENT$",
|
|
97
|
+
# "Deploy": "$PREFIX$ - $DEPLOY_ENVIRONMENT$",
|
|
98
|
+
# "Stage": "$STAGE_ID$"
|
|
99
|
+
# }
|
|
100
|
+
# }
|
|
101
|
+
# perform a search and replace of config utilizing environment variables,
|
|
102
|
+
# for example: replace $PREFIX$ with the value of the PREFIX environment variable
|
|
103
|
+
# Replacements may come anywhere in the value of the tag, not just at the beginning or end.
|
|
104
|
+
# set tags to the value of the tags property in config
|
|
105
|
+
if 'Tags' in config:
|
|
106
|
+
config_tags = config['Tags']
|
|
107
|
+
for key, value in config_tags.items():
|
|
108
|
+
# Find all instances of variable placeholders in the value
|
|
109
|
+
# Place holders are in the format $VARIABLE_NAME$
|
|
110
|
+
# Replace them with the corresponding environment variable values
|
|
111
|
+
if isinstance(value, str):
|
|
112
|
+
# Find variable placeholders in the format $VARIABLE_NAME$
|
|
113
|
+
placeholders = re.findall(r'\$([A-Z_][A-Z0-9_]*)\$', value)
|
|
114
|
+
for placeholder in placeholders:
|
|
115
|
+
# Check if the placeholder corresponds to an environment variable
|
|
116
|
+
env_value = os.getenv(placeholder)
|
|
117
|
+
if env_value is not None:
|
|
118
|
+
# Replace the placeholder with the environment variable value
|
|
119
|
+
value = value.replace(f"${placeholder}$", env_value)
|
|
120
|
+
else:
|
|
121
|
+
print(f"Environment variable '{placeholder}' not found. Using original value.", file=sys.stderr)
|
|
122
|
+
# Append the tag to the tags list
|
|
123
|
+
tags.append({'Key': key, 'Value': value})
|
|
124
|
+
else:
|
|
125
|
+
print("No Tags found in template-configuration.json. Using default tags.", file=sys.stderr)
|
|
126
|
+
|
|
127
|
+
else:
|
|
128
|
+
raise FileNotFoundError("template-configuration.json not found in current or parent directory")
|
|
129
|
+
|
|
130
|
+
except FileNotFoundError:
|
|
131
|
+
print("template-configuration.json not found in current or parent directory. Using default tags.", file=sys.stderr)
|
|
132
|
+
except json.JSONDecodeError:
|
|
133
|
+
print("Error decoding JSON from template-configuration.json. Using default tags.", file=sys.stderr)
|
|
134
|
+
except Exception as e:
|
|
135
|
+
print(f"Error reading template-configuration.json: {e}", file=sys.stderr)
|
|
136
|
+
|
|
137
|
+
# Find the element in the tags list that has the key 'Provisioner' and replace its value with 'CodeBuild'
|
|
138
|
+
provisioner_tag = next((tag for tag in tags if tag['Key'] == 'Provisioner'), None)
|
|
139
|
+
if provisioner_tag:
|
|
140
|
+
provisioner_tag['Value'] = 'CodeBuild'
|
|
141
|
+
else:
|
|
142
|
+
tags.append({'Key': 'Provisioner', 'Value': 'CodeBuild'})
|
|
143
|
+
|
|
144
|
+
# Find the element in the tags list that has the key 'DeployedUsing' and replace its value with 'CodeBuild'
|
|
145
|
+
deployed_using_tag = next((tag for tag in tags if tag['Key'] == 'DeployedUsing'), None)
|
|
146
|
+
if deployed_using_tag:
|
|
147
|
+
deployed_using_tag['Value'] = 'Build Script'
|
|
148
|
+
else:
|
|
149
|
+
tags.append({'Key': 'DeployedUsing', 'Value': 'Build Script (generate-put-ssm.py)'})
|
|
150
|
+
|
|
151
|
+
# print out the tags
|
|
152
|
+
print("Tags to be used:")
|
|
153
|
+
for tag in tags:
|
|
154
|
+
print(f" {tag['Key']}: {tag['Value']}")
|
|
155
|
+
|
|
156
|
+
return tags
|
|
157
|
+
|
|
158
|
+
def main():
|
|
159
|
+
parser = argparse.ArgumentParser(add_help=False)
|
|
160
|
+
parser.add_argument('param_name', help='The name of the key parameter')
|
|
161
|
+
parser.add_argument('--generate', type=int, help='Generate a random key with specified number of bits')
|
|
162
|
+
parser.add_argument('--value', type=str, help='Use the provided value directly')
|
|
163
|
+
parser.add_argument('--dryrun', action='store_true', help='Check if parameter exists but don\'t create it')
|
|
164
|
+
parser.add_argument('--profile', type=str, help='AWS profile to use for the request')
|
|
165
|
+
|
|
166
|
+
try:
|
|
167
|
+
args = parser.parse_args()
|
|
168
|
+
except:
|
|
169
|
+
usage()
|
|
170
|
+
sys.exit(1)
|
|
171
|
+
|
|
172
|
+
# Check for mutually exclusive options
|
|
173
|
+
if args.generate is not None and args.value is not None:
|
|
174
|
+
print("Error: --generate and --value cannot be used together", file=sys.stderr)
|
|
175
|
+
sys.exit(1)
|
|
176
|
+
|
|
177
|
+
param_name = args.param_name
|
|
178
|
+
if not param_name.startswith('/'):
|
|
179
|
+
print("Error: PARAM_NAME must start with a '/'", file=sys.stderr)
|
|
180
|
+
sys.exit(1)
|
|
181
|
+
|
|
182
|
+
print(f"Parameter Name: {param_name}")
|
|
183
|
+
|
|
184
|
+
# Determine value to store
|
|
185
|
+
if args.value is not None:
|
|
186
|
+
value = args.value
|
|
187
|
+
print(f"Using provided value: {value}")
|
|
188
|
+
elif args.generate is not None:
|
|
189
|
+
value = generate_key(args.generate)
|
|
190
|
+
if (args.dryrun):
|
|
191
|
+
print(f"[DRYRUN] Generated key of bit length {args.generate}: {value}")
|
|
192
|
+
else:
|
|
193
|
+
print(f"Generated key of bit length {args.generate}")
|
|
194
|
+
else:
|
|
195
|
+
value = "BLANK"
|
|
196
|
+
print("No value provided. Using default value: 'BLANK' which needs to be replaced with a real value.")
|
|
197
|
+
|
|
198
|
+
tags = get_tags()
|
|
199
|
+
|
|
200
|
+
try:
|
|
201
|
+
session = boto3.Session(profile_name=args.profile) if args.profile else boto3.Session()
|
|
202
|
+
ssm_client = session.client('ssm')
|
|
203
|
+
put_parameter(ssm_client, f"{param_name}", value, tags, args.dryrun)
|
|
204
|
+
except Exception as e:
|
|
205
|
+
print(f"Error: {e}", file=sys.stderr)
|
|
206
|
+
sys.exit(1)
|
|
207
|
+
|
|
208
|
+
if __name__ == "__main__":
|
|
209
|
+
main()
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
{
|
|
2
|
+
"Parameters": {
|
|
3
|
+
"FunctionMaxMemoryInMB": "2048",
|
|
4
|
+
"FunctionTimeOutInSeconds": "30",
|
|
5
|
+
"CacheDataTimeZoneForInterval": "America/Chicago",
|
|
6
|
+
"CacheDataAWSXRayOn": "true"
|
|
7
|
+
},
|
|
8
|
+
"Tags": {
|
|
9
|
+
"Provisioner": "CloudFormation",
|
|
10
|
+
"Application": "$PREFIX$-$PROJECT_ID$",
|
|
11
|
+
"ApplicationDeploymentId": "$PREFIX$-$PROJECT_ID$-$STAGE_ID$",
|
|
12
|
+
"Repository": "$REPOSITORY$"
|
|
13
|
+
}
|
|
14
|
+
}
|