@63klabs/cache-data 1.2.8 → 1.2.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -1
- package/package.json +1 -1
- package/src/lib/tools/AWS.classes.js +5 -1
- package/src/lib/tools/Connections.classes.js +35 -9
- package/docs/00-example-implementation/README.md +0 -55
- package/docs/00-example-implementation/example-buildspec.yml +0 -15
- package/docs/00-example-implementation/example-config.js +0 -0
- package/docs/00-example-implementation/example-handler.js +0 -0
- package/docs/00-example-implementation/example-template-lambda-function.yml +0 -119
- package/docs/00-example-implementation/example-template-parameters.yml +0 -66
- package/docs/00-example-implementation/example-template-s3-and-dynamodb-cache-store.yml +0 -77
- package/docs/00-example-implementation/generate-put-ssm.py +0 -209
- package/docs/00-example-implementation/template-configuration.json +0 -14
- package/docs/00-quick-start-implementation/README.md +0 -615
- package/docs/01-advanced-implementation-for-web-service/README.md +0 -13
- package/docs/README.md +0 -9
- package/docs/features/README.md +0 -5
- package/docs/features/cache/README.md +0 -3
- package/docs/features/endpoint/README.md +0 -3
- package/docs/features/tools/README.md +0 -341
- package/docs/lambda-optimization/README.md +0 -178
|
@@ -1,615 +0,0 @@
|
|
|
1
|
-
# Quick-Start Implementation
|
|
2
|
-
|
|
3
|
-
Just the basics of using debugging, logging, remote endpoints, and caching.
|
|
4
|
-
|
|
5
|
-
Basics:
|
|
6
|
-
|
|
7
|
-
- [Debug Using `DebugAndLog` and `Timer`](#debug-using-debugandlog-and-timer)
|
|
8
|
-
- [Basic Endpoint Connections](#basic-endpoint-connections)
|
|
9
|
-
- [Connections](#connections)
|
|
10
|
-
|
|
11
|
-
Caching:
|
|
12
|
-
|
|
13
|
-
- Secrets and Parameter Store
|
|
14
|
-
- CacheData
|
|
15
|
-
|
|
16
|
-
Advanced:
|
|
17
|
-
|
|
18
|
-
- Data Access Objects
|
|
19
|
-
- Connection Options
|
|
20
|
-
- CacheData Options
|
|
21
|
-
|
|
22
|
-
In addition to providing basic debugging, connections, and cache, the Cache-Data package also provides methods to receive, validate, route, respond to, and log requests as a web service much like an advanced framework such as ExpressJS.
|
|
23
|
-
|
|
24
|
-
For more information, see [Advanced Implementation for Providing a Web Service](../01-advanced-implementation-for-web-service/README.md)
|
|
25
|
-
|
|
26
|
-
## Debug Using `DebugAndLog` and `Timer`
|
|
27
|
-
|
|
28
|
-
The `DebugAndLog` and `Timer` objects are available from `tools`.
|
|
29
|
-
|
|
30
|
-
```js
|
|
31
|
-
const { tools: {DebugAndLog, Timer} } = require('@63klabs');
|
|
32
|
-
|
|
33
|
-
const timer = new Timer("My Timer 1"); // starts new timer
|
|
34
|
-
DebugAndLog.debug("My verbose message"); // outputs message to console.debug only if log level is 5
|
|
35
|
-
DebugAndLog.warn("My warning message");
|
|
36
|
-
DebugAndLog.log("Log data");
|
|
37
|
-
timer.stop(); // Stops timer and outputs log message with timer data when in debug mode
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
When using `try/catch` blocks it is recommended you include a `DebugAndLog.error` or `DebugAndLog.warn` in the `catch` block, and stop the timer in the `finally` block.
|
|
41
|
-
|
|
42
|
-
```js
|
|
43
|
-
const { tools: {DebugAndLog, Timer} } = require('@63klabs/cache-data');
|
|
44
|
-
const timer = new Timer("My Timer 1"); // starts new timer
|
|
45
|
-
|
|
46
|
-
try {
|
|
47
|
-
DebugAndLog.debug("I'm doing something")
|
|
48
|
-
// ... do something
|
|
49
|
-
}
|
|
50
|
-
catch (error) {
|
|
51
|
-
DebugAndLog.error(`Error: ${error.message}`, error.stack);)
|
|
52
|
-
} finally {
|
|
53
|
-
timer.stop();
|
|
54
|
-
}
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
For additional uses and options see [`tools.DebugAndLog`](../features/tools/README.md#toolsdebugandlog) and [`tools.Timer`](../features/tools/README.md#toolstimer) in the features documentation.
|
|
58
|
-
|
|
59
|
-
## Basic Endpoint Connections
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
## Connections
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
### Config
|
|
67
|
-
|
|
68
|
-
#### Parameters and Secrets
|
|
69
|
-
|
|
70
|
-
Cache-Data requires an 32 character hexadecimal key to encrypt data when at rest. This can be stored in an SSM Parameter named `crypt_secureDataKey`.
|
|
71
|
-
|
|
72
|
-
You have two options for storing and retrieving your SSM Parameters:
|
|
73
|
-
|
|
74
|
-
1. Using the Cache-Data SSM Parameter access function.
|
|
75
|
-
2. Using the AWS Parameter and Secrets Lambda Extension.
|
|
76
|
-
|
|
77
|
-
Both are easily accessed using functions in the Cache-Data toolkit.
|
|
78
|
-
|
|
79
|
-
##### Option 1: Cache-Data SSM Parameter access function
|
|
80
|
-
|
|
81
|
-
This runs in the Config.init() function and can be used to retrieve all of the parameters needed for your application.
|
|
82
|
-
|
|
83
|
-
```javascript
|
|
84
|
-
class Config extends tools._ConfigSuperClass {
|
|
85
|
-
static async init() {
|
|
86
|
-
|
|
87
|
-
tools._ConfigSuperClass._promise = new Promise(async (resolve, reject) => {
|
|
88
|
-
|
|
89
|
-
try {
|
|
90
|
-
|
|
91
|
-
let params = await this._initParameters(
|
|
92
|
-
[
|
|
93
|
-
{
|
|
94
|
-
"group": "app", // so we can do params.app.weatherapikey later
|
|
95
|
-
"path": process.env.PARAM_STORE_PATH
|
|
96
|
-
}
|
|
97
|
-
]
|
|
98
|
-
);
|
|
99
|
-
|
|
100
|
-
// You can access within init() using params.app.crypt_secureDataKey
|
|
101
|
-
|
|
102
|
-
resolve(true);
|
|
103
|
-
} catch(error) {
|
|
104
|
-
reject(null);
|
|
105
|
-
}
|
|
106
|
-
});
|
|
107
|
-
}
|
|
108
|
-
}
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
Accesses the SSM Parameter Store and places any parameters found under `/apps/my_cool_app/` into a `params.app` variable. You'll see that the cache initialization uses `params.app.crypt_secureDataKey` which is the parameter we created under `/apps/my_cool_app/crypt_secureDataKey`.
|
|
112
|
-
|
|
113
|
-
New deployments and new concurrent instances will pick up changes to a Parameter value, but long-running instances will not. If you change the value of a Parameter then you need to redeploy the application in order to clear out any use of the old value.
|
|
114
|
-
|
|
115
|
-
###### Option 2: AWS Parameter and Secrets Lambda Extension
|
|
116
|
-
|
|
117
|
-
This is a more robust option and works with Secrets Manager as well. It requires the installation of a Lambda layer and then use of the `CachedSecret` and/or `CachedSSMParameter` Class from the Cache-Data tool-kit.
|
|
118
|
-
|
|
119
|
-
Another advantage is that unlike the previous method, this method will pick up on any Secret and Parameter value changes and begin using the new values within 5 minutes (unless you set the cache for longer).
|
|
120
|
-
|
|
121
|
-
First, make sure you install the Lambda layer:
|
|
122
|
-
|
|
123
|
-
```yaml
|
|
124
|
-
Resources:
|
|
125
|
-
|
|
126
|
-
AppFunction:
|
|
127
|
-
Type: AWS::Serverless::Function
|
|
128
|
-
Properties:
|
|
129
|
-
# ...
|
|
130
|
-
Layers:
|
|
131
|
-
- !Sub "arn:aws:lambda:${AWS::Region}:${ACCT_ID_FOR_AWS_PARAM_AND_SECRETS_EXT}:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11" # https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html#ps-integration-lambda-extensions-add
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
Next, in your code, create the object to store your Parameter or Secret.
|
|
135
|
-
|
|
136
|
-
```javascript
|
|
137
|
-
const myKey = new tools.CachedSSMParameter('appSecretKey', {refreshAfter: 1600});
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
Finally, make sure you prime() and await the value.
|
|
141
|
-
|
|
142
|
-
```javascript
|
|
143
|
-
myKey.prime(); // request in the background so you can do other things before using it.
|
|
144
|
-
|
|
145
|
-
// ... do many things
|
|
146
|
-
|
|
147
|
-
let password = await myKey.getValue();
|
|
148
|
-
```
|
|
149
|
-
|
|
150
|
-
If you place it within an object you can also stringify that object and it will replace the reference with the secret. (You need to await the .prime() before processing)
|
|
151
|
-
|
|
152
|
-
```javascript
|
|
153
|
-
myKey.prime(); // request in the background so you can do other things before using it.
|
|
154
|
-
|
|
155
|
-
// ... do many things
|
|
156
|
-
const dbconn = { username: myUsername, password: myKey };
|
|
157
|
-
|
|
158
|
-
await myKey.prime();
|
|
159
|
-
connect(JSON.parse(JSON.stringify(dbconn)));
|
|
160
|
-
|
|
161
|
-
// or use toString()
|
|
162
|
-
await myKey.prime();
|
|
163
|
-
connect( {
|
|
164
|
-
username: `${myUsername}`,
|
|
165
|
-
password: `${myKey}`
|
|
166
|
-
})
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
##### Connections, and Cache
|
|
170
|
-
|
|
171
|
-
The cache object acts as an intermediary between your application and your data (whether it be a remote endpoint or other storage/process mechanism).
|
|
172
|
-
|
|
173
|
-
Before you can use Parameter Store, S3, and DynamoDb for the cache, they need to be set up with the proper access granted to your Lambda function.
|
|
174
|
-
|
|
175
|
-
1. Set up an S3 bucket (Your application will store cache data in `/cache`)
|
|
176
|
-
2. Create a DynamoDb table
|
|
177
|
-
3. Create a Parameter in SSM Parameter store `/app/my_cool_app/crypt_secureDataKey` and set the secret text to a 64 character length hex value. (64 hex characters because we are using a 256 bit key and cipher (`aes-256-ofb`)in the example below)
|
|
178
|
-
4. Make sure you set up IAM policies to allow you Lambda function access to the S3 bucket, DynamoDb table, and SSM Parameter store.
|
|
179
|
-
|
|
180
|
-
Once the S3 bucket, DynamoDb table, and SSM Parameter are set up we can focus on your Lambda function.
|
|
181
|
-
|
|
182
|
-
During your application initialization (but not for each request) we need to initialize the Config object.
|
|
183
|
-
|
|
184
|
-
The class below will do the following three things:
|
|
185
|
-
|
|
186
|
-
1. Bring in the secret key (and other parameters) from SSM Parameter Store.
|
|
187
|
-
2. Create connections with cache settings for each connection
|
|
188
|
-
3. Initialize the Cache
|
|
189
|
-
|
|
190
|
-
This code can be put into a separate file and brought in using a `require` statement. It should be scoped to the highest level of your Lambda function and not in the request handler.
|
|
191
|
-
|
|
192
|
-
```js
|
|
193
|
-
/* EXAMPLE USING the this._initParameters method of obtaining parameters during Config.init() */
|
|
194
|
-
|
|
195
|
-
// require cache-data
|
|
196
|
-
const { tools, cache, endpoint } = require('@63klabs/cache-data');
|
|
197
|
-
|
|
198
|
-
/**
|
|
199
|
-
* Extends tools._ConfigSuperClass
|
|
200
|
-
* Used to create a custom Config interface
|
|
201
|
-
* Usage: should be placed near the top of the script file outside
|
|
202
|
-
* of the event handler. It should be global and must be initialized.
|
|
203
|
-
* @example
|
|
204
|
-
* const obj = require("./classes.js");
|
|
205
|
-
* obj.Config.init();
|
|
206
|
-
*/
|
|
207
|
-
class Config extends tools._ConfigSuperClass {
|
|
208
|
-
|
|
209
|
-
/**
|
|
210
|
-
* This is custom inititialization code for the application. Depending
|
|
211
|
-
* upon needs, the _init functions from the super class may be used
|
|
212
|
-
* as needed. Init is async, and a promise is stored, allowing the
|
|
213
|
-
* lambda function to wait until the promise is finished.
|
|
214
|
-
*/
|
|
215
|
-
static async init() {
|
|
216
|
-
|
|
217
|
-
tools._ConfigSuperClass._promise = new Promise(async (resolve, reject) => {
|
|
218
|
-
|
|
219
|
-
try {
|
|
220
|
-
|
|
221
|
-
let params = await this._initParameters(
|
|
222
|
-
[
|
|
223
|
-
{
|
|
224
|
-
"group": "app", // so we can do params.app.weatherapikey later
|
|
225
|
-
"path": "/apps/my_cool_app/" // process.env.PARAM_STORE_PATH // or store as a Lambda environment variable
|
|
226
|
-
}
|
|
227
|
-
]
|
|
228
|
-
);
|
|
229
|
-
|
|
230
|
-
// after we have the params, we can set the connections
|
|
231
|
-
let connections = new tools.Connections();
|
|
232
|
-
|
|
233
|
-
/* NOTE: instead of hard coding connections, you could import
|
|
234
|
-
from a connections file and then add in any additional values
|
|
235
|
-
such as keys from the Param store
|
|
236
|
-
*/
|
|
237
|
-
|
|
238
|
-
// for games demo from api.chadkluck.net
|
|
239
|
-
connections.add( {
|
|
240
|
-
name: "demo",
|
|
241
|
-
host: "api.chadkluck.net",
|
|
242
|
-
path: "/games",
|
|
243
|
-
parameters: {},
|
|
244
|
-
headers: {
|
|
245
|
-
referer: "https://chadkluck.net"
|
|
246
|
-
},
|
|
247
|
-
cache: [
|
|
248
|
-
{
|
|
249
|
-
profile: "games",
|
|
250
|
-
overrideOriginHeaderExpiration: true, // if the endpoint returns an expiration, do we ignore it for our own?
|
|
251
|
-
defaultExpirationInSeconds: (10 * 60),// , // 10 minutes
|
|
252
|
-
expiresIsOnInterval: true, // for example, a 10 min cache can expire on the hour, 10, 20, 30... after. 24 hour cache can expire at midnight. 6 hour cache can expire at 6am, noon, 6pm, and midnight
|
|
253
|
-
headersToRetain: "", // what headers from the endpoint do we want to keep with the cache data?
|
|
254
|
-
hostId: "demo", // log entry friendly (or not)
|
|
255
|
-
pathId: "games", // log entry friendly (or not)
|
|
256
|
-
encrypt: false // you can set this to true and it will use the key from param store and encrypt data at rest in S3 and DynamoDb
|
|
257
|
-
}
|
|
258
|
-
]
|
|
259
|
-
} );
|
|
260
|
-
|
|
261
|
-
tools._ConfigSuperClass._connections = connections;
|
|
262
|
-
|
|
263
|
-
// Cache settings
|
|
264
|
-
cache.Cache.init({
|
|
265
|
-
dynamoDbTable: "yourDynamoDbTable", // replace with the name of a DynamoDb table to store cached data
|
|
266
|
-
s3Bucket: "yourS3Bucket", // replace with a bucket name to store cache data. Data will be stored in /cache in yourS3Bucket
|
|
267
|
-
secureDataAlgorithm: "aes-256-ofb", // how do we encrypt data at rest
|
|
268
|
-
secureDataKey: Buffer.from(params.app.crypt_secureDataKey, "hex"), // using the parameter from above during Config.init()
|
|
269
|
-
//secureDataKey: new tools.CachedSSMParameter('/apps/my_cool_app/CacheData_SecureDataKey', {refreshAfter: 300}), // if using tools.CachedSSMParameter()
|
|
270
|
-
idHashAlgorithm: "RSA-SHA3-512", // the alg used to create a unique hash identifier for requests so we can tell them apart in the cache
|
|
271
|
-
DynamoDbMaxCacheSize_kb: 20, // data larger than this (in KB) will be stored in S3 to keep DynamoDb running efficently
|
|
272
|
-
purgeExpiredCacheEntriesAfterXHours: 24, // expired caches hang around for a while before we purge just in case there is cause to fall back on them
|
|
273
|
-
defaultExpirationExtensionOnErrorInSeconds: 300, // so as to not overwhelm a down endpoint, or to not cache an error for too long, how often should we check back?
|
|
274
|
-
timeZoneForInterval: "America/Chicago" // if caching on interval, we need a timezone to account for calculating hours, days, and weeks. List: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
|
|
275
|
-
});
|
|
276
|
-
|
|
277
|
-
resolve(true);
|
|
278
|
-
} catch (error) {
|
|
279
|
-
tools.DebugAndLog.error("Could not initialize Config", error);
|
|
280
|
-
reject(false);
|
|
281
|
-
};
|
|
282
|
-
|
|
283
|
-
});
|
|
284
|
-
|
|
285
|
-
};
|
|
286
|
-
};
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
The `connection` code above does the following:
|
|
290
|
-
|
|
291
|
-
1. Defines the host and path (and any parameters and headers to send to a remote endpoint)
|
|
292
|
-
2. Defines the cache settings for that remote endpoint (note, these are only cache settings for that remote endpoint and not the overall cache)
|
|
293
|
-
|
|
294
|
-
Additional connections may be added using additional `connections.add()` functions.
|
|
295
|
-
|
|
296
|
-
The `cache` code does the following:
|
|
297
|
-
|
|
298
|
-
1. Sets the DynamoDb table and S3 bucket to store cached data
|
|
299
|
-
2. Sets the algorithm to securely encrypt data at rest in DynamoDb and S3
|
|
300
|
-
3. Sets the hash algorithm used to create a unique id for each unique request
|
|
301
|
-
4. How big of an object do we save in DynamoDb before storing it in S3? (20K objects are ideal, anything bigger is in S3)
|
|
302
|
-
5. How long to wait before purging expired entries (they aren't purged right away but kept in case of errors)
|
|
303
|
-
6. If there is an error getting fresh data, how long do we extend any existing cache? (so we can back off while endpoint is in error)
|
|
304
|
-
7. Set the time zone for intervals. For example, we can expire on the hour (8am, 12pm, 8pm, etc) but if we expire at the end of the day, when is the "end of the day"? Midnight where? If empty it will be UTC.
|
|
305
|
-
|
|
306
|
-
Each of these are described in their own sections below.
|
|
307
|
-
|
|
308
|
-
Note that it is probably best to not hard code values but instead bring them in as environment variables from your Lambda function.
|
|
309
|
-
|
|
310
|
-
Next, we need to call the initialization in our application, and before the handler can be executed, make sure the promise has resolved.
|
|
311
|
-
|
|
312
|
-
```js
|
|
313
|
-
// note that the Config object is defined in the code above
|
|
314
|
-
|
|
315
|
-
/* initialize the Config */
|
|
316
|
-
Config.init(); // we need to await completion in the async call function
|
|
317
|
-
|
|
318
|
-
/**
|
|
319
|
-
* Lambda function handler
|
|
320
|
-
*/
|
|
321
|
-
exports.handler = async (event, context, callback) => {
|
|
322
|
-
|
|
323
|
-
/* wait for CONFIG to be settled as we need it before continuing. */
|
|
324
|
-
await Config.promise();
|
|
325
|
-
|
|
326
|
-
/* Process the request and wait for result */
|
|
327
|
-
const response = await someFunction(event, context); // some code or function that generates a response
|
|
328
|
-
|
|
329
|
-
/* Send the result back to API Gateway */
|
|
330
|
-
callback(null, response);
|
|
331
|
-
|
|
332
|
-
}
|
|
333
|
-
```
|
|
334
|
-
|
|
335
|
-
Note that you will replace `someFunction()` with your own function that will call and process the data from cache as in the example below.
|
|
336
|
-
|
|
337
|
-
Once the `Config` object is initialized, the following code can be used to access data through the cache.
|
|
338
|
-
|
|
339
|
-
```js
|
|
340
|
-
/*
|
|
341
|
-
Note that cache object was already set by the require statement
|
|
342
|
-
assuming:
|
|
343
|
-
const { tools, cache, endpoint } = require('@63klabs/cache-data');
|
|
344
|
-
*/
|
|
345
|
-
|
|
346
|
-
let connection = Config.getConnection("demo"); // corresponds with the name we gave it during connections.add()
|
|
347
|
-
let conn = connection.toObject(); // we'll "extract" the connection data. .toObject() will create a clone of the data so we can modify if need be
|
|
348
|
-
|
|
349
|
-
let cacheProfile = connection.getCacheProfile("games"); // corresponds with the cache profile we gave within demo for connections.add()
|
|
350
|
-
|
|
351
|
-
const cacheObj = await cache.CacheableDataAccess.getData(
|
|
352
|
-
cacheProfile, // this is your cache profile for an endpoint, included from connection object
|
|
353
|
-
endpoint.getDataDirectFromURI, // this is the function you want to invoke to get fresh data if the cache is stale. (conn and null will be passed to it)
|
|
354
|
-
conn, // connection information which will be passed to endpoint.getDataDirectFromURI() to get fresh data. Also used to identify the object in cache
|
|
355
|
-
null // this parameter can be used to pass additional data to endpoint.getDataDirectFromURI (or any other DAO)
|
|
356
|
-
);
|
|
357
|
-
|
|
358
|
-
let games = cacheObj.getBody(true); // return the data as an object (true) instead of a string (false). You could use false if you want to keep the data as a string (as in xml or html or text)
|
|
359
|
-
```
|
|
360
|
-
|
|
361
|
-
In order to do its job it needs to:
|
|
362
|
-
|
|
363
|
-
1. Know how to access the data. We use a Connection object from Config to do this. You can think of a Connection object as all the pieces of an HTTP request. It identifies the protocol, domain, path, query string, headers, etc. (However, it doesn't have to be an HTTP request.)
|
|
364
|
-
2. Know the function to use to access fresh data from the remote endpoint. Using the Connection object, your can either use a built in HTTP request, or define your own method for processing an http request or other data source.
|
|
365
|
-
3. Know the cache policy for the data. We use a Cache object to do this. It is an object that has information on expiration, headers to save with the data, where cache data is stored, stored data encryption protocol,
|
|
366
|
-
|
|
367
|
-
### cache.CacheableDataAccess.getData() without Connection
|
|
368
|
-
|
|
369
|
-
Note that you can use `cache.CacheableDataAccess.getData()` without a Connection object. You'll notice that we "extract" the connection data from `connection` using `.toObject()`. We do this not just because it creates an object that isn't a reference (thus allowing us to ad hoc modify things like path or parameters without changing the original) but also because any object with any structure may be passed (as long as your passed function is expecting it).
|
|
370
|
-
|
|
371
|
-
The `cacheProfile` variable is also just an object, but must adhere to the structure outlined in the cache declaration previously shown.
|
|
372
|
-
|
|
373
|
-
You can create the cache configuration and connection on the fly without the Connection object:
|
|
374
|
-
|
|
375
|
-
```js
|
|
376
|
-
const cacheProfile ={
|
|
377
|
-
overrideOriginHeaderExpiration: true,
|
|
378
|
-
defaultExpirationExtensionOnErrorInSeconds: 3600,
|
|
379
|
-
defaultExpirationInSeconds: (10 * 60), // 10 minutes
|
|
380
|
-
expiresIsOnInterval: true,
|
|
381
|
-
headersToRetain: ['x-data-id', 'x-data-sha1'],
|
|
382
|
-
hostId: "example",
|
|
383
|
-
pathId: "person",
|
|
384
|
-
encrypt: true
|
|
385
|
-
};
|
|
386
|
-
|
|
387
|
-
const conn = {
|
|
388
|
-
host: "api.example.com",
|
|
389
|
-
path: "/person",
|
|
390
|
-
parameters: {id: id, event: event },
|
|
391
|
-
headers: {}
|
|
392
|
-
};
|
|
393
|
-
|
|
394
|
-
const cacheObj = await cache.CacheableDataAccess.getData(
|
|
395
|
-
cacheProfile,
|
|
396
|
-
myCustomDAO_getData,
|
|
397
|
-
conn,
|
|
398
|
-
null
|
|
399
|
-
);
|
|
400
|
-
```
|
|
401
|
-
|
|
402
|
-
### Connections using CachedSSMParameter or CachedSecret
|
|
403
|
-
|
|
404
|
-
Creating a connection is similar to above, we can add an authorization property to the conection:
|
|
405
|
-
|
|
406
|
-
```js
|
|
407
|
-
conn = {
|
|
408
|
-
authentication: {
|
|
409
|
-
parameters: {
|
|
410
|
-
apikey: new tools.CachedSSMParameter('/apps/my_cool_app/demoAPIkey', {refreshAfter: 300}),
|
|
411
|
-
}
|
|
412
|
-
}
|
|
413
|
-
}
|
|
414
|
-
```
|
|
415
|
-
|
|
416
|
-
Learn more about Connection Authentication below.
|
|
417
|
-
|
|
418
|
-
And when calling Cache.init(), pass a CachedSSMParameter (or CachedSecret) to the secureDataKey property:
|
|
419
|
-
|
|
420
|
-
```js
|
|
421
|
-
secureDataKey: new tools.CachedSSMParameter('/apps/my_cool_app/CacheData_SecureDataKey', {refreshAfter: 1600}),
|
|
422
|
-
```
|
|
423
|
-
|
|
424
|
-
```js
|
|
425
|
-
// for games demo from api.chadkluck.net
|
|
426
|
-
connections.add( {
|
|
427
|
-
name: "demo",
|
|
428
|
-
host: "api.chadkluck.net",
|
|
429
|
-
path: "/games",
|
|
430
|
-
parameters: {},
|
|
431
|
-
headers: {
|
|
432
|
-
referer: "https://chadkluck.net"
|
|
433
|
-
},
|
|
434
|
-
authentication: {
|
|
435
|
-
parameters: {
|
|
436
|
-
apikey: new tools.CachedSSMParameter('/apps/my_cool_app/demoAPIkey', {refreshAfter: 300}), // ADDED
|
|
437
|
-
}
|
|
438
|
-
},
|
|
439
|
-
cache: myCacheProfilesArray
|
|
440
|
-
} );
|
|
441
|
-
|
|
442
|
-
tools._ConfigSuperClass._connections = connections;
|
|
443
|
-
|
|
444
|
-
// Cache settings
|
|
445
|
-
cache.Cache.init({
|
|
446
|
-
dynamoDbTable: "yourDynamoDbTable",
|
|
447
|
-
s3Bucket: "yourS3Bucket",
|
|
448
|
-
secureDataAlgorithm: "aes-256-ofb",
|
|
449
|
-
secureDataKey: new tools.CachedSSMParameter('/apps/my_cool_app/CacheData_SecureDataKey', {refreshAfter: 1600}), // CHANGED FROM params.app
|
|
450
|
-
idHashAlgorithm: "RSA-SHA3-512",
|
|
451
|
-
DynamoDbMaxCacheSize_kb: 20,
|
|
452
|
-
purgeExpiredCacheEntriesAfterXHours: 24,
|
|
453
|
-
defaultExpirationExtensionOnErrorInSeconds: 300,
|
|
454
|
-
timeZoneForInterval: "America/Chicago"
|
|
455
|
-
});
|
|
456
|
-
|
|
457
|
-
```
|
|
458
|
-
|
|
459
|
-
### Connections Authentication
|
|
460
|
-
|
|
461
|
-
You can store your authentication methods separate from the headers, parameters, and body properties. You can also use Basic authorization.
|
|
462
|
-
|
|
463
|
-
Just add an `authentication` property to your connection.
|
|
464
|
-
|
|
465
|
-
```js
|
|
466
|
-
// for games demo from api.chadkluck.net
|
|
467
|
-
connections.add( {
|
|
468
|
-
name: "demo",
|
|
469
|
-
host: "api.chadkluck.net",
|
|
470
|
-
path: "/games",
|
|
471
|
-
headers: {
|
|
472
|
-
referer: "https://chadkluck.net"
|
|
473
|
-
},
|
|
474
|
-
authentication: {
|
|
475
|
-
parameters: {
|
|
476
|
-
apikey: new tools.CachedSSMParameter('/apps/my_cool_app/demoAPIkey', {refreshAfter: 1600}), // ADDED
|
|
477
|
-
}
|
|
478
|
-
},
|
|
479
|
-
cache: myCacheProfilesArray
|
|
480
|
-
} );
|
|
481
|
-
|
|
482
|
-
connections.add( {
|
|
483
|
-
name: "demoauthbasic",
|
|
484
|
-
host: "api.chadkluck.net",
|
|
485
|
-
path: "/games",
|
|
486
|
-
headers: {
|
|
487
|
-
referer: "https://chadkluck.net"
|
|
488
|
-
},
|
|
489
|
-
authentication: {
|
|
490
|
-
basic: {
|
|
491
|
-
username: new tools.CachedSSMParameter('/apps/my_cool_app/demoUsername', {refreshAfter: 300}),
|
|
492
|
-
password: new tools.CachedSSMParameter('/apps/my_cool_app/demoPassword', {refreshAfter: 300}),
|
|
493
|
-
}
|
|
494
|
-
},
|
|
495
|
-
cache: myCacheProfilesArray
|
|
496
|
-
} );
|
|
497
|
-
|
|
498
|
-
connections.add( {
|
|
499
|
-
name: "demoauthheaders",
|
|
500
|
-
host: "api.chadkluck.net",
|
|
501
|
-
path: "/games",
|
|
502
|
-
headers: {
|
|
503
|
-
referer: "https://chadkluck.net"
|
|
504
|
-
},
|
|
505
|
-
authentication: {
|
|
506
|
-
headers: {
|
|
507
|
-
'x-api-key': new tools.CachedSSMParameter('/apps/my_cool_app/apiKey', {refreshAfter: 300})
|
|
508
|
-
}
|
|
509
|
-
},
|
|
510
|
-
cache: myCacheProfilesArray
|
|
511
|
-
} );
|
|
512
|
-
|
|
513
|
-
connections.add( {
|
|
514
|
-
name: "demoauthbody",
|
|
515
|
-
host: "api.chadkluck.net",
|
|
516
|
-
path: "/games",
|
|
517
|
-
headers: {
|
|
518
|
-
referer: "https://chadkluck.net"
|
|
519
|
-
},
|
|
520
|
-
authentication: {
|
|
521
|
-
body: {
|
|
522
|
-
'x-api-key': new tools.CachedSSMParameter('/apps/my_cool_app/apiKey', {refreshAfter: 300}),
|
|
523
|
-
'account': new tools.CachedSSMParameter('/apps/my_cool_app/accountId', {refreshAfter: 3600})
|
|
524
|
-
}
|
|
525
|
-
},
|
|
526
|
-
cache: myCacheProfilesArray
|
|
527
|
-
} );
|
|
528
|
-
```
|
|
529
|
-
|
|
530
|
-
### Connections Options
|
|
531
|
-
|
|
532
|
-
Specify a `timeout` in the connection to pass to the http_get command. Default is `8000`.
|
|
533
|
-
|
|
534
|
-
Specify how duplicate parameters in a query string should be handled. This allows you to craft your query string to match what your endpoint expects when it parses the query string.
|
|
535
|
-
|
|
536
|
-
```javascript
|
|
537
|
-
connections.add({
|
|
538
|
-
method: "POST",
|
|
539
|
-
host: "api.chadkluck.net",
|
|
540
|
-
path: "/echo/",
|
|
541
|
-
headers: headers,
|
|
542
|
-
uri: "",
|
|
543
|
-
protocol: "https",
|
|
544
|
-
body: null,
|
|
545
|
-
parameters: {
|
|
546
|
-
greeting: "Hello",
|
|
547
|
-
planets: ["Earth", "Mars"]
|
|
548
|
-
},
|
|
549
|
-
options: {
|
|
550
|
-
timeout: 8000,
|
|
551
|
-
separateDuplicateParameters: false, // default is false
|
|
552
|
-
separateDuplicateParametersAppendToKey: "", // "" "[]", or "0++", "1++"
|
|
553
|
-
combinedDuplicateParameterDelimiter: ','
|
|
554
|
-
}
|
|
555
|
-
})
|
|
556
|
-
```
|
|
557
|
-
|
|
558
|
-
By default the query string used for the request will be:
|
|
559
|
-
|
|
560
|
-
```text
|
|
561
|
-
?greeting=Hello&planets=Earth,Mars
|
|
562
|
-
```
|
|
563
|
-
|
|
564
|
-
However, by changing `separateDuplicateParameters` to `true` and `separateDuplicateParametersAppendToKey` to `[]`:
|
|
565
|
-
|
|
566
|
-
```text
|
|
567
|
-
?greeting=Hello&planets[]=Earth&planets[]=Mars
|
|
568
|
-
```
|
|
569
|
-
|
|
570
|
-
You can also append an index to the end of the parameter:
|
|
571
|
-
|
|
572
|
-
```javascript
|
|
573
|
-
options = {
|
|
574
|
-
separateDuplicateParameters: true,
|
|
575
|
-
separateDuplicateParametersAppendToKey: "0++", // "" "[]", or "0++", "1++"
|
|
576
|
-
}
|
|
577
|
-
// ?greeting=Hello&planets0=Earth&planets1=Mars
|
|
578
|
-
```
|
|
579
|
-
|
|
580
|
-
Similarly, you can start at index 1 instead of 0:
|
|
581
|
-
|
|
582
|
-
```javascript
|
|
583
|
-
options = {
|
|
584
|
-
separateDuplicateParameters: true,
|
|
585
|
-
separateDuplicateParametersAppendToKey: "1++", // "" "[]", or "0++", "1++"
|
|
586
|
-
}
|
|
587
|
-
// ?greeting=Hello&planets1=Earth&planets2=Mars
|
|
588
|
-
```
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
### AWS X-Ray
|
|
592
|
-
|
|
593
|
-
X-Ray can be enabled by making sure you have the Lambda layer installed, appropriate permissions granted to your Lambda execution policy, and setting AWS X-Ray to true in the CloudFormation Template parameters or your Lambda Environment variable.
|
|
594
|
-
|
|
595
|
-
The templates and code under the installation section already have these implemented.
|
|
596
|
-
|
|
597
|
-
## Advanced
|
|
598
|
-
|
|
599
|
-
There are advanced uses for the Cache object such as caching processed data not from an endpoint and creating your own Data Access Object (DAO) classes.
|
|
600
|
-
|
|
601
|
-
### Caching data not from a remote endpoint
|
|
602
|
-
|
|
603
|
-
Cache does not have to be from a remote endpoint.
|
|
604
|
-
|
|
605
|
-
Suppose you gather data from six endpoints and process the data in a resource and time intensive process and would like to cache the result for 6 or 24 hours. You can use the Cache object to store any data from any source, either externally or internally.
|
|
606
|
-
|
|
607
|
-
The function parameter passed to the Cache object is the method used to obtain data. Remember the `endpoint.getDataDirectFromURI` from the code sample above? That is just a function to return a bare bones response from an api endpoint. (You can actually extend the `endpoint.Endpoint` class and create your own DAOs that can pre and post process data before returning to your application's cache.)
|
|
608
|
-
|
|
609
|
-
Instead of passing in the function `endpoint.getDataDirectFromURI` you can create any function that will grab or process data and return an object.
|
|
610
|
-
|
|
611
|
-
Remember, when passing functions for another function to execute, do not include the `()` at the end.
|
|
612
|
-
|
|
613
|
-
### Creating your own Data Access Object (DAO)
|
|
614
|
-
|
|
615
|
-
You can either extend `endpoint.Endpoint` or create your own.
|
|
@@ -1,13 +0,0 @@
|
|
|
1
|
-
# Advanced Implementation for Providing a Web Service
|
|
2
|
-
|
|
3
|
-
> Refer to [Quick-Start Implementation](../00-quick-start-implementation/README.md) for documentation on setting up debugging, remote end points, secrets and parameter store access, and caching.
|
|
4
|
-
|
|
5
|
-
In addition to providing basic debugging, connections, and cache, the Cache-Data package also provides methods to receive, validate, route, respond to, and log requests as a web service much like an advanced framework such as ExpressJS.
|
|
6
|
-
|
|
7
|
-
A full implementation example and tutorial is provided as one of the Atlantis Application Starters available through the [Atlantis Tutorials repository](https://github.com/63klabs/atlantis-tutorials). (Atlantis is a collection of templates and deployment scripts to assist in starting and automating serverless deployments using AWS SAM and CloudFormation.)
|
|
8
|
-
|
|
9
|
-
- Request Validation
|
|
10
|
-
- Routing
|
|
11
|
-
- Data Access Objects
|
|
12
|
-
- Response and Logging
|
|
13
|
-
- Dashboard
|
package/docs/README.md
DELETED
|
@@ -1,9 +0,0 @@
|
|
|
1
|
-
# Cache-Data Documentation
|
|
2
|
-
|
|
3
|
-
A full implementation example and tutorial is provided as one of the Atlantis Application Starters available through the [Atlantis Tutorials repository](https://github.com/63klabs/atlantis-tutorials). (Atlantis is a collection of templates and deployment scripts to assist in starting and automating serverless deployments using AWS SAM and CloudFormation.)
|
|
4
|
-
|
|
5
|
-
- [Quick-Start Implementation](./00-quick-start-implementation/README.md)
|
|
6
|
-
- [Advanced Implementation](./01-advanced-implementation-for-web-service/README.md)
|
|
7
|
-
- [Example Implementation Templates](./00-example-implementation/README.md)
|
|
8
|
-
- [Features](./features/README.md)
|
|
9
|
-
- [Lambda Optimization Best Practices](./lambda-optimization/README.md)
|
package/docs/features/README.md
DELETED