@63klabs/cache-data 1.3.8 → 1.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +109 -5
- package/CONTRIBUTING.md +58 -5
- package/README.md +20 -34
- package/package.json +16 -27
- package/src/lib/dao-cache.js +55 -47
- package/src/lib/dao-endpoint.js +68 -38
- package/src/lib/tools/AWS.classes.js +58 -5
- package/src/lib/tools/{APIRequest.class.js → ApiRequest.class.js} +164 -51
- package/src/lib/tools/CachedParametersSecrets.classes.js +10 -9
- package/src/lib/tools/ClientRequest.class.js +987 -44
- package/src/lib/tools/Connections.classes.js +5 -5
- package/src/lib/tools/Response.class.js +25 -0
- package/src/lib/tools/generic.response.html.js +8 -113
- package/src/lib/tools/generic.response.js +73 -0
- package/src/lib/tools/generic.response.json.js +5 -135
- package/src/lib/tools/generic.response.rss.js +10 -114
- package/src/lib/tools/generic.response.text.js +5 -115
- package/src/lib/tools/generic.response.xml.js +10 -114
- package/src/lib/tools/index.js +113 -40
- package/src/lib/utils/ValidationExecutor.class.js +70 -0
- package/src/lib/utils/ValidationMatcher.class.js +417 -0
- package/AGENTS.md +0 -1012
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,110 @@ To report an issue, or to see proposed and upcoming enhancements, check out [63K
|
|
|
8
8
|
|
|
9
9
|
Report all vulnerabilities under the [Security menu](https://github.com/63Klabs/cache-data/security/advisories) in the Cache-Data GitHub repository.
|
|
10
10
|
|
|
11
|
+
## v1.3.10 (2026-03-15)
|
|
12
|
+
|
|
13
|
+
### Fixes
|
|
14
|
+
|
|
15
|
+
- **Security Fixes** in tests and SSM Parameter Store legacy access [Spec: 1-3-10-security-fixes-for-tests](.kiro/specs/1-3-10-security-fixes-for-tests/)
|
|
16
|
+
- **GitHub Workflow fix** for testing matrix. Node24 was failing while Node 20, 22 were passing [Spec: 1-3-10-github-workflow-test-fix](.kiro/specs/1-3-10-github-workflow-test-fix/)
|
|
17
|
+
|
|
18
|
+
### Changes
|
|
19
|
+
|
|
20
|
+
- **Complete Mocha to Jest Test Migration** [Spec: 1-3-10-test-migration-phase-6](.kiro/specs/1-3-10-test-migration-phase-6/)
|
|
21
|
+
- Migrated all 38 remaining Mocha test files to Jest, completing the full test suite migration
|
|
22
|
+
- Jest is now the sole test framework for the project
|
|
23
|
+
- All tests use the `.jest.mjs` file extension
|
|
24
|
+
- Updated `npm test` to run Jest directly
|
|
25
|
+
- Updated all module-specific test scripts (`test:cache`, `test:config`, `test:endpoint`) to use Jest
|
|
26
|
+
- Updated CI/CD configuration to run Jest only
|
|
27
|
+
- Updated documentation (README, CONTRIBUTING, AGENTS, steering documents) to reflect Jest-only testing
|
|
28
|
+
- **ApiRequest** and **CachedSsmParameter** renaming to conform to naming conventions:
|
|
29
|
+
- `APIRequest` has been renamed to `ApiRequest`.
|
|
30
|
+
- `APIRequest` is now aliased to `ApiRequest` and will still work.
|
|
31
|
+
- `CachedSSMParameter` has been renamed to `CachedSsmParameter`.
|
|
32
|
+
- `CachedSSMParameter` is now aliased to `CachedSsmParameter` and will still work.
|
|
33
|
+
- While there is no timeframe to deprecate and remove the old naming, it will happen with a future **major** version update.
|
|
34
|
+
- New code should use the new conventions. Old code should be updated.
|
|
35
|
+
- **tools.endpoint.get()** deprecated in favor of **tools.endpoint.send()**
|
|
36
|
+
- `tools.endpoint.send()` removes any confusion about it accepting more than just `GET` requests and that it will send immediately when called.
|
|
37
|
+
- **Optimized Generic Response** for easier maintenance
|
|
38
|
+
|
|
39
|
+
### Added
|
|
40
|
+
|
|
41
|
+
- **Enhanced Invalid Request Messaging**
|
|
42
|
+
- **New Response Formatting to ApiRequest**
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
### Removed
|
|
46
|
+
|
|
47
|
+
- **Mocha, Chai, and Sinon Dependencies** [Spec: 1-3-10-test-migration-phase-6](.kiro/specs/1-3-10-test-migration-phase-6/)
|
|
48
|
+
- Removed `mocha` from devDependencies
|
|
49
|
+
- Removed `chai` from devDependencies
|
|
50
|
+
- Removed `sinon` from devDependencies
|
|
51
|
+
- Removed `chai-http` from devDependencies
|
|
52
|
+
- Removed all legacy Mocha test files (`*-tests.mjs`)
|
|
53
|
+
- Removed `test:all` script (no longer needed with single test framework)
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
## v1.3.9 (2026-03-09)
|
|
57
|
+
|
|
58
|
+
### Added
|
|
59
|
+
- **Body Parameter Validation for ClientRequest** [Spec: 1-3-9-body-validation-and-header-format-fix](.kiro/specs/1-3-9-body-validation-and-header-format-fix/)
|
|
60
|
+
- **Body Parameter Validation**: ClientRequest now validates body parameters using the same validation framework as path, query, header, and cookie parameters
|
|
61
|
+
- Automatic JSON parsing with error handling for request bodies
|
|
62
|
+
- Support for both API Gateway v1 and v2 formats
|
|
63
|
+
- Validation using existing ValidationMatcher and ValidationExecutor classes
|
|
64
|
+
- Body validation integrated into validation chain after cookie validation
|
|
65
|
+
- `getBodyParameters()` method now returns validated body parameters
|
|
66
|
+
- **Header Key Conversion Utility**: New static method `convertHeaderKeyToCamelCase(headerKey)` helps developers determine correct validation rule keys
|
|
67
|
+
- Converts HTTP header names from kebab-case to camelCase (e.g., `content-type` → `contentType`)
|
|
68
|
+
- Handles multiple hyphens and uppercase input correctly
|
|
69
|
+
- Documented with examples of common HTTP headers
|
|
70
|
+
- **Enhanced Documentation**: Comprehensive JSDoc documentation for header key conversion behavior
|
|
71
|
+
- Header key conversion reference table showing HTTP headers and their camelCase equivalents
|
|
72
|
+
- Detailed explanation of conversion algorithm and why it's necessary
|
|
73
|
+
- Examples showing validation configuration with converted header keys
|
|
74
|
+
- Complete body validation examples with error handling
|
|
75
|
+
- **Backwards Compatible**: All new features are opt-in and maintain full backwards compatibility
|
|
76
|
+
- Existing code continues to work without modification
|
|
77
|
+
- Body validation only activates when configured
|
|
78
|
+
- No changes to existing validation behavior for other parameter types
|
|
79
|
+
- **Comprehensive Testing**: 8 property-based tests and extensive unit/integration tests validate correctness properties
|
|
80
|
+
- Body validation round-trip property
|
|
81
|
+
- Validation failure propagation
|
|
82
|
+
- JSON parsing precondition
|
|
83
|
+
- Header key conversion correctness
|
|
84
|
+
- Backwards compatibility preservation
|
|
85
|
+
- Common validation pattern support
|
|
86
|
+
- Multi-parameter validation interface
|
|
87
|
+
|
|
88
|
+
### Security
|
|
89
|
+
- **Fixed npm security vulnerabilities in serialize-javascript dependency** [Spec: 1-3-9-npm-security-vulnerabilities-fix](.kiro/specs/1-3-9-npm-security-vulnerabilities-fix/)
|
|
90
|
+
- Fixed 2 high severity vulnerabilities in serialize-javascript (RCE via RegExp.flags and Date.prototype.toISOString)
|
|
91
|
+
- Added npm override for serialize-javascript >=7.0.3 to force secure version
|
|
92
|
+
- No breaking changes to public APIs
|
|
93
|
+
- All existing tests pass
|
|
94
|
+
|
|
95
|
+
### Added
|
|
96
|
+
- **Enhanced Validation System for ClientRequest** [Spec: 1-3-9-improve-validations-object](.kiro/specs/1-3-9-improve-validations-object/)
|
|
97
|
+
- **Route-Specific Validations**: Define different validation rules for the same parameter name in different routes (e.g., `id` in `/product/{id}` vs `/employee/{id}`)
|
|
98
|
+
- **Method-Specific Validations**: Define different validation rules based on HTTP method (e.g., stricter validation for `POST` than `GET`)
|
|
99
|
+
- **Method-and-Route Validations**: Most precise control with validation rules for specific method-route combinations (e.g., `POST:join/{id}`)
|
|
100
|
+
- **Multi-Parameter Validations**: Validate multiple parameters together to enforce cross-parameter constraints
|
|
101
|
+
- **Clear Priority Order**: Four-tier priority system (method-and-route > route-only > method-only > global)
|
|
102
|
+
- **Backwards Compatible**: Existing global parameter validations continue to work without any code changes
|
|
103
|
+
- **Performance Optimized**: Pattern normalization caching and early exit optimization for minimal overhead
|
|
104
|
+
- **Comprehensive Testing**: 15 correctness properties validated through property-based testing
|
|
105
|
+
|
|
106
|
+
### Enhancement
|
|
107
|
+
- **AppConfig Async Initialization Optimization** [Spec: 1-3-9-appconfig-async-init-optimization](.kiro/specs/1-3-9-appconfig-async-init-optimization/)
|
|
108
|
+
- **Parallel Initialization**: All AppConfig.init() operations now execute asynchronously in parallel, improving Lambda cold start performance by 10-20%
|
|
109
|
+
- **Backwards Compatible**: No API changes - existing code continues to work without modifications
|
|
110
|
+
- **Optimized Operations**: Settings, connections, validations, responses, and SSM parameters all initialize concurrently
|
|
111
|
+
- **Error Resilient**: Individual initialization failures don't block other operations
|
|
112
|
+
- **Transparent**: Debug logging and error handling work identically to previous implementation
|
|
113
|
+
- **Performance**: Cold start time reduced from 62-212ms to 50-200ms (sequential to parallel execution)
|
|
114
|
+
|
|
11
115
|
## v1.3.8 (2026-03-02)
|
|
12
116
|
|
|
13
117
|
### Security
|
|
@@ -29,14 +133,14 @@ Report all vulnerabilities under the [Security menu](https://github.com/63Klabs/
|
|
|
29
133
|
- Production dependencies remain secure (0 vulnerabilities)
|
|
30
134
|
|
|
31
135
|
### Added
|
|
32
|
-
- **
|
|
33
|
-
- **Automatic Pagination**:
|
|
136
|
+
- **ApiRequest Pagination, Retry, and X-Ray Enhancements** [Spec: 1-3-8-api-request-pagination-retries-xray](.kiro/specs/1-3-8-api-request-pagination-retries-xray/) addresses [#171](https://github.com/63Klabs/cache-data/issues/171), [#172](https://github.com/63Klabs/cache-data/issues/172), [#173](https://github.com/63Klabs/cache-data/issues/173)
|
|
137
|
+
- **Automatic Pagination**: ApiRequest now supports automatic pagination for APIs that return paginated results
|
|
34
138
|
- Configurable pagination labels for different API response structures
|
|
35
139
|
- Batch processing for concurrent page requests
|
|
36
140
|
- Support for both offset-based and token-based pagination
|
|
37
141
|
- Automatic result combination into single response
|
|
38
142
|
- Pagination metadata in response (total pages, total items, incomplete status)
|
|
39
|
-
- **Automatic Retry Logic**:
|
|
143
|
+
- **Automatic Retry Logic**: ApiRequest now includes built-in retry functionality for transient failures
|
|
40
144
|
- Configurable retry attempts (default: 1 retry after initial attempt)
|
|
41
145
|
- Automatic retry on network errors, empty responses, parse errors, and 5xx status codes
|
|
42
146
|
- Optional retry on 4xx status codes (disabled by default)
|
|
@@ -212,9 +316,9 @@ While `DebugAndLog` has also been optimized, there is no change to its results.
|
|
|
212
316
|
|
|
213
317
|
## 1.1.4 (2025-03-18) Added XRay sub-segment for API requests
|
|
214
318
|
|
|
215
|
-
- Feature: Added XRay Segment for
|
|
319
|
+
- Feature: Added XRay Segment for ApiRequest class
|
|
216
320
|
|
|
217
|
-
When using the tools.
|
|
321
|
+
When using the tools.ApiRequest class, each remote request is now annotated and provided meta data.
|
|
218
322
|
|
|
219
323
|
## 1.1.3 (2025-02-17) Additional Options for Sending Parameters via Query String
|
|
220
324
|
|
package/CONTRIBUTING.md
CHANGED
|
@@ -26,7 +26,7 @@ Spec-Driven, AI-Assisted Engineering (SD-AI) is a software development methodolo
|
|
|
26
26
|
|
|
27
27
|
Code must be reviewed, understood, and tested by a human before being merged.
|
|
28
28
|
|
|
29
|
-
Kiro is the required AI coding assistant for final integrations, documentation, and testing, as it is in the AWS Ecosystem and this project is
|
|
29
|
+
Kiro is the required AI coding assistant for final integrations, documentation, and testing, as it is in the AWS Ecosystem and this project is developed to deploy on the AWS platform. Just like test suites, Kiro ensures the proper tests, documentation, and guardrails are in place. Kiro is as important as commit-hooks and tests as it is a tool that ensures quality checks and should not be bypassed.
|
|
30
30
|
|
|
31
31
|
Ensure [AGENTS](./AGENTS.md) and Kiro steering documents are reviewed, understood, and used by both humans and AI.
|
|
32
32
|
|
|
@@ -120,13 +120,66 @@ Ensure all tests pass before submitting changes:
|
|
|
120
120
|
npm test
|
|
121
121
|
|
|
122
122
|
# Run specific test suites
|
|
123
|
-
npm test -- test/documentation
|
|
124
|
-
npm test -- test/cache
|
|
123
|
+
npm test -- test/documentation
|
|
124
|
+
npm test -- test/cache
|
|
125
125
|
```
|
|
126
126
|
|
|
127
|
+
## Testing
|
|
128
|
+
|
|
129
|
+
This project uses Jest as its test framework. All tests use the `.jest.mjs` file extension.
|
|
130
|
+
|
|
131
|
+
### Test Framework
|
|
132
|
+
|
|
133
|
+
- Test Runner: Jest
|
|
134
|
+
- Assertions: Jest built-in (`expect`)
|
|
135
|
+
- Property Testing: fast-check
|
|
136
|
+
- Mocking: Jest built-in (`jest.spyOn`, `jest.fn`)
|
|
137
|
+
- File Pattern: `*.jest.mjs`
|
|
138
|
+
|
|
139
|
+
### Running Tests
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
# Run all tests
|
|
143
|
+
npm test
|
|
144
|
+
|
|
145
|
+
# Run specific test suites
|
|
146
|
+
npm test -- test/cache
|
|
147
|
+
npm test -- test/endpoint
|
|
148
|
+
npm test -- test/documentation
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
### Writing Tests
|
|
152
|
+
|
|
153
|
+
All tests must use Jest. Example:
|
|
154
|
+
|
|
155
|
+
```javascript
|
|
156
|
+
import { describe, it, expect, jest, afterEach } from '@jest/globals';
|
|
157
|
+
import { Cache } from '../src/lib/dao-cache.js';
|
|
158
|
+
|
|
159
|
+
describe('Cache', () => {
|
|
160
|
+
afterEach(() => {
|
|
161
|
+
jest.restoreAllMocks();
|
|
162
|
+
});
|
|
163
|
+
|
|
164
|
+
it('should generate consistent hash for same input', () => {
|
|
165
|
+
const conn = { host: 'example.com', path: '/api' };
|
|
166
|
+
const hash1 = Cache.generateIdHash(conn);
|
|
167
|
+
const hash2 = Cache.generateIdHash(conn);
|
|
168
|
+
expect(hash1).toBe(hash2);
|
|
169
|
+
});
|
|
170
|
+
});
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
### Test Naming Conventions
|
|
174
|
+
|
|
175
|
+
- Test files: `*-tests.jest.mjs`
|
|
176
|
+
- Property tests: `*-property-tests.jest.mjs`
|
|
177
|
+
- Integration tests: `*-integration-tests.jest.mjs`
|
|
178
|
+
- Unit tests: `*-unit-tests.jest.mjs`
|
|
179
|
+
|
|
127
180
|
## Documentation Standards
|
|
128
181
|
|
|
129
|
-
All public APIs must have complete JSDoc documentation. See [Documentation Standards](.kiro/steering/documentation-standards.md) for detailed requirements.
|
|
182
|
+
All public APIs must have complete JSDoc documentation. See [JSDoc Documentation Standards](.kiro/steering/documentation-standards-jsdoc.md) for detailed requirements.
|
|
130
183
|
|
|
131
184
|
**Required JSDoc tags:**
|
|
132
185
|
- Description of what the function/class does
|
|
@@ -149,7 +202,7 @@ All public APIs must have complete JSDoc documentation. See [Documentation Stand
|
|
|
149
202
|
* @example
|
|
150
203
|
* const result = await CacheableDataAccess.getData(
|
|
151
204
|
* cacheProfile,
|
|
152
|
-
* endpoint.
|
|
205
|
+
* endpoint.send,
|
|
153
206
|
* connection
|
|
154
207
|
* );
|
|
155
208
|
* console.log(result.data);
|
package/README.md
CHANGED
|
@@ -50,7 +50,7 @@ The @63klabs/cache-data package provides three main modules:
|
|
|
50
50
|
|
|
51
51
|
### Requirements
|
|
52
52
|
|
|
53
|
-
- Node.js >=
|
|
53
|
+
- Node.js >=22.0.0 runtime on Lambda
|
|
54
54
|
- AWS Services:
|
|
55
55
|
- **AWS Lambda**: For running your serverless functions
|
|
56
56
|
- **Amazon S3**: For storing large cached objects
|
|
@@ -68,7 +68,7 @@ Install the package using npm:
|
|
|
68
68
|
npm install @63klabs/cache-data
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
-
The simplest way to get started is to use the [63klabs Atlantis Templates and Script
|
|
71
|
+
The simplest way to get started is to use the [63klabs Atlantis Templates and Script Platform](https://github.com/63Klabs/atlantis) to deploy this and other ready-to-run solutions via CI/CD.
|
|
72
72
|
|
|
73
73
|
However, if you want to write your own templates and code, follow the following steps:
|
|
74
74
|
|
|
@@ -102,26 +102,22 @@ It is recommended that you use the quick-start method when implementing for the
|
|
|
102
102
|
### Basic Caching Example
|
|
103
103
|
|
|
104
104
|
```javascript
|
|
105
|
-
const { cache } = require("@63klabs/cache-data");
|
|
105
|
+
const { cache: {CacheableDataAccess, Cache}, endpoint } = require("@63klabs/cache-data");
|
|
106
106
|
|
|
107
107
|
// Initialize cache with your S3 bucket and DynamoDB table
|
|
108
108
|
cache.Cache.init({
|
|
109
|
-
s3Bucket: process.env.CACHE_DATA_S3_BUCKET,
|
|
110
|
-
dynamoDbTable: process.env.CACHE_DATA_DYNAMODB_TABLE,
|
|
111
|
-
securityKey: process.env.CACHE_DATA_SECURITY_KEY
|
|
109
|
+
s3Bucket: process.env.CACHE_DATA_S3_BUCKET, // Cache.init will check this env variable automatically if not provided here
|
|
110
|
+
dynamoDbTable: process.env.CACHE_DATA_DYNAMODB_TABLE, // Cache.init will check this env variable automatically if not provided here
|
|
111
|
+
securityKey: process.env.CACHE_DATA_SECURITY_KEY // don't do this, use SSM Parameter Store - example only
|
|
112
112
|
});
|
|
113
113
|
|
|
114
|
-
//
|
|
115
|
-
const cacheKey = "my-data-key";
|
|
116
|
-
const dataToCache = { message: "Hello, World!", timestamp: Date.now() };
|
|
117
|
-
|
|
118
|
-
await cache.Cache.put(cacheKey, dataToCache, 3600); // Cache for 1 hour
|
|
114
|
+
//
|
|
119
115
|
|
|
120
116
|
// Retrieve cached data
|
|
121
|
-
const
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
117
|
+
const conn = { host: "api.example.com", path: "api/users"};
|
|
118
|
+
const cacheProfile = {/* cache parameters */};
|
|
119
|
+
const cachedData = await CacheableDataAccess.getData(cacheProfile, endpoint.send, conn);
|
|
120
|
+
const data = cachedData.getBody(true);
|
|
125
121
|
```
|
|
126
122
|
|
|
127
123
|
### Making Endpoint Requests
|
|
@@ -130,22 +126,21 @@ if (cachedData) {
|
|
|
130
126
|
const { endpoint } = require("@63klabs/cache-data");
|
|
131
127
|
|
|
132
128
|
// Make a simple GET request to an API
|
|
133
|
-
const response = await endpoint.
|
|
134
|
-
{ host: "api.example.com", path: "/data" }
|
|
135
|
-
{ parameters: { q: "search-term" } }
|
|
129
|
+
const response = await endpoint.send(
|
|
130
|
+
{ host: "api.example.com", path: "/data", parameters: { q: "search-term" } }
|
|
136
131
|
);
|
|
137
132
|
|
|
138
133
|
console.log("API Response:", response.body);
|
|
139
134
|
console.log("Status Code:", response.statusCode);
|
|
140
135
|
```
|
|
141
136
|
|
|
142
|
-
### Using
|
|
137
|
+
### Using ApiRequest with Pagination and Retry
|
|
143
138
|
|
|
144
139
|
```javascript
|
|
145
|
-
const { tools } = require("@63klabs/cache-data");
|
|
140
|
+
const { tools: {ApiRequest} } = require("@63klabs/cache-data");
|
|
146
141
|
|
|
147
142
|
// Make a request with automatic pagination and retry
|
|
148
|
-
const request = new tools.
|
|
143
|
+
const request = new tools.ApiRequest({
|
|
149
144
|
host: "api.example.com",
|
|
150
145
|
path: "/users",
|
|
151
146
|
parameters: {
|
|
@@ -165,23 +160,15 @@ const response = await request.send();
|
|
|
165
160
|
// Response contains all users from all pages
|
|
166
161
|
const allUsers = JSON.parse(response.body).items;
|
|
167
162
|
console.log(`Retrieved ${allUsers.length} total users`);
|
|
168
|
-
|
|
169
|
-
// Check metadata
|
|
170
|
-
if (response.metadata?.pagination?.occurred) {
|
|
171
|
-
console.log(`Fetched ${response.metadata.pagination.totalPages} pages`);
|
|
172
|
-
}
|
|
173
|
-
if (response.metadata?.retries?.occurred) {
|
|
174
|
-
console.log(`Succeeded after ${response.metadata.retries.attempts} attempts`);
|
|
175
|
-
}
|
|
176
163
|
```
|
|
177
164
|
|
|
178
165
|
### Using Utility Tools
|
|
179
166
|
|
|
180
167
|
```javascript
|
|
181
|
-
const { tools } = require("@63klabs/cache-data");
|
|
168
|
+
const { tools: {DebugAndLog, Timer} } = require("@63klabs/cache-data");
|
|
182
169
|
|
|
183
170
|
// Create a timer to measure performance
|
|
184
|
-
const timer = new
|
|
171
|
+
const timer = new Timer("my-operation");
|
|
185
172
|
timer.start();
|
|
186
173
|
|
|
187
174
|
// Your code here...
|
|
@@ -190,9 +177,8 @@ timer.stop();
|
|
|
190
177
|
console.log(`Operation took ${timer.elapsed()}ms`);
|
|
191
178
|
|
|
192
179
|
// Use the logger
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
logger.error("An error occurred", { details: "error info" });
|
|
180
|
+
DebugAndLog.debug("MyApp");
|
|
181
|
+
DebugAndLog.error("Error in Service", error.msg);
|
|
196
182
|
```
|
|
197
183
|
|
|
198
184
|
## Help
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@63klabs/cache-data",
|
|
3
|
-
"version": "1.3.
|
|
4
|
-
"description": "
|
|
5
|
-
"author": "Chad
|
|
3
|
+
"version": "1.3.10",
|
|
4
|
+
"description": "A package for AWS Lambda Node.js applications to access and cache data from remote API endpoints (or other sources) utilizing AWS S3 and DynamoDb for the cache storage. Also provides a simple request handling, routing, and response logging framework for running a web service with minimal dependencies.",
|
|
5
|
+
"author": "Chad Kluck (https://chadkluck.me)",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
@@ -16,45 +16,34 @@
|
|
|
16
16
|
"node": ">=20.0.0"
|
|
17
17
|
},
|
|
18
18
|
"dependencies": {
|
|
19
|
-
"aws-xray-sdk-core": "^3.12.0",
|
|
20
19
|
"moment-timezone": "^0.6.0",
|
|
21
20
|
"object-hash": "^3.0.0"
|
|
22
21
|
},
|
|
23
22
|
"devDependencies": {
|
|
23
|
+
"aws-xray-sdk-core": "^3.12.0",
|
|
24
24
|
"@aws-sdk/client-dynamodb": "^3.995.0",
|
|
25
25
|
"@aws-sdk/client-s3": "^3.995.0",
|
|
26
26
|
"@aws-sdk/client-ssm": "^3.995.0",
|
|
27
27
|
"@aws-sdk/lib-dynamodb": "^3.995.0",
|
|
28
28
|
"@eslint/js": "^10.0.1",
|
|
29
|
-
"chai": "^6.2.2",
|
|
30
|
-
"chai-http": "^5.1.2",
|
|
31
29
|
"eslint": "^10.0.1",
|
|
32
30
|
"fast-check": "^4.5.3",
|
|
33
|
-
"jest": "^30.2.0",
|
|
34
|
-
"mocha": "^11.7.5",
|
|
35
|
-
"sinon": "^21.0.1"
|
|
36
|
-
},
|
|
31
|
+
"jest": "^30.2.0" },
|
|
37
32
|
"overrides": {
|
|
38
33
|
"fast-xml-parser": ">=5.3.4",
|
|
39
|
-
"
|
|
40
|
-
"minimatch": ">=10.2.2"
|
|
41
|
-
"glob": ">=13.0.6"
|
|
34
|
+
"glob": ">=13.0.6",
|
|
35
|
+
"minimatch": ">=10.2.2"
|
|
42
36
|
},
|
|
43
37
|
"scripts": {
|
|
44
|
-
"test": "
|
|
45
|
-
"test:
|
|
46
|
-
"test:
|
|
47
|
-
"test:
|
|
48
|
-
"test:
|
|
49
|
-
"test:
|
|
50
|
-
"test:
|
|
51
|
-
"test:
|
|
52
|
-
"test:
|
|
53
|
-
"test:tools:jest": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/tools",
|
|
54
|
-
"test:logging": "mocha 'test/logging/**/*-tests.mjs'",
|
|
55
|
-
"test:request": "mocha 'test/request/**/*-tests.mjs'",
|
|
56
|
-
"test:response": "mocha 'test/response/**/*-tests.mjs'",
|
|
57
|
-
"test:utils": "mocha 'test/utils/**/*-tests.mjs'",
|
|
38
|
+
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js",
|
|
39
|
+
"test:cache": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/cache",
|
|
40
|
+
"test:config": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/config",
|
|
41
|
+
"test:endpoint": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/endpoint",
|
|
42
|
+
"test:tools": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/tools",
|
|
43
|
+
"test:logging": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/logging",
|
|
44
|
+
"test:request": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/request",
|
|
45
|
+
"test:response": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/response",
|
|
46
|
+
"test:utils": "node --experimental-vm-modules node_modules/jest/bin/jest.js test/utils",
|
|
58
47
|
"lint": "eslint .",
|
|
59
48
|
"lint:fix": "eslint . --fix",
|
|
60
49
|
"lint:ci": "eslint . --max-warnings 0"
|
package/src/lib/dao-cache.js
CHANGED
|
@@ -38,9 +38,10 @@ const moment = require('moment-timezone');
|
|
|
38
38
|
* and retrieval operations - cache expiration logic and data management are
|
|
39
39
|
* handled by the CacheData class.
|
|
40
40
|
*
|
|
41
|
-
*
|
|
42
|
-
*
|
|
41
|
+
* Used by Cache object to write and retrieve data that is larger than what
|
|
42
|
+
* we want to store in DynamoDB
|
|
43
43
|
*
|
|
44
|
+
* @private
|
|
44
45
|
* @example
|
|
45
46
|
* // Initialize S3Cache with bucket name
|
|
46
47
|
* S3Cache.init('my-cache-bucket');
|
|
@@ -57,8 +58,13 @@ class S3Cache {
|
|
|
57
58
|
static #bucket = null;
|
|
58
59
|
static #objPath = "cache/";
|
|
59
60
|
|
|
60
|
-
|
|
61
|
-
|
|
61
|
+
/**
|
|
62
|
+
* Create a new S3Cache instance.
|
|
63
|
+
* Private constructor - instances should be created through static methods.
|
|
64
|
+
*
|
|
65
|
+
* @private
|
|
66
|
+
*/
|
|
67
|
+
constructor () {};
|
|
62
68
|
|
|
63
69
|
/**
|
|
64
70
|
* Get the S3 bucket name configured for cache storage.
|
|
@@ -106,7 +112,6 @@ class S3Cache {
|
|
|
106
112
|
* process.env.CACHE_DATA_S3_BUCKET = 'my-cache-bucket';
|
|
107
113
|
* S3Cache.init();
|
|
108
114
|
*/
|
|
109
|
-
|
|
110
115
|
static init(bucket = null) {
|
|
111
116
|
if ( S3Cache.getBucket() === null ) {
|
|
112
117
|
bucket = (bucket === null) ? process.env?.CACHE_DATA_S3_BUCKET || null : bucket;
|
|
@@ -272,14 +277,14 @@ class S3Cache {
|
|
|
272
277
|
};
|
|
273
278
|
|
|
274
279
|
/**
|
|
275
|
-
* Basic
|
|
280
|
+
* Basic DynamoDB read/write for cache data. Provides low-level storage operations
|
|
276
281
|
* for cached data in Amazon DynamoDB. This class handles only the storage format
|
|
277
282
|
* and retrieval operations - cache expiration logic and data management are
|
|
278
283
|
* handled by the CacheData class.
|
|
279
284
|
*
|
|
280
|
-
*
|
|
281
|
-
* For most use cases, use the Cache or CacheableDataAccess classes instead.
|
|
285
|
+
* Used by Cache object to write and retrieve data in DynamoDB
|
|
282
286
|
*
|
|
287
|
+
* @private
|
|
283
288
|
* @example
|
|
284
289
|
* // Initialize DynamoDbCache with table name
|
|
285
290
|
* DynamoDbCache.init('my-cache-table');
|
|
@@ -455,8 +460,8 @@ class DynamoDbCache {
|
|
|
455
460
|
* coordinates storage between DynamoDB (for small items) and S3 (for large items).
|
|
456
461
|
*
|
|
457
462
|
* This class is used internally by the publicly exposed Cache class and should
|
|
458
|
-
* not be used directly
|
|
459
|
-
*
|
|
463
|
+
* not be used directly. Use the CacheableDataAccess to access data for
|
|
464
|
+
* application-level caching.
|
|
460
465
|
*
|
|
461
466
|
* Key responsibilities:
|
|
462
467
|
* - Expiration time calculations and interval-based caching
|
|
@@ -464,6 +469,7 @@ class DynamoDbCache {
|
|
|
464
469
|
* - Automatic routing between DynamoDB and S3 based on size
|
|
465
470
|
* - ETag and Last-Modified header generation
|
|
466
471
|
*
|
|
472
|
+
* @private
|
|
467
473
|
* @example
|
|
468
474
|
* // Initialize CacheData (typically done through Cache.init())
|
|
469
475
|
* CacheData.init({
|
|
@@ -508,7 +514,7 @@ class CacheData {
|
|
|
508
514
|
* @param {string} [parameters.dynamoDbTable] DynamoDB table name (or use CACHE_DATA_DYNAMO_DB_TABLE env var)
|
|
509
515
|
* @param {string} [parameters.s3Bucket] S3 bucket name (or use CACHE_DATA_S3_BUCKET env var)
|
|
510
516
|
* @param {string} [parameters.secureDataAlgorithm='aes-256-cbc'] Encryption algorithm (or use CACHE_DATA_SECURE_DATA_ALGORITHM env var)
|
|
511
|
-
* @param {string|Buffer|tools.Secret|tools.
|
|
517
|
+
* @param {string|Buffer|tools.Secret|tools.CachedSsmParameter|tools.CachedSecret} parameters.secureDataKey Encryption key (required, no env var for security)
|
|
512
518
|
* @param {number} [parameters.DynamoDbMaxCacheSize_kb=10] Max size in KB for DynamoDB storage (or use CACHE_DATA_DYNAMO_DB_MAX_CACHE_SIZE_KB env var)
|
|
513
519
|
* @param {number} [parameters.purgeExpiredCacheEntriesAfterXHours=24] Hours after expiration to purge entries (or use CACHE_DATA_PURGE_EXPIRED_CACHE_ENTRIES_AFTER_X_HRS env var)
|
|
514
520
|
* @param {string} [parameters.timeZoneForInterval='Etc/UTC'] Timezone for interval calculations (or use CACHE_DATA_TIME_ZONE_FOR_INTERVAL env var)
|
|
@@ -1373,26 +1379,18 @@ class CacheData {
|
|
|
1373
1379
|
};
|
|
1374
1380
|
|
|
1375
1381
|
/**
|
|
1376
|
-
* The Cache object handles the settings for the cache system
|
|
1382
|
+
* The Cache object handles the settings for the cache system and is only
|
|
1383
|
+
* accessed by the application during init.
|
|
1377
1384
|
*
|
|
1378
|
-
*
|
|
1385
|
+
* Reading of the cache is done through CacheableDataAccess.getData()
|
|
1386
|
+
*
|
|
1387
|
+
* Before using the Cache, it must be initialized.
|
|
1379
1388
|
*
|
|
1380
1389
|
* Many settings can be set through Environment variables or by
|
|
1381
1390
|
* passing parameters to Cache.init():
|
|
1382
1391
|
*
|
|
1383
1392
|
* Cache.init({parameters});
|
|
1384
1393
|
*
|
|
1385
|
-
* Then you can then make a request, sending it through CacheableDataAccess:
|
|
1386
|
-
*
|
|
1387
|
-
* const { cache } = require("@63klabs/cache-data");
|
|
1388
|
-
*
|
|
1389
|
-
* const cacheObj = await cache.CacheableDataAccess.getData(
|
|
1390
|
-
* cacheCfg,
|
|
1391
|
-
* yourFetchFunction,
|
|
1392
|
-
* conn,
|
|
1393
|
-
* daoQuery
|
|
1394
|
-
* );
|
|
1395
|
-
*
|
|
1396
1394
|
* @example
|
|
1397
1395
|
* // Initialize the cache system
|
|
1398
1396
|
* Cache.init({
|
|
@@ -1401,13 +1399,6 @@ class CacheData {
|
|
|
1401
1399
|
* idHashAlgorithm: 'sha256'
|
|
1402
1400
|
* });
|
|
1403
1401
|
*
|
|
1404
|
-
* // Create a cache instance with connection and profile
|
|
1405
|
-
* const connection = { host: 'api.example.com', path: '/data' };
|
|
1406
|
-
* const cacheProfile = {
|
|
1407
|
-
* defaultExpirationInSeconds: 300,
|
|
1408
|
-
* encrypt: true
|
|
1409
|
-
* };
|
|
1410
|
-
* const cacheInstance = new Cache(connection, cacheProfile);
|
|
1411
1402
|
*/
|
|
1412
1403
|
class Cache {
|
|
1413
1404
|
|
|
@@ -1526,7 +1517,7 @@ class Cache {
|
|
|
1526
1517
|
* @param {string} parameters.dynamoDbTable Can also be set with environment variable CACHE_DATA_DYNAMO_DB_TABLE
|
|
1527
1518
|
* @param {string} parameters.s3Bucket Can also be set with environment variable CACHE_DATA_S3_BUCKET
|
|
1528
1519
|
* @param {string} parameters.secureDataAlgorithm Can also be set with environment variable CACHE_DATA_SECURE_DATA_ALGORITHM
|
|
1529
|
-
* @param {string|Buffer|tools.Secret|tools.
|
|
1520
|
+
* @param {string|Buffer|tools.Secret|tools.CachedSsmParameter|tools.CachedSecret} parameters.secureDataKey Must be passed, will not accept an environment variable for security reasons.
|
|
1530
1521
|
* @param {number} parameters.DynamoDbMaxCacheSize_kb Can also be set with environment variable CACHE_DATA_DYNAMO_DB_MAX_CACHE_SIZE_KB
|
|
1531
1522
|
* @param {number} parameters.purgeExpiredCacheEntriesAfterXHours Can also be set with environment variable CACHE_DATA_PURGE_EXPIRED_CACHE_ENTRIES_AFTER_X_HRS
|
|
1532
1523
|
* @param {string} parameters.timeZoneForInterval Can also be set with environment variable CACHE_DATA_TIME_ZONE_FOR_INTERVAL
|
|
@@ -1545,8 +1536,8 @@ class Cache {
|
|
|
1545
1536
|
("CACHE_DATA_USE_TOOLS_HASH" in process.env ? Cache.bool(process.env.CACHE_DATA_USE_TOOLS_HASH_METHOD) : false);
|
|
1546
1537
|
|
|
1547
1538
|
// Initialize in-memory cache feature flag
|
|
1548
|
-
this.#useInMemoryCache = parameters.useInMemoryCache ||
|
|
1549
|
-
(process.env.CACHE_USE_IN_MEMORY
|
|
1539
|
+
this.#useInMemoryCache = Cache.bool(parameters.useInMemoryCache) ||
|
|
1540
|
+
Cache.bool(process.env.CACHE_USE_IN_MEMORY) ||
|
|
1550
1541
|
false;
|
|
1551
1542
|
|
|
1552
1543
|
// Initialize InMemoryCache if enabled
|
|
@@ -1654,34 +1645,51 @@ class Cache {
|
|
|
1654
1645
|
};
|
|
1655
1646
|
|
|
1656
1647
|
/**
|
|
1657
|
-
* Convert a value to boolean with
|
|
1648
|
+
* Convert a value to boolean with strict handling for string values.
|
|
1658
1649
|
*
|
|
1659
1650
|
* JavaScript's Boolean() function treats any non-empty string as true, including
|
|
1660
|
-
* the string "false". This method
|
|
1661
|
-
* (case-insensitive)
|
|
1662
|
-
*
|
|
1651
|
+
* the string "false". This method provides strict boolean conversion where:
|
|
1652
|
+
* - The string "true" (case-insensitive) returns true
|
|
1653
|
+
* - The string "1" returns true
|
|
1654
|
+
* - The number 1 returns true
|
|
1655
|
+
* - The boolean true returns true
|
|
1656
|
+
* - All other values return false (including "false", "0", "no", whitespace, empty strings, null, undefined)
|
|
1663
1657
|
*
|
|
1664
|
-
*
|
|
1658
|
+
* This is useful when dealing with environment variables, JSON data, query parameters,
|
|
1659
|
+
* or configuration strings where only explicit "true" or "1" should enable a feature.
|
|
1665
1660
|
*
|
|
1666
1661
|
* @param {*} value A value to convert to boolean
|
|
1667
|
-
* @returns {boolean}
|
|
1662
|
+
* @returns {boolean} True only if value is explicitly truthy ("true", "1", 1, or true), false otherwise
|
|
1668
1663
|
* @example
|
|
1669
1664
|
* Cache.bool(true); // true
|
|
1670
1665
|
* Cache.bool(false); // false
|
|
1671
1666
|
* Cache.bool("true"); // true
|
|
1672
|
-
* Cache.bool("
|
|
1673
|
-
* Cache.bool("
|
|
1667
|
+
* Cache.bool("TRUE"); // true (case-insensitive)
|
|
1668
|
+
* Cache.bool("1"); // true
|
|
1674
1669
|
* Cache.bool(1); // true
|
|
1670
|
+
* Cache.bool("false"); // false
|
|
1671
|
+
* Cache.bool("0"); // false
|
|
1672
|
+
* Cache.bool("no"); // false
|
|
1673
|
+
* Cache.bool(" "); // false (whitespace)
|
|
1674
|
+
* Cache.bool(" "); // false (whitespace)
|
|
1675
1675
|
* Cache.bool(0); // false
|
|
1676
1676
|
* Cache.bool(null); // false
|
|
1677
|
+
* Cache.bool(undefined); // false
|
|
1677
1678
|
* Cache.bool(""); // false
|
|
1678
1679
|
*/
|
|
1679
1680
|
static bool (value) {
|
|
1680
1681
|
|
|
1681
|
-
|
|
1682
|
+
// >! Handle strings with strict true/false logic
|
|
1683
|
+
// >! Only "true" and "1" are considered truthy strings
|
|
1684
|
+
if ( typeof value === 'string') {
|
|
1685
|
+
value = value.trim().toLowerCase();
|
|
1686
|
+
return value === 'true' || value === '1';
|
|
1687
|
+
}
|
|
1682
1688
|
|
|
1683
|
-
//
|
|
1684
|
-
|
|
1689
|
+
// >! For non-strings, use standard Boolean conversion
|
|
1690
|
+
// >! This handles: true, 1, objects, arrays
|
|
1691
|
+
// >! And returns false for: false, 0, null, undefined, NaN
|
|
1692
|
+
return Boolean(value);
|
|
1685
1693
|
};
|
|
1686
1694
|
|
|
1687
1695
|
/**
|
|
@@ -2995,7 +3003,7 @@ class Cache {
|
|
|
2995
3003
|
/**
|
|
2996
3004
|
* The CacheableDataAccess object provides an interface to
|
|
2997
3005
|
* the cache. It is responsible for reading from and writing to cache.
|
|
2998
|
-
* All requests to data go through
|
|
3006
|
+
* All requests to data go through CacheableDataAccess.getData
|
|
2999
3007
|
*
|
|
3000
3008
|
* Before using CacheableDataAccess, the Cache must be initialized.
|
|
3001
3009
|
*
|
|
@@ -3082,7 +3090,7 @@ class CacheableDataAccess {
|
|
|
3082
3090
|
*
|
|
3083
3091
|
* const cache = await CacheableDataAccess.getData(
|
|
3084
3092
|
* cachePolicy,
|
|
3085
|
-
* endpoint.
|
|
3093
|
+
* endpoint.send,
|
|
3086
3094
|
* connection,
|
|
3087
3095
|
* null,
|
|
3088
3096
|
* {path: 'users', id: '123'}
|