@adobe/spacecat-shared-tokowaka-client 1.5.3 → 1.5.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +0 -316
- package/package.json +1 -1
- package/src/utils/custom-html-utils.js +24 -8
- package/test/utils/html-utils.test.js +279 -16
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,17 @@
|
|
|
1
|
+
# [@adobe/spacecat-shared-tokowaka-client-v1.5.5](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-tokowaka-client-v1.5.4...@adobe/spacecat-shared-tokowaka-client-v1.5.5) (2026-01-21)
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
### Bug Fixes
|
|
5
|
+
|
|
6
|
+
* remove details from tokowaka readme ([#1277](https://github.com/adobe/spacecat-shared/issues/1277)) ([14f3aa8](https://github.com/adobe/spacecat-shared/commit/14f3aa82cfbde56c6baebd1ad979980cbe24dd12))
|
|
7
|
+
|
|
8
|
+
# [@adobe/spacecat-shared-tokowaka-client-v1.5.4](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-tokowaka-client-v1.5.3...@adobe/spacecat-shared-tokowaka-client-v1.5.4) (2026-01-21)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Bug Fixes
|
|
12
|
+
|
|
13
|
+
* edge preview api headers handling ([#1276](https://github.com/adobe/spacecat-shared/issues/1276)) ([e2a7ba8](https://github.com/adobe/spacecat-shared/commit/e2a7ba88df77ee7e369d39e623966b5760ac9d12))
|
|
14
|
+
|
|
1
15
|
# [@adobe/spacecat-shared-tokowaka-client-v1.5.3](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-tokowaka-client-v1.5.2...@adobe/spacecat-shared-tokowaka-client-v1.5.3) (2026-01-21)
|
|
2
16
|
|
|
3
17
|
|
package/README.md
CHANGED
|
@@ -17,148 +17,6 @@ const tokowakaClient = TokowakaClient.createFrom(context);
|
|
|
17
17
|
const result = await tokowakaClient.deploySuggestions(site, opportunity, suggestions);
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
-
## API Reference
|
|
21
|
-
|
|
22
|
-
### TokowakaClient.createFrom(context)
|
|
23
|
-
|
|
24
|
-
Creates a client instance from a context object.
|
|
25
|
-
|
|
26
|
-
**Required context properties:**
|
|
27
|
-
- `context.s3.s3Client` (S3Client): AWS S3 client instance
|
|
28
|
-
- `context.log` (Object, optional): Logger instance
|
|
29
|
-
- `context.env.TOKOWAKA_SITE_CONFIG_BUCKET` (string): S3 bucket name for deployed configurations
|
|
30
|
-
- `context.env.TOKOWAKA_PREVIEW_BUCKET` (string): S3 bucket name for preview configurations
|
|
31
|
-
- `context.env.TOKOWAKA_CDN_PROVIDER` (string | string[]): CDN provider(s) for cache invalidation
|
|
32
|
-
- `context.env.TOKOWAKA_CDN_CONFIG` (string): JSON configuration for CDN clients
|
|
33
|
-
- `context.env.TOKOWAKA_EDGE_URL` (string): Tokowaka edge URL for preview HTML fetching
|
|
34
|
-
|
|
35
|
-
## Environment Variables
|
|
36
|
-
|
|
37
|
-
**Required:**
|
|
38
|
-
- `TOKOWAKA_SITE_CONFIG_BUCKET` - S3 bucket name for storing deployed configurations
|
|
39
|
-
- `TOKOWAKA_PREVIEW_BUCKET` - S3 bucket name for storing preview configurations
|
|
40
|
-
|
|
41
|
-
**Optional (for CDN invalidation):**
|
|
42
|
-
- `TOKOWAKA_CDN_PROVIDER` - CDN provider name(s). Can be a single provider (e.g., "cloudfront") or multiple providers as comma-separated string (e.g., "cloudfront,fastly") or array
|
|
43
|
-
- `TOKOWAKA_CDN_CONFIG` - JSON string with CDN-specific configuration for all providers:
|
|
44
|
-
```json
|
|
45
|
-
{
|
|
46
|
-
"cloudfront": {
|
|
47
|
-
"distributionId": "<distribution-id>",
|
|
48
|
-
"region": "us-east-1"
|
|
49
|
-
},
|
|
50
|
-
"fastly": {
|
|
51
|
-
"serviceId": "<service-id>",
|
|
52
|
-
"apiToken": "<api-token>",
|
|
53
|
-
"distributionUrl": "https://<cloudfront-distribution>.cloudfront.net"
|
|
54
|
-
}
|
|
55
|
-
}
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
**Note for Fastly**: The `distributionUrl` is required and should be the full CloudFront distribution URL (e.g., `https://deftbrsarcsf4.cloudfront.net`). Fastly uses this to construct full URLs as surrogate keys for cache purging.
|
|
59
|
-
|
|
60
|
-
**Optional (for preview functionality):**
|
|
61
|
-
- `TOKOWAKA_EDGE_URL` - Tokowaka edge URL for fetching HTML content during preview
|
|
62
|
-
|
|
63
|
-
### Main Methods
|
|
64
|
-
|
|
65
|
-
#### `deploySuggestions(site, opportunity, suggestions)`
|
|
66
|
-
|
|
67
|
-
Generates configuration and uploads to S3 **per URL**. **Automatically fetches existing configuration for each URL and merges** new suggestions with it. Invalidates CDN cache after upload. **Updates the metaconfig's `patches` field** to track deployed endpoints.
|
|
68
|
-
|
|
69
|
-
**Architecture Change:** Creates one S3 file per URL instead of a single file with all URLs. This prevents files from growing too large over time.
|
|
70
|
-
|
|
71
|
-
**Metaconfig Tracking:** After successful deployment, the method updates the domain-level metaconfig's `patches` object with the normalized paths of all deployed endpoints (e.g., `{ "/products/item": true }`). This provides a centralized registry of all deployed endpoints for the domain.
|
|
72
|
-
|
|
73
|
-
**Returns:** `Promise<DeploymentResult>` with:
|
|
74
|
-
- `s3Paths` - Array of S3 keys where configs were uploaded (one per URL)
|
|
75
|
-
- `cdnInvalidations` - Array of CDN invalidation results (one per URL per provider)
|
|
76
|
-
- `succeededSuggestions` - Array of deployed suggestions
|
|
77
|
-
- `failedSuggestions` - Array of `{suggestion, reason}` objects for ineligible suggestions
|
|
78
|
-
|
|
79
|
-
#### `rollbackSuggestions(site, opportunity, suggestions)`
|
|
80
|
-
|
|
81
|
-
Rolls back previously deployed suggestions by removing their patches from the configuration. **Automatically fetches existing configuration for each URL** and removes patches matching the provided suggestions. Invalidates CDN cache after upload.
|
|
82
|
-
|
|
83
|
-
**Architecture Change:** Updates one S3 file per URL instead of a single file with all URLs.
|
|
84
|
-
|
|
85
|
-
**Mapper-Specific Rollback Behavior:**
|
|
86
|
-
- Each opportunity mapper handles its own rollback logic via `rollbackPatches()` method
|
|
87
|
-
- **FAQ:** Automatically removes the "FAQs" heading patch if no FAQ suggestions remain for that URL
|
|
88
|
-
- **Headings/Summarization:** Simple removal by suggestion ID (default behavior)
|
|
89
|
-
|
|
90
|
-
**Returns:** `Promise<RollbackResult>` with:
|
|
91
|
-
- `s3Paths` - Array of S3 keys where configs were uploaded (one per URL)
|
|
92
|
-
- `cdnInvalidations` - Array of CDN invalidation results (one per URL per provider)
|
|
93
|
-
- `succeededSuggestions` - Array of rolled back suggestions
|
|
94
|
-
- `failedSuggestions` - Array of `{suggestion, reason}` objects for ineligible suggestions
|
|
95
|
-
- `removedPatchesCount` - Total number of patches removed across all URLs
|
|
96
|
-
|
|
97
|
-
#### `previewSuggestions(site, opportunity, suggestions, options)`
|
|
98
|
-
|
|
99
|
-
Previews suggestions by uploading to preview S3 path and fetching HTML comparison. **All suggestions must belong to the same URL.**
|
|
100
|
-
|
|
101
|
-
**Returns:** `Promise<PreviewResult>` with:
|
|
102
|
-
- `s3Path` - S3 key where preview config was uploaded
|
|
103
|
-
- `config` - Preview configuration object
|
|
104
|
-
- `cdnInvalidations` - Array of CDN invalidation results (one per provider)
|
|
105
|
-
- `succeededSuggestions` - Array of previewed suggestions
|
|
106
|
-
- `failedSuggestions` - Array of `{suggestion, reason}` objects for ineligible suggestions
|
|
107
|
-
- `html` - Object with `url`, `originalHtml`, and `optimizedHtml`
|
|
108
|
-
|
|
109
|
-
#### `fetchConfig(url, isPreview)`
|
|
110
|
-
|
|
111
|
-
Fetches existing Tokowaka configuration from S3 for a specific URL.
|
|
112
|
-
|
|
113
|
-
**Parameters:**
|
|
114
|
-
- `url` - Full URL (e.g., 'https://www.example.com/products/item')
|
|
115
|
-
- `isPreview` - Whether to fetch from preview path (default: false)
|
|
116
|
-
|
|
117
|
-
**Returns:** `Promise<TokowakaConfig | null>` - Configuration object or null if not found
|
|
118
|
-
|
|
119
|
-
#### `mergeConfigs(existingConfig, newConfig)`
|
|
120
|
-
|
|
121
|
-
Merges existing configuration with new configuration. For each URL path, checks if `opportunityId` + `suggestionId` combination exists and either updates or adds patches accordingly.
|
|
122
|
-
|
|
123
|
-
**Returns:** `TokowakaConfig` - Merged configuration
|
|
124
|
-
|
|
125
|
-
#### `generateConfig(url, opportunity, suggestions)`
|
|
126
|
-
|
|
127
|
-
Generates Tokowaka configuration from opportunity suggestions for a specific URL without uploading.
|
|
128
|
-
|
|
129
|
-
**Parameters:**
|
|
130
|
-
- `url` - Full URL for which to generate config
|
|
131
|
-
- `opportunity` - Opportunity entity
|
|
132
|
-
- `suggestions` - Array of suggestion entities
|
|
133
|
-
|
|
134
|
-
#### `uploadConfig(url, config, isPreview)`
|
|
135
|
-
|
|
136
|
-
Uploads configuration to S3 for a specific URL.
|
|
137
|
-
|
|
138
|
-
**Parameters:**
|
|
139
|
-
- `url` - Full URL (e.g., 'https://www.example.com/products/item')
|
|
140
|
-
- `config` - Tokowaka configuration object
|
|
141
|
-
- `isPreview` - Whether to upload to preview path (default: false)
|
|
142
|
-
|
|
143
|
-
**Returns:** `Promise<string>` - S3 key of uploaded configuration
|
|
144
|
-
|
|
145
|
-
## CDN Cache Invalidation
|
|
146
|
-
|
|
147
|
-
The client invalidates CDN cache after uploading configurations. Failures are logged but don't block deployment.
|
|
148
|
-
|
|
149
|
-
## Site Configuration
|
|
150
|
-
|
|
151
|
-
Sites must have the `tokowakaConfig` field in their site-config to confirm that tokowaka is enabled for that particular site.
|
|
152
|
-
|
|
153
|
-
```javascript
|
|
154
|
-
{
|
|
155
|
-
...
|
|
156
|
-
"tokowakaConfig": {
|
|
157
|
-
"enabled": true
|
|
158
|
-
}
|
|
159
|
-
}
|
|
160
|
-
```
|
|
161
|
-
|
|
162
20
|
## Supported Opportunity Types
|
|
163
21
|
|
|
164
22
|
### Headings
|
|
@@ -174,177 +32,3 @@ Sites must have the `tokowakaConfig` field in their site-config to confirm that
|
|
|
174
32
|
### Content Summarization
|
|
175
33
|
|
|
176
34
|
**Deployment Eligibility:** Currently all suggestions for `summarization` opportunity can be deployed.
|
|
177
|
-
|
|
178
|
-
## S3 Storage
|
|
179
|
-
|
|
180
|
-
Configurations are now stored **per URL** with domain-level metadata:
|
|
181
|
-
|
|
182
|
-
### Structure
|
|
183
|
-
```
|
|
184
|
-
s3://{TOKOWAKA_SITE_CONFIG_BUCKET}/opportunities/{normalized-domain}/
|
|
185
|
-
├── config (domain-level metaconfig: siteId, prerender)
|
|
186
|
-
├── {base64-encoded-path-1} (URL-specific patches)
|
|
187
|
-
├── {base64-encoded-path-2} (URL-specific patches)
|
|
188
|
-
└── ...
|
|
189
|
-
```
|
|
190
|
-
|
|
191
|
-
For preview configurations:
|
|
192
|
-
```
|
|
193
|
-
s3://{TOKOWAKA_PREVIEW_BUCKET}/preview/opportunities/{normalized-domain}/
|
|
194
|
-
├── config
|
|
195
|
-
├── {base64-encoded-path-1}
|
|
196
|
-
└── ...
|
|
197
|
-
```
|
|
198
|
-
|
|
199
|
-
**Architecture Change:** Each URL has its own configuration file instead of one file per site. Domain-level metaconfig is stored separately to avoid duplication.
|
|
200
|
-
|
|
201
|
-
**URL Normalization:**
|
|
202
|
-
- Domain: Strips `www.` prefix (e.g., `www.example.com` → `example.com`)
|
|
203
|
-
- Path: Removes trailing slash (except for root `/`), ensures starts with `/`, then base64 URL encodes
|
|
204
|
-
|
|
205
|
-
**Example:**
|
|
206
|
-
- URL: `https://www.example.com/products/item`
|
|
207
|
-
- Metaconfig Path: `opportunities/example.com/config`
|
|
208
|
-
- Patch Config Path: `opportunities/example.com/L3Byb2R1Y3RzL2l0ZW0`
|
|
209
|
-
- Where `L3Byb2R1Y3RzL2l0ZW0` is base64 URL encoding of `/products/item`
|
|
210
|
-
|
|
211
|
-
### Metaconfig File Structure
|
|
212
|
-
Domain-level metaconfig (created once per domain, shared by all URLs):
|
|
213
|
-
```json
|
|
214
|
-
{
|
|
215
|
-
"siteId": "abc-123",
|
|
216
|
-
"apiKeys": ["tokowaka-api-key-1"],
|
|
217
|
-
"tokowakaEnabled": true,
|
|
218
|
-
"enhancements": true,
|
|
219
|
-
"prerender": {
|
|
220
|
-
"allowList": [],
|
|
221
|
-
"denyList": ["/*"]
|
|
222
|
-
},
|
|
223
|
-
"patches": {
|
|
224
|
-
"/products/item": true,
|
|
225
|
-
"/about": true,
|
|
226
|
-
"/contact": true
|
|
227
|
-
}
|
|
228
|
-
}
|
|
229
|
-
```
|
|
230
|
-
|
|
231
|
-
### Configuration File Structure
|
|
232
|
-
Per-URL configuration (flat structure):
|
|
233
|
-
```json
|
|
234
|
-
{
|
|
235
|
-
"url": "https://example.com/products/item",
|
|
236
|
-
"version": "1.0",
|
|
237
|
-
"forceFail": false,
|
|
238
|
-
"prerender": true,
|
|
239
|
-
"patches": [
|
|
240
|
-
{
|
|
241
|
-
"opportunityId": "abc-123",
|
|
242
|
-
"suggestionId": "xyz-789",
|
|
243
|
-
"prerenderRequired": true,
|
|
244
|
-
"lastUpdated": 1234567890,
|
|
245
|
-
"op": "insertAfter",
|
|
246
|
-
"selector": "main",
|
|
247
|
-
"value": { ... },
|
|
248
|
-
"valueFormat": "hast",
|
|
249
|
-
"target": "ai-bots"
|
|
250
|
-
}
|
|
251
|
-
]
|
|
252
|
-
}
|
|
253
|
-
```
|
|
254
|
-
|
|
255
|
-
**Note:**
|
|
256
|
-
- `siteId` is stored only in domain-level `config` (metaconfig)
|
|
257
|
-
- `prerender` is stored in both metaconfig (domain-level) and patch files (URL-level)
|
|
258
|
-
- The `baseURL` field has been renamed to `url`
|
|
259
|
-
- The `tokowakaOptimizations` nested structure has been removed
|
|
260
|
-
- The `tokowakaForceFail` field has been renamed to `forceFail`
|
|
261
|
-
|
|
262
|
-
## CDN Cache Invalidation
|
|
263
|
-
|
|
264
|
-
The client supports multiple CDN providers for cache invalidation with **intelligent batching** for optimal performance.
|
|
265
|
-
|
|
266
|
-
### Batch Optimization
|
|
267
|
-
|
|
268
|
-
When deploying or rolling back multiple URLs:
|
|
269
|
-
- **Before**: Each URL triggered separate CDN invalidation calls (N URLs = N×2 API calls for 2 providers)
|
|
270
|
-
- **After**: All URLs are collected and invalidated in a single batch call per provider (N URLs = 2 API calls for 2 providers)
|
|
271
|
-
|
|
272
|
-
**Example Performance Improvement:**
|
|
273
|
-
- 10 URLs with CloudFront + Fastly
|
|
274
|
-
- Old: 20 API calls (10 per provider)
|
|
275
|
-
- New: **2 API calls** (1 per provider)
|
|
276
|
-
- **90% reduction in API calls!**
|
|
277
|
-
|
|
278
|
-
### Supported CDN Providers
|
|
279
|
-
|
|
280
|
-
#### CloudFront
|
|
281
|
-
AWS CloudFront CDN invalidation using the AWS SDK.
|
|
282
|
-
|
|
283
|
-
**Configuration:**
|
|
284
|
-
```json
|
|
285
|
-
{
|
|
286
|
-
"cloudfront": {
|
|
287
|
-
"distributionId": "E1234567890ABC",
|
|
288
|
-
"region": "us-east-1"
|
|
289
|
-
}
|
|
290
|
-
}
|
|
291
|
-
```
|
|
292
|
-
|
|
293
|
-
#### Fastly
|
|
294
|
-
Fastly CDN purging using surrogate key purging.
|
|
295
|
-
|
|
296
|
-
**Configuration:**
|
|
297
|
-
```json
|
|
298
|
-
{
|
|
299
|
-
"fastly": {
|
|
300
|
-
"serviceId": "abc123xyz",
|
|
301
|
-
"apiToken": "your-fastly-api-token"
|
|
302
|
-
}
|
|
303
|
-
}
|
|
304
|
-
```
|
|
305
|
-
|
|
306
|
-
### Multiple CDN Providers
|
|
307
|
-
|
|
308
|
-
To invalidate cache on multiple CDNs simultaneously:
|
|
309
|
-
|
|
310
|
-
**Environment Variable:**
|
|
311
|
-
```bash
|
|
312
|
-
TOKOWAKA_CDN_PROVIDER="cloudfront,fastly"
|
|
313
|
-
# or as array in code: ["cloudfront", "fastly"]
|
|
314
|
-
|
|
315
|
-
TOKOWAKA_CDN_CONFIG='{"cloudfront":{"distributionId":"...","region":"us-east-1"},"fastly":{"serviceId":"...","apiToken":"...","distributionUrl":"https://xxx.cloudfront.net"}}'
|
|
316
|
-
```
|
|
317
|
-
|
|
318
|
-
**Behavior:**
|
|
319
|
-
- All URLs are batched into a single invalidation request per provider
|
|
320
|
-
- All CDN providers are invalidated sequentially
|
|
321
|
-
- Failures in one CDN don't block others
|
|
322
|
-
- Each CDN returns its own result in the `cdnInvalidations` array
|
|
323
|
-
- Results include status, provider name, and provider-specific details
|
|
324
|
-
|
|
325
|
-
### Fastly Batch Purging
|
|
326
|
-
|
|
327
|
-
The Fastly client uses surrogate key purging with full CloudFront URLs:
|
|
328
|
-
- **Surrogate Keys**: Constructed as full CloudFront URLs (e.g., `https://xxx.cloudfront.net/opportunities/adobe.com/config`)
|
|
329
|
-
- **Batch Purging**: All paths are sent in a single API call using the `Surrogate-Key` header (space-separated URLs)
|
|
330
|
-
- **Configuration**: Requires `distributionUrl` in the Fastly config to construct full URLs
|
|
331
|
-
- Significantly reduces API calls and improves performance for bulk operations
|
|
332
|
-
|
|
333
|
-
**Example Fastly Configuration:**
|
|
334
|
-
```json
|
|
335
|
-
{
|
|
336
|
-
"serviceId": "abc123xyz",
|
|
337
|
-
"apiToken": "your-fastly-api-token",
|
|
338
|
-
"distributionUrl": "https://deftbrsarcsf4.cloudfront.net"
|
|
339
|
-
}
|
|
340
|
-
```
|
|
341
|
-
|
|
342
|
-
This configuration allows Fastly to purge cache entries using URLs like:
|
|
343
|
-
```
|
|
344
|
-
fastly purge --key https://deftbrsarcsf4.cloudfront.net/opportunities/adobe.com/config
|
|
345
|
-
```
|
|
346
|
-
|
|
347
|
-
## Reference Material
|
|
348
|
-
|
|
349
|
-
https://wiki.corp.adobe.com/display/AEMSites/Tokowaka+-+Spacecat+Integration
|
|
350
|
-
https://wiki.corp.adobe.com/display/AEMSites/Tokowaka+Patch+Format
|
package/package.json
CHANGED
|
@@ -24,8 +24,11 @@ function sleep(ms) {
|
|
|
24
24
|
}
|
|
25
25
|
|
|
26
26
|
/**
|
|
27
|
-
* Makes an HTTP request with retry logic
|
|
28
|
-
*
|
|
27
|
+
* Makes an HTTP request with retry logic for both original and optimized HTML.
|
|
28
|
+
* Header validation logic (same for both):
|
|
29
|
+
* - No proxy AND no cache header: Return response immediately (success)
|
|
30
|
+
* - Proxy header present BUT no cache header: Retry until cache header found
|
|
31
|
+
* - Cache header present (regardless of proxy): Return response (success)
|
|
29
32
|
* @param {string} url - URL to fetch
|
|
30
33
|
* @param {Object} options - Fetch options
|
|
31
34
|
* @param {number} maxRetries - Maximum number of retries
|
|
@@ -48,23 +51,35 @@ async function fetchWithRetry(url, options, maxRetries, retryDelayMs, log, fetch
|
|
|
48
51
|
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
|
|
49
52
|
}
|
|
50
53
|
|
|
51
|
-
// Check for
|
|
54
|
+
// Check for edge optimize headers
|
|
52
55
|
const cacheHeader = response.headers.get('x-edge-optimize-cache');
|
|
56
|
+
const proxyHeader = response.headers.get('x-edge-optimize-proxy');
|
|
57
|
+
|
|
58
|
+
log.debug(`Headers - cache: ${cacheHeader || 'none'}, proxy: ${proxyHeader || 'none'}`);
|
|
59
|
+
|
|
60
|
+
// Case 1: Cache header present (regardless of proxy) -> Success
|
|
53
61
|
if (cacheHeader) {
|
|
54
62
|
log.debug(`Cache header found (x-edge-optimize-cache: ${cacheHeader}), stopping retry logic`);
|
|
55
63
|
return response;
|
|
56
64
|
}
|
|
57
65
|
|
|
58
|
-
//
|
|
66
|
+
// Case 2: No cache header AND no proxy header -> Success (return immediately)
|
|
67
|
+
if (!proxyHeader) {
|
|
68
|
+
log.debug('No edge optimize headers found, proceeding as successful flow');
|
|
69
|
+
return response;
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// Case 3: Proxy header present BUT no cache header -> Retry until cache found
|
|
73
|
+
log.debug('Proxy header present without cache header, will retry...');
|
|
74
|
+
|
|
75
|
+
// If we haven't exhausted retries, continue
|
|
59
76
|
if (attempt < maxRetries + 1) {
|
|
60
|
-
log.debug(`No cache header found on attempt ${attempt}, will retry...`);
|
|
61
|
-
// Wait before retrying
|
|
62
77
|
log.debug(`Waiting ${retryDelayMs}ms before retry...`);
|
|
63
78
|
// eslint-disable-next-line no-await-in-loop
|
|
64
79
|
await sleep(retryDelayMs);
|
|
65
80
|
} else {
|
|
66
|
-
// Last attempt
|
|
67
|
-
log.error(`Max retries (${maxRetries}) exhausted
|
|
81
|
+
// Last attempt - throw error
|
|
82
|
+
log.error(`Max retries (${maxRetries}) exhausted. Proxy header present but cache header not found`);
|
|
68
83
|
throw new Error(`Cache header (x-edge-optimize-cache) not found after ${maxRetries} retries`);
|
|
69
84
|
}
|
|
70
85
|
} catch (error) {
|
|
@@ -145,6 +160,7 @@ export async function fetchHtmlWithWarmup(
|
|
|
145
160
|
'x-forwarded-host': forwardedHost,
|
|
146
161
|
'x-edge-optimize-api-key': apiKey,
|
|
147
162
|
'x-edge-optimize-url': urlPath,
|
|
163
|
+
'Accept-Encoding': 'identity', // Disable compression to avoid content-length: 0 issue
|
|
148
164
|
};
|
|
149
165
|
|
|
150
166
|
if (isOptimized) {
|
|
@@ -154,6 +154,150 @@ describe('HTML Utils', () => {
|
|
|
154
154
|
expect(actualUrl).to.equal('https://edge.example.com/page?tokowakaPreview=true');
|
|
155
155
|
});
|
|
156
156
|
|
|
157
|
+
it('should return immediately for optimized HTML when no headers present', async () => {
|
|
158
|
+
// Warmup succeeds
|
|
159
|
+
fetchStub.onCall(0).resolves({
|
|
160
|
+
ok: true,
|
|
161
|
+
status: 200,
|
|
162
|
+
statusText: 'OK',
|
|
163
|
+
headers: {
|
|
164
|
+
get: () => null,
|
|
165
|
+
},
|
|
166
|
+
text: async () => 'warmup',
|
|
167
|
+
});
|
|
168
|
+
// First actual call - no headers, should succeed
|
|
169
|
+
fetchStub.onCall(1).resolves({
|
|
170
|
+
ok: true,
|
|
171
|
+
status: 200,
|
|
172
|
+
statusText: 'OK',
|
|
173
|
+
headers: {
|
|
174
|
+
get: () => null,
|
|
175
|
+
},
|
|
176
|
+
text: async () => '<html>No headers</html>',
|
|
177
|
+
});
|
|
178
|
+
|
|
179
|
+
const html = await fetchHtmlWithWarmup(
|
|
180
|
+
'https://example.com/page',
|
|
181
|
+
'api-key',
|
|
182
|
+
'host',
|
|
183
|
+
'https://edge.example.com',
|
|
184
|
+
log,
|
|
185
|
+
true, // isOptimized
|
|
186
|
+
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
187
|
+
);
|
|
188
|
+
|
|
189
|
+
expect(html).to.equal('<html>No headers</html>');
|
|
190
|
+
// Should succeed immediately (warmup + 1 attempt)
|
|
191
|
+
expect(fetchStub.callCount).to.equal(2);
|
|
192
|
+
});
|
|
193
|
+
|
|
194
|
+
it('should throw error for optimized HTML when proxy present but cache not found after retries', async () => {
|
|
195
|
+
// Warmup succeeds
|
|
196
|
+
fetchStub.onCall(0).resolves({
|
|
197
|
+
ok: true,
|
|
198
|
+
status: 200,
|
|
199
|
+
statusText: 'OK',
|
|
200
|
+
headers: {
|
|
201
|
+
get: () => null,
|
|
202
|
+
},
|
|
203
|
+
text: async () => 'warmup',
|
|
204
|
+
});
|
|
205
|
+
// All actual calls have proxy but no cache header
|
|
206
|
+
fetchStub.onCall(1).resolves({
|
|
207
|
+
ok: true,
|
|
208
|
+
status: 200,
|
|
209
|
+
statusText: 'OK',
|
|
210
|
+
headers: {
|
|
211
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
212
|
+
},
|
|
213
|
+
text: async () => '<html>Proxy only 1</html>',
|
|
214
|
+
});
|
|
215
|
+
fetchStub.onCall(2).resolves({
|
|
216
|
+
ok: true,
|
|
217
|
+
status: 200,
|
|
218
|
+
statusText: 'OK',
|
|
219
|
+
headers: {
|
|
220
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
221
|
+
},
|
|
222
|
+
text: async () => '<html>Proxy only 2</html>',
|
|
223
|
+
});
|
|
224
|
+
fetchStub.onCall(3).resolves({
|
|
225
|
+
ok: true,
|
|
226
|
+
status: 200,
|
|
227
|
+
statusText: 'OK',
|
|
228
|
+
headers: {
|
|
229
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
230
|
+
},
|
|
231
|
+
text: async () => '<html>Proxy only 3</html>',
|
|
232
|
+
});
|
|
233
|
+
|
|
234
|
+
try {
|
|
235
|
+
await fetchHtmlWithWarmup(
|
|
236
|
+
'https://example.com/page',
|
|
237
|
+
'api-key',
|
|
238
|
+
'host',
|
|
239
|
+
'https://edge.example.com',
|
|
240
|
+
log,
|
|
241
|
+
true, // isOptimized
|
|
242
|
+
{ warmupDelayMs: 0, maxRetries: 2, retryDelayMs: 0 },
|
|
243
|
+
);
|
|
244
|
+
expect.fail('Should have thrown error');
|
|
245
|
+
} catch (error) {
|
|
246
|
+
expect(error.message).to.include('Failed to fetch optimized HTML');
|
|
247
|
+
expect(error.message).to.include('Cache header (x-edge-optimize-cache) not found after 2 retries');
|
|
248
|
+
}
|
|
249
|
+
|
|
250
|
+
// Should have tried 3 times (initial + 2 retries) plus warmup
|
|
251
|
+
expect(fetchStub.callCount).to.equal(4);
|
|
252
|
+
});
|
|
253
|
+
|
|
254
|
+
it('should retry for optimized HTML when proxy present until cache found', async () => {
|
|
255
|
+
// Warmup succeeds
|
|
256
|
+
fetchStub.onCall(0).resolves({
|
|
257
|
+
ok: true,
|
|
258
|
+
status: 200,
|
|
259
|
+
statusText: 'OK',
|
|
260
|
+
headers: {
|
|
261
|
+
get: () => null,
|
|
262
|
+
},
|
|
263
|
+
text: async () => 'warmup',
|
|
264
|
+
});
|
|
265
|
+
// First call has only proxy header - should retry
|
|
266
|
+
fetchStub.onCall(1).resolves({
|
|
267
|
+
ok: true,
|
|
268
|
+
status: 200,
|
|
269
|
+
statusText: 'OK',
|
|
270
|
+
headers: {
|
|
271
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
272
|
+
},
|
|
273
|
+
text: async () => '<html>Proxy only</html>',
|
|
274
|
+
});
|
|
275
|
+
// Second call has cache header (proxy might still be there) - should succeed
|
|
276
|
+
fetchStub.onCall(2).resolves({
|
|
277
|
+
ok: true,
|
|
278
|
+
status: 200,
|
|
279
|
+
statusText: 'OK',
|
|
280
|
+
headers: {
|
|
281
|
+
get: (name) => (name === 'x-edge-optimize-cache' ? 'HIT' : null),
|
|
282
|
+
},
|
|
283
|
+
text: async () => '<html>Cached HTML</html>',
|
|
284
|
+
});
|
|
285
|
+
|
|
286
|
+
const html = await fetchHtmlWithWarmup(
|
|
287
|
+
'https://example.com/page',
|
|
288
|
+
'api-key',
|
|
289
|
+
'host',
|
|
290
|
+
'https://edge.example.com',
|
|
291
|
+
log,
|
|
292
|
+
true, // isOptimized
|
|
293
|
+
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
294
|
+
);
|
|
295
|
+
|
|
296
|
+
expect(html).to.equal('<html>Cached HTML</html>');
|
|
297
|
+
// Should retry when only proxy present (warmup + 2 attempts)
|
|
298
|
+
expect(fetchStub.callCount).to.equal(3);
|
|
299
|
+
});
|
|
300
|
+
|
|
157
301
|
it('should throw error when HTTP response is not ok', async () => {
|
|
158
302
|
// Warmup succeeds
|
|
159
303
|
fetchStub.onCall(0).resolves({
|
|
@@ -285,7 +429,7 @@ describe('HTML Utils', () => {
|
|
|
285
429
|
}
|
|
286
430
|
});
|
|
287
431
|
|
|
288
|
-
it('should
|
|
432
|
+
it('should return immediately when no edge optimize headers are present', async () => {
|
|
289
433
|
// Warmup succeeds
|
|
290
434
|
fetchStub.onCall(0).resolves({
|
|
291
435
|
ok: true,
|
|
@@ -296,7 +440,7 @@ describe('HTML Utils', () => {
|
|
|
296
440
|
},
|
|
297
441
|
text: async () => 'warmup',
|
|
298
442
|
});
|
|
299
|
-
// First actual call - no
|
|
443
|
+
// First actual call - no headers, should succeed immediately
|
|
300
444
|
fetchStub.onCall(1).resolves({
|
|
301
445
|
ok: true,
|
|
302
446
|
status: 200,
|
|
@@ -304,17 +448,58 @@ describe('HTML Utils', () => {
|
|
|
304
448
|
headers: {
|
|
305
449
|
get: () => null,
|
|
306
450
|
},
|
|
307
|
-
text: async () => '<html>No
|
|
451
|
+
text: async () => '<html>No headers</html>',
|
|
452
|
+
});
|
|
453
|
+
|
|
454
|
+
const html = await fetchHtmlWithWarmup(
|
|
455
|
+
'https://example.com/page',
|
|
456
|
+
'api-key',
|
|
457
|
+
'host',
|
|
458
|
+
'https://edge.example.com',
|
|
459
|
+
log,
|
|
460
|
+
false,
|
|
461
|
+
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
462
|
+
);
|
|
463
|
+
|
|
464
|
+
expect(html).to.equal('<html>No headers</html>');
|
|
465
|
+
// Should succeed immediately without retry (warmup + 1 attempt)
|
|
466
|
+
expect(fetchStub.callCount).to.equal(2);
|
|
467
|
+
});
|
|
468
|
+
|
|
469
|
+
it('should retry when proxy header present without cache until cache is found', async () => {
|
|
470
|
+
// Warmup succeeds
|
|
471
|
+
fetchStub.onCall(0).resolves({
|
|
472
|
+
ok: true,
|
|
473
|
+
status: 200,
|
|
474
|
+
statusText: 'OK',
|
|
475
|
+
headers: {
|
|
476
|
+
get: () => null,
|
|
477
|
+
},
|
|
478
|
+
text: async () => 'warmup',
|
|
479
|
+
});
|
|
480
|
+
// First call has proxy header but no cache - should retry
|
|
481
|
+
fetchStub.onCall(1).resolves({
|
|
482
|
+
ok: true,
|
|
483
|
+
status: 200,
|
|
484
|
+
statusText: 'OK',
|
|
485
|
+
headers: {
|
|
486
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
487
|
+
},
|
|
488
|
+
text: async () => '<html>Proxy only</html>',
|
|
308
489
|
});
|
|
309
|
-
// Second
|
|
490
|
+
// Second call has both headers - should succeed
|
|
310
491
|
fetchStub.onCall(2).resolves({
|
|
311
492
|
ok: true,
|
|
312
493
|
status: 200,
|
|
313
494
|
statusText: 'OK',
|
|
314
495
|
headers: {
|
|
315
|
-
get: (name) =>
|
|
496
|
+
get: (name) => {
|
|
497
|
+
if (name === 'x-edge-optimize-cache') return 'HIT';
|
|
498
|
+
if (name === 'x-edge-optimize-proxy') return 'true';
|
|
499
|
+
return null;
|
|
500
|
+
},
|
|
316
501
|
},
|
|
317
|
-
text: async () => '<html>
|
|
502
|
+
text: async () => '<html>Both headers</html>',
|
|
318
503
|
});
|
|
319
504
|
|
|
320
505
|
const html = await fetchHtmlWithWarmup(
|
|
@@ -327,12 +512,12 @@ describe('HTML Utils', () => {
|
|
|
327
512
|
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
328
513
|
);
|
|
329
514
|
|
|
330
|
-
expect(html).to.equal('<html>
|
|
331
|
-
// Should
|
|
515
|
+
expect(html).to.equal('<html>Both headers</html>');
|
|
516
|
+
// Should retry when only proxy present (warmup + 2 attempts)
|
|
332
517
|
expect(fetchStub.callCount).to.equal(3);
|
|
333
518
|
});
|
|
334
519
|
|
|
335
|
-
it('should throw error when
|
|
520
|
+
it('should throw error when proxy header present but cache not found after max retries', async () => {
|
|
336
521
|
// Warmup succeeds
|
|
337
522
|
fetchStub.onCall(0).resolves({
|
|
338
523
|
ok: true,
|
|
@@ -343,33 +528,33 @@ describe('HTML Utils', () => {
|
|
|
343
528
|
},
|
|
344
529
|
text: async () => 'warmup',
|
|
345
530
|
});
|
|
346
|
-
// All actual calls
|
|
531
|
+
// All actual calls have proxy but no cache header
|
|
347
532
|
fetchStub.onCall(1).resolves({
|
|
348
533
|
ok: true,
|
|
349
534
|
status: 200,
|
|
350
535
|
statusText: 'OK',
|
|
351
536
|
headers: {
|
|
352
|
-
get: () => null,
|
|
537
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
353
538
|
},
|
|
354
|
-
text: async () => '<html>
|
|
539
|
+
text: async () => '<html>Proxy only 1</html>',
|
|
355
540
|
});
|
|
356
541
|
fetchStub.onCall(2).resolves({
|
|
357
542
|
ok: true,
|
|
358
543
|
status: 200,
|
|
359
544
|
statusText: 'OK',
|
|
360
545
|
headers: {
|
|
361
|
-
get: () => null,
|
|
546
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
362
547
|
},
|
|
363
|
-
text: async () => '<html>
|
|
548
|
+
text: async () => '<html>Proxy only 2</html>',
|
|
364
549
|
});
|
|
365
550
|
fetchStub.onCall(3).resolves({
|
|
366
551
|
ok: true,
|
|
367
552
|
status: 200,
|
|
368
553
|
statusText: 'OK',
|
|
369
554
|
headers: {
|
|
370
|
-
get: () => null,
|
|
555
|
+
get: (name) => (name === 'x-edge-optimize-proxy' ? 'true' : null),
|
|
371
556
|
},
|
|
372
|
-
text: async () => '<html>
|
|
557
|
+
text: async () => '<html>Proxy only 3</html>',
|
|
373
558
|
});
|
|
374
559
|
|
|
375
560
|
try {
|
|
@@ -428,6 +613,84 @@ describe('HTML Utils', () => {
|
|
|
428
613
|
// Should not retry if cache header found on first attempt
|
|
429
614
|
expect(fetchStub.callCount).to.equal(2); // warmup + 1 actual
|
|
430
615
|
});
|
|
616
|
+
|
|
617
|
+
it('should return immediately when cache header is present (with or without proxy)', async () => {
|
|
618
|
+
// Warmup succeeds
|
|
619
|
+
fetchStub.onCall(0).resolves({
|
|
620
|
+
ok: true,
|
|
621
|
+
status: 200,
|
|
622
|
+
statusText: 'OK',
|
|
623
|
+
headers: {
|
|
624
|
+
get: () => null,
|
|
625
|
+
},
|
|
626
|
+
text: async () => 'warmup',
|
|
627
|
+
});
|
|
628
|
+
// First actual call has cache header (proxy may or may not be present)
|
|
629
|
+
fetchStub.onCall(1).resolves({
|
|
630
|
+
ok: true,
|
|
631
|
+
status: 200,
|
|
632
|
+
statusText: 'OK',
|
|
633
|
+
headers: {
|
|
634
|
+
get: (name) => {
|
|
635
|
+
if (name === 'x-edge-optimize-cache') return 'HIT';
|
|
636
|
+
if (name === 'x-edge-optimize-proxy') return 'true';
|
|
637
|
+
return null;
|
|
638
|
+
},
|
|
639
|
+
},
|
|
640
|
+
text: async () => '<html>Cache header present</html>',
|
|
641
|
+
});
|
|
642
|
+
|
|
643
|
+
const html = await fetchHtmlWithWarmup(
|
|
644
|
+
'https://example.com/page',
|
|
645
|
+
'api-key',
|
|
646
|
+
'host',
|
|
647
|
+
'https://edge.example.com',
|
|
648
|
+
log,
|
|
649
|
+
false,
|
|
650
|
+
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
651
|
+
);
|
|
652
|
+
|
|
653
|
+
expect(html).to.equal('<html>Cache header present</html>');
|
|
654
|
+
// Should succeed immediately when cache header present (warmup + 1 attempt)
|
|
655
|
+
expect(fetchStub.callCount).to.equal(2);
|
|
656
|
+
});
|
|
657
|
+
|
|
658
|
+
it('should succeed when only cache header is present (no proxy header)', async () => {
|
|
659
|
+
// Warmup succeeds
|
|
660
|
+
fetchStub.onCall(0).resolves({
|
|
661
|
+
ok: true,
|
|
662
|
+
status: 200,
|
|
663
|
+
statusText: 'OK',
|
|
664
|
+
headers: {
|
|
665
|
+
get: () => null,
|
|
666
|
+
},
|
|
667
|
+
text: async () => 'warmup',
|
|
668
|
+
});
|
|
669
|
+
// First actual call has only cache header
|
|
670
|
+
fetchStub.onCall(1).resolves({
|
|
671
|
+
ok: true,
|
|
672
|
+
status: 200,
|
|
673
|
+
statusText: 'OK',
|
|
674
|
+
headers: {
|
|
675
|
+
get: (name) => (name === 'x-edge-optimize-cache' ? 'HIT' : null),
|
|
676
|
+
},
|
|
677
|
+
text: async () => '<html>Cache only HTML</html>',
|
|
678
|
+
});
|
|
679
|
+
|
|
680
|
+
const html = await fetchHtmlWithWarmup(
|
|
681
|
+
'https://example.com/page',
|
|
682
|
+
'api-key',
|
|
683
|
+
'host',
|
|
684
|
+
'https://edge.example.com',
|
|
685
|
+
log,
|
|
686
|
+
false,
|
|
687
|
+
{ warmupDelayMs: 0, maxRetries: 3, retryDelayMs: 0 },
|
|
688
|
+
);
|
|
689
|
+
|
|
690
|
+
expect(html).to.equal('<html>Cache only HTML</html>');
|
|
691
|
+
// Should succeed immediately with cache header only (warmup + 1 attempt)
|
|
692
|
+
expect(fetchStub.callCount).to.equal(2);
|
|
693
|
+
});
|
|
431
694
|
});
|
|
432
695
|
|
|
433
696
|
describe('calculateForwardedHost', () => {
|