@smart-cloud/publisher-exporter 1.1.65 → 1.1.67
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -8
- package/dist/content-sync.js +15 -14
- package/dist/crawl.js +13 -12
- package/dist/deploy.js +6 -6
- package/dist/invalidate.js +2 -2
- package/dist/queue-runner.js +3 -3
- package/dist/remote-worker-health.js +1 -1
- package/dist/remote-worker.js +3 -3
- package/package.json +2 -1
- package/remote-workers.example.json +3 -0
package/README.md
CHANGED
|
@@ -181,7 +181,7 @@ small `resultKey` pointer. Reusing the same job/task ID is idempotent.
|
|
|
181
181
|
|
|
182
182
|
The companion CDK application is maintained separately at
|
|
183
183
|
[`smartcloudsol/static-publisher-lambda-workers`](https://github.com/smartcloudsol/static-publisher-lambda-workers).
|
|
184
|
-
|
|
184
|
+
Its deploy script emits a `remote-workers.json` containing the
|
|
185
185
|
versioned Lambda alias ARNs, workspace bucket/prefix, region, and generated
|
|
186
186
|
assumable role. The file contains no static AWS access keys.
|
|
187
187
|
|
|
@@ -194,10 +194,12 @@ publisher-exporter remote-worker-health \
|
|
|
194
194
|
--runtime-dir /srv/site/runtime
|
|
195
195
|
```
|
|
196
196
|
|
|
197
|
-
Protocol v1 accepts 1-20 same-origin URLs per render task.
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
197
|
+
Protocol v1 accepts 1-20 same-origin URLs per render task. The default render
|
|
198
|
+
batch size is 5, so a warm Chromium process can render several pages
|
|
199
|
+
sequentially in one invocation while Lambda concurrency supplies parallelism
|
|
200
|
+
between batches. Only failed or deferred URLs are retried. The Lambda response
|
|
201
|
+
is not the rendered body; read the returned S3 result object using the same
|
|
202
|
+
temporary role credentials.
|
|
201
203
|
|
|
202
204
|
Enable unified Lambda processing in the Static Publisher WordPress admin. The
|
|
203
205
|
single Processing concurrency value is dual-written to the legacy per-phase
|
|
@@ -206,6 +208,17 @@ remain in the workspace through rewrite and direct S3-to-S3 deployment; the
|
|
|
206
208
|
coordinator transfers only task/result metadata. Older mixed phase settings
|
|
207
209
|
remain supported until an administrator changes the common controls. A current
|
|
208
210
|
`remote-workers.json` must contain render, asset, rewrite, and deploy-copy aliases.
|
|
211
|
+
When it also contains `status.tableName`, the exporter reads strongly consistent
|
|
212
|
+
DynamoDB heartbeats while a synchronous Lambda invocation is running and
|
|
213
|
+
forwards them to the existing runtime progress snapshot. Workers write those
|
|
214
|
+
rows directly with conditionally increasing sequence numbers.
|
|
215
|
+
|
|
216
|
+
Workers prefer the origin response `Content-Type`, fall back to the resource
|
|
217
|
+
extension only when the header is absent or invalid, and preserve that value
|
|
218
|
+
when writing transformed HTML, CSS, JavaScript, or other text back to S3. The
|
|
219
|
+
deploy worker includes `Content-Type` in its unchanged-object decision. After
|
|
220
|
+
upgrading from an earlier delegated rewrite implementation, republish affected
|
|
221
|
+
targets to replace already incorrect object metadata.
|
|
209
222
|
|
|
210
223
|
The saved runtime configuration remains compatible with older exporters:
|
|
211
224
|
|
|
@@ -217,6 +230,7 @@ The saved runtime configuration remains compatible with older exporters:
|
|
|
217
230
|
"render": {
|
|
218
231
|
"enabled": true,
|
|
219
232
|
"concurrency": 4,
|
|
233
|
+
"batchSize": 5,
|
|
220
234
|
"maxAttempts": 2
|
|
221
235
|
},
|
|
222
236
|
"rewrite": {
|
|
@@ -247,9 +261,9 @@ STATIC_PUBLISHER_RUNTIME_DIR=/srv/site/runtime \
|
|
|
247
261
|
Use `--local-render` only as a legacy render diagnostic override; it disables
|
|
248
262
|
the unified remote data path for that crawl. The coordinator keeps sitemap
|
|
249
263
|
discovery, incremental and safety decisions, compact manifests, and deploy
|
|
250
|
-
planning local.
|
|
251
|
-
stays in S3, while
|
|
252
|
-
reference. Asset workers fetch and inspect bounded URL batches. Final text
|
|
264
|
+
planning local. Admitted pages are grouped into bounded Lambda batches. Their
|
|
265
|
+
HTML stays in S3, while each result returns discovered pages/assets and an
|
|
266
|
+
object reference. Asset workers fetch and inspect bounded URL batches. Final text
|
|
253
267
|
objects are rewritten in place by Lambda, then deploy workers check or copy
|
|
254
268
|
them directly from the workspace to the allowlisted target. Only explicit
|
|
255
269
|
deploy-plan tombstones may be deleted. Each completed
|