@smart-cloud/publisher-exporter 1.1.19 → 1.1.20

package/README.md

@@ -29,9 +29,13 @@ publisher-exporter install-browsers
 If `PLAYWRIGHT_BROWSERS_PATH` points to a shared system directory, create that directory first and make it writable by the same OS user that will run `publisher-exporter install-browsers`. A one-time elevated setup step to create or re-own the directory is fine. The later cron job does not need elevated privileges to use an already installed shared browser cache, but it does need read and execute access to that directory tree.
 
 ## Commands
+- Normal usage:
 
 ```bash
 PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl
+PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --crawl-mode incremental
+PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --resume-rewrite
+PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --retry-timeouts
 PUBLISHER_CONFIG=./publisher.config.json publisher-exporter deploy
 PUBLISHER_CONFIG=./publisher.config.json publisher-exporter invalidate
 publisher-exporter publish
@@ -42,11 +46,40 @@ Equivalent `npx` usage:
 
 ```bash
 PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl
+PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl --crawl-mode incremental
+PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl --resume-rewrite
 PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter deploy
 PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter invalidate
 npx @smart-cloud/publisher-exporter queue-runner --runtime-dir /srv/site/runtime --max-jobs 1
 ```
 
+## Crawl Modes And Repair Workflows
+
+- `crawl` without extra flags performs a normal full crawl, asset download, and final text rewrite.
+- `crawl --crawl-mode incremental` reuses the existing crawl manifest when possible and rewrites only the files affected by re-crawled pages, text assets, or changed asset-map entries.
+- `crawl --resume-rewrite` skips discovery, rendering, and asset download, then reruns the final text rewrite over the existing output tree using the current rewrite rules.
+- `crawl --retry-timeouts` retries timed-out URLs from the latest archived full crawl or publish log snapshot.
+
+Use `--resume-rewrite` when the already exported output is otherwise valid but the rewrite logic changed or a previous crawl stopped during the rewrite phase. This is the fastest repair path for stale output caused by rewrite-only bugs such as escaped JSON replacement issues, protocol-relative asset URLs like `//host/path.css`, or an interrupted final rewrite.
+
+If you need to repair existing exported HTML after a rewrite bug, prefer `--resume-rewrite` over a new incremental crawl. Incremental crawl only rewrites targeted files; `--resume-rewrite` reprocesses every text file in the current output.
+
+During `--resume-rewrite`, the terminal can be quieter than a full crawl. Progress still updates in `runtime/current-progress.json` and the live crawl event snapshot under the configured log directory.
+
+## Direct CLI With Runtime-Managed Config
+
+When the WordPress plugin already wrote `runtime/config.json` into shared publisher storage, direct CLI invocation still needs both the config path and the runtime directory.
+
+Example:
+
+```bash
+STATIC_PUBLISHER_RUNTIME_DIR=/mnt/site/runtime \
+PUBLISHER_CONFIG=/mnt/site/runtime/config.json \
+publisher-exporter crawl --resume-rewrite
+```
+
+`PUBLISHER_CONFIG` selects the main exporter config file. `STATIC_PUBLISHER_RUNTIME_DIR` anchors runtime artifacts such as `current-progress.json`, `crawl-manifest.json`, queue files, and storage-relative output/log paths.
+
 ## Queue Runner
 
 The queue runner reads the runtime JSON files generated by the WordPress plugin.
@@ -123,6 +156,14 @@ Archived files are gzip-compressed per file and listed in `job.json`. WordPress
 
 Prune old archive directories with `publisher-exporter prune-logs --runtime-dir /srv/site/runtime --older-than-days 30`. Add that command to daily cron or another retention scheduler that matches your log policy.
 
+## Troubleshooting
+
+- If exported HTML still contains old rewritten URLs after a rewrite fix, run `crawl --resume-rewrite`. This reprocesses every text file in the current output tree and does not depend on earlier `rewriteComplete: true` entries in `crawl-manifest.json`.
+- If `--resume-rewrite` appears to stop after the initial resume message, check `runtime/current-progress.json` and `<logDir>/current-crawl-event.json` before assuming it is stuck. That phase can stay mostly quiet on stdout while it rewrites files.
+- If direct CLI execution seems to ignore the expected runtime state, verify that `PUBLISHER_CONFIG` and `STATIC_PUBLISHER_RUNTIME_DIR` both point at the same runtime tree. A mismatched config path and runtime dir can make the run look idle or incomplete.
+- Prefer incremental crawl when the existing manifest is trusted and you only need a fast recrawl of changed pages or assets. Prefer a full crawl when discovery rules changed, the manifest is missing or suspect, or you want a clean end-to-end rebuild.
+- For generated 404 capture, set `generated404RequestPath` to a path that really returns HTTP 404 from the origin. The exporter validates that response before reusing it as the generated 404 page.
+
 ## WordPress Integration
 
 The companion WordPress plugin can optionally store an `External exporter dir` setting. Point it at this package root when you want PHP-side diagnostics to verify the local CLI install.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@smart-cloud/publisher-exporter",
-  "version": "1.1.19",
+  "version": "1.1.20",
   "license": "MIT",
   "type": "module",
   "description": "Headless Playwright static publisher for WordPress/Elementor sites with sitemap-only page discovery, strict asset capture, escaped URL rewrite, structured logs, and targeted retry modes.",