retold-remote 0.0.23 → 0.0.26

package/docs/server-setup.md

@@ -21,31 +21,31 @@ The build step bundles the client-side JavaScript with Quackage and copies asset
 
 ## Running the Server
 
-### CLI Command
+### CLI Commands
 
-```bash
-retold-remote serve [content-path] [options]
-```
-
-Or using the short alias:
+There are three bin entries provided by the `retold-remote` package:
 
 ```bash
-rr serve /path/to/media
+retold-remote serve [content-path] [options]   # full form
+rr serve [content-path] [options]               # short alias
+retold-stack [content-path] [options]           # convenience: serve --stack
 ```
 
 If `content-path` is omitted, the current directory is served.
 
+The `retold-stack` shortcut auto-injects `serve --stack`, which spawns Ultravisor as a child process and uses XDG-style data paths. See [Stack Launcher](stack-launcher.md) for the full story.
+
 ### Options
 
 | Flag | Description | Default |
 |------|-------------|---------|
 | `-p, --port [port]` | Port to listen on | Random 7000-7999 |
-| `-H, --hashed-filenames` | Enable hashed filename mode (hides real paths from browser) | Off |
-| `-c, --cache-path [path]` | Root cache directory | `./dist/retold-cache/` |
-| `--cache-thumbnails [path]` | Override thumbnail cache location | `<cache-root>/thumbnails/` |
-| `--cache-archives [path]` | Override archive extraction cache | `<cache-root>/archives/` |
-| `--cache-video-frames [path]` | Override video frame cache | `<cache-root>/video-frames/` |
-| `--cache-audio-waveforms [path]` | Override audio waveform cache | `<cache-root>/audio-waveforms/` |
+| `--no-hash` | Disable hashed filename mode (use plain paths in URLs) | Hashing on |
+| `-c, --cache-path [path]` | Root cache directory | `./dist/retold-cache/` (or `~/.cache/retold-remote/` in stack mode) |
+| `--cache-server [url]` | URL of a remote Parime cache server | None |
+| `-u, --ultravisor [url]` | Connect to an Ultravisor mesh; URL defaults to `http://localhost:54321` if omitted | None |
+| `--stack` | Spawn Ultravisor as a child process and connect to it. Uses XDG-style data paths under `~/.local/share` and `~/.cache`. See [Stack Launcher](stack-launcher.md) | Off |
+| `-l, --logfile [path]` | Write logs to a file (auto-generates timestamped name if path omitted) | Console only |
 
 ### Direct Node.js
 
@@ -65,20 +65,30 @@ Default port is `8086` when using `server.js` directly.
 ### Examples
 
 ```bash
-# Serve current directory on port 8086
-node server.js
+# Serve current directory on a random port
+retold-remote serve
 
 # Serve a specific folder
-node server.js /mnt/nas/media
+retold-remote serve /mnt/nas/media
 
 # CLI with custom port
 retold-remote serve /mnt/nas/media -p 3000
 
-# Hashed filenames for privacy
-retold-remote serve /mnt/nas/media -H
+# Plain paths in URLs (disable hashing)
+retold-remote serve /mnt/nas/media --no-hash
 
 # Custom cache location (useful for Docker volumes)
 retold-remote serve /media -c /cache
+
+# Connect to an existing Ultravisor mesh
+retold-remote serve /media -u http://192.168.1.100:54321
+
+# Full stack: spawn Ultravisor as a child process, embed Orator-Conversion,
+# use XDG paths (~/.local/share/ultravisor, ~/.cache/retold-remote)
+retold-stack /mnt/nas/media
+
+# Same thing via the explicit flag
+retold-remote serve --stack /mnt/nas/media
 ```
 
 ## Configuration
@@ -131,111 +141,132 @@ When enabled (`-H` flag or `RETOLD_HASHED_FILENAMES=true`), real file paths are
 
 ## Docker
 
-### Using the Included Dockerfile
+Retold Remote ships with two Dockerfiles, a `docker-compose.yml`, and a `docker-build-and-save.sh` helper. All images run the full stack (Ultravisor + Retold Remote + embedded Orator-Conversion) as a single container.
+
+> **Running on Synology?** See the dedicated [Synology Container Manager](synology.md) guide for a step-by-step walkthrough including how to build the image on a dev machine and transfer it to the NAS.
 
-A `Dockerfile` is provided in the repository root:
+### Quick Start (any Docker host with the source tree)
+
+The compose file defaults to `image: retold-stack:latest` so it works on hosts that already have the image loaded. To build and run from the source tree in one go:
 
 ```bash
-docker build -t retold-remote .
-docker run -p 8086:8086 -v /path/to/media:/media retold-remote
+cd retold-remote
+
+# Build the image and tag it as retold-stack:latest
+docker build -t retold-stack:latest .
+
+# Start the stack
+MEDIA_PATH=/path/to/your/media docker compose up -d
 ```
 
-### Inline Dockerfile
+Then browse to `http://localhost:7777/`. The Ultravisor coordinator web interface is at `http://localhost:54321/`.
 
-If you need to create a Dockerfile from scratch, here is a complete working version:
+If you'd rather have Compose build the image directly, edit `docker-compose.yml`:
+
+```yaml
+services:
+  retold-stack:
+    # image: retold-stack:latest    # <- comment this out
+    build:                          # <- uncomment this block
+      context: .
+      dockerfile: Dockerfile
+```
 
-```dockerfile
-FROM node:20-slim
+Then `docker compose up -d --build` will build and start in one step.
 
-# Install optional tools for full functionality
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    ffmpeg \
-    imagemagick \
-    p7zip-full \
-    calibre \
-    && rm -rf /var/lib/apt/lists/*
+### Dockerfile Variants
 
-WORKDIR /app
+Both images have been verified to build and run the full stack on ARM64 (Apple Silicon) and should build identically on AMD64.
 
-# Copy package files and install dependencies
-COPY package.json package-lock.json ./
-RUN npm ci --omit=dev && npm cache clean --force
+| File | Measured Size | Includes | Use for |
+|------|---------------|----------|---------|
+| `Dockerfile` | **3.0 GB** | Everything: ffmpeg, ImageMagick, 7z, poppler, pdftk, LibreOffice, Calibre, exiftool, dcraw | Full document conversion (DOC, DOCX, RTF, ODT, WPD, MOBI, PPT, XLS, etc.) |
+| `Dockerfile.slim` | **1.81 GB** | Same but without LibreOffice and Calibre | Image/video/audio/PDF/EPUB only -- no doc/docx/mobi conversion |
 
-# Install sharp (optional image processing)
-RUN npm install sharp || true
+> **Note on audiowaveform:** Neither image installs `audiowaveform` because it is not in the default Debian repositories. Retold Remote automatically falls back to ffprobe + ffmpeg for waveform generation, which is slower but works identically. If you need the BBC `audiowaveform` tool for faster generation, add it via a custom Dockerfile step (build from source or a PPA).
 
-# Copy application source and built assets
-COPY source/ source/
-COPY web-application/ web-application/
-COPY css/ css/
-COPY html/ html/
-COPY server.js ./
+### Building and Saving for Transfer
 
-# Create cache directory
-RUN mkdir -p /cache
+For deploying to a NAS or any host that doesn't have the source tree, use the `docker-build-and-save.sh` helper:
 
-# Default port
-ENV PORT=8086
+```bash
+./docker-build-and-save.sh                # full image, host architecture
+./docker-build-and-save.sh slim           # slim variant
+./docker-build-and-save.sh --amd64        # cross-build for x86_64
+./docker-build-and-save.sh slim --arm64   # slim + arm64
+```
 
-EXPOSE 8086
+This produces a compressed tar file (`retold-stack-image.tar.gz` or `retold-stack-image-slim.tar.gz`) that you can copy to the target host and load with:
 
-# Serve /media with cache at /cache
-CMD ["node", "server.js", "/media"]
+```bash
+docker load -i retold-stack-image.tar.gz
 ```
 
-### Docker Compose
+Then start the stack with the included `docker-compose.yml` (which references `retold-stack:latest` by default). No source tree needed on the target host.
 
-```yaml
-version: "3.8"
-services:
-  retold-remote:
-    build: .
-    ports:
-      - "8086:8086"
-    volumes:
-      - /path/to/media:/media:ro
-      - retold-cache:/cache
-    environment:
-      - PORT=8086
-    restart: unless-stopped
-
-volumes:
-  retold-cache:
+### Manual Docker Run
+
+If you prefer `docker run` over Compose:
+
+```bash
+docker build -t retold-stack .
+
+docker run -d \
+  --name retold-stack \
+  -p 7777:7777 \
+  -p 54321:54321 \
+  -v /path/to/media:/media:ro \
+  -v retold-cache:/cache \
+  -v ultravisor-data:/data \
+  -v retold-config:/config \
+  --restart unless-stopped \
+  retold-stack
 ```
 
-### Docker Usage Notes
+Then browse to `http://localhost:7777/`.
+
+### Volume Mounts
 
-- Mount your media folder to `/media` (read-only is fine with `:ro`)
-- Mount a volume to `/cache` for persistent thumbnail and frame caches
-- The `node:20-slim` base keeps the image small while the `apt-get` packages add full media processing
-- `calibre` is the largest optional package; omit it if you do not need MOBI/AZW ebook conversion
-- `sharp` is installed separately because it has native bindings that may fail on some architectures; the `|| true` ensures the build continues without it
+The stack uses four volume mount points. The first is your media; the other three are for persistent state:
 
-### Minimal Dockerfile (No Optional Tools)
+| Container path | Purpose | Typical host mapping |
+|----------------|---------|---------------------|
+| `/media` | Media folder to browse (read-only) | `/path/to/media:/media:ro` |
+| `/cache` | Thumbnails, frames, waveforms, conversions | Named volume `retold-cache` |
+| `/data` | Ultravisor datastore + staging | Named volume `ultravisor-data` |
+| `/config` | Stack config files | Named volume `retold-config` |
 
-If you only need basic browsing without thumbnails or media probing:
+Inside the container, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, and `XDG_CONFIG_HOME` are set to these paths so the stack launcher uses them automatically.
 
-```dockerfile
-FROM node:20-slim
+### Published Ports
 
-WORKDIR /app
+| Port | Purpose |
+|------|---------|
+| **7777** | Retold Remote web UI (the main thing you browse to) |
+| **54321** | Ultravisor coordinator API + web interface (optional -- useful for monitoring) |
 
-COPY package.json package-lock.json ./
-RUN npm ci --omit=dev && npm cache clean --force
+Both are pinned to fixed ports via the `-p 7777` flag in the default CMD, overriding the usual random 7000-7999 behavior.
 
-COPY source/ source/
-COPY web-application/ web-application/
-COPY css/ css/
-COPY html/ html/
-COPY server.js ./
+### Customizing the CMD
 
-ENV PORT=8086
-EXPOSE 8086
+The default command launches `retold-stack` against `/media` on port 7777 with hashed filenames disabled. To override (e.g., to enable hashed filenames or change the port):
 
-CMD ["node", "server.js", "/media"]
+```yaml
+command: ["node", "source/cli/RetoldRemote-Stack-Run.js", "/media", "-p", "7777"]
 ```
 
-This produces an image under 200MB. Images display directly (no thumbnails), videos play in the browser, and zip/cbz archives use native extraction.
+Remove the `--no-hash` flag to re-enable hashed filenames, or change `/media` if you mounted your media to a different path.
+
+### Image Size Notes
+
+- `node:20-slim` base: ~180 MB
+- Full `Dockerfile` (with LibreOffice + Calibre): ~2.5 GB -- Calibre alone is ~1 GB, LibreOffice is ~600 MB
+- `Dockerfile.slim`: ~1.8 GB -- drops both big dependencies
+- Omit `ffmpeg` and `audiowaveform` too if you don't need video/audio processing, for a ~1 GB image
+
+### Child Process Note
+
+The stack launcher spawns Ultravisor as a child process using `node` and the resolved `ultravisor` bin path. This works inside Docker without any special configuration -- the child runs in the same container PID namespace and its logs are streamed through the main process's stdout with `[ultravisor]` prefixes.
 
 ## Archive Browsing
 

package/docs/stack-launcher.md

@@ -0,0 +1,218 @@
+# Stack Launcher
+
+The `--stack` flag (and the `retold-stack` shortcut) brings up the full Retold stack as a single command: an embedded Ultravisor coordinator, the Retold Remote media browser, and Orator-Conversion (embedded inside Retold Remote). Point it at any directory and it just works -- sane XDG-style data paths, automatic readiness polling, and graceful shutdown of all child processes.
+
+## Quick Start
+
+```bash
+# Run the full stack against the current directory
+retold-stack
+
+# Or against any directory
+retold-stack /mnt/nas/media
+
+# Or via the explicit flag
+retold-remote serve --stack /mnt/nas/media
+```
+
+That's it. No config files, no separate terminals, no manual port wiring.
+
+## What Gets Launched
+
+| Component | Where it runs | Port |
+|-----------|---------------|------|
+| **Ultravisor** | Spawned as a child process | 54321 |
+| **Retold Remote** | Main process (the one you started) | Random 7000-7999 (override with `-p`) |
+| **Orator-Conversion** | Embedded inside Retold Remote (no separate process) | Same as Retold Remote |
+
+The stack launcher:
+1. Resolves XDG-style data paths
+2. Detects whether Ultravisor is already running on port 54321 (and reuses it if so)
+3. Spawns Ultravisor as a child process with a generated config file
+4. Polls until Ultravisor is accepting connections (up to 30 seconds)
+5. Sets the `UltravisorURL` automatically so Retold Remote registers as a beacon
+6. Starts the Retold Remote server with Orator-Conversion embedded
+7. Streams the Ultravisor child's stdout/stderr through the main logger with an `[ultravisor]` prefix
+8. On `SIGINT`/`SIGTERM`, disconnects the beacon, kills the Ultravisor child gracefully, and exits
+
+## Default Data Paths
+
+The stack launcher uses [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defaults so data lives in predictable, user-scoped locations regardless of what directory you launched from.
+
+| Purpose | Default Path | Override |
+|---------|-------------|----------|
+| Ultravisor datastore | `~/.local/share/ultravisor/datastore/` | `$XDG_DATA_HOME/ultravisor/datastore/` |
+| Ultravisor staging | `~/.local/share/ultravisor/staging/` | `$XDG_DATA_HOME/ultravisor/staging/` |
+| Retold Remote cache | `~/.cache/retold-remote/` | `$XDG_CACHE_HOME/retold-remote/`, or `-c` flag |
+| Stack config files | `~/.config/retold-stack/` | `$XDG_CONFIG_HOME/retold-stack/` |
+
+The cache directory holds thumbnails, video frames, audio waveforms, archive extractions, and converted ebooks/PDFs.
+
+The Ultravisor datastore holds the work-queue journal and beacon registry.
+
+These paths are created on first launch if they do not already exist.
+
+## CLI Reference
+
+### `retold-stack`
+
+Convenience entry point that auto-injects `serve --stack`.
+
+```bash
+retold-stack [content-path] [options]
+```
+
+All options accepted by `retold-remote serve` are forwarded.
+
+### `retold-remote serve --stack`
+
+The full form. Useful when you want to combine `--stack` with other flags.
+
+```bash
+retold-remote serve --stack [content-path] [options]
+```
+
+| Flag | Description |
+|------|-------------|
+| `--stack` | Spawn Ultravisor as a child process and connect to it. Sets cache root to `~/.cache/retold-remote/` if `-c` is not passed. Auto-sets the Ultravisor URL to `http://localhost:54321`. |
+
+All other `serve` options still apply (`-p`, `--no-hash`, `-c`, `-l`, etc.).
+
+## Examples
+
+### Browse the current directory with the full stack
+
+```bash
+cd ~/Pictures
+retold-stack
+```
+
+### Browse a specific NAS share
+
+```bash
+retold-stack /mnt/nas/media
+```
+
+### Pin the port and stash logs to a file
+
+```bash
+retold-stack /mnt/nas/media -p 8086 -l ~/retold.log
+```
+
+### Override just the Retold cache location
+
+```bash
+retold-remote serve --stack /mnt/nas/media -c /var/cache/retold
+```
+
+### Reuse an already-running Ultravisor
+
+If port 54321 is already accepting connections, the stack launcher detects this and connects to the existing instance instead of spawning a new one. The log line will read:
+
+```
+[stack] ultravisor already running on port 54321, reusing
+```
+
+This means you can leave Ultravisor running across multiple Retold Remote launches without conflict.
+
+## Logging
+
+The stack launcher streams the Ultravisor child process's output through Retold Remote's logger so you have a single log stream for both processes. Lines from the Ultravisor child are prefixed `[ultravisor]`, and stack-launcher events are prefixed `[stack]`:
+
+```
+[stack] launching ultravisor (port 54321)
+[stack]   data:    /Users/steven/.local/share/ultravisor/datastore
+[stack]   staging: /Users/steven/.local/share/ultravisor/staging
+[ultravisor] UltravisorTaskTypeRegistry: 53 built-in task types registered.
+[stack] ultravisor ready (after 1 attempts)
+==========================================================
+  Retold Remote running on http://localhost:7842
+==========================================================
+  Content: /mnt/nas/media
+  Cache:   /Users/steven/.cache/retold-remote
+  Browse:  http://localhost:7842/
+  Beacon:  registered with Ultravisor at http://localhost:54321
+  Stack:   ultravisor + retold-remote (orator-conversion embedded)
+==========================================================
+```
+
+If you also pass `-l <path>`, both streams are written to the file as well.
+
+## Shutdown
+
+Press `Ctrl+C` once. The launcher will:
+
+1. Log `Shutting down...`
+2. Disconnect the Ultravisor beacon (so Ultravisor knows the worker is leaving cleanly)
+3. Send `SIGTERM` to the Ultravisor child process
+4. Wait up to 1 second for graceful exit
+5. Send `SIGKILL` if it has not exited
+6. Exit the main process
+
+If the launcher reused an already-running Ultravisor (rather than spawning one), it will not kill that Ultravisor -- it only manages processes it started itself.
+
+## Architecture
+
+```
+┌────────────────────────────────────────────────────────────┐
+│  retold-stack /mnt/media                                    │
+│                                                             │
+│  ┌─────────────────────────────────────────────────────┐  │
+│  │ Main process (retold-remote)                         │  │
+│  │  ┌───────────────────────────────────────────────┐  │  │
+│  │  │ Orator HTTP server (port 7000-7999)           │  │  │
+│  │  │  ├─ Retold Remote routes (/api/media/...)     │  │  │
+│  │  │  ├─ Orator-Conversion (/api/conversion/1.0/)  │  │  │
+│  │  │  └─ Static web app (/)                        │  │  │
+│  │  └───────────────────────────────────────────────┘  │  │
+│  │  ┌───────────────────────────────────────────────┐  │  │
+│  │  │ Ultravisor Beacon client                      │  │  │
+│  │  │  └─ Connects to localhost:54321               │  │  │
+│  │  └───────────────────────────────────────────────┘  │  │
+│  └─────────────────────────────────────────────────────┘  │
+│                          ▲                                  │
+│                          │ HTTP                             │
+│                          ▼                                  │
+│  ┌─────────────────────────────────────────────────────┐  │
+│  │ Child process (ultravisor)                           │  │
+│  │  ├─ Coordinator API (port 54321)                     │  │
+│  │  ├─ Beacon registry                                  │  │
+│  │  ├─ Work queue journal                               │  │
+│  │  └─ Web interface                                    │  │
+│  │  Datastore: ~/.local/share/ultravisor/datastore/     │  │
+│  └─────────────────────────────────────────────────────┘  │
+└────────────────────────────────────────────────────────────┘
+```
+
+## Why Stack Mode?
+
+Use `--stack` when you want a single command that brings up the full Retold experience:
+
+- **Heavy media processing offloaded** through the Ultravisor beacon (video frame extraction, audio waveforms, ebook conversion, PDF page rendering)
+- **Document conversion** via the embedded Orator-Conversion service (with the `doc-to-pdf` converter for Word, RTF, ODT, WordPerfect, etc.)
+- **Persistent state** across launches (the Ultravisor datastore lives at a stable XDG location)
+- **No port juggling** -- Ultravisor on its standard port 54321, Retold Remote on a random high port
+
+Use the bare `retold-remote serve` command (without `--stack`) when you just want a quick gallery browser without the coordinator infrastructure.
+
+## Comparison
+
+| Feature | `retold-remote serve` | `retold-stack` |
+|---------|----------------------|----------------|
+| Gallery browser | [x] | [x] |
+| Image/video/audio viewers | [x] | [x] |
+| Document conversion (doc, rtf, etc.) | Embedded only | Embedded + dispatched |
+| Heavy work offloading | No | Yes (via Ultravisor) |
+| Spawns child processes | No | Yes (Ultravisor) |
+| Default cache location | `./dist/retold-cache/` | `~/.cache/retold-remote/` |
+| Survives `cd` | Yes | Yes (XDG paths are absolute) |
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---------|-----|
+| `Could not locate the ultravisor package` | Run `npm install ultravisor` (or `npm install -g retold-remote` to get the bundled version) |
+| `Ultravisor did not become ready within 30000ms` | Check the `[ultravisor]` log lines for an error. Common causes: port 54321 already bound by another process, missing dependencies, or invalid datastore path permissions |
+| Stack mode hangs at `launching ultravisor` | Ultravisor's child process may be waiting on stdin or hitting an interactive prompt. Run with `-l <path>` to capture full logs |
+| Beacon shows "not connected" | Ultravisor came up but the beacon registration failed. Check the Ultravisor child logs for errors and verify nothing is firewalling localhost:54321 |
+| Want to override XDG paths | Set `XDG_DATA_HOME`, `XDG_CACHE_HOME`, or `XDG_CONFIG_HOME` in your environment before launching, or use the `-c` flag for the Retold Remote cache |