retold-remote 0.0.23 → 0.0.26

package/web-application/docs/ebook-reader.md

@@ -79,12 +79,86 @@ RUN apt-get update && apt-get install -y calibre && rm -rf /var/lib/apt/lists/*
 
 ## PDF Files
 
-PDF files are rendered using the browser's native PDF viewer in an iframe. No additional tools or libraries are required. The browser provides its own controls for page navigation, zoom, and search.
+PDF files are rendered using a full **pdf.js** canvas pipeline. The library is loaded from CDN on first use; no server-side dependencies are required for basic display.
+
+### Page Navigation
+
+The controls bar exposes:
+
+- **<- Prev** / **Next ->** -- page navigation
+- **Page input** -- type a page number and press Enter to jump
+- **Zoom In / Out / Fit** -- adjust scale (0.25x - 5x)
+
+### Text Selection
+
+PDF.js renders an invisible text layer over the canvas, so native browser text selection works. Drag to select text, then click ** Save Selection** to capture it as a labeled region.
+
+The capture stores: page number, selected text, optional label.
+
+### Visual Region Selection
+
+Click ** Select Region** to enter visual selection mode. A crosshair cursor appears over the page canvas. Drag a rectangle, release, and a label input appears in the controls bar.
+
+The capture stores: page number, X/Y/Width/Height in PDF units (so the region remaps correctly at any zoom level), optional label.
+
+### Server-Side Text Extraction
+
+`GET /api/media/pdf-text?path=<file>&page=<num>` returns the extracted text for a specific page (via pdf-parse). This is useful for search indexing or for when you want the full text without selecting it manually.
+
+## Convertible Document Formats
+
+Beyond PDF/EPUB/MOBI, the viewer also handles `.doc`, `.docx`, `.rtf`, `.odt`, `.wpd`, `.wps`, `.pages`, `.odp`, `.ppt`, `.pptx`, `.ods`, `.xls`, and `.xlsx` files. These are converted to PDF on the fly via the embedded **Orator-Conversion** service, then displayed in the same pdf.js viewer.
+
+### Conversion Pipeline
+
+1. The browser opens a convertible document
+2. Retold Remote shows a "Converting document to PDF..." spinner
+3. The server calls `GET /api/media/doc-convert?path=<file>` which delegates to `orator-conversion`'s `doc-to-pdf` custom converter
+4. The converter pipes the file through LibreOffice headless (`soffice --headless --convert-to pdf`)
+5. Falls back to Calibre's `ebook-convert` if LibreOffice is unavailable
+6. The resulting PDF is cached in the same Parime cache as ebooks (separate `manifest-pdf.json`)
+7. The browser loads the PDF in the standard pdf.js viewer with full text and region selection
+
+Conversion results are cached per file (keyed by path + mtime), so reopening the same document is instant after the first conversion.
+
+### Required Tools
+
+| Tool | Use |
+|------|-----|
+| **LibreOffice** (`soffice`) | Best fidelity for Word, RTF, ODT, WordPerfect, PowerPoint, Excel |
+| **Calibre** (`ebook-convert`) | Fallback for documents LibreOffice can't handle |
+
+Install at least one of them on the server. On macOS: `brew install --cask libreoffice` or `brew install calibre`.
+
+## eBook Selection Features
+
+The EPUB reader supports the same labeled-region capture pattern as PDF and image viewers.
+
+### Save Text Selection
+
+1. Select text in the rendered EPUB content (works through the epub.js iframe via `getContents()`)
+2. Click ** Save Selection** in the controls bar
+3. Enter a label and save
+
+The capture stores:
+
+- **CFI** (Canonical Fragment Identifier) -- exact location for re-navigating later
+- **Spine index** -- which section of the book
+- **Chapter title** -- best-effort lookup from the TOC
+- **Selected text** -- the actual highlighted prose
+
+Click a saved text selection in the **Regions** sidebar tab to navigate back to its exact location via `rendition.display(cfi)`.
+
+### Visual Region Selection
+
+Click ** Select Region** to overlay a transparent crosshair on the rendered page. Drag a rectangle to capture a visual area. The capture stores X/Y/Width/Height plus the viewport dimensions at capture time (so coordinates can be remapped if the window is resized later).
 
 ## Limitations
 
 - MOBI conversion requires Calibre's `ebook-convert` on the server
+- Document conversion (DOC, DOCX, RTF, etc.) requires LibreOffice or Calibre on the server
 - DRM-protected ebooks cannot be read
 - Complex EPUB layouts (fixed-layout EPUBs, heavy CSS) may not render perfectly
 - The reader uses single-page mode only (no two-page spreads)
 - Page numbers shown in the UI correspond to the reflowed content, not the original book pagination
+- EPUB visual region coordinates are container-relative; they remap reasonably at different window sizes but are not perfectly stable across major layout reflows

package/web-application/docs/image-explorer.md

@@ -51,10 +51,70 @@ If Sharp is not available on the server, the image loads directly regardless of
 | `+` / `=` | Zoom in (1.5x) |
 | `-` | Zoom out (1.5x) |
 | `0` | Reset to home zoom (fit to view) |
-| `a` | Quick-add to active collection |
+| `s` | Toggle region selection mode |
+| `a` | Quick-add to active collection (current region if one is selected) |
 | `b` | Toggle collection panel |
 | `h` | Toggle favorite |
-| `Esc` | Back to image viewer |
+| `Esc` | Unwind one layer: exit edit mode -> exit selection mode -> back to viewer |
+
+`Esc` is layered: if you are editing a saved region, the first `Esc` exits edit mode; if you are drawing a new region, it cancels the draw; otherwise it closes the explorer.
+
+## Subimage Regions
+
+Press `s` (or click the ** Select** button in the header) to enter region selection mode. The cursor changes to a crosshair and OpenSeadragon's drag-to-pan is temporarily disabled.
+
+Drag a rectangle on the image. On release, an inline label input appears in the controls bar -- type a name and hit Enter. The region is saved with these coordinates:
+
+- **X, Y** -- top-left corner in original image pixels
+- **Width, Height** -- dimensions in original image pixels
+
+Saved regions render as gold-bordered overlays with floating label badges. They persist per file via the SubimageService and appear in the **Regions** sidebar tab. Overlays are restored automatically whenever you reopen the image, including after the tile-viewer swap for DZI-mode images.
+
+### Editing Existing Regions
+
+Double-click any saved region overlay to enter **edit mode**. The selected region is highlighted, the other regions dim to 35% opacity, and eight drag handles appear (four corners + four edges). Each handle has a larger invisible hit area around a small visible dot so it's easy to grab on both mouse and touch.
+
+In edit mode you can:
+
+- **Resize** -- drag a corner or edge handle. Dragging past the opposite edge flips the rectangle (so you can go from tl-drag to br-drag without re-entering edit mode).
+- **Move** -- drag the body of the region.
+- **Edit label** -- the inline label input in the controls bar is pre-filled with the current label. Change it and press `Enter` (or click Save).
+- **Cancel** -- click the Cancel button, press `Esc`, or double-click a different region.
+
+Geometry and label changes are **optimistic**: they apply locally the instant you release the drag or press Enter, then fire a `PUT /api/media/subimage-regions/:id` in the background. If the server request fails (e.g., because the file's mtime changed since the region was loaded), the change reverts and a toast explains the failure.
+
+While in edit mode OpenSeadragon's mouse navigation (pan/zoom/click-to-zoom) is suspended inside the region rectangle so drags on the handles or body don't also pan the viewer. Clicks outside any region still pan normally.
+
+Edit mode is mutually exclusive with new-region draw mode: entering one automatically exits the other.
+
+### Region Actions
+
+From the **Regions** sidebar tab you can:
+
+-  **Navigate** -- zoom the explorer to the saved region (`viewport.fitBounds(imageRect)`)
+-  **Add to Collection** -- saves as an `image-crop` collection item with the original-pixel `CropRegion`
+-  **Delete** -- removes the region
+
+The Regions sidebar auto-refreshes when you navigate between files -- you no longer have to click the Regions tab after switching images.
+
+### Multiple Regions
+
+You can save unlimited labeled regions per image. They never lose resolution because the coordinates are stored in the original image pixel space, not in viewport pixels. When the collection is exported, each region is cropped at full resolution via `sharp.extract()`.
+
+This same labeling pattern works on CBZ/CBR comic pages (each page is a regular image inside an archive -- no special handling needed) and is the foundation for the document region system in the EPUB and PDF viewers.
+
+### Regions Browser
+
+The topbar **▣ Regions Browser** button opens a full-screen folder-scoped view of every saved region across every file. This scales cleanly to thousands of regions because the server maintains an in-memory cache keyed on the Bibliograph enumeration, invalidated only on mutations.
+
+Layout:
+
+- **Left** -- folder tree with per-folder region counts. Click a folder to filter the right pane. "All folders" shows every region across the content root.
+- **Right** -- files in the selected folder, each with a header showing the filename and region count, followed by the regions as clickable chips (label + dimensions).
+
+Click a region chip to close the browser, open the file in the appropriate viewer (image explorer for images, media viewer for PDFs/EPUBs), and jump to the region's coordinates.
+
+The initial folder selection is seeded from the gallery's current browsing location, so opening the browser from a folder you're browsing shows just that folder's regions by default. Press `Esc` or the ** Close** button to dismiss.
 
 ## Coordinate Display
 

package/web-application/docs/server-setup.md

@@ -21,31 +21,31 @@ The build step bundles the client-side JavaScript with Quackage and copies asset
 
 ## Running the Server
 
-### CLI Command
+### CLI Commands
 
-```bash
-retold-remote serve [content-path] [options]
-```
-
-Or using the short alias:
+There are three bin entries provided by the `retold-remote` package:
 
 ```bash
-rr serve /path/to/media
+retold-remote serve [content-path] [options]   # full form
+rr serve [content-path] [options]               # short alias
+retold-stack [content-path] [options]           # convenience: serve --stack
 ```
 
 If `content-path` is omitted, the current directory is served.
 
+The `retold-stack` shortcut auto-injects `serve --stack`, which spawns Ultravisor as a child process and uses XDG-style data paths. See [Stack Launcher](stack-launcher.md) for the full story.
+
 ### Options
 
 | Flag | Description | Default |
 |------|-------------|---------|
 | `-p, --port [port]` | Port to listen on | Random 7000-7999 |
-| `-H, --hashed-filenames` | Enable hashed filename mode (hides real paths from browser) | Off |
-| `-c, --cache-path [path]` | Root cache directory | `./dist/retold-cache/` |
-| `--cache-thumbnails [path]` | Override thumbnail cache location | `<cache-root>/thumbnails/` |
-| `--cache-archives [path]` | Override archive extraction cache | `<cache-root>/archives/` |
-| `--cache-video-frames [path]` | Override video frame cache | `<cache-root>/video-frames/` |
-| `--cache-audio-waveforms [path]` | Override audio waveform cache | `<cache-root>/audio-waveforms/` |
+| `--no-hash` | Disable hashed filename mode (use plain paths in URLs) | Hashing on |
+| `-c, --cache-path [path]` | Root cache directory | `./dist/retold-cache/` (or `~/.cache/retold-remote/` in stack mode) |
+| `--cache-server [url]` | URL of a remote Parime cache server | None |
+| `-u, --ultravisor [url]` | Connect to an Ultravisor mesh; URL defaults to `http://localhost:54321` if omitted | None |
+| `--stack` | Spawn Ultravisor as a child process and connect to it. Uses XDG-style data paths under `~/.local/share` and `~/.cache`. See [Stack Launcher](stack-launcher.md) | Off |
+| `-l, --logfile [path]` | Write logs to a file (auto-generates timestamped name if path omitted) | Console only |
 
 ### Direct Node.js
 
@@ -65,20 +65,30 @@ Default port is `8086` when using `server.js` directly.
 ### Examples
 
 ```bash
-# Serve current directory on port 8086
-node server.js
+# Serve current directory on a random port
+retold-remote serve
 
 # Serve a specific folder
-node server.js /mnt/nas/media
+retold-remote serve /mnt/nas/media
 
 # CLI with custom port
 retold-remote serve /mnt/nas/media -p 3000
 
-# Hashed filenames for privacy
-retold-remote serve /mnt/nas/media -H
+# Plain paths in URLs (disable hashing)
+retold-remote serve /mnt/nas/media --no-hash
 
 # Custom cache location (useful for Docker volumes)
 retold-remote serve /media -c /cache
+
+# Connect to an existing Ultravisor mesh
+retold-remote serve /media -u http://192.168.1.100:54321
+
+# Full stack: spawn Ultravisor as a child process, embed Orator-Conversion,
+# use XDG paths (~/.local/share/ultravisor, ~/.cache/retold-remote)
+retold-stack /mnt/nas/media
+
+# Same thing via the explicit flag
+retold-remote serve --stack /mnt/nas/media
 ```
 
 ## Configuration
@@ -131,111 +141,132 @@ When enabled (`-H` flag or `RETOLD_HASHED_FILENAMES=true`), real file paths are
 
 ## Docker
 
-### Using the Included Dockerfile
+Retold Remote ships with two Dockerfiles, a `docker-compose.yml`, and a `docker-build-and-save.sh` helper. All images run the full stack (Ultravisor + Retold Remote + embedded Orator-Conversion) as a single container.
+
+> **Running on Synology?** See the dedicated [Synology Container Manager](synology.md) guide for a step-by-step walkthrough including how to build the image on a dev machine and transfer it to the NAS.
 
-A `Dockerfile` is provided in the repository root:
+### Quick Start (any Docker host with the source tree)
+
+The compose file defaults to `image: retold-stack:latest` so it works on hosts that already have the image loaded. To build and run from the source tree in one go:
 
 ```bash
-docker build -t retold-remote .
-docker run -p 8086:8086 -v /path/to/media:/media retold-remote
+cd retold-remote
+
+# Build the image and tag it as retold-stack:latest
+docker build -t retold-stack:latest .
+
+# Start the stack
+MEDIA_PATH=/path/to/your/media docker compose up -d
 ```
 
-### Inline Dockerfile
+Then browse to `http://localhost:7777/`. The Ultravisor coordinator web interface is at `http://localhost:54321/`.
 
-If you need to create a Dockerfile from scratch, here is a complete working version:
+If you'd rather have Compose build the image directly, edit `docker-compose.yml`:
+
+```yaml
+services:
+  retold-stack:
+    # image: retold-stack:latest    # <- comment this out
+    build:                          # <- uncomment this block
+      context: .
+      dockerfile: Dockerfile
+```
 
-```dockerfile
-FROM node:20-slim
+Then `docker compose up -d --build` will build and start in one step.
 
-# Install optional tools for full functionality
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    ffmpeg \
-    imagemagick \
-    p7zip-full \
-    calibre \
-    && rm -rf /var/lib/apt/lists/*
+### Dockerfile Variants
 
-WORKDIR /app
+Both images have been verified to build and run the full stack on ARM64 (Apple Silicon) and should build identically on AMD64.
 
-# Copy package files and install dependencies
-COPY package.json package-lock.json ./
-RUN npm ci --omit=dev && npm cache clean --force
+| File | Measured Size | Includes | Use for |
+|------|---------------|----------|---------|
+| `Dockerfile` | **3.0 GB** | Everything: ffmpeg, ImageMagick, 7z, poppler, pdftk, LibreOffice, Calibre, exiftool, dcraw | Full document conversion (DOC, DOCX, RTF, ODT, WPD, MOBI, PPT, XLS, etc.) |
+| `Dockerfile.slim` | **1.81 GB** | Same but without LibreOffice and Calibre | Image/video/audio/PDF/EPUB only -- no doc/docx/mobi conversion |
 
-# Install sharp (optional image processing)
-RUN npm install sharp || true
+> **Note on audiowaveform:** Neither image installs `audiowaveform` because it is not in the default Debian repositories. Retold Remote automatically falls back to ffprobe + ffmpeg for waveform generation, which is slower but works identically. If you need the BBC `audiowaveform` tool for faster generation, add it via a custom Dockerfile step (build from source or a PPA).
 
-# Copy application source and built assets
-COPY source/ source/
-COPY web-application/ web-application/
-COPY css/ css/
-COPY html/ html/
-COPY server.js ./
+### Building and Saving for Transfer
 
-# Create cache directory
-RUN mkdir -p /cache
+For deploying to a NAS or any host that doesn't have the source tree, use the `docker-build-and-save.sh` helper:
 
-# Default port
-ENV PORT=8086
+```bash
+./docker-build-and-save.sh                # full image, host architecture
+./docker-build-and-save.sh slim           # slim variant
+./docker-build-and-save.sh --amd64        # cross-build for x86_64
+./docker-build-and-save.sh slim --arm64   # slim + arm64
+```
 
-EXPOSE 8086
+This produces a compressed tar file (`retold-stack-image.tar.gz` or `retold-stack-image-slim.tar.gz`) that you can copy to the target host and load with:
 
-# Serve /media with cache at /cache
-CMD ["node", "server.js", "/media"]
+```bash
+docker load -i retold-stack-image.tar.gz
 ```
 
-### Docker Compose
+Then start the stack with the included `docker-compose.yml` (which references `retold-stack:latest` by default). No source tree needed on the target host.
 
-```yaml
-version: "3.8"
-services:
-  retold-remote:
-    build: .
-    ports:
-      - "8086:8086"
-    volumes:
-      - /path/to/media:/media:ro
-      - retold-cache:/cache
-    environment:
-      - PORT=8086
-    restart: unless-stopped
-
-volumes:
-  retold-cache:
+### Manual Docker Run
+
+If you prefer `docker run` over Compose:
+
+```bash
+docker build -t retold-stack .
+
+docker run -d \
+  --name retold-stack \
+  -p 7777:7777 \
+  -p 54321:54321 \
+  -v /path/to/media:/media:ro \
+  -v retold-cache:/cache \
+  -v ultravisor-data:/data \
+  -v retold-config:/config \
+  --restart unless-stopped \
+  retold-stack
 ```
 
-### Docker Usage Notes
+Then browse to `http://localhost:7777/`.
+
+### Volume Mounts
 
-- Mount your media folder to `/media` (read-only is fine with `:ro`)
-- Mount a volume to `/cache` for persistent thumbnail and frame caches
-- The `node:20-slim` base keeps the image small while the `apt-get` packages add full media processing
-- `calibre` is the largest optional package; omit it if you do not need MOBI/AZW ebook conversion
-- `sharp` is installed separately because it has native bindings that may fail on some architectures; the `|| true` ensures the build continues without it
+The stack uses four volume mount points. The first is your media; the other three are for persistent state:
 
-### Minimal Dockerfile (No Optional Tools)
+| Container path | Purpose | Typical host mapping |
+|----------------|---------|---------------------|
+| `/media` | Media folder to browse (read-only) | `/path/to/media:/media:ro` |
+| `/cache` | Thumbnails, frames, waveforms, conversions | Named volume `retold-cache` |
+| `/data` | Ultravisor datastore + staging | Named volume `ultravisor-data` |
+| `/config` | Stack config files | Named volume `retold-config` |
 
-If you only need basic browsing without thumbnails or media probing:
+Inside the container, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, and `XDG_CONFIG_HOME` are set to these paths so the stack launcher uses them automatically.
 
-```dockerfile
-FROM node:20-slim
+### Published Ports
 
-WORKDIR /app
+| Port | Purpose |
+|------|---------|
+| **7777** | Retold Remote web UI (the main thing you browse to) |
+| **54321** | Ultravisor coordinator API + web interface (optional -- useful for monitoring) |
 
-COPY package.json package-lock.json ./
-RUN npm ci --omit=dev && npm cache clean --force
+Both are pinned to fixed ports via the `-p 7777` flag in the default CMD, overriding the usual random 7000-7999 behavior.
 
-COPY source/ source/
-COPY web-application/ web-application/
-COPY css/ css/
-COPY html/ html/
-COPY server.js ./
+### Customizing the CMD
 
-ENV PORT=8086
-EXPOSE 8086
+The default command launches `retold-stack` against `/media` on port 7777 with hashed filenames disabled. To override (e.g., to enable hashed filenames or change the port):
 
-CMD ["node", "server.js", "/media"]
+```yaml
+command: ["node", "source/cli/RetoldRemote-Stack-Run.js", "/media", "-p", "7777"]
 ```
 
-This produces an image under 200MB. Images display directly (no thumbnails), videos play in the browser, and zip/cbz archives use native extraction.
+Remove the `--no-hash` flag to re-enable hashed filenames, or change `/media` if you mounted your media to a different path.
+
+### Image Size Notes
+
+- `node:20-slim` base: ~180 MB
+- Full `Dockerfile` (with LibreOffice + Calibre): ~2.5 GB -- Calibre alone is ~1 GB, LibreOffice is ~600 MB
+- `Dockerfile.slim`: ~1.8 GB -- drops both big dependencies
+- Omit `ffmpeg` and `audiowaveform` too if you don't need video/audio processing, for a ~1 GB image
+
+### Child Process Note
+
+The stack launcher spawns Ultravisor as a child process using `node` and the resolved `ultravisor` bin path. This works inside Docker without any special configuration -- the child runs in the same container PID namespace and its logs are streamed through the main process's stdout with `[ultravisor]` prefixes.
 
 ## Archive Browsing