kryten-webqueue 0.8.1__tar.gz → 0.9.0__tar.gz

{kryten_webqueue-0.8.1 → kryten_webqueue-0.9.0}/CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [0.9.0] — 2026-06-09
+
+### Added
+
+- **Reimplemented content jobs (vendored).** Five tools are vendored into `kryten_webqueue/integrations/` and driven in-process as parameterized jobs, replacing the Windows-script workflow:
+  - `enrichtitles`, `enrichmeta`, `enrichtv` — clean titles / enrich movie & TV metadata via TMDb/OMDb, pushing to MediaCMS. Params: `dry_run`, `limit`, `days` (+ tool-specific `tubi_upgrade`, `min_score`, `min_duration`, `max_duration`, `delay`).
+  - `fetch` — download a yt-dlp-supported URL (single or playlist) to MediaCMS with `quality` (`best`/`good`/`medium`) and an optional **Add to playlist** that appends the uploaded item(s) to a saved playlist.
+  - `fetchurls` — read the upcoming **weekend's** Channel Z workbook from a local `.xlsx` (file-only, no SharePoint/Graph in v1), resolve off-site URLs via the in-process downloader, and import each section as a saved playlist named `{sheet}-{section}` (idempotent re-runs replace items). The target sheet is always the imminent Fri/Sat (Friday → today; Sat/Sun → next Friday).
+  Each blocking tool runs off the event loop via `asyncio.to_thread`, bridging progress back to the `job_runs` row. Jobs whose optional dependencies are missing register normally but fail the run fast with a clear "dependency not installed" message. New optional extra `jobs` (`yt-dlp`, `openpyxl`, `requests`, `pyyaml`); `fetch` additionally needs `ffmpeg` on the host. New config: `fetch_cookies_path`, `fetchurls.workbook_path`.
+- **Parameterized background jobs.** The job framework now supports declarative parameter schemas: `JobManager.register(name, func, *, label, schema)` and `run(name, *, triggered_by, params)`. Job functions receive `(params, ctx)` where `ctx` exposes `db`, `api_gate`, `config`, `triggered_by`, and an async `progress(detail)` callback that writes live progress to the run's `job_runs` row. Submitted params are validated/coerced against the schema (required, defaults, `string`/`int`/`float`/`bool`/`enum`/`playlist` types) and persisted to the new `job_runs.params` column. The admin **Run** button opens a schema-driven modal for jobs that declare parameters (and runs immediately for those that don't); each job also shows a last-run summary. New endpoint `GET /admin/jobs/{name}/schema`; `GET /admin/jobs` now includes each job's `schema` + `last_run`; `POST /admin/jobs/{name}/run` accepts a JSON `{params}` body (400 on invalid params).
+- **Browse sort control.** Browse and search now offer a *Sort by* dropdown — `Default` (quality-weighted), `Title A–Z`, `Title Z–A`, `Newest first`, `Oldest first` — available to everyone. The choice carries through pagination and the facet form and is remembered per-browser via `localStorage`. `Newest`/`Oldest` order by the catalog `added_at`, which is now populated from the MediaCMS `add_date` on sync (and backfilled from `synced_at` for existing rows).
+- **Branded placeholder art with hover-to-thumbnail.** Tiles without a real poster match (no TMDB/OMDB art) now show a random branded placeholder from `placeholder_dir` instead of the raw MediaCMS thumbnail; hovering reveals the real thumbnail via a CSS crossfade. The placeholder list is cached in memory and rescanned periodically.
+- **Admin tile actions.** Catalog tiles now offer admins (rank ≥ 3) *Add to playlist* (picker modal), *+ Recent* (append to the admin's most recently created playlist, no modal), and *Hide* (tags the item `kryten-hidden` in MediaCMS and hides it locally immediately). New endpoints: `POST /admin/playlists/{id}/append`, `POST /admin/playlists/recent/append`, `POST /admin/catalog/{token}/hide`, and `POST /admin/catalog/{token}/unhide`.
+
+### Changed
+
+- **Tile action buttons stack vertically**, full-width, for clearer affordance at narrow tile widths.
+
+### Fixed
+
+- **Job-run history no longer shows phantom `running` rows.** A job's running flag lived only in memory, so a restart or killed worker mid-run left the `job_runs` row stuck at `running` forever. On startup such orphans are now reconciled to a new `interrupted` status (styled in the admin dashboard).
+
+### Removed
+
+- **Live queue page no longer shows the order number or a drag handle.** The `qi-pos` index and the non-functional `qi-drag` (☰) affordance were removed; reordering lives in the playlist editor, not the live queue.
+
+## [0.8.2] - 2026-06-08
+
+### Fixed
+
+- **Queue ETAs no longer depend on the server clock/timezone.** Predicted start times were emitted as absolute UTC timestamps, so any host clock skew or timezone misconfiguration shifted every ETA by the offset (the persistent "TZ issue"). The shadow now also emits a clock-independent relative offset (`estimated_start_in_sec` = seconds-from-now until an item plays), and the queue page computes the wall-clock time from the **browser's** own clock (`Date.now() + offset`). The absolute timestamp is retained for compatibility and as a fallback. Result: ETAs are correct regardless of server clock/timezone.
+
 ## [0.8.1] - 2026-06-08
 
 ### Added

{kryten_webqueue-0.8.1 → kryten_webqueue-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kryten-webqueue
-Version: 0.8.1
+Version: 0.9.0
 Summary: Netflix/Tubi-style catalog browser and pay-to-play queue management for CyTube
 Author: grobertson
 License-Expression: MIT
@@ -23,6 +23,11 @@ Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: jobs
+Requires-Dist: openpyxl>=3.1; extra == 'jobs'
+Requires-Dist: pyyaml>=6.0; extra == 'jobs'
+Requires-Dist: requests>=2.31; extra == 'jobs'
+Requires-Dist: yt-dlp>=2024.1; extra == 'jobs'
 Description-Content-Type: text/markdown
 
 # kryten-webqueue

{kryten_webqueue-0.8.1 → kryten_webqueue-0.9.0}/config.example.json

@@ -14,6 +14,11 @@
   "tmdb_api_key": "",
   "omdb_api_key": "",
 
+  "fetch_cookies_path": "",
+  "fetchurls": {
+    "workbook_path": ""
+  },
+
   "db_path": "/var/lib/kryten-webqueue/webqueue.db",
 
   "image_dir": "/var/lib/kryten-webqueue/images",

{kryten_webqueue-0.8.1 → kryten_webqueue-0.9.0}/docs/IMPLEMENTATION_SPEC.md

@@ -1,4 +1,4 @@
-# kryten-webqueue — Implementation Specification
+                        # kryten-webqueue — Implementation Specification
 
 **Version:** 1.0  
 **Date:** 2026-05-29  

kryten_webqueue-0.9.0/docs/SPEC_JOBS_AND_BROWSE.md

@@ -0,0 +1,353 @@
+# kryten-webqueue — Jobs & Browse Enhancements Spec
+
+**Version:** 1.1
+**Date:** 2026-06-08
+**Status:** Design — open questions resolved — ready for phased implementation
+**Author direction:** self-authored implementation plan (GitHub Copilot)
+
+---
+
+## 0. Decisions captured (from clarification)
+
+| # | Question | Decision |
+|---|----------|----------|
+| 1 | How to run fetch/fetchurls/enrich* (Windows scripts vs Linux service) | **Reimplement the tools' logic inside webqueue** by vendoring the existing Python modules into an internal `integrations/` package and driving them in-process (see §A2). |
+| 2 | Tool identity | Confirmed: `d:\devel\cmsutils\{fetchurls,enrichtitles,enrichmeta,enrichtv}.py` + `d:\devel\yt-pipe` downloader (`youtube_to_mediacms.py`, invoked today via `fetch.ps1`). The original request's "cmstools / enhance*" names map to these. |
+| 3 | "Jobs never run at the same time" | **Per-job lock only** (already enforced in memory). The real defect the user is seeing is the **job-run history list** showing phantom `running` rows — fix that (see §A1.2). |
+| 4 | Hide Item tag + write target | Write tag **`kryten-hidden`** to MediaCMS via the API token (MediaCMS is the source of truth). Hide **immediately in the local catalog**; the next sync confirms it. |
+| 5 | Browse sort options + scope | `Default (quality)`, `Title A–Z`, `Title Z–A`, `Newest first`, `Oldest first`. **`Newest first` is available to everyone** (not admin-only). |
+| 6 | "Most recent playlist" | **Most recently *created* saved playlist by the current admin** (`saved_playlists` where `created_by = user ORDER BY created_at DESC LIMIT 1`). |
+| 7 | fetchurls weekend | **Always the upcoming weekend**: compute the next Friday and target the sheet named `M.D-M.D` (e.g. `3.6-3.7`), overriding the tool's current/just-past auto-select. |
+
+### 0.1 Resolved open questions (was §I)
+
+| OQ | Question | **Resolution** |
+|----|----------|----------------|
+| OQ-1 | fetchurls SharePoint auth in a headless service | **v1 = local file only.** The job reads the workbook from a configured/uploaded `.xlsx` path (`sharepoint.workbook_path` or an admin upload), reusing the tool's existing `--file` code path. **No Microsoft Graph / MSAL device-code in v1.** A future phase MAY add Graph with the device code surfaced in the admin UI; spec'd but not built now. Column-F writeback is **disabled** in file-only mode unless the file is writable in place. |
+| OQ-2 | Vendor vs packaged dependency | **Vendor-and-adapt** into `kryten_webqueue/integrations/` (accept drift from `d:\devel\cmsutils`). Record the upstream commit/date in each vendored file's header. A future option to repackage `cmsutils` as an installable dependency is noted but not pursued now. |
+| OQ-3 | "Upcoming weekend" when run on a Friday | **Use today's weekend** (the imminent Fri/Sat). `friday = today + ((4 - weekday) mod 7)` yields today when run on Friday; only Sat/Sun roll forward to next Friday. |
+| OQ-4 | Random branded art stability | **Per server-render.** The browse route picks `random.choice(placeholders)` per affected tile when building the page; the src is stable for that page load (no client reshuffle, no layout thrash). Hover still reveals the real thumbnail. |
+| OQ-5 | Include `unhide`? | **Yes.** Ship `POST /admin/catalog/{token}/unhide` (removes `kryten-hidden` in MediaCMS + locally) so a mis-hide is reversible from the admin "show hidden" view. |
+| OQ-6 | MediaCMS tag-write endpoint | **Reuse the enrich tools' MediaCMS edit path.** Tags are written via the media-edit call to `POST /api/v1/media/{friendly_token}` with the `tags` field and the API token — the same mechanism `enrichmeta`/`enrichtv` use. Extract into `integrations/cmsutils/_common.py:MediaCMSClient.set_tags(token, tags)` and **read-modify-write** (fetch current tags, add/remove `kryten-hidden`, submit) to preserve existing tags. **Verify exact field/verb against the live instance during B6** with a round-trip integration test on a disposable item before wiring the UI. |
+
+---
+
+## 1. Scope
+
+Two feature areas plus a jobs-framework fix:
+
+- **A. Admin → Jobs**: framework fix (history reconciliation), and five new jobs that reimplement `fetch`, `fetchurls`, `enrichtitles`, `enrichmeta`, `enrichtv` with reasonable defaults and a small amount of new wiring (fetch → playlist, fetchurls → imported saved playlists).
+- **B. Browse / Results**: sort control, random branded art with hover-to-real-thumbnail, vertically stacked tile buttons, admin "Add to playlist" / "Add to most-recent playlist" / "Hide Item".
+- **C. Queue page**: hide the order number, remove the drag handle and all drag-reorder affordance.
+
+Out of scope: changing the economy/pay flow, the scheduler, or the public catalog filtering rules beyond the new hide tag.
+
+---
+
+## A. Admin → Jobs
+
+### A1. Job framework changes
+
+#### A1.1 Per-job concurrency (confirm existing)
+`JobManager.run()` already rejects a second start of the same job while one is in `self._running`. **No change required** to the guard itself. Document it: a job that is running returns `{"started": false, "reason": "already_running"}` and the UI disables its Run button (already implemented in `admin/index.html`).
+
+#### A1.2 Fix the job-run history list (the actual bug)
+**Problem:** `start_job_run()` inserts a row with `status='running'`; `_running` is in-memory only. If the service restarts (or the worker is killed) mid-run, the row is **never updated** and shows `running` forever in the history table. Long-running jobs (fetch/enrich) make this common.
+
+**Fix (required):**
+1. **Startup reconciliation.** On app startup, before registering jobs, run:
+   ```sql
+   UPDATE job_runs SET status='interrupted',
+       ended_at = COALESCE(ended_at, CURRENT_TIMESTAMP)
+   WHERE status='running';
+   ```
+   Add a `Database.reconcile_orphaned_job_runs()` method; call it in the `lifespan` startup (after `run_migrations`, before background workers).
+2. **Status vocabulary.** Add `interrupted` to the known statuses; style it in CSS (`.job-status-interrupted { color: var(--warning); }`).
+3. **History query.** `get_job_runs` ordering by `id DESC` is fine; ensure the admin dashboard groups/labels by `job_name` and shows `triggered_by`. (Optional polish: a per-job "last run" summary above the raw history.)
+4. **Heartbeat (optional, phase 2).** For very long jobs, periodically `UPDATE job_runs SET detail=? WHERE id=?` with progress (e.g. `{"processed": N, "total": M}`) so the UI can show live progress; the dashboard already polls.
+
+**Acceptance:** after a hard restart during a job, the history shows that run as `interrupted`, never a perpetual `running`; a fresh run of the same job is allowed.
+
+#### A1.3 Parameterized jobs
+The current `JobManager.register(name, func)` takes a zero-arg coroutine. New jobs need **parameters** (URL, quality, playlist id, limits, dry-run, etc.). Extend the framework:
+
+- `register(name, func, *, label, schema=None)` where `schema` is a small declarative list of fields (name, type, default, label, required, options) used to render a parameter form in the admin UI and to validate input.
+- `run(name, *, triggered_by, params: dict | None = None)` passes validated `params` to the job function: `func(params, ctx)` where `ctx` exposes `db`, `api_gate`, `config`, and an async `progress(detail: dict)` callback.
+- Persist the submitted `params` into the `job_runs.detail` (or a new `params` column) so history shows what was run.
+- Back-compat: existing `catalog_sync` registers with no schema and a `func(params, ctx)` that ignores params.
+
+**Admin UI:** the Run button opens a small modal generated from `schema` (reusing the shared admin modal/`field` CSS from v0.8.0). Jobs with no schema run immediately as today.
+
+**New endpoint:** `GET /admin/jobs/{name}/schema` (or include schema in `GET /admin/jobs`) so the UI can render the form. `POST /admin/jobs/{name}/run` accepts a JSON `params` body.
+
+### A2. Reimplementation strategy ("logic inside webqueue")
+
+The enrich tools are large (`enrichtv.py` ≈ 1700 lines) and battle-tested. **Do not rewrite from scratch.** Instead:
+
+1. **Vendor** the source modules into `kryten_webqueue/integrations/`:
+   ```
+   kryten_webqueue/integrations/
+     __init__.py
+     cmsutils/            # vendored from d:\devel\cmsutils
+       enrichtitles.py
+       enrichmeta.py
+       enrichtv.py
+       fetchurls.py
+       _common.py         # shared MediaCMS client / scoring helpers if extracted
+     ytpipe/              # vendored from d:\devel\yt-pipe
+       downloader.py      # from youtube_to_mediacms.py
+   ```
+2. **Refactor each module's entry point** from `main(argv)` (argparse + console-UTF8 + `print` + interactive prompts) into a **callable** `run(params: dict, *, config, progress) -> dict`:
+   - Remove `argparse`, `sys.stdin` prompts, and `-i/--interactive` paths (service is headless — interactive enrich modes are **disabled**).
+   - Replace `print(...)` with the `progress()` callback + `logging`.
+   - Accept config (MediaCMS URL/token, TMDb/OMDb keys) from webqueue `Config`, not from `config.yaml`/CLI.
+   - Return a result dict (counts, errors) for `job_runs.detail`.
+3. **Run blocking work off the event loop.** These modules use synchronous `requests`, `openpyxl`, `yt_dlp`, `msal`. Each job function is an `async def` that does `await asyncio.to_thread(module.run, params, config=..., progress=thread_safe_progress)`. The `progress` callback must hop back to the loop (`asyncio.run_coroutine_threadsafe` or a queue) to write `job_runs`.
+4. **Config reuse.** MediaCMS token and TMDb/OMDb keys already exist in webqueue `Config` (`mediacms_token`, `tmdb_api_key`, `omdb_api_key`). The enrich tools accept exactly these. fetchurls needs **new** SharePoint config (§A4).
+5. **System dependencies.** `fetch` needs `yt-dlp` + `ffmpeg` on the host; document in deploy notes. `fetchurls` needs `openpyxl` + `msal`. Add Python deps to `pyproject.toml` as an **optional extra** (`jobs`) so a minimal deployment without these tools still installs:
+   ```toml
+   [project.optional-dependencies]
+   jobs = ["yt-dlp>=2024.1", "openpyxl>=3.1", "msal>=1.28", "requests>=2.31"]
+   ```
+   Jobs whose deps are missing register but fail fast with a clear "dependency not installed" message (don't crash startup).
+
+### A3. Job: `fetch` (yt-pipe downloader)
+
+Reimplements `youtube_to_mediacms.py` (today wrapped by `fetch.ps1`). Downloads a yt-dlp-supported URL and uploads it to MediaCMS.
+
+**Parameters (schema):**
+
+| Field | Type | Default | Notes |
+|-------|------|---------|-------|
+| `url` | string (required) | — | Source URL (YouTube/Tubi/etc.); apply the existing Mix/Radio playlist cleanup. |
+| `quality` | enum | `medium` | `best` \| `good` \| `medium` (same mapping as `fetch.ps1`). |
+| `max_videos` | int | `50` | For playlist URLs. |
+| `add_to_playlist` | playlist picker | none | **New.** If set, append the resulting MediaCMS item(s) to this saved playlist after upload. |
+
+**Config:** `mediacms_url`, `mediacms_token` (already present). Cookies: optional `fetch_cookies_path` config for age/region-gated sources (mirrors `cookies.txt`).
+
+**Add-to-playlist behavior (new):**
+- After a successful upload, the downloader returns the new `friendly_token`(s).
+- For each, append `{media_type: 'cm', media_id: <manifest_url or token>, title, duration_sec}` to the chosen `saved_playlist` via the existing `replace_playlist_items` (read → append → write) or a new `append_playlist_item` helper.
+- If `add_to_playlist` is set but upload yields no token, record a non-fatal warning in the run detail.
+
+**Result detail:** `{"downloaded": N, "uploaded": N, "tokens": [...], "added_to_playlist": <id|null>, "errors": [...]}`.
+
+### A4. Job: `fetchurls`
+
+Reimplements `fetchurls.py`: read the Channel Z Excel workbook from SharePoint, resolve each source URL (validate dropsugar.co with HEAD; download YouTube/Tubi via the `fetch` downloader), and produce per-section playlists.
+
+**Always-upcoming-weekend rule (new, overrides tool):**
+- Compute the **next Friday** from today: `friday = today + ((4 - today.weekday()) % 7)`. When run **on a Friday this yields today** (the imminent weekend, per OQ-3); Sat/Sun roll forward to the next Friday. Saturday = friday + 1.
+- Sheet name = `f"{friday.month}.{friday.day}-{saturday.month}.{saturday.day}"` (e.g. `3.6-3.7`), matching `_SHEET_DATE_RE`. Pass this explicitly as the target sheet rather than calling `_auto_select_sheet` (which selects current/just-past).
+- If the computed sheet is absent from the workbook, fail the run with a clear message listing available sheet names.
+
+**Parameters (schema):**
+
+| Field | Type | Default | Notes |
+|-------|------|---------|-------|
+| `section` | enum | `all` | `all` \| `friday` \| `saturday-night` \| `saturday-morning`. |
+| `dry_run` | bool | `false` | Resolve/preview only; no downloads, no writeback, no import. |
+| `writeback` | bool | `true` | Write resolved URLs back to column F. |
+| `validate` | bool | `true` | HEAD-check existing dropsugar.co URLs. |
+
+**Import resulting playlists (new):**
+- The tool produces `playlists/{sheet}-friday.txt`, `{sheet}-saturday-night.txt`, `{sheet}-saturday-morning.txt`, and `{sheet}-failures.txt`.
+- After a successful (non-dry-run) run, **import each non-failures file as a saved playlist** named exactly like the file stem: `{sheet}-friday`, `{sheet}-saturday-night`, `{sheet}-saturday-morning`. Reuse `import_playlist_text()` to resolve lines, then `create_saved_playlist(name=stem, created_by=triggered_by)` + `replace_playlist_items()`.
+- If a playlist of that name already exists, **replace its items** (idempotent re-runs) rather than creating a duplicate. (Match by exact name + `created_by`.)
+- The `{sheet}-failures` file is **not** imported; surface its count in the run detail.
+
+**Workbook source (OQ-1 resolved — local file only in v1):** `fetchurls` reads the Channel Z workbook from a **configured/uploaded `.xlsx`**, reusing the tool's existing `--file` path. **No SharePoint/Graph/MSAL in v1.** New config:
+  ```jsonc
+  "fetchurls": {
+    "workbook_path": ""   // absolute path to a synced/exported Channel Z Playlist .xlsx
+  }
+  ```
+  Optionally the admin Run modal accepts a one-off file upload that overrides `workbook_path` for that run. Column-F writeback is **disabled** in file-only mode unless the configured file is writable in place (toggle `writeback`). A future phase may add Graph auth (device code surfaced in the admin UI) — out of scope here.
+
+**Result detail:** `{"sheet": "3.6-3.7", "resolved": N, "downloaded": N, "failures": N, "imported_playlists": ["3.6-3.7-friday", ...]}`.
+
+### A5. Jobs: `enrichtitles`, `enrichmeta`, `enrichtv`
+
+Reimplement the three enrich tools. All three already accept the same core config webqueue has (`--token`, `--tmdb-key`, `--omdb-key`, `--api-url`) and default to **dry-run** unless `--commit`. For one-click admin jobs we default to **commit on** (the point is to apply enrichment) with a `dry_run` toggle for safety. **Interactive mode is disabled.**
+
+Common parameters:
+
+| Field | Type | Default | Applies to |
+|-------|------|---------|-----------|
+| `dry_run` | bool | `false` | all (true = report/scan only, no writes) |
+| `limit` | int | none | all |
+| `days` | int | none | all (only items uploaded in last N days) |
+
+Tool-specific defaults (mirror the CLIs):
+
+- **enrichtitles** — params: `dry_run`, `limit`, `days`. Cleans/normalizes titles. Default commit.
+- **enrichmeta** — params: `dry_run`, `limit`, `days`, `tubi_upgrade` (bool, default false), `min_score` (default = tool's `MIN_SCORE_THRESHOLD`), `min_duration` (default `MIN_DURATION`), `delay` (default `REQUEST_DELAY` 0.25s). Uses TMDb/OMDb keys from config.
+- **enrichtv** — params: `dry_run`, `limit`, `days`, `min_score` (default 50), `min_duration` (default 600), `max_duration` (default 3599), `delay` (default 0.25s). Uses TMDb/OMDb keys from config.
+
+**MediaCMS writes:** these tools push enriched metadata to MediaCMS via the API token — this is the existing, working write path we also rely on for the Hide tag (§B6). When vendoring, **extract the MediaCMS edit call into `integrations/cmsutils/_common.py`** so Hide and enrich share one client.
+
+**Result detail:** per tool, `{"scanned": N, "matched": N, "committed": N, "skipped": N, "errors": [...]}`.
+
+**Phasing note:** enrich jobs are the lowest-risk reimplementations (pure HTTP, no auth dance, no large downloads). Do these **first** to prove the vendor+thread pattern, then `fetch`, then `fetchurls` last.
+
+---
+
+## B. Browse / Results
+
+Applies to both `catalog/browse.html` (browse + search results share this template) and the JSON `/catalog/browse` + `/catalog/search` routes.
+
+### B1. Sort control
+- Add a **Sort by** `<select>` beside the existing Category/Tag facets: `Default` (current quality-weighted order), `Title A–Z`, `Title Z–A`, `Newest first`, `Oldest first`. All options available to everyone (per decision #5).
+- **DB:** add a `sort` parameter to `db.browse(...)` and `db.search(...)` mapping to an `ORDER BY`:
+  - `default` → existing quality-weighted clause.
+  - `title_asc` / `title_desc` → `c.title ASC|DESC`.
+  - `newest` / `oldest` → `c.added_at DESC|ASC` (with `synced_at` tiebreaker).
+- **Data gap (must fix):** `catalog.added_at` is currently **not populated** on insert (it has no default and `insert_catalog` omits it), so `Newest first` would be unreliable. During sync, **populate `added_at` from MediaCMS `add_date`** (the media list includes `add_date`). Backfill existing rows in a migration: `UPDATE catalog SET added_at = synced_at WHERE added_at IS NULL` as a stopgap, then let sync overwrite with the true `add_date`.
+- **Routes/UI:** carry `sort` as a query param through pagination and the facet form (extend `applyFacets()`); persist the user's choice in `localStorage` for convenience.
+
+### B2. Random branded art with hover-to-thumbnail
+Goal: stop showing "shitty" MediaCMS thumbnails as the primary tile art; show a random branded placeholder instead, but reveal the real thumbnail on hover (sometimes useful).
+
+- **When it applies:** a tile whose best art is *not* a real poster — i.e. `cover_art_source` is `null`/`thumbnail` (no TMDB/OMDB match). Tiles with a real `cover_art_path` from `tmdb`/`omdb` are unchanged.
+- **Front art:** a **random** image from the branded placeholder folder (`config.placeholder_dir`, served under `/images/placeholders/`), chosen **per server-render** (OQ-4): the browse route picks `random.choice(...)` per affected tile when building the page, so each tile has a stable src for that response (no client reshuffle, no layout thrash).
+- **Hover:** on pointer-over, swap/overlay the real `thumbnail_url` (the "shitty" art) so it can still be inspected; restore the placeholder on pointer-out. Implement as two stacked `<img>`s with a CSS hover crossfade (no JS needed), guarded so it only renders when a `thumbnail_url` exists.
+- **Expose the placeholder list:** add a tiny endpoint or template-context helper that lists files in `placeholder_dir` (filename only). The browse route picks `random.choice(...)` per affected tile. Cache the directory listing in memory (refresh on an interval) to avoid disk scans per request.
+- **Fallback:** if `placeholder_dir` is empty, keep the current letter placeholder.
+
+### B3. Vertical tile button stack
+- The tile `.card-actions` (currently a horizontal row of `Queue` / `Queue as Admin`) should **stack vertically**, full-width buttons. CSS only: `.card-actions { flex-direction: column; align-items: stretch; gap: 0.4rem; }` plus `.card-actions .btn { width: 100%; }`. Verify it reads well at the grid's min tile width (180px).
+
+### B4. Admin "Add to playlist" (per tile)
+- Add an admin-only button to each tile: **Add to playlist** (visible when `user.rank >= 3`).
+- Opens the shared admin modal with a playlist `<select>` (populated from `GET /admin/playlists/`) and an "Add" action.
+- **Endpoint:** `POST /admin/playlists/{id}/append` `{friendly_token}` — resolves the catalog item (`get_item_admin`) and appends `{media_type:'cm', media_id: manifest_url, title, duration_sec}` to the playlist (read → append → `replace_playlist_items`, or a dedicated `append_playlist_item`). Returns the new item count.
+- Toast on success/failure (existing pattern).
+
+### B5. Admin "Add to most-recent playlist" (no modal)
+- Add a second admin-only tile button: **+ Recent** (or "Add to <name>"), which appends to the admin's most-recently-**created** playlist **without a modal**.
+- **Endpoint:** `POST /admin/playlists/recent/append` `{friendly_token}`:
+  - Resolve most-recent: `SELECT * FROM saved_playlists WHERE created_by=? ORDER BY created_at DESC LIMIT 1`.
+  - If none exists, return a 409 with a message the UI shows as a toast ("Create a playlist first").
+  - Otherwise append as in B4 and return `{playlist_id, name, count}` so the toast can say `Added to "<name>"`.
+- Optional polish: label the button with the resolved playlist name if cheaply available (e.g. fetched once on page load and cached client-side).
+
+### B6. Admin "Hide Item" → MediaCMS `kryten-hidden` tag
+- Add an admin-only tile button **Hide** with a confirm step ("Hide this item from the catalog? It will be tagged `kryten-hidden` in MediaCMS.").
+- **Source of truth = MediaCMS.** On confirm:
+  1. **Write** the `kryten-hidden` tag to the item in MediaCMS via the API token, using the same edit mechanism the enrich tools use (`integrations/cmsutils/_common.py` MediaCMS client). Verify the exact endpoint/payload for tag editing against MediaCMS (the enrich tools already PATCH/POST media edits — reuse that). 
+  2. **Immediately hide locally** so the admin sees it disappear without waiting for sync: either (a) add `kryten-hidden` to the local `catalog_tags` join for that token, or (b) maintain a local `hidden` flag column. Prefer (a) so the existing hidden-tag filter (v0.7.5) applies uniformly. Ensure `kryten-hidden` is in the hidden-tags exclusion set.
+  3. The next catalog sync re-reads tags from MediaCMS and the hide persists (and propagates to any other consumer).
+- **Endpoint:** `POST /admin/catalog/{friendly_token}/hide` (admin). Returns success; the tile is removed from the grid client-side on success.
+- **Unhide (recommended, low cost):** since admins can already reveal hidden items (`?show_hidden=1` from v0.7.5), add the inverse `POST /admin/catalog/{friendly_token}/unhide` that removes the `kryten-hidden` tag in MediaCMS + locally, so a mis-hide is reversible from the admin "show hidden" view. (See OQ-5.)
+- **Config:** ensure `kryten-hidden` is registered in `HIDDEN_TAG_NAMES` (or a dedicated constant) so browse/search/facets exclude it for non-admins.
+
+---
+
+## C. Queue page (`queue/index.html`)
+
+### C1. Hide the order number
+- Remove the `qi-pos` index number from each queue item in `renderQueue()` (and its CSS), or hide via CSS (`.qi-pos { display: none; }`). Prefer removing the element from the template render to keep the DOM clean.
+
+### C2. Remove the drag handle / no drag-drop
+- Remove the `qi-drag` (☰) handle element from each queue item render. There is **no** drag-reorder implemented on this page and none should be added — drop the handle, its title tooltip, and any related CSS. (Admin reordering lives in the playlist editor, not the live queue.)
+
+---
+
+## D. Data model & migrations
+
+New migration(s):
+
+1. **Job-run reconciliation** is runtime, not schema; but add `params` column to `job_runs` (nullable TEXT/JSON) to record submitted parameters:
+   ```sql
+   ALTER TABLE job_runs ADD COLUMN params TEXT;
+   ```
+2. **catalog.added_at backfill** (stopgap before sync repopulates from `add_date`):
+   ```sql
+   UPDATE catalog SET added_at = synced_at WHERE added_at IS NULL;
+   ```
+3. No new tables required for B4/B5 (reuse `saved_playlists`/`saved_playlist_items`). B6 reuses `catalog_tags`.
+
+Sync change (not a migration): set `added_at` from MediaCMS `add_date` in `_process_item`/`insert_catalog`/`update_catalog`.
+
+---
+
+## E. Config additions
+
+```jsonc
+{
+  // existing: mediacms_url, mediacms_token, tmdb_api_key, omdb_api_key, image_dir, placeholder_dir ...
+
+  "fetch_cookies_path": "",                 // optional yt-dlp cookies for gated sources
+  "fetchurls": {                             // fetchurls (v1 = local file only, OQ-1)
+    "workbook_path": ""                      // absolute path to a synced Channel Z Playlist .xlsx
+  }
+}
+```
+
+All new config is optional; jobs whose config/deps are absent register but fail fast with a clear message. (SharePoint/Graph config is intentionally omitted in v1.)
+
+---
+
+## F. API surface additions
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| `GET` | `/admin/jobs` | (extend) include each job's `schema` + last-run summary |
+| `POST` | `/admin/jobs/{name}/run` | (extend) accept `{params}` body |
+| `POST` | `/admin/playlists/{id}/append` | Append one catalog item to a playlist (B4) |
+| `POST` | `/admin/playlists/recent/append` | Append to the admin's most-recent playlist (B5) |
+| `POST` | `/admin/catalog/{friendly_token}/hide` | Tag `kryten-hidden` in MediaCMS + hide locally (B6) |
+| `POST` | `/admin/catalog/{friendly_token}/unhide` | Remove `kryten-hidden` (B6, recommended) |
+| `GET` | `/catalog/browse`, `/catalog/search` | (extend) accept `sort` param (B1) |
+
+All `/admin/*` routes use the existing `require_admin` dependency.
+
+---
+
+## G. Dependencies
+
+- New optional extra `jobs`: `yt-dlp`, `openpyxl`, `requests`. (`msal` is **not** needed in v1 — SharePoint/Graph is deferred per OQ-1.)
+- System: `ffmpeg` on the host for `fetch`. Document in `deploy/` notes.
+- No new deps for Browse/Queue work.
+
+---
+
+## H. Phasing / sequencing
+
+1. **Phase 1 — Quick wins (no external tools):**
+   - A1.2 job-history reconciliation (`interrupted` status).
+   - C1/C2 queue page cleanup.
+   - B3 vertical buttons, B1 sort (incl. `added_at` backfill + sync populate), B2 random art + hover.
+   - B4/B5/B6 admin tile actions (B6 needs the MediaCMS write client — see Phase 3 note).
+2. **Phase 2 — Job framework:** A1.3 parameterized jobs + schema-driven Run modal.
+3. **Phase 3 — Reimplemented jobs (in dependency order of risk):**
+   - `enrichtitles`, `enrichmeta`, `enrichtv` (proves vendor+thread pattern; yields the shared MediaCMS write client used by B6).
+   - `fetch` (+ add-to-playlist); needs yt-dlp/ffmpeg.
+   - `fetchurls` (+ playlist import, upcoming-weekend, **local-file workbook per OQ-1**) — **last**, highest risk.
+
+> If B6 must ship in Phase 1 before the enrich vendor work, write a minimal standalone MediaCMS tag-write helper now and fold it into `_common.py` later.
+
+---
+
+## I. Residual risks
+
+All open questions are resolved in §0.1. Remaining implementation risks:
+
+- **MediaCMS tag write (B6 / OQ-6):** the exact edit verb/field must be confirmed against the live instance. Mitigation: a round-trip integration test on a disposable item before wiring the Hide UI; reuse the enrich tools' proven edit path.
+- **Vendoring drift (OQ-2):** vendored enrich/fetch logic (~3k lines) will diverge from `d:\devel\cmsutils` over time. Mitigation: header-stamp the upstream commit/date; keep adapters thin so re-vendoring is mechanical.
+- **Long-job UX:** fetch/fetchurls can run for many minutes. The in-memory per-job lock + history `interrupted` reconciliation cover correctness; the optional heartbeat (A1.2.4) gives admins live progress.
+- **`fetch`/`fetchurls` host deps:** require `yt-dlp` + `ffmpeg` (and a readable workbook for fetchurls). Mitigation: jobs register but fail fast with a clear "dependency/config missing" message instead of crashing startup.
+- **`added_at` accuracy (B1):** the stopgap backfill sets `added_at = synced_at`; true `add_date` only becomes correct after the next full sync. Mitigation: trigger a sync after deploy, or document that "Newest first" sharpens once sync runs.
+
+---
+
+## J. Validation plan
+
+- **Jobs framework:** unit-test reconciliation (insert a `running` row → startup → becomes `interrupted`); schema validation rejects bad params; a parameterized job receives params and records them.
+- **enrich jobs:** run against a disposable MediaCMS item in `dry_run` and assert "scanned/matched" counts without writes; one `commit` run asserts the metadata changed and `job_runs.detail` is populated.
+- **fetch:** dry-ish test with a short known clip; assert a `friendly_token` returns and (with `add_to_playlist`) the item appears in the playlist.
+- **fetchurls:** unit-test the upcoming-weekend sheet-name computation across weekdays (incl. Friday → today per OQ-3); integration test against a fixture `.xlsx` via the configured file path; assert imported saved playlists are named `{sheet}-{section}` and re-runs replace rather than duplicate.
+- **Browse:** SQLite fixture tests for each `sort` ordering incl. `added_at` newest/oldest; verify hidden-tag exclusion still applies; render checks for vertical buttons and the hover art markup.
+- **Hide:** assert the MediaCMS write is attempted, the local `catalog_tags` gains `kryten-hidden`, and the item drops from a non-admin browse query.
+- **Queue:** assert rendered items contain neither `qi-pos` nor `qi-drag`.

{kryten_webqueue-0.8.1 → kryten_webqueue-0.9.0}/kryten_webqueue/app.py

@@ -28,6 +28,7 @@ from .routes.admin_playlists import router as admin_playlists_router
 from .routes.admin_schedules import router as admin_schedules_router
 from .routes.admin_queue import router as admin_queue_router
 from .routes.admin_jobs import router as admin_jobs_router
+from .routes.admin_catalog import router as admin_catalog_router
 from .routes.pages import router as pages_router
 from .ws.handler import router as ws_router
 
@@ -42,6 +43,11 @@ async def lifespan(app: FastAPI):
     db = Database(config.db_path)
     await db.connect()
     await db.run_migrations()
+    # Any job run still marked 'running' is an orphan from a prior crash/restart
+    # (the running flag is in-memory only). Reconcile before registering jobs.
+    orphaned = await db.reconcile_orphaned_job_runs()
+    if orphaned:
+        logger.warning("Reconciled %d orphaned job run(s) to 'interrupted'", orphaned)
     app.state.db = db
 
     # API Gate client
@@ -67,12 +73,38 @@ async def lifespan(app: FastAPI):
     app.state.catalog_sync = catalog_sync
 
     # Generic background job runner (records run history to job_runs)
-    job_manager = JobManager(db)
+    job_manager = JobManager(db, api_gate=api_gate, config=config)
+
+    async def _catalog_sync_job(params, ctx):
+        # catalog_sync registers with no schema; params/ctx are ignored. The
+        # adapter keeps the zero-arg sync compatible with the params/ctx job API.
+        return await catalog_sync.sync()
+
     job_manager.register(
         "catalog_sync",
-        catalog_sync.sync,
+        _catalog_sync_job,
         label="Catalog Sync",
     )
+
+    # Reimplemented cmsutils enrichment jobs (vendored, run off-loop). These
+    # register regardless of optional deps; a missing dep fails the run fast
+    # with a clear message rather than crashing startup.
+    from .jobs.tasks import (
+        enrichtitles_job, enrichmeta_job, enrichtv_job,
+        fetch_job, fetchurls_job,
+        ENRICHTITLES_SCHEMA, ENRICHMETA_SCHEMA, ENRICHTV_SCHEMA,
+        FETCH_SCHEMA, FETCHURLS_SCHEMA,
+    )
+    job_manager.register("enrichtitles", enrichtitles_job,
+                         label="Enrich Titles", schema=ENRICHTITLES_SCHEMA)
+    job_manager.register("enrichmeta", enrichmeta_job,
+                         label="Enrich Movie Metadata", schema=ENRICHMETA_SCHEMA)
+    job_manager.register("enrichtv", enrichtv_job,
+                         label="Enrich TV Metadata", schema=ENRICHTV_SCHEMA)
+    job_manager.register("fetch", fetch_job,
+                         label="Fetch (download → MediaCMS)", schema=FETCH_SCHEMA)
+    job_manager.register("fetchurls", fetchurls_job,
+                         label="Fetch URLs (weekend workbook)", schema=FETCHURLS_SCHEMA)
     app.state.job_manager = job_manager
 
     # WebSocket manager
@@ -184,6 +216,7 @@ def create_app(config: Config) -> FastAPI:
     app.include_router(admin_schedules_router)
     app.include_router(admin_queue_router)
     app.include_router(admin_jobs_router)
+    app.include_router(admin_catalog_router)
     app.include_router(ws_router)
 
     # Health check