kryten-webqueue 0.9.7__tar.gz → 0.9.9__tar.gz

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/CHANGELOG.md

@@ -6,6 +6,28 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [Unreleased]
 
+## [0.9.9] — 2026-06-12
+
+### Changed
+
+- **Graceful handling of expected job failures.** Misconfiguration and bad-input errors (e.g. `fetchurls` not finding this weekend's worksheet, a missing/unauthenticated workbook, or a missing optional dependency) are now recorded as a clean, actionable message in the job-run history and logged at WARNING — no stack trace. A new internal `JobError` distinguishes these expected failures from unexpected bugs (which still log a full traceback and keep their exception-type prefix).
+- The `fetchurls` "sheet not found" message now reads as guidance ("This weekend's worksheet 'M.D-M.D' was not found…") and lists only the date-format weekend sheets instead of every tab in the workbook.
+- The admin Jobs history table now shows a **Detail** column — the failure message for failed runs, or a compact summary (sheet, imported playlists, counts) for successful ones.
+
+## [0.9.8] — 2026-06-12
+
+### Added
+
+- **`fetchurls` job now reads the Channel Z workbook from SharePoint** (Microsoft Graph) and **writes resolved dropsugar URLs back to column F**, restoring the original tool's full round-trip. The service authenticates *silently* from a pre-seeded MSAL token cache and never prompts interactively. A one-time sign-in helper seeds the cache: `python -m kryten_webqueue.jobs.fetchurls_auth` (also installed as `kryten-webqueue-fetchurls-auth`). It prints a device-code URL; sign in once and the ~90-day refresh token is cached for unattended runs. If the cache is missing/expired the job fails with a clear "run fetchurls_auth" message.
+  - New config under `fetchurls`: `sharepoint_tenant_id`, `sharepoint_client_id`, `sharepoint_sharing_url`, `token_cache_path`. A local `workbook_path` (or a per-run `workbook_path` param) still works as a fallback/override.
+  - New `writeback` job parameter (default on) controls column-F writeback; `dry_run` and local-file mode skip it.
+  - Added `msal` as a core dependency.
+- **`fetchurls` imports into three fixed, well-known playlists** — `Friday Night`, `Saturday Morning`, `Saturday Night` — instead of date-prefixed names. The playlists are matched by name (any creator), their items replaced in place on every run (idempotent), and their existing immutability preserved; if a playlist is missing it is (re)created as immutable so the reserved status is retained.
+
+### Fixed
+
+- Bulk/`cm:` playlist imports for items not yet in the local catalog (e.g. freshly downloaded by `fetch`/`fetchurls` before a sync) now construct the CyTube manifest URL from the token instead of leaving a bare token that wouldn't play.
+
 ## [0.9.7] — 2026-06-11
 
 ### Changed
@@ -19,7 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - **Richer Bulk Text Import on the admin Playlists editor.** The text import now accepts, one entry per line:
-  - **dropsugar.co / dropsugar.com links** (watch `?m=TOKEN` or manifest `/api/v1/media/cytube/TOKEN.json`) — resolved against the catalog for title/duration, falling back to a constructed manifest URL when the token isn't catalogued yet.
+  - **dropsugar.co links** (watch `?m=TOKEN` or manifest `/api/v1/media/cytube/TOKEN.json`) — resolved against the catalog for title/duration, falling back to a constructed manifest URL when the token isn't catalogued yet.
   - **YouTube / youtu.be links** — playlist (`list=`), start-time (`t`/`start`) and all other arguments are stripped, leaving a clean `yt:VIDEOID` item.
   - Legacy `cm:token`, `yt:id`, and bare catalog tokens (unchanged).
   - Trailing free text after a URL (e.g. `URL - My Title`) is used as a title hint.

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kryten-webqueue
-Version: 0.9.7
+Version: 0.9.9
 Summary: Netflix/Tubi-style catalog browser and pay-to-play queue management for CyTube
 Author: grobertson
 License-Expression: MIT
@@ -10,6 +10,7 @@ Requires-Dist: apscheduler>=3.10
 Requires-Dist: fastapi>=0.115
 Requires-Dist: httpx>=0.27
 Requires-Dist: jinja2>=3.1
+Requires-Dist: msal>=1.28
 Requires-Dist: openpyxl>=3.1
 Requires-Dist: pillow>=10.0
 Requires-Dist: pydantic>=2.0

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/config.example.json

@@ -16,7 +16,11 @@
 
   "fetch_cookies_path": "",
   "fetchurls": {
-    "workbook_path": ""
+    "workbook_path": "",
+    "sharepoint_tenant_id": "",
+    "sharepoint_client_id": "",
+    "sharepoint_sharing_url": "",
+    "token_cache_path": "/var/lib/kryten-webqueue/.fetchurls_tokens.bin"
   },
 
   "db_path": "/var/lib/kryten-webqueue/webqueue.db",

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/catalog/db.py

@@ -956,6 +956,19 @@ class Database:
             [name, created_by],
         )
 
+    async def get_playlist_by_name_any(self, name: str) -> dict | None:
+        """Match an existing saved playlist by exact name, any creator.
+
+        Used for the fetchurls fixed section playlists ("Friday Night",
+        "Saturday Morning", "Saturday Night") which pre-exist (possibly created
+        by a different admin) and must be reused/replaced in place rather than
+        duplicated.
+        """
+        return await self._fetch_one(
+            "SELECT * FROM saved_playlists WHERE name=? ORDER BY id LIMIT 1",
+            [name],
+        )
+
     # --- Schedules ---
 
     async def get_schedules(self) -> list[dict]:

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/config.py

@@ -4,9 +4,22 @@ import json
 
 
 class FetchUrlsConfig(BaseModel):
-    """Settings for the fetchurls job (v1 = local file only, OQ-1)."""
+    """Settings for the fetchurls job.
 
-    workbook_path: str = ""  # absolute path to a synced Channel Z Playlist .xlsx
+    Reads the Channel Z workbook from SharePoint (Microsoft Graph) when the
+    SharePoint fields are configured, otherwise falls back to a local ``.xlsx``
+    at ``workbook_path``. SharePoint auth uses a pre-seeded MSAL token cache
+    (see ``python -m kryten_webqueue.jobs.fetchurls_auth``); the service only
+    acquires tokens *silently* from that cache and never prompts interactively.
+    """
+
+    workbook_path: str = ""  # local .xlsx fallback (used when SharePoint unset)
+
+    # SharePoint / Microsoft Graph (read workbook + write resolved URLs to col F)
+    sharepoint_tenant_id: str = ""
+    sharepoint_client_id: str = ""
+    sharepoint_sharing_url: str = ""
+    token_cache_path: str = ""  # MSAL cache file, pre-seeded out-of-band
 
 
 class Config(BaseModel):

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/integrations/cmsutils/fetchurls.py

@@ -287,6 +287,43 @@ def acquire_graph_token(tenant_id: str, client_id: str, cache_path: str = "") ->
     return result["access_token"]
 
 
+def acquire_graph_token_silent(tenant_id: str, client_id: str, cache_path: str) -> Optional[str]:
+    """Return a Graph access token from a pre-seeded MSAL cache, or None.
+
+    Unlike :func:`acquire_graph_token`, this NEVER prompts interactively — it is
+    safe to call from the headless webqueue service. The cache must have been
+    seeded out-of-band (``python -m kryten_webqueue.jobs.fetchurls_auth``). A
+    None return means the admin must (re)authenticate.
+    """
+    if not _HAS_MSAL:
+        raise RuntimeError(
+            "fetchurls SharePoint integration requires 'msal'. "
+            "Install the optional extra: pip install 'kryten-webqueue[jobs]'"
+        )
+    if not cache_path:
+        return None
+    cache_file = Path(cache_path)
+    if not cache_file.exists():
+        return None
+    token_cache = msal.SerializableTokenCache()
+    token_cache.deserialize(cache_file.read_text(encoding="utf-8"))
+    app = msal.PublicClientApplication(
+        client_id,
+        authority=f"https://login.microsoftonline.com/{tenant_id}",
+        token_cache=token_cache,
+    )
+    accounts = app.get_accounts()
+    if not accounts:
+        return None
+    result = app.acquire_token_silent(GRAPH_SCOPES, account=accounts[0])
+    if token_cache.has_state_changed:
+        cache_file.write_text(token_cache.serialize(), encoding="utf-8")
+    if result and "access_token" in result:
+        return result["access_token"]
+    return None
+
+
+
 def _encode_sharing_url(url: str) -> str:
     """Encode a SharePoint sharing/document URL for use with the Graph shares API."""
     encoded = base64.urlsafe_b64encode(url.encode()).decode().rstrip("=")
@@ -1269,13 +1306,21 @@ def _make_inprocess_fetch(config):
 
 
 def run(params: dict, *, config, progress=None) -> dict:
-    """Resolve the upcoming weekend's Channel Z workbook into playlist files.
+    """Resolve the upcoming weekend's Channel Z workbook into playlist sections.
+
+    Source precedence:
+      1. ``params['workbook_path']`` (one-off local override, e.g. for tests)
+      2. SharePoint via Microsoft Graph when ``config.fetchurls.sharepoint_*``
+         are set (token acquired *silently* from the pre-seeded MSAL cache)
+      3. ``config.fetchurls.workbook_path`` (local file fallback)
 
-    File-only (OQ-1): reads a local ``.xlsx`` (``params['workbook_path']`` or
-    ``config.fetchurls.workbook_path``); no SharePoint/Graph/MSAL, no column-F
-    writeback. Off-site URLs are downloaded via the in-process yt-pipe
-    downloader. Returns counts + per-section resolved ``cm:`` lines for the job
-    wrapper to import as saved playlists.
+    When the source is SharePoint and ``writeback`` is on (and not a dry run),
+    resolved dropsugar URLs are written back to column F via the Graph Excel
+    API. Off-site URLs are downloaded via the in-process yt-pipe downloader.
+
+    Returns counts plus, per section slug, the resolved ``cm:`` lines and the
+    human label ("Friday Night", etc.) so the job wrapper can import each into a
+    fixed, well-known saved playlist.
     """
     _check_openpyxl()
     import openpyxl as _oxl  # noqa: F811 - explicit local import for clarity
@@ -1283,32 +1328,64 @@ def run(params: dict, *, config, progress=None) -> dict:
     section = params.get("section") or "all"
     dry_run = bool(params.get("dry_run", False))
     validate = bool(params.get("validate", True))
-
-    cfg_fetchurls = getattr(config, "fetchurls", None)
-    cfg_path = ""
-    if cfg_fetchurls is not None:
-        cfg_path = getattr(cfg_fetchurls, "workbook_path", "") or ""
-    workbook_path = params.get("workbook_path") or cfg_path
-    if not workbook_path:
-        raise RuntimeError("fetchurls requires a workbook path (config.fetchurls.workbook_path)")
-    wb_file = Path(workbook_path)
-    if not wb_file.exists():
-        raise RuntimeError(f"Workbook not found: {wb_file}")
+    writeback = bool(params.get("writeback", True))
 
     def _emit(detail):
         if progress:
             progress(detail)
 
-    wb_bytes = wb_file.read_bytes()
+    cfg = getattr(config, "fetchurls", None)
+    local_override = params.get("workbook_path") or ""
+    sp_tenant = getattr(cfg, "sharepoint_tenant_id", "") if cfg else ""
+    sp_client = getattr(cfg, "sharepoint_client_id", "") if cfg else ""
+    sp_share = getattr(cfg, "sharepoint_sharing_url", "") if cfg else ""
+    sp_cache = getattr(cfg, "token_cache_path", "") if cfg else ""
+    cfg_local = getattr(cfg, "workbook_path", "") if cfg else ""
+
+    use_sharepoint = bool(sp_tenant and sp_client and sp_share) and not local_override
+
+    wb_ctx = None
+    drive_id = item_id = ""
+    graph_token = ""
+
+    if use_sharepoint:
+        _emit({"phase": "auth", "source": "sharepoint"})
+        graph_token = acquire_graph_token_silent(sp_tenant, sp_client, sp_cache)
+        if not graph_token:
+            raise RuntimeError(
+                "SharePoint authentication unavailable: no valid token in the MSAL "
+                "cache. Run a one-time sign-in on the server: "
+                "python -m kryten_webqueue.jobs.fetchurls_auth"
+            )
+        _emit({"phase": "download", "source": "sharepoint"})
+        try:
+            wb_bytes, drive_id, item_id = download_sharepoint_xlsx(graph_token, sp_share)
+        except SystemExit as exc:  # the vendored reader uses sys.exit on failure
+            raise RuntimeError(f"SharePoint download failed: {exc}") from exc
+    else:
+        workbook_path = local_override or cfg_local
+        if not workbook_path:
+            raise RuntimeError(
+                "fetchurls needs a workbook source: configure "
+                "fetchurls.sharepoint_* (recommended) or fetchurls.workbook_path."
+            )
+        wb_file = Path(workbook_path)
+        if not wb_file.exists():
+            raise RuntimeError(f"Workbook not found: {wb_file}")
+        wb_bytes = wb_file.read_bytes()
 
     sheet_name, friday, saturday = upcoming_weekend_sheet()
     wb_peek = _oxl.load_workbook(io.BytesIO(wb_bytes), read_only=True)
     all_sheets = wb_peek.sheetnames
     wb_peek.close()
     if sheet_name not in all_sheets:
+        # Suggest only date-format weekend sheets (ignore Sheet1/Played Movies/etc).
+        weekend_sheets = [s for s in all_sheets if _SHEET_DATE_RE.match(s.strip())]
+        available = ", ".join(weekend_sheets) if weekend_sheets else ", ".join(all_sheets)
         raise RuntimeError(
-            f"Sheet '{sheet_name}' (upcoming weekend) not found. "
-            f"Available: {', '.join(all_sheets)}"
+            f"This weekend's worksheet '{sheet_name}' was not found in the workbook. "
+            f"Add a sheet named '{sheet_name}' (Friday.date-Saturday.date), or check "
+            f"the sheet name matches. Available weekend sheets: {available}"
         )
 
     _emit({"phase": "parsing", "sheet": sheet_name})
@@ -1316,6 +1393,14 @@ def run(params: dict, *, config, progress=None) -> dict:
     if section and section != "all":
         sections = {k: v for k, v in sections.items() if k == section}
 
+    # Build the writeback context (SharePoint only, non-dry-run, writeback on).
+    if use_sharepoint and writeback and not dry_run:
+        wb_ctx = WritebackContext(
+            token=graph_token, drive_id=drive_id, item_id=item_id,
+            sheet_name=sheet_name, dry_run=False,
+            tenant_id=sp_tenant, client_id=sp_client, cache_path=sp_cache,
+        )
+
     # Swap in the in-process fetch (no fetch.ps1 subprocess in a service).
     global run_fetch  # noqa: PLW0603
     original_run_fetch = run_fetch
@@ -1328,15 +1413,17 @@ def run(params: dict, *, config, progress=None) -> dict:
     downloaded = 0
     failures = 0
     section_lines: dict[str, list[str]] = {}
+    section_labels: dict[str, str] = {}
     all_results: dict[str, list[ProcessResult]] = {}
 
     try:
         for slug, url_rows in sections.items():
             if not url_rows:
                 continue
-            label = SECTION_LABELS.get(slug, slug) if "SECTION_LABELS" in globals() else slug
+            label = SECTION_LABELS.get(slug, slug)
+            section_labels[slug] = label
             results = process_section(
-                label, url_rows, Path("."), dry_run, validate, wb_ctx=None,
+                label, url_rows, Path("."), dry_run, validate, wb_ctx=wb_ctx,
             )
             all_results[slug] = results
             write_playlist(out_dir / f"{sheet_name}-{slug}.txt", results)
@@ -1354,14 +1441,26 @@ def run(params: dict, *, config, progress=None) -> dict:
         write_failures(out_dir / f"{sheet_name}-failures.txt", all_results)
     finally:
         run_fetch = original_run_fetch
+        if wb_ctx is not None:
+            try:
+                _close_workbook_session(wb_ctx)
+            except Exception:  # noqa: BLE001 - best-effort cleanup
+                pass
+
+    writeback_stats = None
+    if wb_ctx is not None:
+        writeback_stats = {"ok": wb_ctx.writes_ok, "failed": wb_ctx.writes_fail}
 
     _emit({"phase": "done", "sheet": sheet_name, "resolved": resolved, "failures": failures})
     return {
         "sheet": sheet_name,
+        "source": "sharepoint" if use_sharepoint else "local",
         "resolved": resolved,
         "downloaded": downloaded,
         "failures": failures,
+        "writeback": writeback_stats,
         "section_lines": section_lines,
+        "section_labels": section_labels,
         "imported_playlists": [],  # filled in by the async job wrapper
         "dry_run": dry_run,
     }

kryten_webqueue-0.9.9/kryten_webqueue/jobs/fetchurls_auth.py

@@ -0,0 +1,62 @@
+"""One-time SharePoint sign-in for the fetchurls job.
+
+The webqueue service only acquires Microsoft Graph tokens *silently* from a
+pre-seeded MSAL cache (it never prompts). Run this once on the server to create
+that cache via the device-code flow:
+
+    python -m kryten_webqueue.jobs.fetchurls_auth [--config /path/to/config.json]
+
+It prints a URL + code; sign in with the curiousmotors account and grant the
+Files.ReadWrite.All consent. The refresh token is cached (~90 days) at
+``fetchurls.token_cache_path`` so subsequent unattended job runs authenticate
+silently. Re-run this when the cache expires.
+"""
+
+import argparse
+import os
+import sys
+
+from ..config import Config
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description="One-time SharePoint sign-in for fetchurls")
+    parser.add_argument(
+        "--config",
+        default=os.environ.get("WQ_CONFIG", "/etc/kryten-webqueue/config.json"),
+        help="Path to the webqueue config.json (default: $WQ_CONFIG or /etc/kryten-webqueue/config.json)",
+    )
+    args = parser.parse_args(argv)
+
+    config = Config.from_file(args.config)
+    fu = config.fetchurls
+    if not (fu.sharepoint_tenant_id and fu.sharepoint_client_id):
+        print(
+            "ERROR: fetchurls.sharepoint_tenant_id and sharepoint_client_id must be "
+            "set in the config before authenticating.",
+            file=sys.stderr,
+        )
+        return 2
+    if not fu.token_cache_path:
+        print(
+            "ERROR: fetchurls.token_cache_path must be set so the token can be cached "
+            "for the service to reuse.",
+            file=sys.stderr,
+        )
+        return 2
+
+    from ..integrations.cmsutils.fetchurls import acquire_graph_token
+
+    token = acquire_graph_token(
+        fu.sharepoint_tenant_id, fu.sharepoint_client_id, fu.token_cache_path
+    )
+    if token:
+        print(f"\n✓ Authenticated. Token cached at: {fu.token_cache_path}")
+        print("  The fetchurls job can now run unattended until the cache expires.")
+        return 0
+    print("ERROR: Authentication did not complete.", file=sys.stderr)
+    return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/jobs/manager.py

@@ -24,6 +24,15 @@ logger = logging.getLogger(__name__)
 JobFunc = Callable[[dict, "JobContext"], Awaitable[dict | None]]
 
 
+class JobError(Exception):
+    """An expected, user-facing job failure (bad input / config, not a bug).
+
+    Raising this from a job records a clean ``failed`` run with the message and
+    logs it at WARNING without a stack trace, so misconfiguration (e.g. a
+    missing workbook sheet) reads as actionable guidance rather than a crash.
+    """
+
+
 def _option_values(field: dict) -> list:
     """Return the allowed values for an enum field's ``options``.
 
@@ -184,6 +193,13 @@ class JobManager:
         except asyncio.CancelledError:
             await self._db.finish_job_run(run_id, "cancelled", None)
             raise
+        except JobError as exc:
+            # Expected, user-facing failure (bad input/config): record a clean
+            # message and log without a stack trace.
+            logger.warning("Job '%s' failed: %s", name, exc)
+            await self._db.finish_job_run(
+                run_id, "failed", json.dumps({"error": str(exc)})
+            )
         except Exception as exc:  # noqa: BLE001 - record any failure
             logger.exception("Job '%s' failed", name)
             await self._db.finish_job_run(

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/jobs/tasks.py

@@ -15,6 +15,8 @@ import functools
 import importlib
 import logging
 
+from .manager import JobError
+
 logger = logging.getLogger(__name__)
 
 
@@ -44,14 +46,23 @@ def _thread_safe_progress(ctx, loop):
 
 
 async def _run_vendored(module_path: str, params: dict, ctx, *, deps: list[str]):
-    """Import a vendored module, verify deps, and run its ``run()`` off-loop."""
-    for dep in deps:
-        _require(dep)
-    module = importlib.import_module(module_path)
-    loop = asyncio.get_running_loop()
-    progress = _thread_safe_progress(ctx, loop)
-    fn = functools.partial(module.run, params, config=ctx.config, progress=progress)
-    return await asyncio.to_thread(fn)
+    """Import a vendored module, verify deps, and run its ``run()`` off-loop.
+
+    The vendored tools raise ``RuntimeError`` for expected, user-facing failures
+    (missing/unauthenticated workbook, a sheet that isn't present, a missing
+    optional dependency). Surface those as :class:`JobError` so the run history
+    shows a clean, actionable message instead of a stack trace.
+    """
+    try:
+        for dep in deps:
+            _require(dep)
+        module = importlib.import_module(module_path)
+        loop = asyncio.get_running_loop()
+        progress = _thread_safe_progress(ctx, loop)
+        fn = functools.partial(module.run, params, config=ctx.config, progress=progress)
+        return await asyncio.to_thread(fn)
+    except RuntimeError as exc:
+        raise JobError(str(exc)) from exc
 
 
 # ── Enrich jobs ────────────────────────────────────────────────────────────────
@@ -121,33 +132,43 @@ async def fetch_job(params: dict, ctx):
 # ── fetchurls job ──────────────────────────────────────────────────────────────
 
 async def _import_section_as_playlist(ctx, name: str, lines: list[str], triggered_by: str) -> dict | None:
-    """Import resolved ``cm:`` lines as a saved playlist, idempotent by name.
+    """Import resolved ``cm:`` lines into a fixed, well-known saved playlist.
 
-    Re-running replaces the items of an existing same-named playlist (matched
-    by name + creator) rather than creating a duplicate.
+    The three fetchurls playlists ("Friday Night", "Saturday Morning",
+    "Saturday Night") pre-exist and are immutable. We match by **name only**
+    (any creator), replace their items in place (idempotent re-runs), and
+    preserve their existing immutability. If a playlist is missing it is created
+    as immutable so a recreated playlist keeps the reserved status.
     """
     from ..playlists.importer import import_playlist_text
 
     if not lines:
         return None
-    parsed = await import_playlist_text(ctx.db, "\n".join(lines))
+    parsed = await import_playlist_text(
+        ctx.db, "\n".join(lines), mediacms_url=ctx.config.mediacms_url
+    )
     items = parsed.get("items") or []
     if not items:
         return None
 
-    existing = await ctx.db.get_playlist_by_name(name, triggered_by)
+    existing = await ctx.db.get_playlist_by_name_any(name)
     if existing:
         playlist_id = existing["id"]
     else:
         playlist_id = await ctx.db.create_saved_playlist(
-            name=name, description=None, is_immutable=False, created_by=triggered_by,
+            name=name, description=None, is_immutable=True, created_by=triggered_by,
         )
     await ctx.db.replace_playlist_items(playlist_id, items)
     return {"id": playlist_id, "name": name, "count": len(items)}
 
 
 async def fetchurls_job(params: dict, ctx):
-    """Resolve the upcoming-weekend workbook, then import each section playlist."""
+    """Resolve the upcoming-weekend workbook, then import each section playlist.
+
+    Sections map to the fixed playlists by their human label:
+      friday → "Friday Night", saturday-morning → "Saturday Morning",
+      saturday-night → "Saturday Night".
+    """
     result = await _run_vendored(
         "kryten_webqueue.integrations.cmsutils.fetchurls", params, ctx,
         deps=["openpyxl", "yaml", "requests"],
@@ -156,16 +177,17 @@ async def fetchurls_job(params: dict, ctx):
     if result.get("dry_run"):
         return result
 
-    sheet = result.get("sheet")
     triggered_by = ctx.triggered_by or "system"
+    labels = result.get("section_labels") or {}
     imported = []
     for slug, lines in (result.get("section_lines") or {}).items():
-        name = f"{sheet}-{slug}"
+        name = labels.get(slug) or slug
         info = await _import_section_as_playlist(ctx, name, lines, triggered_by)
         if info:
             imported.append(info["name"])
     result["imported_playlists"] = imported
     result.pop("section_lines", None)  # keep the persisted detail compact
+    result.pop("section_labels", None)
     return result
 
 
@@ -210,5 +232,6 @@ FETCHURLS_SCHEMA = [
      "options": ["all", "friday", "saturday-night", "saturday-morning"], "label": "Section"},
     {"name": "dry_run", "type": "bool", "default": False, "label": "Dry run (resolve only)"},
     {"name": "validate", "type": "bool", "default": True, "label": "Validate existing URLs"},
-    {"name": "workbook_path", "type": "string", "default": None, "label": "Workbook path (override)"},
+    {"name": "writeback", "type": "bool", "default": True, "label": "Write resolved URLs back to SharePoint (col F)"},
+    {"name": "workbook_path", "type": "string", "default": None, "label": "Local workbook path (override SharePoint)"},
 ]

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/playlists/importer.py

@@ -179,9 +179,16 @@ async def import_playlist_text(db, text: str, *, mediacms_url: str | None = None
         if line.startswith("cm:"):
             media_id = line[3:].strip()
             catalog_item = await db.get_item_admin(media_id)
+            if catalog_item:
+                resolved_media = catalog_item["manifest_url"]
+            else:
+                # Not yet in the local catalog (e.g. freshly downloaded by the
+                # fetch/fetchurls job before a sync) — construct the CyTube
+                # manifest URL from the token so it still plays.
+                resolved_media = _manifest_url_for_token(media_id, mediacms_url) or media_id
             items.append({
                 "media_type": "cm",
-                "media_id": catalog_item["manifest_url"] if catalog_item else media_id,
+                "media_id": resolved_media,
                 "title": catalog_item["title"] if catalog_item else None,
                 "duration_sec": catalog_item["duration_sec"] if catalog_item else None,
             })

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/kryten_webqueue/templates/admin/index.html

@@ -212,13 +212,14 @@ async function loadJobs() {
         const el = document.getElementById('job-runs');
         if (runs.length > 0) {
             el.innerHTML = `<table class="admin-table">
-                <tr><th>Job</th><th>Started</th><th>Ended</th><th>Status</th></tr>
+                <tr><th>Job</th><th>Started</th><th>Ended</th><th>Status</th><th>Detail</th></tr>
                 ${runs.map(r => `
                     <tr>
                         <td>${escapeHtml(r.job_name || '')}</td>
                         <td>${formatLocalDateTime(r.started_at)}</td>
                         <td>${r.ended_at ? formatLocalDateTime(r.ended_at) : '—'}</td>
                         <td><span class="job-status job-status-${escapeHtml(r.status || '')}">${escapeHtml(r.status || '')}</span></td>
+                        <td class="job-detail">${escapeHtml(summarizeRunDetail(r.detail))}</td>
                     </tr>
                 `).join('')}
             </table>`;
@@ -269,6 +270,27 @@ function escapeHtml(str) {
     return div.innerHTML;
 }
 
+// Summarize a job_runs.detail JSON blob for the history table. Prefers a
+// failure message, otherwise a compact success summary.
+function summarizeRunDetail(detail) {
+    if (!detail) return '';
+    let d;
+    try { d = typeof detail === 'string' ? JSON.parse(detail) : detail; }
+    catch { return String(detail).slice(0, 160); }
+    if (d && d.error) return d.error;
+    if (d && typeof d === 'object') {
+        const parts = [];
+        if (d.sheet) parts.push(d.sheet);
+        if (d.imported_playlists && d.imported_playlists.length) parts.push(`imported: ${d.imported_playlists.join(', ')}`);
+        if (typeof d.resolved === 'number') parts.push(`resolved ${d.resolved}`);
+        if (typeof d.failures === 'number' && d.failures) parts.push(`${d.failures} failed`);
+        if (typeof d.committed === 'number') parts.push(`committed ${d.committed}`);
+        if (typeof d.added_to_playlist !== 'undefined' && d.added_to_playlist) parts.push(`→ playlist ${d.added_to_playlist}`);
+        if (parts.length) return parts.join(' · ');
+    }
+    return '';
+}
+
 loadAdminData();
 loadJobs();
 </script>

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "kryten-webqueue"
-version = "0.9.7"
+version = "0.9.9"
 description = "Netflix/Tubi-style catalog browser and pay-to-play queue management for CyTube"
 readme = "README.md"
 license = "MIT"
@@ -25,6 +25,7 @@ dependencies = [
     "requests>=2.31",
     "pyyaml>=6.0",
     "python-slugify>=8.0",
+    "msal>=1.28",
 ]
 
 [project.optional-dependencies]
@@ -39,6 +40,7 @@ dev = [
 
 [project.scripts]
 kryten-webqueue = "kryten_webqueue.__main__:main"
+kryten-webqueue-fetchurls-auth = "kryten_webqueue.jobs.fetchurls_auth:main"
 
 [build-system]
 requires = ["hatchling"]

kryten_webqueue-0.9.9/tests/test_fetchurls_sharepoint.py

@@ -0,0 +1,150 @@
+"""fetchurls SharePoint + fixed-name playlist behaviour (v0.9.8).
+
+Covers the silent-token guard, source selection / clear error when the MSAL
+cache is unseeded, the fixed section-label playlist names, immutable-preserve on
+re-import, and the importer ``cm:`` manifest-URL fallback.
+"""
+
+from types import SimpleNamespace
+
+import pytest
+
+from kryten_webqueue.catalog.db import Database
+from kryten_webqueue.jobs.manager import JobContext
+from kryten_webqueue.jobs import tasks
+from kryten_webqueue.integrations.cmsutils import fetchurls
+from kryten_webqueue.playlists.importer import import_playlist_text
+
+
+@pytest.fixture
+async def db(tmp_path):
+    database = Database(str(tmp_path / "fu.db"))
+    await database.connect()
+    await database.run_migrations()
+    yield database
+    await database.close()
+
+
+def _config(**fu):
+    return SimpleNamespace(
+        mediacms_url="https://cms.example",
+        mediacms_token="tok",
+        tmdb_api_key="", omdb_api_key="",
+        image_dir="/tmp/kqimg",
+        fetch_cookies_path="",
+        fetchurls=SimpleNamespace(
+            workbook_path=fu.get("workbook_path", ""),
+            sharepoint_tenant_id=fu.get("sharepoint_tenant_id", ""),
+            sharepoint_client_id=fu.get("sharepoint_client_id", ""),
+            sharepoint_sharing_url=fu.get("sharepoint_sharing_url", ""),
+            token_cache_path=fu.get("token_cache_path", ""),
+        ),
+    )
+
+
+def _ctx(db, config=None):
+    return JobContext(db=db, api_gate=None, config=config or _config(), run_id=1, triggered_by="admin")
+
+
+# --- silent token guard ---
+
+def test_silent_token_none_without_cache(tmp_path):
+    missing = str(tmp_path / "nope.bin")
+    assert fetchurls.acquire_graph_token_silent("tenant", "client", missing) is None
+    assert fetchurls.acquire_graph_token_silent("tenant", "client", "") is None
+
+
+# --- run() source selection / clear error ---
+
+def test_run_sharepoint_without_token_raises_clear_error(tmp_path):
+    config = _config(
+        sharepoint_tenant_id="t", sharepoint_client_id="c",
+        sharepoint_sharing_url="https://sp/share",
+        token_cache_path=str(tmp_path / "absent.bin"),
+    )
+    with pytest.raises(RuntimeError, match="fetchurls_auth"):
+        fetchurls.run({}, config=config, progress=None)
+
+
+def test_run_no_source_raises(tmp_path):
+    with pytest.raises(RuntimeError, match="workbook source"):
+        fetchurls.run({}, config=_config(), progress=None)
+
+
+# --- vendored RuntimeError → JobError translation ---
+
+async def test_run_vendored_translates_runtimeerror_to_joberror(db, monkeypatch):
+    from kryten_webqueue.jobs.manager import JobError
+
+    def boom(params, *, config, progress):
+        raise RuntimeError("This weekend's worksheet was not found.")
+
+    fake_mod = SimpleNamespace(run=boom)
+    monkeypatch.setattr(tasks.importlib, "import_module", lambda name, *a, **k: fake_mod)
+
+    with pytest.raises(JobError, match="worksheet was not found"):
+        await tasks._run_vendored("whatever", {}, _ctx(db), deps=[])
+
+
+# --- fixed section-label playlist names + immutable preserve ---
+
+async def _add_catalog(db, token, title="T"):
+    await db.insert_catalog({
+        "friendly_token": token, "title": title, "description": "",
+        "duration_sec": 600, "manifest_url": f"https://cms/api/v1/media/cytube/{token}.json",
+        "thumbnail_url": "", "synced_at": "2026-01-01T00:00:00+00:00",
+    })
+
+
+async def test_fetchurls_job_uses_fixed_label_names(db, monkeypatch):
+    await _add_catalog(db, "f1")
+    await _add_catalog(db, "s1")
+
+    async def fake_run_vendored(module_path, params, ctx, *, deps):
+        return {
+            "sheet": "3.6-3.7", "dry_run": False, "resolved": 2, "failures": 0,
+            "section_lines": {"friday": ["cm:f1"], "saturday-night": ["cm:s1"]},
+            "section_labels": {"friday": "Friday Night", "saturday-night": "Saturday Night"},
+        }
+
+    monkeypatch.setattr(tasks, "_run_vendored", fake_run_vendored)
+    result = await tasks.fetchurls_job({}, _ctx(db))
+
+    assert sorted(result["imported_playlists"]) == ["Friday Night", "Saturday Night"]
+    names = {p["name"] for p in await db.get_saved_playlists()}
+    assert {"Friday Night", "Saturday Night"} <= names
+    # Created playlists are immutable (reserved).
+    fri = await db.get_playlist_by_name_any("Friday Night")
+    assert fri["is_immutable"] == 1
+
+
+async def test_fetchurls_import_preserves_existing_immutability(db, monkeypatch):
+    await _add_catalog(db, "f1")
+    await _add_catalog(db, "f2")
+    # Pre-existing MUTABLE playlist with the fixed name, created by someone else.
+    pid = await db.create_saved_playlist(
+        name="Friday Night", description=None, is_immutable=False, created_by="other-admin",
+    )
+
+    info = await tasks._import_section_as_playlist(_ctx(db), "Friday Night", ["cm:f1", "cm:f2"], "admin")
+    assert info["id"] == pid                      # reused, not duplicated
+    pl = await db.get_playlist_by_name_any("Friday Night")
+    assert pl["is_immutable"] == 0                # existing flag preserved (not forced)
+    assert pl["created_by"] == "other-admin"      # original owner kept
+    items = await db.get_saved_playlist_items(pid)
+    assert len(items) == 2
+
+
+# --- importer cm: manifest fallback for not-yet-synced items ---
+
+async def test_importer_cm_fallback_builds_manifest(db):
+    out = await import_playlist_text(db, "cm:freshtoken", mediacms_url="https://cms.example")
+    assert len(out["items"]) == 1
+    assert out["items"][0]["media_id"] == "https://cms.example/api/v1/media/cytube/freshtoken.json?format=json"
+
+
+async def test_importer_cm_uses_catalog_when_present(db):
+    await _add_catalog(db, "known", "Known Title")
+    out = await import_playlist_text(db, "cm:known", mediacms_url="https://cms.example")
+    assert out["items"][0]["media_id"] == "https://cms/api/v1/media/cytube/known.json"
+    assert out["items"][0]["title"] == "Known Title"

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/tests/test_phase2_jobs.py

@@ -135,6 +135,36 @@ async def test_unknown_job_raises_keyerror(db):
         await jm.run("nope")
 
 
+async def test_job_error_records_clean_message(db):
+    from kryten_webqueue.jobs.manager import JobError
+
+    async def job(params, ctx):
+        raise JobError("This weekend's worksheet '6.12-6.13' was not found.")
+
+    jm = JobManager(db)
+    jm.register("fetchurls", job)
+    await jm.run("fetchurls")
+    run = await _wait_terminal(db, "fetchurls")
+    assert run["status"] == "failed"
+    # Clean message, no "RuntimeError:"/"JobError:" type prefix or traceback.
+    assert json.loads(run["detail"]) == {
+        "error": "This weekend's worksheet '6.12-6.13' was not found."
+    }
+
+
+async def test_unexpected_error_keeps_type_prefix(db):
+    async def job(params, ctx):
+        raise ValueError("boom")
+
+    jm = JobManager(db)
+    jm.register("crashy", job)
+    await jm.run("crashy")
+    run = await _wait_terminal(db, "crashy")
+    assert run["status"] == "failed"
+    # Unexpected (bug) failures retain the exception type for debugging.
+    assert json.loads(run["detail"]) == {"error": "ValueError: boom"}
+
+
 async def test_already_running_guard(db):
     started = asyncio.Event()
     release = asyncio.Event()

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/tests/test_phase3_jobs.py

@@ -69,6 +69,7 @@ def test_extract_manifest_token():
 async def test_fetch_job_missing_ytdlp_fails_fast(db, monkeypatch):
     # Simulate yt_dlp absent regardless of the host environment.
     import importlib
+    from kryten_webqueue.jobs.manager import JobError
     real_import = importlib.import_module
 
     def fake_import(name, *a, **k):
@@ -77,7 +78,8 @@ async def test_fetch_job_missing_ytdlp_fails_fast(db, monkeypatch):
         return real_import(name, *a, **k)
 
     monkeypatch.setattr(tasks.importlib, "import_module", fake_import)
-    with pytest.raises(RuntimeError, match="yt_dlp"):
+    # A missing optional dependency is a clean, user-facing JobError.
+    with pytest.raises(JobError, match="yt_dlp"):
         await tasks.fetch_job({"url": "http://x"}, _ctx(db))
 
 

{kryten_webqueue-0.9.7 → kryten_webqueue-0.9.9}/tests/test_playlist_import.py

@@ -129,8 +129,9 @@ async def test_legacy_tokens_still_work(db):
     yt = [i for i in out["items"] if i["media_type"] == "yt"]
     cm = [i for i in out["items"] if i["media_type"] == "cm"]
     assert yt[0]["media_id"] == "dQw4w9WgXcQ"
-    # bare1 resolves to its manifest URL; cm:cmtoken passes through.
+    # bare1 resolves to its manifest URL; cm:cmtoken (not in catalog) now falls
+    # back to a constructed manifest URL so not-yet-synced items still play.
     assert any(i["media_id"].endswith("bare1.json?format=json") for i in cm)
-    assert any(i["media_id"] == "cmtoken" for i in cm)
+    assert any(i["media_id"] == f"{MEDIACMS}/api/v1/media/cytube/cmtoken.json?format=json" for i in cm)
     # unknownbare is not in catalog -> error
     assert any(e["token"] == "unknownbare" for e in out["errors"])