kryten-webqueue 0.43.0__tar.gz → 0.44.1__tar.gz

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/CHANGELOG.md

@@ -2,6 +2,68 @@
 
 ## [Unreleased]
 
+### Fixed
+
+- **Static assets are now cache-busted by version.** JS/CSS includes gained a
+  `?v={app_version}` query string, so browsers fetch fresh assets after every
+  release instead of serving a stale cached copy. This fixes the admin Download
+  Queue rendering as empty immediately after the 0.44.0 upgrade: the new endpoint
+  returns `{items, …}` but a cached pre-0.44.0 `admin.js` read the response as a
+  bare array, so `.length` was `undefined` and it showed "Queue is empty."
+
+## [0.44.0]
+
+### Added
+
+- **Admin Jobs tab: full-text run logs + humanized summaries.** Every background
+  job run now captures the Python log records it emits (scoped to that run via a
+  `contextvars` filter, so concurrent jobs and web requests don't bleed in) into
+  a new uncapped `job_run_logs` table (schema **v29**, forward-only migration —
+  kept as a separate table rather than a `job_runs` column so it ports cleanly to
+  Postgres). The run-detail modal now renders the JSON summary as **humanized,
+  labelled rows** (friendly `Snake Case → Title Case` labels, `Yes/No` for
+  booleans, thousands separators) instead of a raw blob, and shows the **full log**
+  beneath it. Both the JSON summary and the full-text log are **downloadable** per
+  run (`GET /admin/jobs/runs/{id}/detail/download` and `.../log/download`) for
+  offline study.
+- **Per-job run selector.** Each job row's "Last Run" text is replaced by a
+  dropdown of that job's recent runs (by start time) and a clickable status cell
+  (`completed` / `failed` / …) sitting just before the action buttons; picking a
+  run and clicking its status opens that run's summary + log.
+- **Download queue pagination + 24h success expiry.** The download-queue list is
+  paginated at **20 items/page**, and successful (`done`) downloads finished more
+  than **24h ago are hidden** to keep the list manageable. Failures and
+  pending/running items are always shown regardless of age. This is a
+  **visibility filter only** — no rows are ever deleted, so the audit trail is
+  preserved (`GET /admin/jobs/fetch-queue` now returns
+  `{items, total, page, pages, limit}`).
+
+- **Failed MediaCMS uploads are re-queued instead of lost.** When a fetch-queue
+  item's upload fails with a broken connection (`IncompleteRead`), the drain now
+  re-queues it to the **back** of the queue for another attempt (up to
+  **3 attempts** total) rather than marking it `failed` on the first hit. The
+  retry runs *after* everything else currently pending, and the existing
+  inter-item cooldown naturally spaces it out. Only the **upload stage** is
+  retried — download/validation failures still fail immediately (they need
+  case-by-case handling). A new `fetch_queue.attempts` column (schema **v28**,
+  forward-only migration) tracks attempts; the last failure is recorded as
+  `failed` with an "after N attempts" note.
+
+### Fixed
+
+- **MediaCMS upload no longer fails hard on a broken response.** The final upload
+  POST previously had **no timeout** and no handling for a dropped connection, so
+  a truncated response (`Connection broken: IncompleteRead(...)` — MediaCMS
+  serializes a large media object into the response *after* accepting the file)
+  surfaced as a permanent `Failed to upload to MediaCMS` even though the item had
+  usually already been created server-side. The POST now uses a `(connect, read)`
+  timeout of `(30, 600)`, and on a connection-broken/`IncompleteRead`/timeout it
+  attempts to **recover the just-created item** by its "Original URL" description
+  marker (applying tags to it) rather than blindly re-POSTing and creating a
+  duplicate. If it can't be recovered yet (e.g. search-index lag) it raises a
+  distinct, retryable error — the next attempt's up-front duplicate check skips it
+  if it did land.
+
 ## [0.43.0] - 2026-08-24
 
 ### Added

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: kryten-webqueue
-Version: 0.43.0
+Version: 0.44.1
 Summary: Netflix/Tubi-style catalog browser and pay-to-play queue management for CyTube
 Author: grobertson
 License-Expression: MIT

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/catalog/db/_catalog.py

@@ -1146,6 +1146,36 @@ class _CatalogMixin:
         """Update only a running job's detail column (used for live progress)."""
         await self._execute("UPDATE job_runs SET detail=? WHERE id=?", [detail, run_id])
 
+    async def add_job_run_logs(
+        self, run_id: int, records: list[tuple[str, str, str, str]]
+    ) -> None:
+        """Bulk-append captured log lines for a run.
+
+        ``records`` is a list of ``(logged_at_iso, level, logger, message)``
+        tuples in emission order; ``seq`` is assigned from the list index so the
+        original ordering survives even when timestamps collide.
+        """
+        if not records:
+            return
+        rows = [
+            (run_id, seq, logged_at, level, logger, message)
+            for seq, (logged_at, level, logger, message) in enumerate(records)
+        ]
+        await self._db.executemany(
+            "INSERT INTO job_run_logs (run_id, seq, logged_at, level, logger, message) "
+            "VALUES (?, ?, ?, ?, ?, ?)",
+            rows,
+        )
+        await self._db.commit()
+
+    async def get_job_run_logs(self, run_id: int) -> list[dict]:
+        return await self._fetch_all(
+            "SELECT * FROM job_run_logs WHERE run_id=? ORDER BY seq ASC", [run_id]
+        )
+
+    async def get_job_run(self, run_id: int) -> dict | None:
+        return await self._fetch_one("SELECT * FROM job_runs WHERE id=?", [run_id])
+
     async def get_job_runs(
         self, job_name: str | None = None, limit: int = 10
     ) -> list[dict]:

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/catalog/db/_connection.py

@@ -509,6 +509,30 @@ MIGRATIONS = [
     CREATE INDEX IF NOT EXISTS idx_device_api_keys_user ON device_api_keys(username);
     CREATE INDEX IF NOT EXISTS idx_device_api_keys_hash ON device_api_keys(key_hash);
     """,
+    # v28: Retry counter for the fetch queue. A MediaCMS upload that fails with a
+    # broken connection (IncompleteRead) is re-queued to the back of the queue up
+    # to a bounded number of attempts instead of failing permanently; this tracks
+    # how many upload attempts an item has consumed.
+    """
+    ALTER TABLE fetch_queue ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0;
+    """,
+    # v29: Full-text run logs. Each background job run captures the Python log
+    # records it emits (scoped to the run via a contextvars filter) as one row
+    # per line, so the admin UI can show and download a complete log alongside
+    # the JSON summary. Kept in a separate, uncapped table (Postgres-friendly)
+    # rather than a column on job_runs.
+    """
+    CREATE TABLE IF NOT EXISTS job_run_logs (
+        id        INTEGER PRIMARY KEY AUTOINCREMENT,
+        run_id    INTEGER NOT NULL,
+        seq       INTEGER NOT NULL,
+        logged_at TIMESTAMP NOT NULL,
+        level     TEXT,
+        logger    TEXT,
+        message   TEXT NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS idx_job_run_logs_run ON job_run_logs(run_id, seq);
+    """,
 ]
 
 

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/catalog/db/_fetch_queue.py

@@ -46,11 +46,13 @@ class _FetchQueueMixin:
         status: str,
         result_json: str | None = None,
         error: str | None = None,
+        bump_attempts: bool = False,
     ) -> None:
+        attempts_sql = ", attempts = attempts + 1" if bump_attempts else ""
         await self._db.execute(
-            """
+            f"""
             UPDATE fetch_queue
-            SET status = ?, finished_at = datetime('now'), result_json = ?, error = ?
+            SET status = ?, finished_at = datetime('now'), result_json = ?, error = ?{attempts_sql}
             WHERE id = ?
             """,
             [status, result_json, error, item_id],
@@ -71,6 +73,30 @@ class _FetchQueueMixin:
         )
         await self._db.commit()
 
+    async def requeue_fetch_item_for_retry(
+        self, item_id: int, *, error: str | None = None
+    ) -> int:
+        """Re-queue an item to the *back* of the queue for another upload attempt.
+
+        Increments ``attempts``, sets ``added_at`` strictly after every existing
+        row so it sorts last (retry after everything else currently pending), and
+        clears the run timestamps. Keeps ``error`` as a breadcrumb of why it was
+        re-queued. Returns the new ``attempts`` count.
+        """
+        await self._db.execute(
+            "UPDATE fetch_queue"
+            " SET status = 'pending', attempts = attempts + 1,"
+            " added_at = (SELECT datetime(MAX(added_at), '+1 second') FROM fetch_queue),"
+            " started_at = NULL, finished_at = NULL, error = ?"
+            " WHERE id = ?",
+            [error, item_id],
+        )
+        await self._db.commit()
+        row = await self._fetch_one(
+            "SELECT attempts FROM fetch_queue WHERE id = ?", [item_id]
+        )
+        return int(row["attempts"]) if row else 0
+
     async def reset_running_fetch_items(self) -> int:
         """Reset any 'running' items to 'pending'; return how many were reset.
 
@@ -88,16 +114,54 @@ class _FetchQueueMixin:
         return cursor.rowcount or 0
 
     async def get_fetch_queue(
-        self, *, limit: int = 100, status: str | None = None
+        self,
+        *,
+        limit: int = 100,
+        offset: int = 0,
+        status: str | None = None,
+        hide_expired_done: bool = False,
     ) -> list[dict]:
         sql = "SELECT * FROM fetch_queue"
+        where, params = self._fetch_queue_filter(status, hide_expired_done)
+        if where:
+            sql += " WHERE " + where
+        sql += " ORDER BY added_at DESC LIMIT ? OFFSET ?"
+        params.extend([limit, offset])
+        return await self._fetch_all(sql, params)
+
+    async def count_fetch_queue(
+        self, *, status: str | None = None, hide_expired_done: bool = False
+    ) -> int:
+        """Count rows matching the same visibility filter as ``get_fetch_queue``."""
+        sql = "SELECT COUNT(*) AS c FROM fetch_queue"
+        where, params = self._fetch_queue_filter(status, hide_expired_done)
+        if where:
+            sql += " WHERE " + where
+        row = await self._fetch_one(sql, params)
+        return int(row["c"]) if row else 0
+
+    @staticmethod
+    def _fetch_queue_filter(
+        status: str | None, hide_expired_done: bool
+    ) -> tuple[str, list]:
+        """Build the shared WHERE clause for fetch-queue list/count queries.
+
+        When ``hide_expired_done`` is set, successful downloads finished more than
+        24h ago are excluded from view (rows are retained for audit — this is a
+        visibility filter only). Failed / pending / running items are never
+        hidden regardless of age.
+        """
+        clauses: list[str] = []
         params: list = []
         if status:
-            sql += " WHERE status = ?"
+            clauses.append("status = ?")
             params.append(status)
-        sql += " ORDER BY added_at DESC LIMIT ?"
-        params.append(limit)
-        return await self._fetch_all(sql, params)
+        if hide_expired_done:
+            clauses.append(
+                "NOT (status = 'done' AND finished_at IS NOT NULL "
+                "AND finished_at < datetime('now', '-24 hours'))"
+            )
+        return " AND ".join(clauses), params
 
     async def count_fetch_queue_pending(self) -> int:
         row = await self._fetch_one(

kryten_webqueue-0.44.1/kryten_webqueue/integrations/ytpipe/__init__.py

@@ -0,0 +1,11 @@
+"""Vendored from d:\\Devel\\yt-pipe (youtube_to_mediacms.py), vendored 2026-06-09."""
+
+
+class RetryableUploadError(Exception):
+    """A MediaCMS upload failed transiently (connection broken / IncompleteRead).
+
+    Raised only from the upload stage after an on-the-spot recovery attempt
+    fails, so the fetch-queue drain can re-queue the item rather than marking it
+    permanently failed. Lives here (not in ``downloader``) so importers can catch
+    it without pulling in the heavy ``yt_dlp`` optional dependency.
+    """

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/integrations/ytpipe/downloader.py

@@ -32,6 +32,8 @@ import requests
 import yt_dlp
 from slugify import slugify
 
+from . import RetryableUploadError
+
 
 # yt-dlp needs an external JavaScript runtime to solve YouTube's JS challenges.
 # Only "deno" is enabled by default; many hosts only have Node installed, which
@@ -2865,8 +2867,11 @@ class MediaDownloaderToMediaCMS:
         # Prepare file for upload
         file_name = os.path.basename(file_path)
 
+        # Remove Content-Type header for multipart upload
+        upload_session = requests.Session()
+        upload_session.headers.update({"Authorization": f"Token {self.api_token}"})
+
         try:
-            # First, upload the file
             with open(file_path, "rb") as video_file:
                 files = {"media_file": (file_name, video_file, "video/mp4")}
                 data = {
@@ -2880,16 +2885,45 @@ class MediaDownloaderToMediaCMS:
                     f"[DEBUG] Tags: {len(existing_tags)} via API, {len(missing_tags)} in description"
                 )
 
-                # Remove Content-Type header for multipart upload
-                upload_session = requests.Session()
-                upload_session.headers.update(
-                    {"Authorization": f"Token {self.api_token}"}
-                )
-
+                # (connect, read) timeout. The read leg is generous because
+                # MediaCMS serializes a large media object (encodings, thumbnails)
+                # into the response after accepting the file.
                 response = upload_session.post(
-                    f"{self.api_url}/media/", files=files, data=data
+                    f"{self.api_url}/media/",
+                    files=files,
+                    data=data,
+                    timeout=(30, 600),
                 )
+        except (
+            requests.exceptions.ConnectionError,
+            requests.exceptions.ChunkedEncodingError,
+            requests.exceptions.Timeout,
+        ) as e:
+            # The connection dropped mid-transfer — most often an IncompleteRead
+            # while reading MediaCMS's (large) JSON response, meaning the item was
+            # already created server-side before the socket broke. Blindly
+            # re-POSTing here would duplicate it, so try to recover the created
+            # item by its "Original URL" description marker instead.
+            print(f"[DEBUG] Upload connection error (item may already exist): {e}")
+            recovered = self.check_media_already_imported(original_url, title)
+            if recovered:
+                print(
+                    "✅ Recovered upload after broken response — already in MediaCMS: "
+                    f"{recovered.get('friendly_token', 'Unknown')}"
+                )
+                if existing_tags:
+                    media_id = recovered.get("friendly_token")
+                    if media_id:
+                        self.add_tags_to_media(media_id, existing_tags)
+                return recovered
+            # Not recoverable yet (e.g. search index lag). Surface a distinct,
+            # retryable error so the caller/queue can re-attempt later; the next
+            # attempt's up-front duplicate check will skip it if it did land.
+            raise RetryableUploadError(
+                f"Failed to upload to MediaCMS (connection broken, item not recovered): {e}"
+            ) from e
 
+        try:
             if response.status_code == 201:
                 result = response.json()
 

kryten_webqueue-0.44.1/kryten_webqueue/jobs/log_capture.py

@@ -0,0 +1,73 @@
+"""Per-run capture of Python log records emitted during a background job.
+
+A single :class:`logging.Handler` is installed on the root logger. It is gated
+by a :class:`contextvars.ContextVar`, so a record is captured only when it is
+emitted from within a job run's async context — the buffer is set on the run's
+task and inherited by any child tasks / ``asyncio.to_thread`` calls it spawns.
+Concurrent web requests (and other jobs) run in their own contexts with their
+own buffer, so their logs never bleed into a run's log.
+
+The captured records are drained by :class:`~kryten_webqueue.jobs.manager.JobManager`
+when a run finishes and persisted one row per line to ``job_run_logs``.
+"""
+
+import contextvars
+import logging
+from datetime import datetime, UTC
+
+# One in-flight log buffer per run context. Each entry is
+# ``(logged_at_iso, level, logger_name, message)``.
+_current_buffer: contextvars.ContextVar[list[tuple[str, str, str, str]] | None] = (
+    contextvars.ContextVar("kryten_job_run_log_buffer", default=None)
+)
+
+_handler: "logging.Handler | None" = None
+_formatter = logging.Formatter()
+
+
+class _RunLogHandler(logging.Handler):
+    """Append formatted records to the current context's run buffer, if any."""
+
+    def emit(self, record: logging.LogRecord) -> None:
+        buffer = _current_buffer.get()
+        if buffer is None:
+            return
+        try:
+            logged_at = datetime.fromtimestamp(record.created, UTC).isoformat()
+            buffer.append(
+                (logged_at, record.levelname, record.name, record.getMessage())
+            )
+            if record.exc_info:
+                buffer.append(
+                    (
+                        logged_at,
+                        record.levelname,
+                        record.name,
+                        _formatter.formatException(record.exc_info),
+                    )
+                )
+        except Exception:  # noqa: BLE001 - logging must never raise
+            pass
+
+
+def install() -> None:
+    """Attach the capture handler to the root logger (idempotent)."""
+    global _handler
+    if _handler is not None:
+        return
+    handler = _RunLogHandler()
+    handler.setLevel(logging.DEBUG)
+    logging.getLogger().addHandler(handler)
+    _handler = handler
+
+
+def start_capture() -> list[tuple[str, str, str, str]]:
+    """Begin capturing in the current context; returns the run's log buffer."""
+    buffer: list[tuple[str, str, str, str]] = []
+    _current_buffer.set(buffer)
+    return buffer
+
+
+def stop_capture() -> None:
+    """Stop capturing in the current context."""
+    _current_buffer.set(None)

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/jobs/manager.py

@@ -17,6 +17,8 @@ import json
 import logging
 from typing import Any, Awaitable, Callable
 
+from . import log_capture
+
 logger = logging.getLogger(__name__)
 
 # A job is an async callable taking validated params + a context, returning an
@@ -150,6 +152,7 @@ class JobManager:
         self._cover_art = cover_art
         self._jobs: dict[str, dict] = {}
         self._running: dict[str, asyncio.Task] = {}
+        log_capture.install()
 
     def register(
         self,
@@ -237,6 +240,7 @@ class JobManager:
             cover_art=getattr(self, "_cover_art", None),
             job_manager=self,
         )
+        log_buffer = log_capture.start_capture()
         try:
             result = await func(params, ctx)
             detail = json.dumps(result) if result is not None else None
@@ -278,3 +282,13 @@ class JobManager:
                 )
         finally:
             self._running.pop(name, None)
+            log_capture.stop_capture()
+            if log_buffer:
+                try:
+                    await self._db.add_job_run_logs(run_id, log_buffer)
+                except Exception:  # noqa: BLE001 - log persistence is best-effort
+                    logger.debug(
+                        "Could not persist %d captured log line(s) for job '%s'",
+                        len(log_buffer),
+                        name,
+                    )

{kryten_webqueue-0.43.0 → kryten_webqueue-0.44.1}/kryten_webqueue/jobs/tasks.py

@@ -18,10 +18,16 @@ import logging
 import random
 import time
 
+from ..integrations.ytpipe import RetryableUploadError
 from .manager import JobError
 
 logger = logging.getLogger(__name__)
 
+# Max MediaCMS upload attempts per fetch-queue item before it is failed for good.
+# A broken-connection upload (IncompleteRead) re-queues to the back of the queue;
+# after this many total attempts it is recorded as ``failed``.
+MAX_FETCH_UPLOAD_ATTEMPTS = 3
+
 
 def _require(module_name: str, *, extra: str = "jobs"):
     """Import a job dependency, raising a clear error if it's not installed."""
@@ -972,6 +978,42 @@ async def fetch_queue_drain_job(params: dict, ctx) -> dict:
                 "fetch_queue_drain: id=%d interrupted — re-queued for retry", item["id"]
             )
             raise
+        except RetryableUploadError as exc:
+            # The MediaCMS upload dropped its connection (IncompleteRead). Re-queue
+            # to the back of the queue for another attempt, bounded by
+            # MAX_FETCH_UPLOAD_ATTEMPTS; the next attempt's up-front duplicate
+            # check skips it cleanly if the item actually landed server-side.
+            attempts = (item.get("attempts") or 0) + 1
+            if attempts < MAX_FETCH_UPLOAD_ATTEMPTS:
+                await ctx.db.requeue_fetch_item_for_retry(
+                    item["id"],
+                    error=f"upload retry {attempts}/{MAX_FETCH_UPLOAD_ATTEMPTS}: {exc}",
+                )
+                logger.warning(
+                    "fetch_queue_drain: id=%d (%s) upload failed (attempt %d/%d) — "
+                    "re-queued: %s",
+                    item["id"],
+                    item["url"],
+                    attempts,
+                    MAX_FETCH_UPLOAD_ATTEMPTS,
+                    exc,
+                )
+            else:
+                await ctx.db.finish_fetch_item(
+                    item["id"],
+                    status="failed",
+                    error=f"upload failed after {attempts} attempts: {exc}",
+                    bump_attempts=True,
+                )
+                failed += 1
+                logger.warning(
+                    "fetch_queue_drain: id=%d (%s) upload failed after %d attempts — "
+                    "giving up: %s",
+                    item["id"],
+                    item["url"],
+                    attempts,
+                    exc,
+                )
         except Exception as exc:  # noqa: BLE001
             logger.warning(
                 "fetch_queue_drain: id=%d (%s) failed: %s", item["id"], item["url"], exc