kryten-webqueue 0.42.0__tar.gz → 0.44.0__tar.gz

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/CHANGELOG.md

@@ -2,6 +2,82 @@
 
 ## [Unreleased]
 
+### Added
+
+- **Admin Jobs tab: full-text run logs + humanized summaries.** Every background
+  job run now captures the Python log records it emits (scoped to that run via a
+  `contextvars` filter, so concurrent jobs and web requests don't bleed in) into
+  a new uncapped `job_run_logs` table (schema **v29**, forward-only migration —
+  kept as a separate table rather than a `job_runs` column so it ports cleanly to
+  Postgres). The run-detail modal now renders the JSON summary as **humanized,
+  labelled rows** (friendly `Snake Case → Title Case` labels, `Yes/No` for
+  booleans, thousands separators) instead of a raw blob, and shows the **full log**
+  beneath it. Both the JSON summary and the full-text log are **downloadable** per
+  run (`GET /admin/jobs/runs/{id}/detail/download` and `.../log/download`) for
+  offline study.
+- **Per-job run selector.** Each job row's "Last Run" text is replaced by a
+  dropdown of that job's recent runs (by start time) and a clickable status cell
+  (`completed` / `failed` / …) sitting just before the action buttons; picking a
+  run and clicking its status opens that run's summary + log.
+- **Download queue pagination + 24h success expiry.** The download-queue list is
+  paginated at **20 items/page**, and successful (`done`) downloads finished more
+  than **24h ago are hidden** to keep the list manageable. Failures and
+  pending/running items are always shown regardless of age. This is a
+  **visibility filter only** — no rows are ever deleted, so the audit trail is
+  preserved (`GET /admin/jobs/fetch-queue` now returns
+  `{items, total, page, pages, limit}`).
+
+- **Failed MediaCMS uploads are re-queued instead of lost.** When a fetch-queue
+  item's upload fails with a broken connection (`IncompleteRead`), the drain now
+  re-queues it to the **back** of the queue for another attempt (up to
+  **3 attempts** total) rather than marking it `failed` on the first hit. The
+  retry runs *after* everything else currently pending, and the existing
+  inter-item cooldown naturally spaces it out. Only the **upload stage** is
+  retried — download/validation failures still fail immediately (they need
+  case-by-case handling). A new `fetch_queue.attempts` column (schema **v28**,
+  forward-only migration) tracks attempts; the last failure is recorded as
+  `failed` with an "after N attempts" note.
+
+### Fixed
+
+- **MediaCMS upload no longer fails hard on a broken response.** The final upload
+  POST previously had **no timeout** and no handling for a dropped connection, so
+  a truncated response (`Connection broken: IncompleteRead(...)` — MediaCMS
+  serializes a large media object into the response *after* accepting the file)
+  surfaced as a permanent `Failed to upload to MediaCMS` even though the item had
+  usually already been created server-side. The POST now uses a `(connect, read)`
+  timeout of `(30, 600)`, and on a connection-broken/`IncompleteRead`/timeout it
+  attempts to **recover the just-created item** by its "Original URL" description
+  marker (applying tags to it) rather than blindly re-POSTing and creating a
+  duplicate. If it can't be recovered yet (e.g. search-index lag) it raises a
+  distinct, retryable error — the next attempt's up-front duplicate check skips it
+  if it did land.
+
+## [0.43.0] - 2026-08-24
+
+### Added
+
+- **Fetch-queue drain cooldown (anti-bot pacing).** The **Fetch Queue — Process**
+  job now waits a randomized cooldown *between* items instead of downloading
+  back-to-back, so it stops tripping the source's bot detection (the MediaCMS
+  encoder is the real bottleneck anyway). The wait is drawn uniformly from
+  `cooldown_mean_minutes ± cooldown_jitter_minutes` (default **42 ± 8 min**).
+  New optional `fetch_queue` config block: `cooldown_mean_minutes`,
+  `cooldown_jitter_minutes`, `cooldown_enabled`. Settings are **re-read from the
+  config file before every wait**, so editing the config retunes a *running*
+  drain — no restart needed. A `cooldown` progress heartbeat exposes the
+  remaining wait to the admin UI.
+
+### Changed
+
+- **Interrupted fetch-queue downloads now resume.** On startup any item left
+  `running` by a crash/restart is reset to `pending`, and the drain auto-starts
+  if anything is pending — so a download cut off mid-flight is re-attempted
+  without an admin re-triggering it. A drain cancelled by shutdown re-queues its
+  in-flight item (rather than marking it `failed`) for the same reason. Retries
+  re-download the whole file; a leftover yt-dlp `.part` file may let yt-dlp
+  resume where it left off.
+
 ## [0.42.0] - 2026-08-24
 
 ### Added

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: kryten-webqueue
-Version: 0.42.0
+Version: 0.44.0
 Summary: Netflix/Tubi-style catalog browser and pay-to-play queue management for CyTube
 Author: grobertson
 License-Expression: MIT
@@ -122,6 +122,16 @@ sudo nginx -t && sudo systemctl reload nginx
 ### WebSocket
 - `ws://host/ws` — Real-time queue updates (auth via session cookie)
 
+### Public API (third-party clients)
+- `POST /api/public/v1/link` — Exchange a one-time device code for an API key
+- `GET /api/public/v1/current` — Now playing
+- `GET /api/public/v1/queue` — Live queue with predicted start times
+- `GET /api/public/v1/events` — Upcoming scheduled events
+
+Key-authenticated (`Authorization: Bearer <key>`) endpoints for Smart TV /
+tablet apps. Users link a device at `/link`. Full contract, JSON schemas, and a
+developer's guide: [docs/PUBLIC_API.md](docs/PUBLIC_API.md).
+
 ## Dependencies
 
 - Python ≥ 3.12

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/README.md

@@ -89,6 +89,16 @@ sudo nginx -t && sudo systemctl reload nginx
 ### WebSocket
 - `ws://host/ws` — Real-time queue updates (auth via session cookie)
 
+### Public API (third-party clients)
+- `POST /api/public/v1/link` — Exchange a one-time device code for an API key
+- `GET /api/public/v1/current` — Now playing
+- `GET /api/public/v1/queue` — Live queue with predicted start times
+- `GET /api/public/v1/events` — Upcoming scheduled events
+
+Key-authenticated (`Authorization: Bearer <key>`) endpoints for Smart TV /
+tablet apps. Users link a device at `/link`. Full contract, JSON schemas, and a
+developer's guide: [docs/PUBLIC_API.md](docs/PUBLIC_API.md).
+
 ## Dependencies
 
 - Python ≥ 3.12

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/config.example.json

@@ -30,6 +30,12 @@
     "token_cache_path": "/var/lib/kryten-webqueue/.fetchurls_tokens.bin"
   },
 
+  "fetch_queue": {
+    "cooldown_mean_minutes": 42.0,
+    "cooldown_jitter_minutes": 8.0,
+    "cooldown_enabled": true
+  },
+
   "presence_refund": {
     "enabled": true,
     "on_leave": true,

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/docs/PUBLIC_API.md

@@ -21,7 +21,7 @@ Keys are minted through a short, user-driven pairing flow so a device never
 handles the user's credentials.
 
 ```
-┌────────────┐   1. names device, gets code    ┌──────────────────┐
+┌────────────┐   1. names device, gets code     ┌──────────────────┐
 │  Browser   │ ───────────────────────────────▶│  queue.dropsugar │
 │ (logged in)│   GET /link  →  "AB2CD"          │      .co         │
 └────────────┘                                  └──────────────────┘
@@ -275,10 +275,12 @@ Errors use FastAPI's standard shape:
 ## 5. Client implementation notes
 
 - **Polling:** these endpoints return point-in-time snapshots. Poll `/current`
-  and `/queue` every ~10–15s; `/events` changes rarely (every few minutes is
-  plenty).
+  and `/queue` no more often than once per minute, it is advised that you
+  locally calculate time remaining and poll even less, if an item is in progress
+  the only thing that will change about it's API data is the time elapsed and
+  time remaining values; `/events` changes rarely (every 15m-1h is plenty).
 - **Countdowns:** compute live countdowns on-device from `remaining_sec` /
-  `estimated_start_in_sec` rather than re-polling every second.
+  `estimated_start_in_sec` rather than re-polling every minute.
 - **Cover art:** `cover_art_url` is absolute and ready to load. It may be
   `null`; show a placeholder.
 - **Key loss / revocation:** if requests start returning 401, the key was

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/kryten_webqueue/app.py

@@ -189,6 +189,26 @@ async def lifespan(app: FastAPI):
     )
     app.state.job_manager = job_manager
 
+    # Recover interrupted fetch-queue downloads. Any item left 'running' was cut
+    # off by a crash/restart (the running flag isn't durable); flip it back to
+    # 'pending' so the drain re-attempts it, then auto-start the drain if
+    # anything is pending so downloads resume without an admin re-triggering it.
+    requeued = await db.reset_running_fetch_items()
+    if requeued:
+        logger.warning(
+            "Re-queued %d interrupted fetch-queue item(s) after restart", requeued
+        )
+    if await db.count_fetch_queue_pending() > 0 and not job_manager.is_running(
+        "fetch_queue_drain"
+    ):
+        try:
+            await job_manager.run("fetch_queue_drain", triggered_by="startup-recovery")
+            logger.info("Auto-started fetch_queue_drain to resume pending downloads")
+        except Exception:  # noqa: BLE001 - best-effort; admin can re-trigger
+            logger.debug(
+                "Could not auto-start fetch_queue_drain at startup", exc_info=True
+            )
+
     # Cron-based job scheduler (persists schedules to job_schedules table)
     job_scheduler = JobScheduler(db, job_manager)
     # Seed a default schedule for the security-sensitive ban-reconcile job so it

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/kryten_webqueue/catalog/db/_catalog.py

@@ -1146,6 +1146,36 @@ class _CatalogMixin:
         """Update only a running job's detail column (used for live progress)."""
         await self._execute("UPDATE job_runs SET detail=? WHERE id=?", [detail, run_id])
 
+    async def add_job_run_logs(
+        self, run_id: int, records: list[tuple[str, str, str, str]]
+    ) -> None:
+        """Bulk-append captured log lines for a run.
+
+        ``records`` is a list of ``(logged_at_iso, level, logger, message)``
+        tuples in emission order; ``seq`` is assigned from the list index so the
+        original ordering survives even when timestamps collide.
+        """
+        if not records:
+            return
+        rows = [
+            (run_id, seq, logged_at, level, logger, message)
+            for seq, (logged_at, level, logger, message) in enumerate(records)
+        ]
+        await self._db.executemany(
+            "INSERT INTO job_run_logs (run_id, seq, logged_at, level, logger, message) "
+            "VALUES (?, ?, ?, ?, ?, ?)",
+            rows,
+        )
+        await self._db.commit()
+
+    async def get_job_run_logs(self, run_id: int) -> list[dict]:
+        return await self._fetch_all(
+            "SELECT * FROM job_run_logs WHERE run_id=? ORDER BY seq ASC", [run_id]
+        )
+
+    async def get_job_run(self, run_id: int) -> dict | None:
+        return await self._fetch_one("SELECT * FROM job_runs WHERE id=?", [run_id])
+
     async def get_job_runs(
         self, job_name: str | None = None, limit: int = 10
     ) -> list[dict]:

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/kryten_webqueue/catalog/db/_connection.py

@@ -509,6 +509,30 @@ MIGRATIONS = [
     CREATE INDEX IF NOT EXISTS idx_device_api_keys_user ON device_api_keys(username);
     CREATE INDEX IF NOT EXISTS idx_device_api_keys_hash ON device_api_keys(key_hash);
     """,
+    # v28: Retry counter for the fetch queue. A MediaCMS upload that fails with a
+    # broken connection (IncompleteRead) is re-queued to the back of the queue up
+    # to a bounded number of attempts instead of failing permanently; this tracks
+    # how many upload attempts an item has consumed.
+    """
+    ALTER TABLE fetch_queue ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0;
+    """,
+    # v29: Full-text run logs. Each background job run captures the Python log
+    # records it emits (scoped to the run via a contextvars filter) as one row
+    # per line, so the admin UI can show and download a complete log alongside
+    # the JSON summary. Kept in a separate, uncapped table (Postgres-friendly)
+    # rather than a column on job_runs.
+    """
+    CREATE TABLE IF NOT EXISTS job_run_logs (
+        id        INTEGER PRIMARY KEY AUTOINCREMENT,
+        run_id    INTEGER NOT NULL,
+        seq       INTEGER NOT NULL,
+        logged_at TIMESTAMP NOT NULL,
+        level     TEXT,
+        logger    TEXT,
+        message   TEXT NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS idx_job_run_logs_run ON job_run_logs(run_id, seq);
+    """,
 ]
 
 

kryten_webqueue-0.44.0/kryten_webqueue/catalog/db/_fetch_queue.py

@@ -0,0 +1,179 @@
+class _FetchQueueMixin:
+    """Persistent download queue for the fetch_queue_add / fetch_queue_drain jobs."""
+
+    async def enqueue_fetch(
+        self,
+        *,
+        url: str,
+        quality: str = "medium",
+        max_videos: int = 50,
+        add_to_playlist: int | None = None,
+        added_by: str | None = None,
+    ) -> int:
+        cursor = await self._db.execute(
+            """
+            INSERT INTO fetch_queue (url, quality, max_videos, add_to_playlist, added_by)
+            VALUES (?, ?, ?, ?, ?)
+            """,
+            [url, quality, max_videos, add_to_playlist, added_by],
+        )
+        await self._db.commit()
+        return cursor.lastrowid
+
+    async def claim_next_fetch_item(self) -> dict | None:
+        """Atomically claim the oldest pending item (sets status to running)."""
+        row = await self._fetch_one(
+            "SELECT * FROM fetch_queue WHERE status = 'pending' ORDER BY added_at ASC LIMIT 1"
+        )
+        if not row:
+            return None
+        cursor = await self._db.execute(
+            "UPDATE fetch_queue SET status = 'running', started_at = datetime('now')"
+            " WHERE id = ? AND status = 'pending'",
+            [row["id"]],
+        )
+        await self._db.commit()
+        if (cursor.rowcount or 0) == 0:
+            return None  # lost the claim (shouldn't happen; drain is single-flight)
+        row = dict(row)
+        row["status"] = "running"
+        return row
+
+    async def finish_fetch_item(
+        self,
+        item_id: int,
+        *,
+        status: str,
+        result_json: str | None = None,
+        error: str | None = None,
+        bump_attempts: bool = False,
+    ) -> None:
+        attempts_sql = ", attempts = attempts + 1" if bump_attempts else ""
+        await self._db.execute(
+            f"""
+            UPDATE fetch_queue
+            SET status = ?, finished_at = datetime('now'), result_json = ?, error = ?{attempts_sql}
+            WHERE id = ?
+            """,
+            [status, result_json, error, item_id],
+        )
+        await self._db.commit()
+
+    async def requeue_fetch_item(self, item_id: int) -> None:
+        """Return a single item to 'pending' (e.g. interrupted by shutdown).
+
+        Clears the run timestamps and any prior error so the drain re-attempts it
+        cleanly on the next pass.
+        """
+        await self._db.execute(
+            "UPDATE fetch_queue"
+            " SET status = 'pending', started_at = NULL, finished_at = NULL, error = NULL"
+            " WHERE id = ?",
+            [item_id],
+        )
+        await self._db.commit()
+
+    async def requeue_fetch_item_for_retry(
+        self, item_id: int, *, error: str | None = None
+    ) -> int:
+        """Re-queue an item to the *back* of the queue for another upload attempt.
+
+        Increments ``attempts``, sets ``added_at`` strictly after every existing
+        row so it sorts last (retry after everything else currently pending), and
+        clears the run timestamps. Keeps ``error`` as a breadcrumb of why it was
+        re-queued. Returns the new ``attempts`` count.
+        """
+        await self._db.execute(
+            "UPDATE fetch_queue"
+            " SET status = 'pending', attempts = attempts + 1,"
+            " added_at = (SELECT datetime(MAX(added_at), '+1 second') FROM fetch_queue),"
+            " started_at = NULL, finished_at = NULL, error = ?"
+            " WHERE id = ?",
+            [error, item_id],
+        )
+        await self._db.commit()
+        row = await self._fetch_one(
+            "SELECT attempts FROM fetch_queue WHERE id = ?", [item_id]
+        )
+        return int(row["attempts"]) if row else 0
+
+    async def reset_running_fetch_items(self) -> int:
+        """Reset any 'running' items to 'pending'; return how many were reset.
+
+        Called at startup to recover from a crash or hard kill that left the
+        in-flight item marked 'running' forever (the running flag is process
+        state, not durable). Flipping it back to 'pending' lets the drain
+        re-attempt it — a fresh download, though a yt-dlp ``.part`` file left in
+        the work dir may let yt-dlp resume where it left off.
+        """
+        cursor = await self._db.execute(
+            "UPDATE fetch_queue SET status = 'pending', started_at = NULL"
+            " WHERE status = 'running'"
+        )
+        await self._db.commit()
+        return cursor.rowcount or 0
+
+    async def get_fetch_queue(
+        self,
+        *,
+        limit: int = 100,
+        offset: int = 0,
+        status: str | None = None,
+        hide_expired_done: bool = False,
+    ) -> list[dict]:
+        sql = "SELECT * FROM fetch_queue"
+        where, params = self._fetch_queue_filter(status, hide_expired_done)
+        if where:
+            sql += " WHERE " + where
+        sql += " ORDER BY added_at DESC LIMIT ? OFFSET ?"
+        params.extend([limit, offset])
+        return await self._fetch_all(sql, params)
+
+    async def count_fetch_queue(
+        self, *, status: str | None = None, hide_expired_done: bool = False
+    ) -> int:
+        """Count rows matching the same visibility filter as ``get_fetch_queue``."""
+        sql = "SELECT COUNT(*) AS c FROM fetch_queue"
+        where, params = self._fetch_queue_filter(status, hide_expired_done)
+        if where:
+            sql += " WHERE " + where
+        row = await self._fetch_one(sql, params)
+        return int(row["c"]) if row else 0
+
+    @staticmethod
+    def _fetch_queue_filter(
+        status: str | None, hide_expired_done: bool
+    ) -> tuple[str, list]:
+        """Build the shared WHERE clause for fetch-queue list/count queries.
+
+        When ``hide_expired_done`` is set, successful downloads finished more than
+        24h ago are excluded from view (rows are retained for audit — this is a
+        visibility filter only). Failed / pending / running items are never
+        hidden regardless of age.
+        """
+        clauses: list[str] = []
+        params: list = []
+        if status:
+            clauses.append("status = ?")
+            params.append(status)
+        if hide_expired_done:
+            clauses.append(
+                "NOT (status = 'done' AND finished_at IS NOT NULL "
+                "AND finished_at < datetime('now', '-24 hours'))"
+            )
+        return " AND ".join(clauses), params
+
+    async def count_fetch_queue_pending(self) -> int:
+        row = await self._fetch_one(
+            "SELECT COUNT(*) AS c FROM fetch_queue WHERE status = 'pending'"
+        )
+        return int(row["c"]) if row else 0
+
+    async def delete_fetch_queue_item(self, item_id: int) -> bool:
+        """Remove a non-running item. Returns False if not found or currently running."""
+        cursor = await self._db.execute(
+            "DELETE FROM fetch_queue WHERE id = ? AND status != 'running'",
+            [item_id],
+        )
+        await self._db.commit()
+        return (cursor.rowcount or 0) > 0

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/kryten_webqueue/config.py

@@ -51,6 +51,26 @@ class FetchUrlsConfig(BaseModel):
     token_cache_path: str = ""  # MSAL cache file, pre-seeded out-of-band
 
 
+class FetchQueueConfig(BaseModel):
+    """Settings for the persistent fetch-queue drain job.
+
+    The drain processes queued downloads one at a time and waits a randomized
+    cooldown *between* items so it doesn't hammer the source and trip bot
+    detection (the MediaCMS encoder is the real bottleneck anyway, so a long
+    gap costs little). Each wait is drawn uniformly from
+    ``cooldown_mean_minutes ± cooldown_jitter_minutes`` (clamped at >= 0).
+
+    These are re-read from the config file before every wait, so editing the
+    config and reloading retunes a *running* drain without a restart.
+    """
+
+    # Randomized inter-item cooldown, centered on ~42 min with jitter.
+    cooldown_mean_minutes: float = 42.0
+    cooldown_jitter_minutes: float = 8.0
+    # Set false to drain back-to-back (e.g. for a one-off bulk import).
+    cooldown_enabled: bool = True
+
+
 class PresenceRefundConfig(BaseModel):
     """Settings for presence-based cancel/refund of pending paid items.
 
@@ -168,9 +188,9 @@ class Config(BaseModel):
     # Jobs (optional; jobs whose config/deps are absent fail fast at run time)
     fetch_cookies_path: str = ""  # optional yt-dlp cookies for gated sources
     fetchurls: FetchUrlsConfig = FetchUrlsConfig()
+    fetch_queue: FetchQueueConfig = FetchQueueConfig()
     emote_rehost: EmoteRehostConfig = EmoteRehostConfig()
     motd: MOTDConfig = MOTDConfig()
-
     # Presence-based cancel/refund of pending paid items
     presence_refund: PresenceRefundConfig = PresenceRefundConfig()
 
@@ -213,6 +233,23 @@ class Config(BaseModel):
         cfg._source_path = Path(path)
         return cfg
 
+    def reload_fetch_queue(self) -> "FetchQueueConfig":
+        """Re-read just the ``fetch_queue`` section from the source file.
+
+        Enables live cooldown tuning: edit the config file and the running drain
+        picks up new pacing before its next wait — no restart needed. On any
+        error (no known source path, unreadable or invalid file) falls back to
+        the section loaded at startup so the drain never breaks on a bad edit.
+        """
+        if self._source_path is None:
+            return self.fetch_queue
+        try:
+            with open(self._source_path, encoding="utf-8") as f:
+                raw = json.load(f)
+            return FetchQueueConfig(**(raw.get("fetch_queue") or {}))
+        except Exception:  # noqa: BLE001 - never let a bad edit break the drain
+            return self.fetch_queue
+
     def save(self) -> None:
         """Persist the current config back to the file it was loaded from.
 

kryten_webqueue-0.44.0/kryten_webqueue/integrations/ytpipe/__init__.py

@@ -0,0 +1,11 @@
+"""Vendored from d:\\Devel\\yt-pipe (youtube_to_mediacms.py), vendored 2026-06-09."""
+
+
+class RetryableUploadError(Exception):
+    """A MediaCMS upload failed transiently (connection broken / IncompleteRead).
+
+    Raised only from the upload stage after an on-the-spot recovery attempt
+    fails, so the fetch-queue drain can re-queue the item rather than marking it
+    permanently failed. Lives here (not in ``downloader``) so importers can catch
+    it without pulling in the heavy ``yt_dlp`` optional dependency.
+    """

{kryten_webqueue-0.42.0 → kryten_webqueue-0.44.0}/kryten_webqueue/integrations/ytpipe/downloader.py

@@ -32,6 +32,8 @@ import requests
 import yt_dlp
 from slugify import slugify
 
+from . import RetryableUploadError
+
 
 # yt-dlp needs an external JavaScript runtime to solve YouTube's JS challenges.
 # Only "deno" is enabled by default; many hosts only have Node installed, which
@@ -2865,8 +2867,11 @@ class MediaDownloaderToMediaCMS:
         # Prepare file for upload
         file_name = os.path.basename(file_path)
 
+        # Remove Content-Type header for multipart upload
+        upload_session = requests.Session()
+        upload_session.headers.update({"Authorization": f"Token {self.api_token}"})
+
         try:
-            # First, upload the file
             with open(file_path, "rb") as video_file:
                 files = {"media_file": (file_name, video_file, "video/mp4")}
                 data = {
@@ -2880,16 +2885,45 @@ class MediaDownloaderToMediaCMS:
                     f"[DEBUG] Tags: {len(existing_tags)} via API, {len(missing_tags)} in description"
                 )
 
-                # Remove Content-Type header for multipart upload
-                upload_session = requests.Session()
-                upload_session.headers.update(
-                    {"Authorization": f"Token {self.api_token}"}
-                )
-
+                # (connect, read) timeout. The read leg is generous because
+                # MediaCMS serializes a large media object (encodings, thumbnails)
+                # into the response after accepting the file.
                 response = upload_session.post(
-                    f"{self.api_url}/media/", files=files, data=data
+                    f"{self.api_url}/media/",
+                    files=files,
+                    data=data,
+                    timeout=(30, 600),
                 )
+        except (
+            requests.exceptions.ConnectionError,
+            requests.exceptions.ChunkedEncodingError,
+            requests.exceptions.Timeout,
+        ) as e:
+            # The connection dropped mid-transfer — most often an IncompleteRead
+            # while reading MediaCMS's (large) JSON response, meaning the item was
+            # already created server-side before the socket broke. Blindly
+            # re-POSTing here would duplicate it, so try to recover the created
+            # item by its "Original URL" description marker instead.
+            print(f"[DEBUG] Upload connection error (item may already exist): {e}")
+            recovered = self.check_media_already_imported(original_url, title)
+            if recovered:
+                print(
+                    "✅ Recovered upload after broken response — already in MediaCMS: "
+                    f"{recovered.get('friendly_token', 'Unknown')}"
+                )
+                if existing_tags:
+                    media_id = recovered.get("friendly_token")
+                    if media_id:
+                        self.add_tags_to_media(media_id, existing_tags)
+                return recovered
+            # Not recoverable yet (e.g. search index lag). Surface a distinct,
+            # retryable error so the caller/queue can re-attempt later; the next
+            # attempt's up-front duplicate check will skip it if it did land.
+            raise RetryableUploadError(
+                f"Failed to upload to MediaCMS (connection broken, item not recovered): {e}"
+            ) from e
 
+        try:
             if response.status_code == 201:
                 result = response.json()
 

kryten_webqueue-0.44.0/kryten_webqueue/jobs/log_capture.py

@@ -0,0 +1,73 @@
+"""Per-run capture of Python log records emitted during a background job.
+
+A single :class:`logging.Handler` is installed on the root logger. It is gated
+by a :class:`contextvars.ContextVar`, so a record is captured only when it is
+emitted from within a job run's async context — the buffer is set on the run's
+task and inherited by any child tasks / ``asyncio.to_thread`` calls it spawns.
+Concurrent web requests (and other jobs) run in their own contexts with their
+own buffer, so their logs never bleed into a run's log.
+
+The captured records are drained by :class:`~kryten_webqueue.jobs.manager.JobManager`
+when a run finishes and persisted one row per line to ``job_run_logs``.
+"""
+
+import contextvars
+import logging
+from datetime import datetime, UTC
+
+# One in-flight log buffer per run context. Each entry is
+# ``(logged_at_iso, level, logger_name, message)``.
+_current_buffer: contextvars.ContextVar[list[tuple[str, str, str, str]] | None] = (
+    contextvars.ContextVar("kryten_job_run_log_buffer", default=None)
+)
+
+_handler: "logging.Handler | None" = None
+_formatter = logging.Formatter()
+
+
+class _RunLogHandler(logging.Handler):
+    """Append formatted records to the current context's run buffer, if any."""
+
+    def emit(self, record: logging.LogRecord) -> None:
+        buffer = _current_buffer.get()
+        if buffer is None:
+            return
+        try:
+            logged_at = datetime.fromtimestamp(record.created, UTC).isoformat()
+            buffer.append(
+                (logged_at, record.levelname, record.name, record.getMessage())
+            )
+            if record.exc_info:
+                buffer.append(
+                    (
+                        logged_at,
+                        record.levelname,
+                        record.name,
+                        _formatter.formatException(record.exc_info),
+                    )
+                )
+        except Exception:  # noqa: BLE001 - logging must never raise
+            pass
+
+
+def install() -> None:
+    """Attach the capture handler to the root logger (idempotent)."""
+    global _handler
+    if _handler is not None:
+        return
+    handler = _RunLogHandler()
+    handler.setLevel(logging.DEBUG)
+    logging.getLogger().addHandler(handler)
+    _handler = handler
+
+
+def start_capture() -> list[tuple[str, str, str, str]]:
+    """Begin capturing in the current context; returns the run's log buffer."""
+    buffer: list[tuple[str, str, str, str]] = []
+    _current_buffer.set(buffer)
+    return buffer
+
+
+def stop_capture() -> None:
+    """Stop capturing in the current context."""
+    _current_buffer.set(None)