yt-briefing 0.10.0 → 0.11.1

package/.claude/skills/yt-search/SKILL.md

@@ -1,12 +1,12 @@
 ---
 name: yt-search
 description: Search WITHIN one YouTube channel by intent — name a channel and what you're after; the engine lists that channel's uploads, ranks them against your intent (metadata only, no transcript yet), then lazily yields ONE matching video at a time with a rich summary. You keep or skip each; at the end it synthesizes a comparison from everything you kept. Channel-scoped, not whole-YouTube. Same transcript engine + proxy as /yt; lazy on purpose (no transcript bursts → no IP block). Summaries and prompts use the language chosen at onboarding.
-argument-hint: A channel (@handle or URL) and a descriptive intent, e.g. "@t3dotgg which terminal for AI coding". Optional --top N (default 10), --scan N, --since YYYY-MM-DD.
+argument-hint: A channel (@handle or URL) and a descriptive intent, e.g. "@t3dotgg which terminal for AI coding". Searches the channel's whole upload history. Optional --top N (default 10).
 ---
 
 ## How it works
 
-`src/yt-search.ts` is the whole engine: list one channel's uploads (cheap — `playlistItems`, ~1 quota unit/page; NOT `search.list`) → re-rank them against your intent on metadata only (title/description, no transcript) → **lazy** one-candidate-at-a-time yield with a rich summary → record keep/skip → on demand synthesize a comparison from everything kept. **Channel-scoped on purpose** — you choose where to look; it does NOT search all of YouTube. Matching is descriptive: the LLM filters the channel's videos by intent. This skill is a thin loop — paste the summary, collect keep/skip, show the final comparison.
+`src/yt-search.ts` is the whole engine: list the channel's **full upload history** (cheap — `playlistItems`, ~1 quota unit/page; NOT `search.list`) → re-rank it against your intent on metadata only (title/description, no transcript; ranked in chunks and merged so a 1000+ video channel never overflows the LLM) → **lazy** one-candidate-at-a-time yield with a rich summary → record keep/skip → on demand synthesize a comparison from everything kept. **Channel-scoped on purpose** — you choose where to look; it does NOT search all of YouTube. Matching is descriptive: the LLM filters the channel's videos by intent. This skill is a thin loop — paste the summary, collect keep/skip, show the final comparison.
 
 **Lazy on purpose:** one transcript per step, never a burst — a burst looks like scraping and gets the IP blocked (same reason `/yt` is lazy). Run the engine bare — stdout is a single JSON line, stderr empty; never redirect.
 
@@ -53,5 +53,5 @@ while true:
 
 - **Verbatim:** paste `summary` and `comparison` exactly as returned; never paste a raw transcript.
 - **Language:** question text + option descriptions follow `output_lang`; button labels stay `Keep` / `Skip`.
-- **Scope:** one channel per search. Listing is cheap; the cost is the lazy transcript fetches, so let the user keep/skip rather than pulling everything. A bare resume (no `--reset`) continues the same ranked queue. `--scan N` (default 50) caps how many recent uploads are considered; `--since` widens by date.
+- **Scope:** one channel per search, ranked across its **whole upload history**. Listing is cheap; the cost is the lazy transcript fetches, so let the user keep/skip rather than pulling everything. A bare resume (no `--reset`) continues the same ranked queue. `--top N` (default 10) caps how many of the top re-ranked matches get triaged — it is the only flag.
 - **Stateless triage:** independent of `/yt` (no channel profiles, no ratings written). For the recurring multi-channel briefing use `/yt`; for one known video use `/yt-transcribe`.

package/dist/yt-search.js

@@ -5,8 +5,9 @@
  *
  * You point it at a channel and describe what you're after ("which terminal does he recommend
  * for AI coding"); the engine:
- *   1. lists that channel's uploads (cheap — playlistItems, 1 quota unit/page; NOT search.list),
- *   2. re-ranks them against your intent on metadata only — title/description, NO transcript yet,
+ *   1. lists that channel's FULL upload history (cheap — playlistItems, 1 quota unit/page; NOT search.list),
+ *   2. re-ranks it against your intent on metadata only — title/description, NO transcript yet
+ *      (the history is ranked in chunks and merged, so a 1000+ video channel never overflows the LLM),
  *   3. yields ONE matching video at a time with a rich summary, lazily — never a burst of
  *      transcript fetches (a burst looks like scraping and gets the IP blocked),
  *   4. records your keep/skip decision; kept summaries accumulate in a cache,
@@ -16,7 +17,7 @@
  * the channel's videos by intent (no exact-keyword needed).
  *
  * Usage (the skill / CLI drives these; one JSON line per call):
- *   yt-search "<intent>" --channel <@handle|url> [--reset] [--top N] [--scan N] [--since DATE] [--lang auto]
+ *   yt-search "<intent>" --channel <@handle|url> [--reset] [--top N] [--lang auto]
  *   yt-search --keep        record the pending candidate, advance, yield next
  *   yt-search --skip        drop the pending candidate, advance, yield next
  *   yt-search --compare     synthesize a comparison from everything kept
@@ -45,7 +46,7 @@ mkdirSync(CACHE_DIR, { recursive: true });
 const RUNTIME = process.execPath;
 const LANG = outputLang();
 const argv = process.argv.slice(2);
-const VALUE_FLAGS = new Set(['--channel', '--top', '--scan', '--since', '--lang']);
+const VALUE_FLAGS = new Set(['--channel', '--top', '--lang']);
 const has = (f) => argv.includes(f);
 const flagVal = (f) => {
     const i = argv.indexOf(f);
@@ -73,8 +74,6 @@ const SKIP = has('--skip');
 const COMPARE = has('--compare');
 const CHANNEL = flagVal('--channel');
 const TOP = Math.max(1, parseInt(flagVal('--top') || '10', 10));
-const SCAN = Math.max(1, parseInt(flagVal('--scan') || '50', 10)); // recent uploads to consider when no --since
-const SINCE = flagVal('--since');
 const LANGTRACK = flagVal('--lang') || 'auto';
 function emit(obj) {
     process.stdout.write(JSON.stringify(obj));
@@ -116,8 +115,12 @@ function parseJsonArray(out) {
         return null;
     }
 }
-/** Re-rank a channel's videos against the intent (metadata only — no transcript). */
-async function rerank(intent, items) {
+// The whole channel history is ranked, so the pool can be large (a years-old channel is 1000+
+// uploads). One LLM prompt would overflow context, so the pool is ranked in chunks and merged.
+const RERANK_CHUNK = 200;
+/** Re-rank ONE chunk against the intent (metadata only). Returns [] on a parse/LLM failure so a
+ *  bad chunk is dropped rather than flooding the queue with unranked videos. */
+async function rerankBatch(intent, items) {
     if (items.length === 0)
         return [];
     const compact = items.map(h => ({ id: h.videoId, title: h.title, published: h.publishedAt, desc: (h.description || '').slice(0, 280) }));
@@ -135,7 +138,7 @@ Set keep=false for anything not relevant to the intent.`;
         const out = await chat(prompt, { system: 'You output ONLY a raw JSON array as instructed.', temperature: 0 });
         const arr = parseJsonArray(out);
         if (!arr)
-            return items; // fall back: keep all, original (newest-first) order
+            return [];
         const byId = new Map(items.map(h => [h.videoId, h]));
         const ranked = [];
         for (const r of arr) {
@@ -145,11 +148,26 @@ Set keep=false for anything not relevant to the intent.`;
             if (h)
                 ranked.push({ ...h, score: typeof r.score === 'number' ? r.score : undefined, reason: r.reason });
         }
-        return ranked.length ? ranked : items;
+        return ranked;
     }
     catch {
-        return items;
+        return [];
+    }
+}
+/** Re-rank the channel's full history against the intent: chunk the pool, rank each chunk, merge
+ *  the keepers globally by score (highest first), dedup by id. */
+async function rerank(intent, items) {
+    if (items.length === 0)
+        return [];
+    const merged = [];
+    for (let i = 0; i < items.length; i += RERANK_CHUNK) {
+        merged.push(...await rerankBatch(intent, items.slice(i, i + RERANK_CHUNK)));
     }
+    const byId = new Map();
+    for (const c of merged)
+        if (!byId.has(c.videoId))
+            byId.set(c.videoId, c);
+    return [...byId.values()].sort((a, b) => (b.score ?? -1) - (a.score ?? -1));
 }
 /** Rich, standalone summary for one candidate — the triage artifact + compare input. */
 async function megaSummary(c, transcript, intent) {
@@ -278,7 +296,7 @@ async function main() {
         emit({ status: 'error', error: `Could not read a channel handle from "${CHANNEL}" — use @name or the channel URL.` });
     let videos;
     try {
-        videos = await fetchChannelVideos(handle, { since: SINCE, limit: SINCE ? null : SCAN, enrich: false });
+        videos = await fetchChannelVideos(handle, { limit: null, enrich: false }); // full channel history
     }
     catch (e) {
         emit({ status: 'error', error: e.message });

package/dist/yt-transcript.js

@@ -91,6 +91,7 @@ const ytdlpArgs = [
     '--sub-langs', subLangs,
     '--sub-format', 'vtt',
     '--ignore-errors', // continue if one language track fails (e.g. 429 on pl for en-only video)
+    '--extractor-retries', '3', // yt-dlp's own retry on the extractor-level bot-check / 429
     '--no-warnings',
     '--quiet',
     '-o', outTemplate,
@@ -98,7 +99,26 @@ const ytdlpArgs = [
 if (proxyUrl)
     ytdlpArgs.push('--proxy', proxyUrl);
 ytdlpArgs.push(`https://www.youtube.com/watch?v=${videoId}`);
-const result = spawnSync(YT_DLP, ytdlpArgs, { encoding: 'utf8', timeout: 60_000 });
+// The first request through a cold WARP/shared egress IP intermittently trips YouTube's
+// "Sign in to confirm you're not a bot" check; the very next request seconds later succeeds
+// (the extractor session warms up). Without a retry that transient flake exits 2, which the
+// sweep treats as fatal (emit → exit) — so a cold `/yt` needed two calls to get going. Retry
+// ONLY on rate-limit-with-no-subtitles: a genuine IP block still exhausts the attempts and
+// exits 2, and real tooling errors / captionless videos fall straight through untouched.
+const sleepSync = (ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
+let result;
+for (let attempt = 1; attempt <= 3; attempt++) {
+    result = spawnSync(YT_DLP, ytdlpArgs, { encoding: 'utf8', timeout: 60_000 });
+    if (result.error)
+        break; // spawn / ENOENT — retrying won't help
+    const stderr = (result.stderr ?? '').toLowerCase();
+    const limited = stderr.includes('sign in') || stderr.includes('429') || stderr.includes('too many') || stderr.includes('captcha');
+    const gotSubs = existsSync(tmpDir) && readdirSync(tmpDir).some(f => f.endsWith('.vtt'));
+    if (gotSubs || !limited)
+        break; // success, or a non-rate-limit outcome → stop
+    if (attempt < 3)
+        sleepSync(attempt * 2000); // 2s, then 4s backoff between cold retries
+}
 const cleanup = () => { try {
     rmSync(tmpDir, { recursive: true });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "yt-briefing",
-  "version": "0.10.0",
+  "version": "0.11.1",
   "description": "A self-learning YouTube briefing engine: it sweeps the channels you follow, filters noise in two stages (title, then transcript), summarizes the rest in your language, and adapts to your ratings — one video at a time.",
   "type": "module",
   "bin": {