@stephane-klein/youtube-to-markdown 0.2.0 → 0.4.0

package/README.md

@@ -7,8 +7,8 @@ I use it to search through the content of popular-science videos and shows of
 all kinds. I also use it to feed that corpus into a RAG.
 
 Technically, it is fairly simple: the project relies on
-[yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch a YouTube channel's metadata
-and then to download its transcripts.
+[yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch the metadata of a YouTube
+channel or playlist and then to download its transcripts.
 
 In the final step, I use an LLM through a standard OpenAI-compatible API to
 reconstruct a complete text from the video transcripts.
@@ -46,8 +46,8 @@ $ youtube-to-markdown --help
 youtube-to-markdown <command>
 
 Commands:
-  extract-video-metadata  Fetch the channel videos from YouTube into the
-                          configuration file
+  extract-video-metadata  Fetch the videos of each feed source (channel or
+                          playlist) from YouTube into the configuration file
   download-vtt            Download the VTT transcripts of the marked videos
   generate-markdown       Turn the VTT transcripts into Markdown prose with an
                           LLM
@@ -75,7 +75,7 @@ Use `~/.bashrc` on bash.
 
 ## Getting started
 
-**1. Create `youtube_to_markdown.yaml`** with the LLM settings and your channels.
+**1. Create `youtube_to_markdown.yaml`** with the LLM settings and your sources.
 The example below uses [OpenCode Go](https://opencode.ai/en/go), an OpenAI-compatible API:
 
 ```yaml
@@ -115,15 +115,16 @@ model     mimo-v2.5                                       ok
 endpoint  https://opencode.ai/zen/go/v1/chat/completions  ok
 api key   OPENAI_API_KEY                                  ok
 session   set                                             info
+pricing   models.dev                                      ok ($0.14 in / $0.28 out / $0.0028 cached per 1M)
 llm       1.0s                                            ok (248 in / 8 out)
 
-summary: 8 ok, 0 warning, 0 error
+summary: 9 ok, 0 warning, 0 error
 ```
 
 `doctor` exits with a non-zero status as soon as one line is an `error`, so it
 can gate a CI job.
 
-**4. Fetch the channel videos** into the file:
+**4. Fetch the source videos** into the file:
 
 ```sh
 $ youtube-to-markdown extract-video-metadata
@@ -137,21 +138,22 @@ The command is idempotent: a new run only adds the missing videos.
 other commands are `markdown-stats`, `reorganize` and `doctor`; run
 `youtube-to-markdown <command> --help` for their options.
 
-## Configure the settings and the channels
+## Configure the settings and the sources
 
 The commands read `youtube_to_markdown.yaml` from the current directory; override
-it with `--config` or `YT_TO_MD_CONFIG`. It holds the settings and the channels.
+it with `--config` or `YT_TO_MD_CONFIG`. It holds the settings and the sources.
 Settings resolve in this order: command-line flags, then `YT_TO_MD_*` environment
 variables, then the settings written in the YAML file, then the built-in
 defaults. The API key stays a secret: `--api-key` or `OPENAI_API_KEY`. The
 settings are `x_opencode_session`, `model_id`, `openaiapi_endpoint`,
 `concurrency` and `ytdlp_concurrency` (plus the optional `api_key`,
-`retry_delays` and `max_output_tokens`). I only set the channel URLs, and the
-`extract-video-metadata` command fills the rest:
+`retry_delays`, `max_output_tokens` and `chunk_target_tokens`). I only set the
+source URLs, and the `extract-video-metadata` command fills the rest:
 
-`x_opencode_session` is sent as the `X-OpenCode-Session` header; it is optional
-and specific to OpenCode. `model_id` and `openaiapi_endpoint` are not: point them
-at any OpenAI-compatible provider.
+`x_opencode_session` is the base of the `X-OpenCode-Session` header; it is
+optional and specific to OpenCode. A stable per-video suffix is appended to it,
+so each video is routed as its own conversation. `model_id` and
+`openaiapi_endpoint` are not: point them at any OpenAI-compatible provider.
 
 ```yaml
 x_opencode_session: "youtube-to-markdown/0.1"
@@ -181,13 +183,35 @@ other settings, it can only be set in the YAML file — there is no flag or
 environment variable for it. `.gitignore` only ignores the default `contents/`,
 so add your own path there if you move it.
 
-`folder_slug` (optional, on a channel) stores that channel's transcripts and
+`folder_slug` (optional, on a feed entry) stores that source's transcripts and
 Markdown in a subdirectory of `contents_path`, for example `science-4-all`.
-It is a relative path, so it stays inside `contents_path`; a channel without
+It is a relative path, so it stays inside `contents_path`; a source without
 `folder_slug` keeps its files at the root of `contents_path`. The reserved name
 `_orphans` is rejected. Change it whenever you like, then run
 `youtube-to-markdown reorganize` to move the existing files.
 
+## Playlists
+
+A feed entry can point to a YouTube playlist instead of a channel:
+
+```yaml
+feed:
+  - title: Contre-histoire de la philosophie, vol. 9
+    folder_slug: contre-histoire-philo-9
+    url: https://www.youtube.com/playlist?list=OLAK5uy_lEyjyS840R2OkHNVgOt1fGjPzeBTs1xKI
+    videos: []
+```
+
+The commands detect a playlist from its URL (the `/playlist` path);
+`extract-video-metadata` uses it as is instead of resolving the channel uploads
+playlist. `title` is filled with the playlist title when it is empty, and the
+videos keep the playlist order instead of being sorted by date. Each video gets
+an `index` field (`1`, `2`, …) which becomes the filename prefix:
+`contents/contre-histoire-philo-9/1-2019-01-04_quel-xixeme-siecle-i-un-temps-nouveau.fr.vtt`.
+The `index` field is written by `extract-video-metadata`; do not edit it by
+hand. A video removed from the playlist is kept as an extra, without `index`,
+at the end of the list.
+
 ## Download the transcripts of the videos
 
 I mark the videos I want as transcripts with a `download_vtt` field:
@@ -278,6 +302,25 @@ from the server is honored when longer than the schedule. Each retry is shown in
 the phase output; after the last attempt the file is reported as an error and is
 retried on the next run.
 
+### Output budget and long transcripts
+
+The model must rewrite the whole transcript, so the command asks it for roughly
+twice the input size (`maxOutputTokens`). That budget is capped per call: the
+command reads the model's advertised output limit from the pricing catalogs and
+uses it as the ceiling; if the catalog has no entry it falls back to 32768
+tokens. `--max-output-tokens` (or `YT_TO_MD_MAX_OUTPUT_TOKENS`) overrides the
+ceiling by hand.
+
+When the projected output would exceed the ceiling — a long video on a model
+with a small output limit — the transcript is split into chunks that each fit
+the budget, generated independently, then stitched together: immediately
+repeated headings are dropped and each junction is smoothed by a small extra
+LLM call that rewrites the beginning of the next section so it flows from the
+previous one. `--chunk-target-tokens` (or `YT_TO_MD_CHUNK_TARGET_TOKENS`,
+default 8000) sets the target input size of each chunk. The generation of a
+chunk is guaranteed to stop with `stop`; otherwise the file is reported as an
+error and nothing is written. The frontmatter records how many chunks were used.
+
 ### Generated frontmatter
 
 Each generated file starts with a YAML frontmatter describing the run:
@@ -289,20 +332,37 @@ video_title: Pourquoi π est-il si fou ? Relativité 1
 generated_at: 2026-09-19T10:12:33.456Z
 llm:
   model: mimo-v2.5
+  chunks: 1
   duration_seconds: 87.1
   input_tokens: 4428
   cached_input_tokens: 0
   output_tokens: 3262
   estimated_cost_usd: 0.000927
   finish_reason: stop
+  pricing:
+    source: models.dev
+    peak: null
 ---
 ```
 
 `video_title` is the original video title in the file's language and
-`generated_at` is the UTC time of the generation. `estimated_cost_usd` is `null`
-when the model is not in the price table. Existing files are left untouched: run
-`youtube-to-markdown generate-markdown --force` to regenerate every file, which
-is also the way to add the frontmatter to files generated before it existed.
+`generated_at` is the UTC time of the generation. `pricing.source` names the
+catalog the rates came from and `pricing.peak` tells whether the peak rate
+applied (`null` when the source has no peak/off-peak pricing). Existing files
+are left untouched: run `youtube-to-markdown generate-markdown --force` to
+regenerate every file, which is also the way to add the frontmatter to files
+generated before it existed.
+
+### Pricing
+
+The estimated cost uses rates fetched at run time from
+[models.dev](https://models.dev) and
+[LiteLLM](https://github.com/BerriAI/litellm). The provider is resolved from
+`openaiapi_endpoint`; LiteLLM is preferred when it prices the model, because it
+carries DeepSeek's peak/off-peak windows, otherwise models.dev is used. Both
+catalogs are cached for one day under `$XDG_CACHE_HOME/youtube-to-markdown`
+(`~/.cache/youtube-to-markdown` by default). When a catalog or the model is
+unavailable, `estimated_cost_usd` is `null` and the run continues.
 
 ## Organize the files into folders
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stephane-klein/youtube-to-markdown",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "description": "Collect YouTube channel videos into a YAML config with yt-dlp, download transcripts and turn them into Markdown prose with an LLM.",
   "type": "module",
   "license": "MIT",
@@ -22,7 +22,8 @@
     "access": "public"
   },
   "scripts": {
-    "youtube-to-markdown": "node src/cli.js"
+    "youtube-to-markdown": "node src/cli.js",
+    "test": "node --test"
   },
   "dependencies": {
     "@ai-sdk/openai-compatible": "~3.0.51",

package/src/chunk.js

@@ -0,0 +1,68 @@
+export const CHARS_PER_TOKEN = 3.5;
+
+export function estimateTokens(text) {
+  return Math.ceil(`${text ?? ""}`.length / CHARS_PER_TOKEN);
+}
+
+export function chunkTranscript(transcript, { targetTokens }) {
+  const target = Math.max(1, Math.floor(Number(targetTokens) || 1));
+  const lines = `${transcript ?? ""}`.split("\n");
+  const chunks = [];
+  let current = [];
+  let currentTokens = 0;
+
+  for (const line of lines) {
+    const lineTokens = estimateTokens(line) + 1;
+    if (current.length > 0 && currentTokens + lineTokens > target) {
+      chunks.push(current.join("\n"));
+      current = [];
+      currentTokens = 0;
+    }
+    current.push(line);
+    currentTokens += lineTokens;
+  }
+
+  if (current.length > 0) chunks.push(current.join("\n"));
+  return chunks.length > 0 ? chunks : [""];
+}
+
+export function splitBlocks(markdown) {
+  return `${markdown ?? ""}`.split(/\n{2,}/);
+}
+
+export function firstBlock(markdown) {
+  return splitBlocks(markdown)[0]?.trim() ?? "";
+}
+
+export function lastBlock(markdown) {
+  const blocks = splitBlocks(markdown).filter((block) => block.trim());
+  return blocks[blocks.length - 1]?.trim() ?? "";
+}
+
+export function replaceFirstBlock(markdown, replacement) {
+  const blocks = splitBlocks(markdown);
+  if (blocks.length === 0) return replacement;
+  blocks[0] = replacement;
+  return blocks.join("\n\n");
+}
+
+export function dedupeAdjacentHeadings(markdown) {
+  const lines = `${markdown ?? ""}`.split("\n");
+  const out = [];
+  let lastHeading = null;
+
+  for (const line of lines) {
+    const heading = line.match(/^\s*#{1,6}\s+(.*\S)\s*$/);
+    if (heading) {
+      const key = heading[1].trim().toLowerCase();
+      if (key === lastHeading) continue;
+      lastHeading = key;
+      out.push(line);
+      continue;
+    }
+    if (line.trim() !== "") lastHeading = null;
+    out.push(line);
+  }
+
+  return out.join("\n").replace(/\n{3,}/g, "\n\n");
+}

package/src/cli.js

@@ -23,6 +23,7 @@ const SETTING_KEYS = [
   "ytdlpConcurrency",
   "retryDelays",
   "maxOutputTokens",
+  "chunkTargetTokens",
 ];
 
 const handle = (run) => async (argv) => {
@@ -57,7 +58,7 @@ const cli = yargs(hideBin(process.argv))
   .option("config", config)
   .command(
     "extract-video-metadata",
-    "Fetch the channel videos from YouTube into the configuration file",
+    "Fetch the videos of each feed source (channel or playlist) from YouTube into the configuration file",
     (yargs) =>
       yargs
         .option("force-dates", {
@@ -116,6 +117,10 @@ const cli = yargs(hideBin(process.argv))
           describe: "Override the computed output token budget",
           type: "number",
         })
+        .option("chunk-target-tokens", {
+          describe: "Target input tokens per chunk when a transcript is split (default: 8000)",
+          type: "number",
+        })
         .option("force", {
           describe: "Regenerate every Markdown file",
           type: "boolean",

package/src/config.js

@@ -14,6 +14,7 @@ const YAML_KEYS = {
   ytdlpConcurrency: "ytdlp_concurrency",
   retryDelays: "retry_delays",
   maxOutputTokens: "max_output_tokens",
+  chunkTargetTokens: "chunk_target_tokens",
 };
 
 export function settingsFromConfig(config) {
@@ -34,6 +35,7 @@ const ENV = {
   ytdlpConcurrency: "YT_TO_MD_YTDLP_CONCURRENCY",
   retryDelays: "YT_TO_MD_RETRY_DELAYS",
   maxOutputTokens: "YT_TO_MD_MAX_OUTPUT_TOKENS",
+  chunkTargetTokens: "YT_TO_MD_CHUNK_TARGET_TOKENS",
 };
 
 export function loadEnv(env = process.env) {

package/src/doctor.js

@@ -3,6 +3,7 @@ import { dirname, resolve } from "node:path";
 import { generateText } from "ai";
 import { loadConfig, settingsFromConfig } from "./config.js";
 import { createModel, describeError } from "./generate-markdown.js";
+import { loadPricing } from "./pricing.js";
 import { exists } from "./vtt.js";
 import { runYtDlp } from "./ytdlp.js";
 
@@ -86,7 +87,7 @@ async function loadConfigSafe(configPath) {
       row: {
         status: "ok",
         value: configPath,
-        note: `${channels} channel(s), ${videos} video(s)`,
+        note: `${channels} source(s), ${videos} video(s)`,
       },
     };
   } catch (error) {
@@ -141,6 +142,31 @@ async function checkLlm({ llm, settings, key, timeoutMs }) {
   }
 }
 
+async function checkPricing({ settings }) {
+  if (!settings.endpoint) {
+    return { status: "warn", value: "-", note: "endpoint not set" };
+  }
+  if (!settings.model) {
+    return { status: "warn", value: "-", note: "model not set" };
+  }
+
+  const pricing = await loadPricing({ endpoint: settings.endpoint });
+  if (!pricing) return { status: "warn", value: "-", note: "no pricing source" };
+
+  const price = pricing.priceAt(settings.model);
+  if (!price) {
+    return { status: "warn", value: pricing.provider, note: "model not priced" };
+  }
+
+  const peak = price.peak === null ? "" : price.peak ? " (peak)" : " (off-peak)";
+  const usd = (value) => (typeof value === "number" ? `$${value}` : "-");
+  return {
+    status: "ok",
+    value: `${price.source}${peak}`,
+    note: `${usd(price.input)} in / ${usd(price.output)} out / ${usd(price.cachedInput)} cached per 1M`,
+  };
+}
+
 function render(rows) {
   const labelWidth = Math.max(...rows.map((row) => row.label.length));
   const valueWidth = Math.max(...rows.map((row) => row.value.length));
@@ -192,6 +218,7 @@ export async function runDoctor({
   if (settings.session) {
     rows.push({ label: "session", value: "set", status: "info", note: null });
   }
+  rows.push({ label: "pricing", ...(await checkPricing({ settings })) });
   const llmRow = await checkLlm({ llm, settings, key, timeoutMs });
   rows.push({ label: "llm", ...llmRow });
 

package/src/extract-video-metadata.js

@@ -32,11 +32,19 @@ async function uploadsPlaylistUrl(channelUrl) {
   return `https://www.youtube.com/playlist?list=UU${channelId.slice(2)}`;
 }
 
+const isPlaylistUrl = (url) => {
+  try {
+    return new URL(url).pathname === "/playlist";
+  } catch {
+    return false;
+  }
+};
+
 async function listEntries(playlistUrl, lang) {
   const args = [
     "--flat-playlist",
     "-O",
-    '{"titre": %(title)j, "url": %(webpage_url)j}',
+    '{"titre": %(title)j, "url": %(webpage_url)j, "playlist": %(playlist_title)j, "index": %(playlist_index)j}',
     "--extractor-args",
     `youtube:lang=${lang}`,
   ];
@@ -81,8 +89,9 @@ async function fetchExactDates(urls) {
   return dates;
 }
 
-async function fetchVideos(channelUrl, knownDates) {
-  const playlistUrl = await uploadsPlaylistUrl(channelUrl);
+async function fetchVideos(sourceUrl, knownDates) {
+  const playlist = isPlaylistUrl(sourceUrl);
+  const playlistUrl = playlist ? sourceUrl : await uploadsPlaylistUrl(sourceUrl);
   const en = await listEntries(playlistUrl, "en");
   const fr = await listEntries(playlistUrl, "fr");
   const frTitles = new Map(fr.map((entry) => [entry.url, entry.titre]));
@@ -92,11 +101,15 @@ async function fetchVideos(channelUrl, knownDates) {
     .map((entry) => {
       const enTitle = entry.titre.trim();
       const frTitle = (frTitles.get(entry.url) ?? enTitle).trim();
-      const title =
-        frTitle && enTitle && frTitle !== enTitle
-          ? { fr: frTitle, en: enTitle }
-          : { fr: frTitle || enTitle };
-      return { title, url: entry.url };
+      const video = {
+        title:
+          frTitle && enTitle && frTitle !== enTitle
+            ? { fr: frTitle, en: enTitle }
+            : { fr: frTitle || enTitle },
+        url: entry.url,
+      };
+      if (playlist && Number.isInteger(entry.index)) video.index = entry.index;
+      return video;
     });
 
   const missing = videos
@@ -109,9 +122,20 @@ async function fetchVideos(channelUrl, knownDates) {
     video.date = knownDates.get(id) ?? fetched.get(id) ?? "";
   }
 
-  return videos
-    .map((video) => ({ title: video.title, date: video.date, url: video.url }))
-    .reverse();
+  const ordered = playlist
+    ? [...videos].sort((a, b) => (a.index ?? 0) - (b.index ?? 0))
+    : [...videos].reverse();
+
+  return {
+    title: playlist ? (en[0]?.playlist ?? "") : "",
+    playlist,
+    videos: ordered.map((video) => {
+      const fields = { title: video.title, date: video.date, url: video.url };
+      return video.index !== undefined
+        ? { index: video.index, ...fields }
+        : fields;
+    }),
+  };
 }
 
 function normalizeEntry(entry) {
@@ -122,7 +146,7 @@ function normalizeEntry(entry) {
   return entry;
 }
 
-const MANAGED_KEYS = new Set(["title", "date", "url", "titre"]);
+const MANAGED_KEYS = new Set(["title", "date", "url", "titre", "index"]);
 
 function userFields(entry) {
   return Object.fromEntries(
@@ -130,7 +154,7 @@ function userFields(entry) {
   );
 }
 
-function mergeVideos(existing, fetched) {
+function mergeVideos(existing, fetched, { sort = true } = {}) {
   const existingByUrl = new Map(
     existing.filter((v) => v.url).map((v) => [v.url, v]),
   );
@@ -143,10 +167,12 @@ function mergeVideos(existing, fetched) {
   const fetchedUrls = new Set(fetched.map((v) => v.url));
   const extras = existing
     .filter((v) => v.url && !fetchedUrls.has(v.url))
-    .map(normalizeEntry);
-  return [...merged, ...extras].sort((a, b) =>
-    (a.date ?? "").localeCompare(b.date ?? ""),
-  );
+    .map(normalizeEntry)
+    .map(({ index, ...video }) => video);
+  const all = [...merged, ...extras];
+  return sort
+    ? all.sort((a, b) => (a.date ?? "").localeCompare(b.date ?? ""))
+    : all;
 }
 
 export async function runExtract({
@@ -166,10 +192,11 @@ export async function runExtract({
           .map((video) => [videoId(video.url), video.date]),
       );
     console.error(`→ ${channel.url}`);
-    channel.videos = mergeVideos(
-      videos,
-      await fetchVideos(channel.url, knownDates),
-    );
+    const fetched = await fetchVideos(channel.url, knownDates);
+    if (fetched.title && !channel.title) channel.title = fetched.title;
+    channel.videos = mergeVideos(videos, fetched.videos, {
+      sort: !fetched.playlist,
+    });
   }
 
   await writeFile(path, stringify(config, { lineWidth: 0 }));

package/src/generate-markdown.js

@@ -1,9 +1,19 @@
+import { createHash } from "node:crypto";
 import { mkdir, readFile, writeFile } from "node:fs/promises";
 import { dirname, relative } from "node:path";
 import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
 import { APICallError, generateText } from "ai";
 import { Listr, ListrLogger, ProcessOutput } from "listr2";
 import { stringify } from "yaml";
+import {
+  chunkTranscript,
+  dedupeAdjacentHeadings,
+  estimateTokens,
+  firstBlock,
+  lastBlock,
+  replaceFirstBlock,
+} from "./chunk.js";
+import { loadPricing } from "./pricing.js";
 import {
   DEFAULT_DIR,
   downloadVtt,
@@ -24,19 +34,23 @@ const SYSTEM = [
   "- Output only the Markdown body.",
 ].join("\n");
 
-const WIDTH = 80;
+const SMOOTH_SYSTEM = [
+  "You improve the transition between two consecutive Markdown sections that were written independently.",
+  "- You are given the end of the previous section and the beginning of the next one.",
+  "- Rewrite ONLY the beginning of the next section so it flows naturally from the previous one.",
+  "- Keep the same language, the same facts and roughly the same length.",
+  "- Keep the Markdown formatting of the beginning.",
+  "- Output only the rewritten beginning, without any preamble or code fences.",
+].join("\n");
 
-const PRICES = {
-  "mimo-v2.5": { input: 0.14, output: 0.28, cachedInput: 0.0028 }
-};
+const WIDTH = 80;
 
 function cachedInputTokens(usage) {
   const details = usage?.inputTokenDetails ?? {};
   return details.cacheReadTokens ?? usage?.cachedInputTokens ?? 0;
 }
 
-function estimateCost(modelId, usage) {
-  const price = PRICES[modelId];
+function estimateCost(price, usage) {
   if (!price || !usage) return null;
 
   const details = usage.inputTokenDetails ?? {};
@@ -53,20 +67,26 @@ function estimateCost(modelId, usage) {
 
 const OUTPUT_FACTOR = 2;
 const OUTPUT_FLOOR = 2048;
-const OUTPUT_CEILING = 32768;
+const DEFAULT_CEILING = 32768;
+const DEFAULT_CHUNK_TARGET_TOKENS = 8000;
 
-function outputBudget(transcript, override) {
-  if (override !== undefined && override !== null && override !== "") {
-    return Number(override);
-  }
+function isSet(value) {
+  return value !== undefined && value !== null && value !== "";
+}
 
-  const inputTokens = Math.ceil(transcript.length / 3.5);
+function plannedBudget(transcript, ceiling) {
+  const inputTokens = estimateTokens(transcript);
   return Math.min(
-    OUTPUT_CEILING,
+    ceiling,
     Math.max(OUTPUT_FLOOR, Math.ceil(inputTokens * OUTPUT_FACTOR)),
   );
 }
 
+function budgetForPart(text, { maxOutputTokens, ceiling }) {
+  if (isSet(maxOutputTokens)) return Number(maxOutputTokens);
+  return plannedBudget(text, ceiling);
+}
+
 function metricsLine(modelId, usage, elapsed, cost) {
   return [
     modelId,
@@ -80,19 +100,21 @@ function metricsLine(modelId, usage, elapsed, cost) {
     .join("  ");
 }
 
-function frontmatter(job, { modelId, usage, elapsed, cost, finishReason }) {
+function frontmatter(job, { modelId, usage, elapsed, cost, finishReason, price, chunks }) {
   const data = {
     source_url: job.url,
     video_title: titleForLang(job.titles, job.lang),
     generated_at: new Date().toISOString(),
     llm: {
       model: modelId,
+      chunks: chunks ?? 1,
       duration_seconds: Number(elapsed.toFixed(1)),
       input_tokens: usage?.inputTokens ?? null,
       cached_input_tokens: usage ? cachedInputTokens(usage) : null,
       output_tokens: usage?.outputTokens ?? null,
       estimated_cost_usd: cost === null ? null : Number(cost.toFixed(6)),
       finish_reason: finishReason,
+      pricing: price ? { source: price.source, peak: price.peak } : null,
     },
   };
 
@@ -201,44 +223,83 @@ function parseRetryDelays(value) {
     .filter((item) => Number.isFinite(item) && item >= 0);
 }
 
+function convertUsage(usage) {
+  const prompt = usage?.prompt_tokens ?? 0;
+  const completion = usage?.completion_tokens ?? 0;
+  const cacheRead =
+    usage?.prompt_cache_hit_tokens ??
+    usage?.prompt_tokens_details?.cached_tokens ??
+    0;
+  const reasoning = usage?.completion_tokens_details?.reasoning_tokens ?? 0;
+  return {
+    inputTokens: {
+      total: prompt,
+      noCache: Math.max(prompt - cacheRead, 0),
+      cacheRead,
+      cacheWrite: undefined,
+    },
+    outputTokens: {
+      total: completion,
+      text: Math.max(completion - reasoning, 0),
+      reasoning,
+    },
+    raw: usage,
+  };
+}
+
 export function createModel({ modelId, endpoint, apiKey, session }) {
   const provider = createOpenAICompatible({
     name: "opencode",
     apiKey,
     baseURL: endpoint.replace(/\/chat\/completions\/?$/, ""),
     headers: session ? { "X-OpenCode-Session": session } : {},
+    convertUsage,
   });
   return provider(modelId);
 }
 
-async function toMarkdown(
-  { model, retryDelays, maxOutputTokens },
-  title,
-  transcript,
-  onRetry,
+function videoSession(base, job) {
+  const id = createHash("sha256")
+    .update(`${job.url}#${job.lang}`)
+    .digest("hex")
+    .slice(0, 16);
+  return `${base || "youtube-to-markdown"}-${id}`;
+}
+
+function cleanBody(text) {
+  return text
+    .trim()
+    .replace(/^```[a-z]*\n?/, "")
+    .replace(/\n?```$/, "")
+    .trim();
+}
+
+async function generateOnce(
+  { model, retryDelays, session, limiter },
+  { system, prompt, maxOutputTokens, onRetry },
 ) {
   const totalAttempts = retryDelays.length + 1;
 
   for (let attempt = 1;; attempt++) {
     const started = performance.now();
     try {
-      const { text, usage, finishReason } = await generateText({
-        model,
-        system: SYSTEM,
-        prompt: `Title: ${title}\n\nTranscript:\n${transcript}`,
-        temperature: 0.2,
-        maxOutputTokens: outputBudget(transcript, maxOutputTokens),
-        maxRetries: 0,
-      });
+      const call = () =>
+        generateText({
+          model,
+          system,
+          prompt,
+          temperature: 0.2,
+          reasoning: "low",
+          maxOutputTokens,
+          maxRetries: 0,
+          headers: { "X-OpenCode-Session": session },
+        });
+      const { text, usage, finishReason } = await (
+        limiter ? limiter(call) : call()
+      );
       const elapsed = (performance.now() - started) / 1000;
 
-      const body = text
-        .trim()
-        .replace(/^```[a-z]*\n?/, "")
-        .replace(/\n?```$/, "")
-        .trim();
-
-      return { body, usage, finishReason, elapsed };
+      return { text, usage, finishReason, elapsed };
     } catch (error) {
       if (!isRetryable(error) || attempt > retryDelays.length) {
         throw new Error(describeError(error, attempt), { cause: error });
@@ -250,6 +311,182 @@ async function toMarkdown(
   }
 }
 
+function chunkPrompt(title, transcript, index, total) {
+  if (total <= 1) return `Title: ${title}\n\nTranscript:\n${transcript}`;
+  return [
+    `Title: ${title}`,
+    "",
+    `This is part ${index + 1} of ${total} of the transcript.`,
+    "Output only the Markdown body for this part, without the main title.",
+    "",
+    `Transcript (part ${index + 1}/${total}):`,
+    transcript,
+  ].join("\n");
+}
+
+function smoothPrompt(previousTail, nextHead) {
+  return [
+    "End of the previous section:",
+    "",
+    previousTail,
+    "",
+    "Beginning of the next section:",
+    "",
+    nextHead,
+  ].join("\n");
+}
+
+function sumUsage(usages) {
+  const present = usages.filter(Boolean);
+  if (present.length === 0) return null;
+
+  const total = {
+    inputTokens: 0,
+    outputTokens: 0,
+    totalTokens: 0,
+    inputTokenDetails: { noCacheTokens: 0, cacheReadTokens: 0 },
+    outputTokenDetails: { textTokens: 0, reasoningTokens: 0 },
+  };
+
+  for (const usage of present) {
+    total.inputTokens += usage.inputTokens ?? 0;
+    total.outputTokens += usage.outputTokens ?? 0;
+    total.totalTokens += usage.totalTokens ?? 0;
+    total.inputTokenDetails.noCacheTokens +=
+      usage.inputTokenDetails?.noCacheTokens ?? 0;
+    total.inputTokenDetails.cacheReadTokens +=
+      usage.inputTokenDetails?.cacheReadTokens ?? 0;
+    total.outputTokenDetails.textTokens +=
+      usage.outputTokenDetails?.textTokens ?? 0;
+    total.outputTokenDetails.reasoningTokens +=
+      usage.outputTokenDetails?.reasoningTokens ?? 0;
+  }
+
+  return total;
+}
+
+function createLimiter(limit) {
+  const max = Math.max(1, Number(limit) || 1);
+  let active = 0;
+  const queue = [];
+
+  const pump = () => {
+    while (active < max && queue.length > 0) {
+      active++;
+      const { task, resolve, reject } = queue.shift();
+      Promise.resolve()
+        .then(task)
+        .then(resolve, reject)
+        .finally(() => {
+          active--;
+          pump();
+        });
+    }
+  };
+
+  return (task) =>
+    new Promise((resolve, reject) => {
+      queue.push({ task, resolve, reject });
+      pump();
+    });
+}
+
+async function toMarkdown(
+  {
+    model,
+    retryDelays,
+    maxOutputTokens,
+    ceiling,
+    chunkTargetTokens,
+    session,
+    limiter,
+  },
+  title,
+  transcript,
+  onRetry,
+  onProgress,
+) {
+  const budgetCeiling = isSet(maxOutputTokens)
+    ? Number(maxOutputTokens)
+    : ceiling;
+  const desired = Math.max(
+    OUTPUT_FLOOR,
+    Math.ceil(estimateTokens(transcript) * OUTPUT_FACTOR),
+  );
+  const parts =
+    desired > budgetCeiling
+      ? chunkTranscript(transcript, { targetTokens: chunkTargetTokens })
+      : [transcript];
+
+  const total = parts.length;
+  const bodies = new Array(total);
+  const usages = [];
+  let elapsed = 0;
+  let finishReason = "stop";
+
+  await Promise.all(
+    parts.map(async (part, index) => {
+      onProgress?.(index + 1, total);
+      const result = await generateOnce(
+        { model, retryDelays, session, limiter },
+        {
+          system: SYSTEM,
+          prompt: chunkPrompt(title, part, index, total),
+          maxOutputTokens: budgetForPart(part, { maxOutputTokens, ceiling }),
+          onRetry,
+        },
+      );
+      bodies[index] = cleanBody(result.text);
+      usages.push(result.usage);
+      elapsed += result.elapsed;
+      if (result.finishReason !== "stop" && finishReason === "stop") {
+        finishReason = result.finishReason;
+      }
+    }),
+  );
+
+  if (finishReason === "stop" && total > 1) {
+    for (let index = 0; index + 1 < total; index++) {
+      const previousTail = lastBlock(bodies[index]);
+      const nextHead = firstBlock(bodies[index + 1]);
+      if (!previousTail || !nextHead) continue;
+
+      const smoothing = await generateOnce(
+        { model, retryDelays, session, limiter },
+        {
+          system: SMOOTH_SYSTEM,
+          prompt: smoothPrompt(previousTail, nextHead),
+          maxOutputTokens: plannedBudget(nextHead, budgetCeiling),
+          onRetry,
+        },
+      );
+      usages.push(smoothing.usage);
+      elapsed += smoothing.elapsed;
+      if (smoothing.finishReason === "stop") {
+        const rewritten = cleanBody(smoothing.text);
+        if (rewritten) {
+          bodies[index + 1] = replaceFirstBlock(bodies[index + 1], rewritten);
+        }
+      }
+    }
+  }
+
+  const body = dedupeAdjacentHeadings(
+    bodies
+      .join("\n\n")
+      .replace(/\n{3,}/g, "\n\n")
+      .trim(),
+  );
+
+  return {
+    body,
+    usage: sumUsage(usages),
+    finishReason,
+    elapsed,
+    chunks: total,
+  };
+}
+
 async function forEachConcurrent(items, limit, worker) {
   let next = 0;
   await Promise.all(
@@ -267,11 +504,12 @@ export async function runGenerateMarkdown({
   model: modelId,
   endpoint,
   apiKey,
-  session,
+  sessionBase,
   concurrency = 6,
   ytdlpConcurrency = 2,
   retryDelays = "10,30,60",
   maxOutputTokens,
+  chunkTargetTokens = DEFAULT_CHUNK_TARGET_TOKENS,
   force = false,
   dir = DEFAULT_DIR,
 } = {}) {
@@ -283,8 +521,9 @@ export async function runGenerateMarkdown({
   const RETRY_DELAYS = parseRetryDelays(retryDelays);
   const llmConcurrency = Number(concurrency);
   const vttConcurrency = Number(ytdlpConcurrency);
+  const limiter = createLimiter(llmConcurrency);
 
-  const model = createModel({ modelId, endpoint, apiKey: key, session });
+  const model = createModel({ modelId, endpoint, apiKey: key, session: sessionBase });
 
   await mkdir(dir, { recursive: true });
 
@@ -335,6 +574,17 @@ export async function runGenerateMarkdown({
     else pending.push(job);
   }
 
+  const prices = pending.length > 0 ? await loadPricing({ endpoint }) : null;
+  const modelOutputLimit = prices?.outputLimitAt(modelId) ?? null;
+  const ceiling = isSet(maxOutputTokens)
+    ? Number(maxOutputTokens)
+    : isSet(modelOutputLimit)
+      ? Number(modelOutputLimit)
+      : DEFAULT_CEILING;
+  console.error(
+    `output budget: up to ${ceiling} tokens per call, chunk target ${chunkTargetTokens} tokens`,
+  );
+
   const downloadTask = {
     title: "Download transcripts",
     rendererOptions: phaseOptions,
@@ -410,20 +660,34 @@ export async function runGenerateMarkdown({
         const name = relative(dir, md);
         try {
           const transcript = vttToText(await readFile(job.vtt, "utf8"));
-          const { body, usage, finishReason, elapsed } = await toMarkdown(
-            { model, retryDelays: RETRY_DELAYS, maxOutputTokens },
+          const price = prices ? prices.priceAt(modelId) : null;
+          const { body, usage, finishReason, elapsed, chunks } = await toMarkdown(
+            {
+              model,
+              retryDelays: RETRY_DELAYS,
+              maxOutputTokens,
+              ceiling,
+              chunkTargetTokens,
+              session: videoSession(sessionBase, job),
+              limiter,
+            },
             job.title,
             transcript,
             (nextAttempt, totalAttempts, waitMs) => {
               task.output = `retrying in ${Math.round(waitMs / 1000)}s ` +
                 `(attempt ${nextAttempt}/${totalAttempts})  ${name}`;
             },
+            (part, totalParts) => {
+              if (totalParts > 1) {
+                task.output = `generating part ${part}/${totalParts}  ${name}`;
+              }
+            },
           );
 
           stats.totalSeconds += elapsed;
           stats.totalInput += usage?.inputTokens ?? 0;
           stats.totalOutput += usage?.outputTokens ?? 0;
-          const cost = estimateCost(modelId, usage);
+          const cost = estimateCost(price, usage);
           if (cost === null) stats.costKnown = false;
           else stats.totalCost += cost;
           const metrics = metricsLine(modelId, usage, elapsed, cost);
@@ -440,6 +704,8 @@ export async function runGenerateMarkdown({
             elapsed,
             cost,
             finishReason,
+            price,
+            chunks,
           });
           await mkdir(dirname(md), { recursive: true });
           await writeFile(
@@ -476,3 +742,13 @@ export async function runGenerateMarkdown({
     }`,
   );
 }
+
+export {
+  budgetForPart,
+  chunkPrompt,
+  cleanBody,
+  createLimiter,
+  plannedBudget,
+  smoothPrompt,
+  sumUsage,
+};

package/src/organize-files.js

@@ -43,7 +43,9 @@ function foldersByPrefix(feed) {
     for (const video of channel.videos ?? []) {
       const slug = slugify(primaryTitle(video));
       if (!video.date || !slug) continue;
-      const prefix = `${video.date}_${slug}`;
+      const prefix = Number.isInteger(video.index)
+        ? `${video.index}-${video.date}_${slug}`
+        : `${video.date}_${slug}`;
       const folders = targets.get(prefix) ?? new Set();
       folders.add(folder);
       targets.set(prefix, folders);

package/src/pricing.js

@@ -0,0 +1,185 @@
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { dirname, join } from "node:path";
+
+const MODELS_DEV_URL = "https://models.dev/api.json";
+const LITELLM_URL =
+  "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json";
+
+const CACHE_TTL_MS = 24 * 60 * 60 * 1000;
+const FETCH_TIMEOUT_MS = 8000;
+
+const CACHE_DIR = join(
+  process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache"),
+  "youtube-to-markdown",
+);
+
+const normalizeBase = (url) => url.replace(/\/+$/, "");
+
+async function fetchText(url) {
+  const response = await fetch(url, {
+    signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
+  });
+  if (!response.ok) throw new Error(`HTTP ${response.status} from ${url}`);
+  return response.text();
+}
+
+async function loadCachedText(url, name) {
+  const file = join(CACHE_DIR, name);
+  try {
+    const info = await stat(file);
+    if (Date.now() - info.mtimeMs < CACHE_TTL_MS) {
+      return await readFile(file, "utf8");
+    }
+  } catch {
+    // no fresh cache
+  }
+
+  try {
+    const text = await fetchText(url);
+    await mkdir(dirname(file), { recursive: true });
+    await writeFile(file, text);
+    return text;
+  } catch (error) {
+    return await readFile(file, "utf8").catch(() => {
+      throw error;
+    });
+  }
+}
+
+async function loadJson(url, name) {
+  return JSON.parse(await loadCachedText(url, name));
+}
+
+function findProvider(catalog, base) {
+  const candidates = base.endsWith("/v1")
+    ? [base, base.slice(0, -"/v1".length)]
+    : [base];
+  for (const candidate of candidates) {
+    for (const provider of Object.values(catalog)) {
+      if (provider?.api && normalizeBase(provider.api) === candidate) {
+        return provider;
+      }
+    }
+  }
+  return null;
+}
+
+function indexLitellm(catalog) {
+  const providers = new Map();
+  for (const [key, entry] of Object.entries(catalog)) {
+    const provider = entry?.litellm_provider;
+    if (!provider) continue;
+    let models = providers.get(provider);
+    if (!models) {
+      models = new Map();
+      providers.set(provider, models);
+    }
+    const name = key.includes("/") ? key.slice(key.lastIndexOf("/") + 1) : key;
+    if (!models.has(name)) models.set(name, entry);
+    if (!models.has(key)) models.set(key, entry);
+  }
+  return providers;
+}
+
+const toMinutes = (value) => {
+  const [hours, minutes] = value.split(":").map(Number);
+  return hours * 60 + minutes;
+};
+
+const isWithinRange = (minutes, start, end) =>
+  start === end
+    ? true
+    : start < end
+      ? minutes >= start && minutes < end
+      : minutes >= start || minutes < end;
+
+function isOffPeak(windows, at) {
+  const day = at.getUTCDay();
+  const weekday = day === 0 ? 7 : day;
+  const minutes = at.getUTCHours() * 60 + at.getUTCMinutes();
+  for (const window of windows ?? []) {
+    if (!window?.weekdays?.includes(weekday)) continue;
+    const ranges = Array.isArray(window.hours_utc)
+      ? window.hours_utc
+      : [window.hours_utc];
+    for (const range of ranges) {
+      if (typeof range !== "string" || !range.includes("-")) continue;
+      const [start, end] = range.split("-").map(toMinutes);
+      if (isWithinRange(minutes, start, end)) return true;
+    }
+  }
+  return false;
+}
+
+const perMillion = (perToken) =>
+  typeof perToken === "number" ? perToken * 1e6 : undefined;
+
+function litellmPrice(entry, at) {
+  const offPeak =
+    entry.off_peak_pricing && isOffPeak(entry.off_peak_pricing.windows, at);
+  const rates = offPeak ? entry.off_peak_pricing : entry;
+  return {
+    input: perMillion(rates.input_cost_per_token),
+    output: perMillion(rates.output_cost_per_token),
+    cachedInput: perMillion(
+      rates.cache_read_input_token_cost ?? rates.input_cost_per_token_cache_hit,
+    ),
+    peak: entry.off_peak_pricing ? !offPeak : null,
+    source: "litellm",
+  };
+}
+
+function modelsDevPrice(model) {
+  const cost = model?.cost;
+  if (!cost) return null;
+  return {
+    input: cost.input,
+    output: cost.output,
+    cachedInput: cost.cache_read ?? 0,
+    peak: null,
+    source: "models.dev",
+  };
+}
+
+export async function loadPricing({ endpoint } = {}) {
+  if (!endpoint) return null;
+  const base = normalizeBase(endpoint.replace(/\/chat\/completions\/?$/, ""));
+
+  let catalog;
+  try {
+    catalog = await loadJson(MODELS_DEV_URL, "models-dev.json");
+  } catch (error) {
+    console.error(`warning: pricing catalog unavailable (${error.message})`);
+    return null;
+  }
+
+  const provider = findProvider(catalog, base);
+  if (!provider) return null;
+
+  let litellm = null;
+  try {
+    litellm = indexLitellm(await loadJson(LITELLM_URL, "litellm.json")).get(
+      provider.id,
+    ) ?? null;
+  } catch {
+    litellm = null;
+  }
+
+  return {
+    provider: provider.id,
+    priceAt(modelId, at = new Date()) {
+      const entry = litellm?.get(modelId);
+      if (entry) return litellmPrice(entry, at);
+      return modelsDevPrice(provider.models?.[modelId]);
+    },
+    outputLimitAt(modelId) {
+      const entry = litellm?.get(modelId);
+      const limit =
+        entry?.max_output_tokens ??
+        provider.models?.[modelId]?.limit?.output ??
+        entry?.max_tokens;
+      return typeof limit === "number" && limit > 0 ? limit : null;
+    },
+  };
+}

package/src/vtt.js

@@ -121,8 +121,10 @@ export function videoJobs(video, dir = DEFAULT_DIR, folderSlug = null) {
   const slug = slugify(primaryTitle(video));
   if (!slug) return [];
 
+  const prefix = Number.isInteger(video.index) ? `${video.index}-` : "";
+
   return langs.map((lang) => {
-    const name = `${video.date}_${slug}.${lang}`;
+    const name = `${prefix}${video.date}_${slug}.${lang}`;
     const base = folder ? join(dir, folder, name) : join(dir, name);
     return {
       url: video.url,