@stephane-klein/youtube-to-markdown 0.1.0 → 0.3.0

package/README.md

@@ -1,8 +1,17 @@
 # youtube-feed-to-markdown
 
-I collect the videos of a few YouTube channels into `youtube_to_markdown.yaml`,
-using [yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch the metadata. It is
-published on npmjs as `@stephane-klein/youtube-to-markdown`.
+The `youtube-to-markdown` CLI generates Markdown files from YouTube video
+transcripts.
+
+I use it to search through the content of popular-science videos and shows of
+all kinds. I also use it to feed that corpus into a RAG.
+
+Technically, it is fairly simple: the project relies on
+[yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch the metadata of a YouTube
+channel or playlist and then to download its transcripts.
+
+In the final step, I use an LLM through a standard OpenAI-compatible API to
+reconstruct a complete text from the video transcripts.
 
 ## Requirements
 
@@ -28,11 +37,94 @@ Or run it without installing it, with `npx`:
 $ npx @stephane-klein/youtube-to-markdown extract-video-metadata
 ```
 
-Either way, `yt-dlp` must be on your `PATH` (see Requirements). The commands
-read `youtube_to_markdown.yaml` from the current directory unless you pass
-`--config`.
+Either way, `yt-dlp` must be on your `PATH` (see Requirements).
+
+Run `youtube-to-markdown --help` to list the available commands:
+
+```sh
+$ youtube-to-markdown --help
+youtube-to-markdown <command>
+
+Commands:
+  extract-video-metadata  Fetch the videos of each feed source (channel or
+                          playlist) from YouTube into the configuration file
+  download-vtt            Download the VTT transcripts of the marked videos
+  generate-markdown       Turn the VTT transcripts into Markdown prose with an
+                          LLM
+  markdown-stats          Report global tokens, cost and time from the Markdown
+                          frontmatter
+  reorganize              Move the transcripts and Markdown into their
+                          folder_slug folders
+  doctor                  Check the runtime dependencies and the LLM access
+  completion              Generate a shell completion script for bash or zsh
+
+Options:
+  --config   Path to the YAML configuration file
+                                  [string] [default: "youtube_to_markdown.yaml"]
+  --version  Show version number                                       [boolean]
+  --help     Show help                                                 [boolean]
+```
+
+Install the shell completion for bash or zsh (`$SHELL` selects the script):
+
+```sh
+$ youtube-to-markdown completion >> ~/.zshrc
+```
+
+Use `~/.bashrc` on bash.
+
+## Getting started
+
+**1. Create `youtube_to_markdown.yaml`** with the LLM settings and your sources.
+The example below uses [OpenCode Go](https://opencode.ai/en/go), an OpenAI-compatible API:
+
+```yaml
+x_opencode_session: "youtube-to-markdown/0.1"
+model_id: "mimo-v2.5"
+openaiapi_endpoint: "https://opencode.ai/zen/go/v1/chat/completions"
+feed:
+  - title: Science4All
+    url: https://www.youtube.com/@le_science4all
+    videos: []
+```
+
+Only the `url` is required: `extract-video-metadata` fills the titles, the dates
+and the video lists. `model_id` and `openaiapi_endpoint` are not specific to
+OpenCode Go — point them at any OpenAI-compatible provider. The other settings
+are documented below.
+
+**2. Store your API key** in `.secret.sh` and source it. Write it with your editor
+so the key never lands in your shell history:
+
+```sh
+$ $EDITOR .secret.sh
+$ cat .secret.sh
+export OPENAI_API_KEY="sk-…"
+$ source .secret.sh
+```
+
+**3. Check the setup** with `doctor` and fix every `error` (a `warn` is fine):
+
+```sh
+$ youtube-to-markdown doctor
+node      v24.21.0                                        ok
+yt-dlp    2026.08.19                                      ok
+config    youtube_to_markdown.yaml                        ok (2 channel(s), 636 video(s))
+contents  contents                                        ok
+model     mimo-v2.5                                       ok
+endpoint  https://opencode.ai/zen/go/v1/chat/completions  ok
+api key   OPENAI_API_KEY                                  ok
+session   set                                             info
+pricing   models.dev                                      ok ($0.14 in / $0.28 out / $0.0028 cached per 1M)
+llm       1.0s                                            ok (248 in / 8 out)
+
+summary: 9 ok, 0 warning, 0 error
+```
+
+`doctor` exits with a non-zero status as soon as one line is an `error`, so it
+can gate a CI job.
 
-## Run
+**4. Fetch the source videos** into the file:
 
 ```sh
 $ youtube-to-markdown extract-video-metadata
@@ -40,29 +132,29 @@ $ youtube-to-markdown extract-video-metadata
 → https://www.youtube.com/@MonsieurPhi
 ```
 
-`youtube-to-markdown` exposes five subcommands: `extract-video-metadata`,
-`download-vtt`, `generate-markdown`, `markdown-stats` and `reorganize`. Run
-`youtube-to-markdown <command> --help` for the available options.
+The command is idempotent: a new run only adds the missing videos.
 
-The configuration file is `youtube_to_markdown.yaml` (override it with
-`--config` or `YT_TO_MD_CONFIG`). Its settings resolve in this order: command-line
-flags, then `YT_TO_MD_*` environment variables, then the settings written in the
-YAML file, then the built-in defaults. The API key stays a secret: `--api-key` or
-`OPENAI_API_KEY`.
+**5. Generate the Markdown** with `download-vtt`, then `generate-markdown`. The
+other commands are `markdown-stats`, `reorganize` and `doctor`; run
+`youtube-to-markdown <command> --help` for their options.
 
-The `extract-video-metadata` command reads the channel URLs from the
-configuration file, asks yt-dlp for each channel's videos, and writes the result
-back into the file. I run it from time to time: it is idempotent, so a new run
-only adds the videos that are missing.
+## Configure the settings and the sources
 
-## Configuration
-
-`youtube_to_markdown.yaml` holds the settings and the channels. Settings are
-`x_opencode_session`, `model_id`, `openaiapi_endpoint`, `concurrency` and
-`ytdlp_concurrency` (plus the optional `api_key`, `retry_delays` and
-`max_output_tokens`). I only set the channel URLs, and the
+The commands read `youtube_to_markdown.yaml` from the current directory; override
+it with `--config` or `YT_TO_MD_CONFIG`. It holds the settings and the sources.
+Settings resolve in this order: command-line flags, then `YT_TO_MD_*` environment
+variables, then the settings written in the YAML file, then the built-in
+defaults. The API key stays a secret: `--api-key` or `OPENAI_API_KEY`. The
+settings are `x_opencode_session`, `model_id`, `openaiapi_endpoint`,
+`concurrency` and `ytdlp_concurrency` (plus the optional `api_key`,
+`retry_delays` and `max_output_tokens`). I only set the source URLs, and the
 `extract-video-metadata` command fills the rest:
 
+`x_opencode_session` is the base of the `X-OpenCode-Session` header; it is
+optional and specific to OpenCode. A stable per-video suffix is appended to it,
+so each video is routed as its own conversation. `model_id` and
+`openaiapi_endpoint` are not: point them at any OpenAI-compatible provider.
+
 ```yaml
 x_opencode_session: "youtube-to-markdown/0.1"
 model_id: "mimo-v2.5"
@@ -91,14 +183,36 @@ other settings, it can only be set in the YAML file — there is no flag or
 environment variable for it. `.gitignore` only ignores the default `contents/`,
 so add your own path there if you move it.
 
-`folder_slug` (optional, on a channel) stores that channel's transcripts and
+`folder_slug` (optional, on a feed entry) stores that source's transcripts and
 Markdown in a subdirectory of `contents_path`, for example `science-4-all`.
-It is a relative path, so it stays inside `contents_path`; a channel without
+It is a relative path, so it stays inside `contents_path`; a source without
 `folder_slug` keeps its files at the root of `contents_path`. The reserved name
 `_orphans` is rejected. Change it whenever you like, then run
 `youtube-to-markdown reorganize` to move the existing files.
 
-## Transcripts
+## Playlists
+
+A feed entry can point to a YouTube playlist instead of a channel:
+
+```yaml
+feed:
+  - title: Contre-histoire de la philosophie, vol. 9
+    folder_slug: contre-histoire-philo-9
+    url: https://www.youtube.com/playlist?list=OLAK5uy_lEyjyS840R2OkHNVgOt1fGjPzeBTs1xKI
+    videos: []
+```
+
+The commands detect a playlist from its URL (the `/playlist` path);
+`extract-video-metadata` uses it as is instead of resolving the channel uploads
+playlist. `title` is filled with the playlist title when it is empty, and the
+videos keep the playlist order instead of being sorted by date. Each video gets
+an `index` field (`1`, `2`, …) which becomes the filename prefix:
+`contents/contre-histoire-philo-9/1-2019-01-04_quel-xixeme-siecle-i-un-temps-nouveau.fr.vtt`.
+The `index` field is written by `extract-video-metadata`; do not edit it by
+hand. A video removed from the playlist is kept as an extra, without `index`,
+at the end of the list.
+
+## Download the transcripts of the videos
 
 I mark the videos I want as transcripts with a `download_vtt` field:
 
@@ -135,7 +249,7 @@ field.
 When a video has no subtitle at all, the command writes a `<…>.vtt.missing`
 marker and stops trying. I delete that marker to force a new attempt.
 
-## Markdown
+## Generate Markdown text from the transcripts
 
 I mark the videos I want as Markdown with `generate_markdown`. The transcript is
 downloaded first if it is not there yet:
@@ -188,7 +302,7 @@ from the server is honored when longer than the schedule. Each retry is shown in
 the phase output; after the last attempt the file is reported as an error and is
 retried on the next run.
 
-### Frontmatter
+### Generated frontmatter
 
 Each generated file starts with a YAML frontmatter describing the run:
 
@@ -205,16 +319,32 @@ llm:
   output_tokens: 3262
   estimated_cost_usd: 0.000927
   finish_reason: stop
+  pricing:
+    source: models.dev
+    peak: null
 ---
 ```
 
 `video_title` is the original video title in the file's language and
-`generated_at` is the UTC time of the generation. `estimated_cost_usd` is `null`
-when the model is not in the price table. Existing files are left untouched: run
-`youtube-to-markdown generate-markdown --force` to regenerate every file, which
-is also the way to add the frontmatter to files generated before it existed.
-
-## Folders
+`generated_at` is the UTC time of the generation. `pricing.source` names the
+catalog the rates came from and `pricing.peak` tells whether the peak rate
+applied (`null` when the source has no peak/off-peak pricing). Existing files
+are left untouched: run `youtube-to-markdown generate-markdown --force` to
+regenerate every file, which is also the way to add the frontmatter to files
+generated before it existed.
+
+### Pricing
+
+The estimated cost uses rates fetched at run time from
+[models.dev](https://models.dev) and
+[LiteLLM](https://github.com/BerriAI/litellm). The provider is resolved from
+`openaiapi_endpoint`; LiteLLM is preferred when it prices the model, because it
+carries DeepSeek's peak/off-peak windows, otherwise models.dev is used. Both
+catalogs are cached for one day under `$XDG_CACHE_HOME/youtube-to-markdown`
+(`~/.cache/youtube-to-markdown` by default). When a catalog or the model is
+unavailable, `estimated_cost_usd` is `null` and the run continues.
+
+## Organize the files into folders
 
 A channel may set `folder_slug` to keep its files in their own subdirectory of
 `contents_path`. When you add, change or remove that setting, `reorganize` moves
@@ -234,7 +364,7 @@ moves it under `contents_path/_orphans/`, keeping its relative path. A file whos
 target already exists, or a filename shared by two channels with different
 `folder_slug`, is reported as a conflict and left untouched.
 
-## Stats
+## Report the tokens, cost and time
 
 `markdown-stats` reads the frontmatter of every `*.md` under `contents_path`
 (recursively) and prints global totals — tokens, estimated cost, processing
@@ -280,21 +410,23 @@ folders
 Files without frontmatter (generated before the frontmatter existed) are counted
 but left out of the totals.
 
-## Development
+## Set up the development environment
 
 To work on this project, [mise](https://mise.jdx.dev/) installs both yt-dlp and
-Node.js and defines the `youtube-to-markdown` shell alias; `npm install` fetches
-the JavaScript dependencies:
+Node.js; `npm install` fetches the JavaScript dependencies:
 
 ```sh
 $ mise install
 $ npm install
 ```
 
-The shell alias maps `youtube-to-markdown` to `node src/cli.js`. It is set when
-you enter the project from an interactive bash, zsh or fish shell, so it is only
-available in interactive shells; in scripts and CI, call
-`node src/cli.js <command>` instead.
+The `enter` hook defines a `youtube-to-markdown` shell function that calls
+`node src/cli.js`, and loads the shell completion for bash and zsh; the `leave`
+hook removes both. Everything is set when you enter the project from an
+interactive bash or zsh shell, so it is only available in interactive shells; in
+scripts and CI, call `node src/cli.js <command>` instead. A function, rather than
+a shell alias, is what lets `youtube-to-markdown <TAB>` complete the subcommands:
+zsh expands aliases before dispatching completion, which would bypass `compdef`.
 
 To generate Markdown without exporting `OPENAI_API_KEY` in your shell, copy
 `.secret.sh.example` to `.secret.sh` and put your API key in it; mise sources
@@ -305,7 +437,7 @@ $ cp .secret.sh.example .secret.sh
 $ $EDITOR .secret.sh
 ```
 
-## Publish
+## Publish the package to npmjs
 
 I publish the package to npmjs from this repository with mise:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stephane-klein/youtube-to-markdown",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "description": "Collect YouTube channel videos into a YAML config with yt-dlp, download transcripts and turn them into Markdown prose with an LLM.",
   "type": "module",
   "license": "MIT",

package/src/cli.js

@@ -10,6 +10,7 @@ import { runDownloadVtt } from "./download-vtt.js";
 import { runGenerateMarkdown } from "./generate-markdown.js";
 import { runMarkdownStats } from "./markdown-stats.js";
 import { runReorganize } from "./organize-files.js";
+import { runDoctor } from "./doctor.js";
 
 const { version } = createRequire(import.meta.url)("../package.json");
 
@@ -49,13 +50,14 @@ const project = (argv) => {
   return { path: argv.config, config: yaml, feed: yaml.feed ?? [], settings, dir };
 };
 
-yargs(hideBin(process.argv))
-  .scriptName("youtube-to-markdown")
+const cli = yargs(hideBin(process.argv))
+  .scriptName("")
+  .usage("youtube-to-markdown <command>")
   .config(loadEnv())
   .option("config", config)
   .command(
     "extract-video-metadata",
-    "Fetch the channel videos from YouTube into the configuration file",
+    "Fetch the videos of each feed source (channel or playlist) from YouTube into the configuration file",
     (yargs) =>
       yargs
         .option("force-dates", {
@@ -147,9 +149,65 @@ yargs(hideBin(process.argv))
       await runReorganize({ feed, dir, dryRun: argv.dryRun });
     }),
   )
-  .demandCommand(1, "Use one of the available commands")
+  .command(
+    "doctor",
+    "Check the runtime dependencies and the LLM access",
+    (yargs) =>
+      yargs
+        .option("model", {
+          describe: "LLM model id to test (default: model_id in the configuration)",
+          type: "string",
+        })
+        .option("endpoint", {
+          describe: "OpenAI-compatible chat completions endpoint to test",
+          type: "string",
+        })
+        .option("api-key", {
+          describe: "API key to test (defaults to OPENAI_API_KEY)",
+          type: "string",
+        })
+        .option("session", {
+          describe: "Value of the X-OpenCode-Session header",
+          type: "string",
+        })
+        .option("llm", {
+          describe: "Run a minimal LLM request to test the API access",
+          type: "boolean",
+          default: true,
+        })
+        .option("timeout", {
+          describe: "Timeout in seconds for the LLM request (default: 30)",
+          type: "number",
+          default: 30,
+        }),
+    handle(async (argv) => {
+      await runDoctor({
+        configPath: argv.config,
+        overrides: {
+          model: argv.model,
+          endpoint: argv.endpoint,
+          apiKey: argv.apiKey,
+          session: argv.session,
+        },
+        llm: argv.llm,
+        timeoutMs: argv.timeout * 1000,
+      });
+    }),
+  )
+  .command(
+    "completion",
+    "Generate a shell completion script for bash or zsh",
+    () => {},
+    () => cli.showCompletionScript("youtube-to-markdown", "completion"),
+  )
+  .demandCommand(
+    1,
+    "Use one of the available commands, or run `youtube-to-markdown doctor` " +
+      "first.",
+  )
   .strictCommands()
   .recommendCommands()
   .version(version)
-  .help()
-  .parse();
+  .help();
+
+cli.parse();

package/src/doctor.js

@@ -0,0 +1,238 @@
+import { access, constants } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+import { generateText } from "ai";
+import { loadConfig, settingsFromConfig } from "./config.js";
+import { createModel, describeError } from "./generate-markdown.js";
+import { loadPricing } from "./pricing.js";
+import { exists } from "./vtt.js";
+import { runYtDlp } from "./ytdlp.js";
+
+const MIN_NODE_MAJOR = 22;
+
+function resolveApiKey(overrides, config, env = process.env) {
+  if (overrides.apiKey !== undefined) {
+    const envKey = env.YT_TO_MD_API_KEY;
+    const source = envKey !== undefined && envKey === overrides.apiKey
+      ? "YT_TO_MD_API_KEY"
+      : "--api-key";
+    return { key: overrides.apiKey, source };
+  }
+  if (config?.api_key !== undefined) {
+    return { key: config.api_key, source: "api_key (config)" };
+  }
+  if (env.OPENAI_API_KEY) {
+    return { key: env.OPENAI_API_KEY, source: "OPENAI_API_KEY" };
+  }
+  return { key: undefined, source: null };
+}
+
+async function checkContents(dir) {
+  try {
+    await access(dir, constants.W_OK);
+    return { status: "ok", note: null };
+  } catch (error) {
+    if (error.code !== "ENOENT") {
+      return { status: "error", note: "not writable" };
+    }
+  }
+  let parent = dirname(dir);
+  while (true) {
+    try {
+      await access(parent, constants.W_OK);
+      return { status: "warn", note: "not created yet" };
+    } catch (error) {
+      if (error.code !== "ENOENT") {
+        return { status: "error", note: "not writable" };
+      }
+      const next = dirname(parent);
+      if (next === parent) return { status: "error", note: "not writable" };
+      parent = next;
+    }
+  }
+}
+
+function checkNode() {
+  const version = process.versions.node;
+  const major = Number(version.split(".")[0]);
+  return major >= MIN_NODE_MAJOR
+    ? { status: "ok", value: `v${version}`, note: null }
+    : { status: "error", value: `v${version}`, note: `requires >= ${MIN_NODE_MAJOR}` };
+}
+
+async function checkYtDlp() {
+  try {
+    const { stdout } = await runYtDlp(["--version"]);
+    return { status: "ok", value: stdout.trim() || "found", note: null };
+  } catch (error) {
+    return { status: "error", value: "-", note: error.message };
+  }
+}
+
+async function loadConfigSafe(configPath) {
+  if (!(await exists(configPath))) {
+    return {
+      config: null,
+      row: { status: "warn", value: configPath, note: "not found" },
+    };
+  }
+  try {
+    const config = loadConfig(configPath);
+    const channels = config.feed?.length ?? 0;
+    const videos = (config.feed ?? []).reduce(
+      (total, channel) => total + (channel.videos?.length ?? 0),
+      0,
+    );
+    return {
+      config,
+      row: {
+        status: "ok",
+        value: configPath,
+        note: `${channels} source(s), ${videos} video(s)`,
+      },
+    };
+  } catch (error) {
+    return {
+      config: null,
+      row: { status: "error", value: configPath, note: error.message },
+    };
+  }
+}
+
+function checkEndpoint(endpoint) {
+  if (!endpoint) return { status: "error", value: "-", note: "not set" };
+  try {
+    new URL(endpoint);
+    return { status: "ok", value: endpoint, note: null };
+  } catch {
+    return { status: "error", value: endpoint, note: "not a valid URL" };
+  }
+}
+
+async function checkLlm({ llm, settings, key, timeoutMs }) {
+  if (!llm) {
+    return { status: "warn", value: "-", note: "skipped (--no-llm)" };
+  }
+  if (!settings.model) return { status: "error", value: "-", note: "model not set" };
+  if (!settings.endpoint) return { status: "error", value: "-", note: "endpoint not set" };
+  if (!key) return { status: "error", value: "-", note: "api key not set" };
+
+  try {
+    const model = createModel({
+      modelId: settings.model,
+      endpoint: settings.endpoint,
+      apiKey: key,
+      session: settings.session,
+    });
+    const started = performance.now();
+    const { usage } = await generateText({
+      model,
+      prompt: "ping",
+      maxOutputTokens: 8,
+      maxRetries: 0,
+      abortSignal: AbortSignal.timeout(timeoutMs),
+    });
+    const elapsed = (performance.now() - started) / 1000;
+    return {
+      status: "ok",
+      value: `${elapsed.toFixed(1)}s`,
+      note: `${usage?.inputTokens ?? 0} in / ${usage?.outputTokens ?? 0} out`,
+    };
+  } catch (error) {
+    return { status: "error", value: "-", note: describeError(error, 1) };
+  }
+}
+
+async function checkPricing({ settings }) {
+  if (!settings.endpoint) {
+    return { status: "warn", value: "-", note: "endpoint not set" };
+  }
+  if (!settings.model) {
+    return { status: "warn", value: "-", note: "model not set" };
+  }
+
+  const pricing = await loadPricing({ endpoint: settings.endpoint });
+  if (!pricing) return { status: "warn", value: "-", note: "no pricing source" };
+
+  const price = pricing.priceAt(settings.model);
+  if (!price) {
+    return { status: "warn", value: pricing.provider, note: "model not priced" };
+  }
+
+  const peak = price.peak === null ? "" : price.peak ? " (peak)" : " (off-peak)";
+  const usd = (value) => (typeof value === "number" ? `$${value}` : "-");
+  return {
+    status: "ok",
+    value: `${price.source}${peak}`,
+    note: `${usd(price.input)} in / ${usd(price.output)} out / ${usd(price.cachedInput)} cached per 1M`,
+  };
+}
+
+function render(rows) {
+  const labelWidth = Math.max(...rows.map((row) => row.label.length));
+  const valueWidth = Math.max(...rows.map((row) => row.value.length));
+  return rows.map((row) => {
+    const note = row.note ? ` (${row.note})` : "";
+    return `${row.label.padEnd(labelWidth)}  ${row.value.padEnd(valueWidth)}  ${row.status}${note}`;
+  });
+}
+
+export async function runDoctor({
+  configPath = "youtube_to_markdown.yaml",
+  overrides = {},
+  llm = true,
+  timeoutMs = 30000,
+} = {}) {
+  const { config, row: configRow } = await loadConfigSafe(configPath);
+
+  const settings = { ...settingsFromConfig(config ?? {}) };
+  for (const [key, value] of Object.entries(overrides)) {
+    if (value !== undefined) settings[key] = value;
+  }
+
+  const { key, source } = resolveApiKey(overrides, config);
+  const dir = resolve(dirname(configPath), config?.contents_path ?? "contents");
+
+  const rows = [
+    { label: "node", ...checkNode() },
+    { label: "yt-dlp", ...(await checkYtDlp()) },
+  ];
+  rows.push({ label: "config", ...configRow });
+  rows.push({
+    label: "contents",
+    value: config?.contents_path ?? "contents",
+    ...(await checkContents(dir)),
+  });
+  rows.push({
+    label: "model",
+    value: settings.model ?? "-",
+    status: settings.model ? "ok" : "error",
+    note: settings.model ? null : "not set",
+  });
+  rows.push({ label: "endpoint", ...checkEndpoint(settings.endpoint) });
+  rows.push({
+    label: "api key",
+    value: source ?? "-",
+    status: key ? "ok" : "error",
+    note: key ? null : "not set",
+  });
+  if (settings.session) {
+    rows.push({ label: "session", value: "set", status: "info", note: null });
+  }
+  rows.push({ label: "pricing", ...(await checkPricing({ settings })) });
+  const llmRow = await checkLlm({ llm, settings, key, timeoutMs });
+  rows.push({ label: "llm", ...llmRow });
+
+  const counts = { ok: 0, warn: 0, error: 0 };
+  for (const row of rows) {
+    if (row.status in counts) counts[row.status]++;
+  }
+
+  console.log(`youtube-to-markdown doctor`);
+  console.log(render(rows).join("\n"));
+  console.log();
+  console.log(
+    `summary: ${counts.ok} ok, ${counts.warn} warning, ${counts.error} error`,
+  );
+
+  if (counts.error > 0) process.exitCode = 1;
+}

package/src/extract-video-metadata.js

@@ -32,11 +32,19 @@ async function uploadsPlaylistUrl(channelUrl) {
   return `https://www.youtube.com/playlist?list=UU${channelId.slice(2)}`;
 }
 
+const isPlaylistUrl = (url) => {
+  try {
+    return new URL(url).pathname === "/playlist";
+  } catch {
+    return false;
+  }
+};
+
 async function listEntries(playlistUrl, lang) {
   const args = [
     "--flat-playlist",
     "-O",
-    '{"titre": %(title)j, "url": %(webpage_url)j}',
+    '{"titre": %(title)j, "url": %(webpage_url)j, "playlist": %(playlist_title)j, "index": %(playlist_index)j}',
     "--extractor-args",
     `youtube:lang=${lang}`,
   ];
@@ -81,8 +89,9 @@ async function fetchExactDates(urls) {
   return dates;
 }
 
-async function fetchVideos(channelUrl, knownDates) {
-  const playlistUrl = await uploadsPlaylistUrl(channelUrl);
+async function fetchVideos(sourceUrl, knownDates) {
+  const playlist = isPlaylistUrl(sourceUrl);
+  const playlistUrl = playlist ? sourceUrl : await uploadsPlaylistUrl(sourceUrl);
   const en = await listEntries(playlistUrl, "en");
   const fr = await listEntries(playlistUrl, "fr");
   const frTitles = new Map(fr.map((entry) => [entry.url, entry.titre]));
@@ -92,11 +101,15 @@ async function fetchVideos(channelUrl, knownDates) {
     .map((entry) => {
       const enTitle = entry.titre.trim();
       const frTitle = (frTitles.get(entry.url) ?? enTitle).trim();
-      const title =
-        frTitle && enTitle && frTitle !== enTitle
-          ? { fr: frTitle, en: enTitle }
-          : { fr: frTitle || enTitle };
-      return { title, url: entry.url };
+      const video = {
+        title:
+          frTitle && enTitle && frTitle !== enTitle
+            ? { fr: frTitle, en: enTitle }
+            : { fr: frTitle || enTitle },
+        url: entry.url,
+      };
+      if (playlist && Number.isInteger(entry.index)) video.index = entry.index;
+      return video;
     });
 
   const missing = videos
@@ -109,9 +122,20 @@ async function fetchVideos(channelUrl, knownDates) {
     video.date = knownDates.get(id) ?? fetched.get(id) ?? "";
   }
 
-  return videos
-    .map((video) => ({ title: video.title, date: video.date, url: video.url }))
-    .reverse();
+  const ordered = playlist
+    ? [...videos].sort((a, b) => (a.index ?? 0) - (b.index ?? 0))
+    : [...videos].reverse();
+
+  return {
+    title: playlist ? (en[0]?.playlist ?? "") : "",
+    playlist,
+    videos: ordered.map((video) => {
+      const fields = { title: video.title, date: video.date, url: video.url };
+      return video.index !== undefined
+        ? { index: video.index, ...fields }
+        : fields;
+    }),
+  };
 }
 
 function normalizeEntry(entry) {
@@ -122,7 +146,7 @@ function normalizeEntry(entry) {
   return entry;
 }
 
-const MANAGED_KEYS = new Set(["title", "date", "url", "titre"]);
+const MANAGED_KEYS = new Set(["title", "date", "url", "titre", "index"]);
 
 function userFields(entry) {
   return Object.fromEntries(
@@ -130,7 +154,7 @@ function userFields(entry) {
   );
 }
 
-function mergeVideos(existing, fetched) {
+function mergeVideos(existing, fetched, { sort = true } = {}) {
   const existingByUrl = new Map(
     existing.filter((v) => v.url).map((v) => [v.url, v]),
   );
@@ -143,10 +167,12 @@ function mergeVideos(existing, fetched) {
   const fetchedUrls = new Set(fetched.map((v) => v.url));
   const extras = existing
     .filter((v) => v.url && !fetchedUrls.has(v.url))
-    .map(normalizeEntry);
-  return [...merged, ...extras].sort((a, b) =>
-    (a.date ?? "").localeCompare(b.date ?? ""),
-  );
+    .map(normalizeEntry)
+    .map(({ index, ...video }) => video);
+  const all = [...merged, ...extras];
+  return sort
+    ? all.sort((a, b) => (a.date ?? "").localeCompare(b.date ?? ""))
+    : all;
 }
 
 export async function runExtract({
@@ -166,10 +192,11 @@ export async function runExtract({
           .map((video) => [videoId(video.url), video.date]),
       );
     console.error(`→ ${channel.url}`);
-    channel.videos = mergeVideos(
-      videos,
-      await fetchVideos(channel.url, knownDates),
-    );
+    const fetched = await fetchVideos(channel.url, knownDates);
+    if (fetched.title && !channel.title) channel.title = fetched.title;
+    channel.videos = mergeVideos(videos, fetched.videos, {
+      sort: !fetched.playlist,
+    });
   }
 
   await writeFile(path, stringify(config, { lineWidth: 0 }));

package/src/generate-markdown.js

@@ -1,9 +1,11 @@
+import { createHash } from "node:crypto";
 import { mkdir, readFile, writeFile } from "node:fs/promises";
 import { dirname, relative } from "node:path";
 import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
 import { APICallError, generateText } from "ai";
 import { Listr, ListrLogger, ProcessOutput } from "listr2";
 import { stringify } from "yaml";
+import { loadPricing } from "./pricing.js";
 import {
   DEFAULT_DIR,
   downloadVtt,
@@ -26,17 +28,12 @@ const SYSTEM = [
 
 const WIDTH = 80;
 
-const PRICES = {
-  "mimo-v2.5": { input: 0.14, output: 0.28, cachedInput: 0.0028 }
-};
-
 function cachedInputTokens(usage) {
   const details = usage?.inputTokenDetails ?? {};
   return details.cacheReadTokens ?? usage?.cachedInputTokens ?? 0;
 }
 
-function estimateCost(modelId, usage) {
-  const price = PRICES[modelId];
+function estimateCost(price, usage) {
   if (!price || !usage) return null;
 
   const details = usage.inputTokenDetails ?? {};
@@ -80,7 +77,7 @@ function metricsLine(modelId, usage, elapsed, cost) {
     .join("  ");
 }
 
-function frontmatter(job, { modelId, usage, elapsed, cost, finishReason }) {
+function frontmatter(job, { modelId, usage, elapsed, cost, finishReason, price }) {
   const data = {
     source_url: job.url,
     video_title: titleForLang(job.titles, job.lang),
@@ -93,6 +90,7 @@ function frontmatter(job, { modelId, usage, elapsed, cost, finishReason }) {
       output_tokens: usage?.outputTokens ?? null,
       estimated_cost_usd: cost === null ? null : Number(cost.toFixed(6)),
       finish_reason: finishReason,
+      pricing: price ? { source: price.source, peak: price.peak } : null,
     },
   };
 
@@ -183,7 +181,7 @@ function retryDelayMs(error, fallbackMs) {
   return Math.max(headerMs, fallbackMs);
 }
 
-function describeError(error, attempts) {
+export function describeError(error, attempts) {
   const suffix = ` after ${attempts} attempt(s)`;
   if (!APICallError.isInstance(error)) return `${error.message}${suffix}`;
 
@@ -201,8 +199,51 @@ function parseRetryDelays(value) {
     .filter((item) => Number.isFinite(item) && item >= 0);
 }
 
+function convertUsage(usage) {
+  const prompt = usage?.prompt_tokens ?? 0;
+  const completion = usage?.completion_tokens ?? 0;
+  const cacheRead =
+    usage?.prompt_cache_hit_tokens ??
+    usage?.prompt_tokens_details?.cached_tokens ??
+    0;
+  const reasoning = usage?.completion_tokens_details?.reasoning_tokens ?? 0;
+  return {
+    inputTokens: {
+      total: prompt,
+      noCache: Math.max(prompt - cacheRead, 0),
+      cacheRead,
+      cacheWrite: undefined,
+    },
+    outputTokens: {
+      total: completion,
+      text: Math.max(completion - reasoning, 0),
+      reasoning,
+    },
+    raw: usage,
+  };
+}
+
+export function createModel({ modelId, endpoint, apiKey, session }) {
+  const provider = createOpenAICompatible({
+    name: "opencode",
+    apiKey,
+    baseURL: endpoint.replace(/\/chat\/completions\/?$/, ""),
+    headers: session ? { "X-OpenCode-Session": session } : {},
+    convertUsage,
+  });
+  return provider(modelId);
+}
+
+function videoSession(base, job) {
+  const id = createHash("sha256")
+    .update(`${job.url}#${job.lang}`)
+    .digest("hex")
+    .slice(0, 16);
+  return `${base || "youtube-to-markdown"}-${id}`;
+}
+
 async function toMarkdown(
-  { model, retryDelays, maxOutputTokens },
+  { model, retryDelays, maxOutputTokens, session },
   title,
   transcript,
   onRetry,
@@ -217,8 +258,10 @@ async function toMarkdown(
         system: SYSTEM,
         prompt: `Title: ${title}\n\nTranscript:\n${transcript}`,
         temperature: 0.2,
+        reasoning: "low",
         maxOutputTokens: outputBudget(transcript, maxOutputTokens),
         maxRetries: 0,
+        headers: { "X-OpenCode-Session": session },
       });
       const elapsed = (performance.now() - started) / 1000;
 
@@ -257,7 +300,7 @@ export async function runGenerateMarkdown({
   model: modelId,
   endpoint,
   apiKey,
-  session,
+  sessionBase,
   concurrency = 6,
   ytdlpConcurrency = 2,
   retryDelays = "10,30,60",
@@ -274,14 +317,7 @@ export async function runGenerateMarkdown({
   const llmConcurrency = Number(concurrency);
   const vttConcurrency = Number(ytdlpConcurrency);
 
-  const provider = createOpenAICompatible({
-    name: "opencode",
-    apiKey: key,
-    baseURL: endpoint.replace(/\/chat\/completions\/?$/, ""),
-    headers: session ? { "X-OpenCode-Session": session } : {},
-  });
-
-  const model = provider(modelId);
+  const model = createModel({ modelId, endpoint, apiKey: key, session: sessionBase });
 
   await mkdir(dir, { recursive: true });
 
@@ -332,6 +368,8 @@ export async function runGenerateMarkdown({
     else pending.push(job);
   }
 
+  const prices = pending.length > 0 ? await loadPricing({ endpoint }) : null;
+
   const downloadTask = {
     title: "Download transcripts",
     rendererOptions: phaseOptions,
@@ -407,8 +445,14 @@ export async function runGenerateMarkdown({
         const name = relative(dir, md);
         try {
           const transcript = vttToText(await readFile(job.vtt, "utf8"));
+          const price = prices ? prices.priceAt(modelId) : null;
           const { body, usage, finishReason, elapsed } = await toMarkdown(
-            { model, retryDelays: RETRY_DELAYS, maxOutputTokens },
+            {
+              model,
+              retryDelays: RETRY_DELAYS,
+              maxOutputTokens,
+              session: videoSession(sessionBase, job),
+            },
             job.title,
             transcript,
             (nextAttempt, totalAttempts, waitMs) => {
@@ -420,7 +464,7 @@ export async function runGenerateMarkdown({
           stats.totalSeconds += elapsed;
           stats.totalInput += usage?.inputTokens ?? 0;
           stats.totalOutput += usage?.outputTokens ?? 0;
-          const cost = estimateCost(modelId, usage);
+          const cost = estimateCost(price, usage);
           if (cost === null) stats.costKnown = false;
           else stats.totalCost += cost;
           const metrics = metricsLine(modelId, usage, elapsed, cost);
@@ -437,6 +481,7 @@ export async function runGenerateMarkdown({
             elapsed,
             cost,
             finishReason,
+            price,
           });
           await mkdir(dirname(md), { recursive: true });
           await writeFile(

package/src/organize-files.js

@@ -43,7 +43,9 @@ function foldersByPrefix(feed) {
     for (const video of channel.videos ?? []) {
       const slug = slugify(primaryTitle(video));
       if (!video.date || !slug) continue;
-      const prefix = `${video.date}_${slug}`;
+      const prefix = Number.isInteger(video.index)
+        ? `${video.index}-${video.date}_${slug}`
+        : `${video.date}_${slug}`;
       const folders = targets.get(prefix) ?? new Set();
       folders.add(folder);
       targets.set(prefix, folders);

package/src/pricing.js

@@ -0,0 +1,177 @@
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { dirname, join } from "node:path";
+
+const MODELS_DEV_URL = "https://models.dev/api.json";
+const LITELLM_URL =
+  "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json";
+
+const CACHE_TTL_MS = 24 * 60 * 60 * 1000;
+const FETCH_TIMEOUT_MS = 8000;
+
+const CACHE_DIR = join(
+  process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache"),
+  "youtube-to-markdown",
+);
+
+const normalizeBase = (url) => url.replace(/\/+$/, "");
+
+async function fetchText(url) {
+  const response = await fetch(url, {
+    signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
+  });
+  if (!response.ok) throw new Error(`HTTP ${response.status} from ${url}`);
+  return response.text();
+}
+
+async function loadCachedText(url, name) {
+  const file = join(CACHE_DIR, name);
+  try {
+    const info = await stat(file);
+    if (Date.now() - info.mtimeMs < CACHE_TTL_MS) {
+      return await readFile(file, "utf8");
+    }
+  } catch {
+    // no fresh cache
+  }
+
+  try {
+    const text = await fetchText(url);
+    await mkdir(dirname(file), { recursive: true });
+    await writeFile(file, text);
+    return text;
+  } catch (error) {
+    return await readFile(file, "utf8").catch(() => {
+      throw error;
+    });
+  }
+}
+
+async function loadJson(url, name) {
+  return JSON.parse(await loadCachedText(url, name));
+}
+
+function findProvider(catalog, base) {
+  const candidates = base.endsWith("/v1")
+    ? [base, base.slice(0, -"/v1".length)]
+    : [base];
+  for (const candidate of candidates) {
+    for (const provider of Object.values(catalog)) {
+      if (provider?.api && normalizeBase(provider.api) === candidate) {
+        return provider;
+      }
+    }
+  }
+  return null;
+}
+
+function indexLitellm(catalog) {
+  const providers = new Map();
+  for (const [key, entry] of Object.entries(catalog)) {
+    const provider = entry?.litellm_provider;
+    if (!provider) continue;
+    let models = providers.get(provider);
+    if (!models) {
+      models = new Map();
+      providers.set(provider, models);
+    }
+    const name = key.includes("/") ? key.slice(key.lastIndexOf("/") + 1) : key;
+    if (!models.has(name)) models.set(name, entry);
+    if (!models.has(key)) models.set(key, entry);
+  }
+  return providers;
+}
+
+const toMinutes = (value) => {
+  const [hours, minutes] = value.split(":").map(Number);
+  return hours * 60 + minutes;
+};
+
+const isWithinRange = (minutes, start, end) =>
+  start === end
+    ? true
+    : start < end
+      ? minutes >= start && minutes < end
+      : minutes >= start || minutes < end;
+
+function isOffPeak(windows, at) {
+  const day = at.getUTCDay();
+  const weekday = day === 0 ? 7 : day;
+  const minutes = at.getUTCHours() * 60 + at.getUTCMinutes();
+  for (const window of windows ?? []) {
+    if (!window?.weekdays?.includes(weekday)) continue;
+    const ranges = Array.isArray(window.hours_utc)
+      ? window.hours_utc
+      : [window.hours_utc];
+    for (const range of ranges) {
+      if (typeof range !== "string" || !range.includes("-")) continue;
+      const [start, end] = range.split("-").map(toMinutes);
+      if (isWithinRange(minutes, start, end)) return true;
+    }
+  }
+  return false;
+}
+
+const perMillion = (perToken) =>
+  typeof perToken === "number" ? perToken * 1e6 : undefined;
+
+function litellmPrice(entry, at) {
+  const offPeak =
+    entry.off_peak_pricing && isOffPeak(entry.off_peak_pricing.windows, at);
+  const rates = offPeak ? entry.off_peak_pricing : entry;
+  return {
+    input: perMillion(rates.input_cost_per_token),
+    output: perMillion(rates.output_cost_per_token),
+    cachedInput: perMillion(
+      rates.cache_read_input_token_cost ?? rates.input_cost_per_token_cache_hit,
+    ),
+    peak: entry.off_peak_pricing ? !offPeak : null,
+    source: "litellm",
+  };
+}
+
+function modelsDevPrice(model) {
+  const cost = model?.cost;
+  if (!cost) return null;
+  return {
+    input: cost.input,
+    output: cost.output,
+    cachedInput: cost.cache_read ?? 0,
+    peak: null,
+    source: "models.dev",
+  };
+}
+
+export async function loadPricing({ endpoint } = {}) {
+  if (!endpoint) return null;
+  const base = normalizeBase(endpoint.replace(/\/chat\/completions\/?$/, ""));
+
+  let catalog;
+  try {
+    catalog = await loadJson(MODELS_DEV_URL, "models-dev.json");
+  } catch (error) {
+    console.error(`warning: pricing catalog unavailable (${error.message})`);
+    return null;
+  }
+
+  const provider = findProvider(catalog, base);
+  if (!provider) return null;
+
+  let litellm = null;
+  try {
+    litellm = indexLitellm(await loadJson(LITELLM_URL, "litellm.json")).get(
+      provider.id,
+    ) ?? null;
+  } catch {
+    litellm = null;
+  }
+
+  return {
+    provider: provider.id,
+    priceAt(modelId, at = new Date()) {
+      const entry = litellm?.get(modelId);
+      if (entry) return litellmPrice(entry, at);
+      return modelsDevPrice(provider.models?.[modelId]);
+    },
+  };
+}

package/src/vtt.js

@@ -121,8 +121,10 @@ export function videoJobs(video, dir = DEFAULT_DIR, folderSlug = null) {
   const slug = slugify(primaryTitle(video));
   if (!slug) return [];
 
+  const prefix = Number.isInteger(video.index) ? `${video.index}-` : "";
+
   return langs.map((lang) => {
-    const name = `${video.date}_${slug}.${lang}`;
+    const name = `${prefix}${video.date}_${slug}.${lang}`;
     const base = folder ? join(dir, folder, name) : join(dir, name);
     return {
       url: video.url,