@stephane-klein/youtube-to-markdown 0.1.0 → 0.2.0

package/README.md

@@ -1,8 +1,17 @@
 # youtube-feed-to-markdown
 
-I collect the videos of a few YouTube channels into `youtube_to_markdown.yaml`,
-using [yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch the metadata. It is
-published on npmjs as `@stephane-klein/youtube-to-markdown`.
+The `youtube-to-markdown` CLI generates Markdown files from YouTube video
+transcripts.
+
+I use it to search through the content of popular-science videos and shows of
+all kinds. I also use it to feed that corpus into a RAG.
+
+Technically, it is fairly simple: the project relies on
+[yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch a YouTube channel's metadata
+and then to download its transcripts.
+
+In the final step, I use an LLM through a standard OpenAI-compatible API to
+reconstruct a complete text from the video transcripts.
 
 ## Requirements
 
@@ -28,11 +37,93 @@ Or run it without installing it, with `npx`:
 $ npx @stephane-klein/youtube-to-markdown extract-video-metadata
 ```
 
-Either way, `yt-dlp` must be on your `PATH` (see Requirements). The commands
-read `youtube_to_markdown.yaml` from the current directory unless you pass
-`--config`.
+Either way, `yt-dlp` must be on your `PATH` (see Requirements).
+
+Run `youtube-to-markdown --help` to list the available commands:
+
+```sh
+$ youtube-to-markdown --help
+youtube-to-markdown <command>
+
+Commands:
+  extract-video-metadata  Fetch the channel videos from YouTube into the
+                          configuration file
+  download-vtt            Download the VTT transcripts of the marked videos
+  generate-markdown       Turn the VTT transcripts into Markdown prose with an
+                          LLM
+  markdown-stats          Report global tokens, cost and time from the Markdown
+                          frontmatter
+  reorganize              Move the transcripts and Markdown into their
+                          folder_slug folders
+  doctor                  Check the runtime dependencies and the LLM access
+  completion              Generate a shell completion script for bash or zsh
+
+Options:
+  --config   Path to the YAML configuration file
+                                  [string] [default: "youtube_to_markdown.yaml"]
+  --version  Show version number                                       [boolean]
+  --help     Show help                                                 [boolean]
+```
 
-## Run
+Install the shell completion for bash or zsh (`$SHELL` selects the script):
+
+```sh
+$ youtube-to-markdown completion >> ~/.zshrc
+```
+
+Use `~/.bashrc` on bash.
+
+## Getting started
+
+**1. Create `youtube_to_markdown.yaml`** with the LLM settings and your channels.
+The example below uses [OpenCode Go](https://opencode.ai/en/go), an OpenAI-compatible API:
+
+```yaml
+x_opencode_session: "youtube-to-markdown/0.1"
+model_id: "mimo-v2.5"
+openaiapi_endpoint: "https://opencode.ai/zen/go/v1/chat/completions"
+feed:
+  - title: Science4All
+    url: https://www.youtube.com/@le_science4all
+    videos: []
+```
+
+Only the `url` is required: `extract-video-metadata` fills the titles, the dates
+and the video lists. `model_id` and `openaiapi_endpoint` are not specific to
+OpenCode Go — point them at any OpenAI-compatible provider. The other settings
+are documented below.
+
+**2. Store your API key** in `.secret.sh` and source it. Write it with your editor
+so the key never lands in your shell history:
+
+```sh
+$ $EDITOR .secret.sh
+$ cat .secret.sh
+export OPENAI_API_KEY="sk-…"
+$ source .secret.sh
+```
+
+**3. Check the setup** with `doctor` and fix every `error` (a `warn` is fine):
+
+```sh
+$ youtube-to-markdown doctor
+node      v24.21.0                                        ok
+yt-dlp    2026.08.19                                      ok
+config    youtube_to_markdown.yaml                        ok (2 channel(s), 636 video(s))
+contents  contents                                        ok
+model     mimo-v2.5                                       ok
+endpoint  https://opencode.ai/zen/go/v1/chat/completions  ok
+api key   OPENAI_API_KEY                                  ok
+session   set                                             info
+llm       1.0s                                            ok (248 in / 8 out)
+
+summary: 8 ok, 0 warning, 0 error
+```
+
+`doctor` exits with a non-zero status as soon as one line is an `error`, so it
+can gate a CI job.
+
+**4. Fetch the channel videos** into the file:
 
 ```sh
 $ youtube-to-markdown extract-video-metadata
@@ -40,29 +131,28 @@ $ youtube-to-markdown extract-video-metadata
 → https://www.youtube.com/@MonsieurPhi
 ```
 
-`youtube-to-markdown` exposes five subcommands: `extract-video-metadata`,
-`download-vtt`, `generate-markdown`, `markdown-stats` and `reorganize`. Run
-`youtube-to-markdown <command> --help` for the available options.
+The command is idempotent: a new run only adds the missing videos.
 
-The configuration file is `youtube_to_markdown.yaml` (override it with
-`--config` or `YT_TO_MD_CONFIG`). Its settings resolve in this order: command-line
-flags, then `YT_TO_MD_*` environment variables, then the settings written in the
-YAML file, then the built-in defaults. The API key stays a secret: `--api-key` or
-`OPENAI_API_KEY`.
+**5. Generate the Markdown** with `download-vtt`, then `generate-markdown`. The
+other commands are `markdown-stats`, `reorganize` and `doctor`; run
+`youtube-to-markdown <command> --help` for their options.
 
-The `extract-video-metadata` command reads the channel URLs from the
-configuration file, asks yt-dlp for each channel's videos, and writes the result
-back into the file. I run it from time to time: it is idempotent, so a new run
-only adds the videos that are missing.
+## Configure the settings and the channels
 
-## Configuration
-
-`youtube_to_markdown.yaml` holds the settings and the channels. Settings are
-`x_opencode_session`, `model_id`, `openaiapi_endpoint`, `concurrency` and
-`ytdlp_concurrency` (plus the optional `api_key`, `retry_delays` and
-`max_output_tokens`). I only set the channel URLs, and the
+The commands read `youtube_to_markdown.yaml` from the current directory; override
+it with `--config` or `YT_TO_MD_CONFIG`. It holds the settings and the channels.
+Settings resolve in this order: command-line flags, then `YT_TO_MD_*` environment
+variables, then the settings written in the YAML file, then the built-in
+defaults. The API key stays a secret: `--api-key` or `OPENAI_API_KEY`. The
+settings are `x_opencode_session`, `model_id`, `openaiapi_endpoint`,
+`concurrency` and `ytdlp_concurrency` (plus the optional `api_key`,
+`retry_delays` and `max_output_tokens`). I only set the channel URLs, and the
 `extract-video-metadata` command fills the rest:
 
+`x_opencode_session` is sent as the `X-OpenCode-Session` header; it is optional
+and specific to OpenCode. `model_id` and `openaiapi_endpoint` are not: point them
+at any OpenAI-compatible provider.
+
 ```yaml
 x_opencode_session: "youtube-to-markdown/0.1"
 model_id: "mimo-v2.5"
@@ -98,7 +188,7 @@ It is a relative path, so it stays inside `contents_path`; a channel without
 `_orphans` is rejected. Change it whenever you like, then run
 `youtube-to-markdown reorganize` to move the existing files.
 
-## Transcripts
+## Download the transcripts of the videos
 
 I mark the videos I want as transcripts with a `download_vtt` field:
 
@@ -135,7 +225,7 @@ field.
 When a video has no subtitle at all, the command writes a `<…>.vtt.missing`
 marker and stops trying. I delete that marker to force a new attempt.
 
-## Markdown
+## Generate Markdown text from the transcripts
 
 I mark the videos I want as Markdown with `generate_markdown`. The transcript is
 downloaded first if it is not there yet:
@@ -188,7 +278,7 @@ from the server is honored when longer than the schedule. Each retry is shown in
 the phase output; after the last attempt the file is reported as an error and is
 retried on the next run.
 
-### Frontmatter
+### Generated frontmatter
 
 Each generated file starts with a YAML frontmatter describing the run:
 
@@ -214,7 +304,7 @@ when the model is not in the price table. Existing files are left untouched: run
 `youtube-to-markdown generate-markdown --force` to regenerate every file, which
 is also the way to add the frontmatter to files generated before it existed.
 
-## Folders
+## Organize the files into folders
 
 A channel may set `folder_slug` to keep its files in their own subdirectory of
 `contents_path`. When you add, change or remove that setting, `reorganize` moves
@@ -234,7 +324,7 @@ moves it under `contents_path/_orphans/`, keeping its relative path. A file whos
 target already exists, or a filename shared by two channels with different
 `folder_slug`, is reported as a conflict and left untouched.
 
-## Stats
+## Report the tokens, cost and time
 
 `markdown-stats` reads the frontmatter of every `*.md` under `contents_path`
 (recursively) and prints global totals — tokens, estimated cost, processing
@@ -280,21 +370,23 @@ folders
 Files without frontmatter (generated before the frontmatter existed) are counted
 but left out of the totals.
 
-## Development
+## Set up the development environment
 
 To work on this project, [mise](https://mise.jdx.dev/) installs both yt-dlp and
-Node.js and defines the `youtube-to-markdown` shell alias; `npm install` fetches
-the JavaScript dependencies:
+Node.js; `npm install` fetches the JavaScript dependencies:
 
 ```sh
 $ mise install
 $ npm install
 ```
 
-The shell alias maps `youtube-to-markdown` to `node src/cli.js`. It is set when
-you enter the project from an interactive bash, zsh or fish shell, so it is only
-available in interactive shells; in scripts and CI, call
-`node src/cli.js <command>` instead.
+The `enter` hook defines a `youtube-to-markdown` shell function that calls
+`node src/cli.js`, and loads the shell completion for bash and zsh; the `leave`
+hook removes both. Everything is set when you enter the project from an
+interactive bash or zsh shell, so it is only available in interactive shells; in
+scripts and CI, call `node src/cli.js <command>` instead. A function, rather than
+a shell alias, is what lets `youtube-to-markdown <TAB>` complete the subcommands:
+zsh expands aliases before dispatching completion, which would bypass `compdef`.
 
 To generate Markdown without exporting `OPENAI_API_KEY` in your shell, copy
 `.secret.sh.example` to `.secret.sh` and put your API key in it; mise sources
@@ -305,7 +397,7 @@ $ cp .secret.sh.example .secret.sh
 $ $EDITOR .secret.sh
 ```
 
-## Publish
+## Publish the package to npmjs
 
 I publish the package to npmjs from this repository with mise:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stephane-klein/youtube-to-markdown",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Collect YouTube channel videos into a YAML config with yt-dlp, download transcripts and turn them into Markdown prose with an LLM.",
   "type": "module",
   "license": "MIT",

package/src/cli.js

@@ -10,6 +10,7 @@ import { runDownloadVtt } from "./download-vtt.js";
 import { runGenerateMarkdown } from "./generate-markdown.js";
 import { runMarkdownStats } from "./markdown-stats.js";
 import { runReorganize } from "./organize-files.js";
+import { runDoctor } from "./doctor.js";
 
 const { version } = createRequire(import.meta.url)("../package.json");
 
@@ -49,8 +50,9 @@ const project = (argv) => {
   return { path: argv.config, config: yaml, feed: yaml.feed ?? [], settings, dir };
 };
 
-yargs(hideBin(process.argv))
-  .scriptName("youtube-to-markdown")
+const cli = yargs(hideBin(process.argv))
+  .scriptName("")
+  .usage("youtube-to-markdown <command>")
   .config(loadEnv())
   .option("config", config)
   .command(
@@ -147,9 +149,65 @@ yargs(hideBin(process.argv))
       await runReorganize({ feed, dir, dryRun: argv.dryRun });
     }),
   )
-  .demandCommand(1, "Use one of the available commands")
+  .command(
+    "doctor",
+    "Check the runtime dependencies and the LLM access",
+    (yargs) =>
+      yargs
+        .option("model", {
+          describe: "LLM model id to test (default: model_id in the configuration)",
+          type: "string",
+        })
+        .option("endpoint", {
+          describe: "OpenAI-compatible chat completions endpoint to test",
+          type: "string",
+        })
+        .option("api-key", {
+          describe: "API key to test (defaults to OPENAI_API_KEY)",
+          type: "string",
+        })
+        .option("session", {
+          describe: "Value of the X-OpenCode-Session header",
+          type: "string",
+        })
+        .option("llm", {
+          describe: "Run a minimal LLM request to test the API access",
+          type: "boolean",
+          default: true,
+        })
+        .option("timeout", {
+          describe: "Timeout in seconds for the LLM request (default: 30)",
+          type: "number",
+          default: 30,
+        }),
+    handle(async (argv) => {
+      await runDoctor({
+        configPath: argv.config,
+        overrides: {
+          model: argv.model,
+          endpoint: argv.endpoint,
+          apiKey: argv.apiKey,
+          session: argv.session,
+        },
+        llm: argv.llm,
+        timeoutMs: argv.timeout * 1000,
+      });
+    }),
+  )
+  .command(
+    "completion",
+    "Generate a shell completion script for bash or zsh",
+    () => {},
+    () => cli.showCompletionScript("youtube-to-markdown", "completion"),
+  )
+  .demandCommand(
+    1,
+    "Use one of the available commands, or run `youtube-to-markdown doctor` " +
+      "first.",
+  )
   .strictCommands()
   .recommendCommands()
   .version(version)
-  .help()
-  .parse();
+  .help();
+
+cli.parse();

package/src/doctor.js

@@ -0,0 +1,211 @@
+import { access, constants } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+import { generateText } from "ai";
+import { loadConfig, settingsFromConfig } from "./config.js";
+import { createModel, describeError } from "./generate-markdown.js";
+import { exists } from "./vtt.js";
+import { runYtDlp } from "./ytdlp.js";
+
+const MIN_NODE_MAJOR = 22;
+
+function resolveApiKey(overrides, config, env = process.env) {
+  if (overrides.apiKey !== undefined) {
+    const envKey = env.YT_TO_MD_API_KEY;
+    const source = envKey !== undefined && envKey === overrides.apiKey
+      ? "YT_TO_MD_API_KEY"
+      : "--api-key";
+    return { key: overrides.apiKey, source };
+  }
+  if (config?.api_key !== undefined) {
+    return { key: config.api_key, source: "api_key (config)" };
+  }
+  if (env.OPENAI_API_KEY) {
+    return { key: env.OPENAI_API_KEY, source: "OPENAI_API_KEY" };
+  }
+  return { key: undefined, source: null };
+}
+
+async function checkContents(dir) {
+  try {
+    await access(dir, constants.W_OK);
+    return { status: "ok", note: null };
+  } catch (error) {
+    if (error.code !== "ENOENT") {
+      return { status: "error", note: "not writable" };
+    }
+  }
+  let parent = dirname(dir);
+  while (true) {
+    try {
+      await access(parent, constants.W_OK);
+      return { status: "warn", note: "not created yet" };
+    } catch (error) {
+      if (error.code !== "ENOENT") {
+        return { status: "error", note: "not writable" };
+      }
+      const next = dirname(parent);
+      if (next === parent) return { status: "error", note: "not writable" };
+      parent = next;
+    }
+  }
+}
+
+function checkNode() {
+  const version = process.versions.node;
+  const major = Number(version.split(".")[0]);
+  return major >= MIN_NODE_MAJOR
+    ? { status: "ok", value: `v${version}`, note: null }
+    : { status: "error", value: `v${version}`, note: `requires >= ${MIN_NODE_MAJOR}` };
+}
+
+async function checkYtDlp() {
+  try {
+    const { stdout } = await runYtDlp(["--version"]);
+    return { status: "ok", value: stdout.trim() || "found", note: null };
+  } catch (error) {
+    return { status: "error", value: "-", note: error.message };
+  }
+}
+
+async function loadConfigSafe(configPath) {
+  if (!(await exists(configPath))) {
+    return {
+      config: null,
+      row: { status: "warn", value: configPath, note: "not found" },
+    };
+  }
+  try {
+    const config = loadConfig(configPath);
+    const channels = config.feed?.length ?? 0;
+    const videos = (config.feed ?? []).reduce(
+      (total, channel) => total + (channel.videos?.length ?? 0),
+      0,
+    );
+    return {
+      config,
+      row: {
+        status: "ok",
+        value: configPath,
+        note: `${channels} channel(s), ${videos} video(s)`,
+      },
+    };
+  } catch (error) {
+    return {
+      config: null,
+      row: { status: "error", value: configPath, note: error.message },
+    };
+  }
+}
+
+function checkEndpoint(endpoint) {
+  if (!endpoint) return { status: "error", value: "-", note: "not set" };
+  try {
+    new URL(endpoint);
+    return { status: "ok", value: endpoint, note: null };
+  } catch {
+    return { status: "error", value: endpoint, note: "not a valid URL" };
+  }
+}
+
+async function checkLlm({ llm, settings, key, timeoutMs }) {
+  if (!llm) {
+    return { status: "warn", value: "-", note: "skipped (--no-llm)" };
+  }
+  if (!settings.model) return { status: "error", value: "-", note: "model not set" };
+  if (!settings.endpoint) return { status: "error", value: "-", note: "endpoint not set" };
+  if (!key) return { status: "error", value: "-", note: "api key not set" };
+
+  try {
+    const model = createModel({
+      modelId: settings.model,
+      endpoint: settings.endpoint,
+      apiKey: key,
+      session: settings.session,
+    });
+    const started = performance.now();
+    const { usage } = await generateText({
+      model,
+      prompt: "ping",
+      maxOutputTokens: 8,
+      maxRetries: 0,
+      abortSignal: AbortSignal.timeout(timeoutMs),
+    });
+    const elapsed = (performance.now() - started) / 1000;
+    return {
+      status: "ok",
+      value: `${elapsed.toFixed(1)}s`,
+      note: `${usage?.inputTokens ?? 0} in / ${usage?.outputTokens ?? 0} out`,
+    };
+  } catch (error) {
+    return { status: "error", value: "-", note: describeError(error, 1) };
+  }
+}
+
+function render(rows) {
+  const labelWidth = Math.max(...rows.map((row) => row.label.length));
+  const valueWidth = Math.max(...rows.map((row) => row.value.length));
+  return rows.map((row) => {
+    const note = row.note ? ` (${row.note})` : "";
+    return `${row.label.padEnd(labelWidth)}  ${row.value.padEnd(valueWidth)}  ${row.status}${note}`;
+  });
+}
+
+export async function runDoctor({
+  configPath = "youtube_to_markdown.yaml",
+  overrides = {},
+  llm = true,
+  timeoutMs = 30000,
+} = {}) {
+  const { config, row: configRow } = await loadConfigSafe(configPath);
+
+  const settings = { ...settingsFromConfig(config ?? {}) };
+  for (const [key, value] of Object.entries(overrides)) {
+    if (value !== undefined) settings[key] = value;
+  }
+
+  const { key, source } = resolveApiKey(overrides, config);
+  const dir = resolve(dirname(configPath), config?.contents_path ?? "contents");
+
+  const rows = [
+    { label: "node", ...checkNode() },
+    { label: "yt-dlp", ...(await checkYtDlp()) },
+  ];
+  rows.push({ label: "config", ...configRow });
+  rows.push({
+    label: "contents",
+    value: config?.contents_path ?? "contents",
+    ...(await checkContents(dir)),
+  });
+  rows.push({
+    label: "model",
+    value: settings.model ?? "-",
+    status: settings.model ? "ok" : "error",
+    note: settings.model ? null : "not set",
+  });
+  rows.push({ label: "endpoint", ...checkEndpoint(settings.endpoint) });
+  rows.push({
+    label: "api key",
+    value: source ?? "-",
+    status: key ? "ok" : "error",
+    note: key ? null : "not set",
+  });
+  if (settings.session) {
+    rows.push({ label: "session", value: "set", status: "info", note: null });
+  }
+  const llmRow = await checkLlm({ llm, settings, key, timeoutMs });
+  rows.push({ label: "llm", ...llmRow });
+
+  const counts = { ok: 0, warn: 0, error: 0 };
+  for (const row of rows) {
+    if (row.status in counts) counts[row.status]++;
+  }
+
+  console.log(`youtube-to-markdown doctor`);
+  console.log(render(rows).join("\n"));
+  console.log();
+  console.log(
+    `summary: ${counts.ok} ok, ${counts.warn} warning, ${counts.error} error`,
+  );
+
+  if (counts.error > 0) process.exitCode = 1;
+}

package/src/generate-markdown.js

@@ -183,7 +183,7 @@ function retryDelayMs(error, fallbackMs) {
   return Math.max(headerMs, fallbackMs);
 }
 
-function describeError(error, attempts) {
+export function describeError(error, attempts) {
   const suffix = ` after ${attempts} attempt(s)`;
   if (!APICallError.isInstance(error)) return `${error.message}${suffix}`;
 
@@ -201,6 +201,16 @@ function parseRetryDelays(value) {
     .filter((item) => Number.isFinite(item) && item >= 0);
 }
 
+export function createModel({ modelId, endpoint, apiKey, session }) {
+  const provider = createOpenAICompatible({
+    name: "opencode",
+    apiKey,
+    baseURL: endpoint.replace(/\/chat\/completions\/?$/, ""),
+    headers: session ? { "X-OpenCode-Session": session } : {},
+  });
+  return provider(modelId);
+}
+
 async function toMarkdown(
   { model, retryDelays, maxOutputTokens },
   title,
@@ -274,14 +284,7 @@ export async function runGenerateMarkdown({
   const llmConcurrency = Number(concurrency);
   const vttConcurrency = Number(ytdlpConcurrency);
 
-  const provider = createOpenAICompatible({
-    name: "opencode",
-    apiKey: key,
-    baseURL: endpoint.replace(/\/chat\/completions\/?$/, ""),
-    headers: session ? { "X-OpenCode-Session": session } : {},
-  });
-
-  const model = provider(modelId);
+  const model = createModel({ modelId, endpoint, apiKey: key, session });
 
   await mkdir(dir, { recursive: true });