@stephane-klein/youtube-to-markdown 0.1.0

package/README.md

@@ -0,0 +1,324 @@
+# youtube-feed-to-markdown
+
+I collect the videos of a few YouTube channels into `youtube_to_markdown.yaml`,
+using [yt-dlp](https://github.com/yt-dlp/yt-dlp) to fetch the metadata. It is
+published on npmjs as `@stephane-klein/youtube-to-markdown`.
+
+## Requirements
+
+- Node.js 22 or later;
+- [yt-dlp](https://github.com/yt-dlp/yt-dlp) on your `PATH`.
+
+The commands call yt-dlp with `--js-runtimes node`: solving YouTube's JavaScript
+challenges requires an external JavaScript runtime, and Node.js — already
+required here — does the job.
+
+## Install
+
+The package is published on npmjs as `@stephane-klein/youtube-to-markdown`.
+Install it globally to get the `youtube-to-markdown` command on your `PATH`:
+
+```sh
+$ npm install -g @stephane-klein/youtube-to-markdown
+```
+
+Or run it without installing it, with `npx`:
+
+```sh
+$ npx @stephane-klein/youtube-to-markdown extract-video-metadata
+```
+
+Either way, `yt-dlp` must be on your `PATH` (see Requirements). The commands
+read `youtube_to_markdown.yaml` from the current directory unless you pass
+`--config`.
+
+## Run
+
+```sh
+$ youtube-to-markdown extract-video-metadata
+→ https://www.youtube.com/@le_science4all
+→ https://www.youtube.com/@MonsieurPhi
+```
+
+`youtube-to-markdown` exposes five subcommands: `extract-video-metadata`,
+`download-vtt`, `generate-markdown`, `markdown-stats` and `reorganize`. Run
+`youtube-to-markdown <command> --help` for the available options.
+
+The configuration file is `youtube_to_markdown.yaml` (override it with
+`--config` or `YT_TO_MD_CONFIG`). Its settings resolve in this order: command-line
+flags, then `YT_TO_MD_*` environment variables, then the settings written in the
+YAML file, then the built-in defaults. The API key stays a secret: `--api-key` or
+`OPENAI_API_KEY`.
+
+The `extract-video-metadata` command reads the channel URLs from the
+configuration file, asks yt-dlp for each channel's videos, and writes the result
+back into the file. I run it from time to time: it is idempotent, so a new run
+only adds the videos that are missing.
+
+## Configuration
+
+`youtube_to_markdown.yaml` holds the settings and the channels. Settings are
+`x_opencode_session`, `model_id`, `openaiapi_endpoint`, `concurrency` and
+`ytdlp_concurrency` (plus the optional `api_key`, `retry_delays` and
+`max_output_tokens`). I only set the channel URLs, and the
+`extract-video-metadata` command fills the rest:
+
+```yaml
+x_opencode_session: "youtube-to-markdown/0.1"
+model_id: "mimo-v2.5"
+openaiapi_endpoint: "https://opencode.ai/zen/go/v1/chat/completions"
+concurrency: "6"
+ytdlp_concurrency: "2"
+feed:
+  - title: Science4All
+    folder_slug: science-4-all
+    url: https://www.youtube.com/@le_science4all
+    videos:
+      - title:
+          fr: Pourquoi π est-il si fou ? Relativité 1
+          en: Why is π so crazy? Relativity 1
+        date: 2016-02-12
+        url: https://www.youtube.com/watch?v=PxRPpdzmbUQ
+  - title: MrPhi
+    url: https://www.youtube.com/@MonsieurPhi
+    videos: []
+```
+
+`contents_path` (optional) chooses the directory that holds the transcripts and
+the generated Markdown. It accepts an absolute path or a path relative to the
+directory of `youtube_to_markdown.yaml`; it defaults to `contents`. Unlike the
+other settings, it can only be set in the YAML file — there is no flag or
+environment variable for it. `.gitignore` only ignores the default `contents/`,
+so add your own path there if you move it.
+
+`folder_slug` (optional, on a channel) stores that channel's transcripts and
+Markdown in a subdirectory of `contents_path`, for example `science-4-all`.
+It is a relative path, so it stays inside `contents_path`; a channel without
+`folder_slug` keeps its files at the root of `contents_path`. The reserved name
+`_orphans` is rejected. Change it whenever you like, then run
+`youtube-to-markdown reorganize` to move the existing files.
+
+## Transcripts
+
+I mark the videos I want as transcripts with a `download_vtt` field:
+
+```yaml
+  - title:
+      fr: Pourquoi π est-il si fou ? Relativité 1
+      en: Why is π so crazy? Relativity 1
+    date: 2016-02-12
+    url: https://www.youtube.com/watch?v=PxRPpdzmbUQ
+    download_vtt:
+      fr: true
+```
+
+Then:
+
+```sh
+$ youtube-to-markdown download-vtt
+21 transcript(s) to check
+[ 1/21] downloaded         2016-07-14_le-sol-accelere-t-il-vraiment-vers-le-haut-debattonsmieux.fr.vtt
+[ 2/21] already downloaded 2016-08-25_les-synonymes-a-connotations-opposees-debattonsmieux.fr.vtt
+[ 3/21] missing            2017-06-26_favoriser-l-honnetete-democratie-18.fr.vtt
+…
+summary: 1 downloaded, 1 already downloaded, 1 missing, 0 error
+```
+
+For each marked video, the command writes the subtitle to
+`contents/<date>_<slug>.<lang>.vtt`, for example
+`contents/2016-02-12_pourquoi-est-il-si-fou-relativite-1.fr.vtt`. It prefers
+manual subtitles and falls back to the auto-generated ones. A file that already
+exists is left untouched, so running the command again only fetches new
+transcripts. The `extract-video-metadata` command keeps the `download_vtt`
+field.
+
+When a video has no subtitle at all, the command writes a `<…>.vtt.missing`
+marker and stops trying. I delete that marker to force a new attempt.
+
+## Markdown
+
+I mark the videos I want as Markdown with `generate_markdown`. The transcript is
+downloaded first if it is not there yet:
+
+```yaml
+    download_vtt:
+      fr: true
+    generate_markdown:
+      fr: true
+```
+
+Before the first run, provide your API key through the `OPENAI_API_KEY`
+environment variable, or pass it with `--api-key`.
+
+Then:
+
+```sh
+$ youtube-to-markdown generate-markdown
+74 markdown(s) to generate with mimo-v2.5
+✔ Download transcripts (73/74)
+  › downloaded  2017-08-04_nietzsche-la-morale-des-winners-genealogie-de-la-morale-1-2.fr.md
+  › skipped (no transcript)  2017-06-26_favoriser-l-honnetete-democratie-18.fr.md
+✔ Generate markdown (73/74, 37 already)
+  › generated  2017-10-13_le-scepticisme-le-trilemme-d-agrippa-grain-de-philo-14-ep-2.fr.md  mimo-v2.5  llm 87.1s  4428 in / 3262 out  ~$0.000927
+  …
+summary: 36 generated, 37 already generated, 1 skipped, 0 error
+total: 3135.6s, 159408 in / 117432 out, ~$0.033372
+```
+
+The command sends each transcript to an LLM through ai-sdk, configured with the
+API key (`OPENAI_API_KEY` or `--api-key`), the model (`YT_TO_MD_MODEL_ID` or
+`--model`) and the endpoint (`YT_TO_MD_OPENAIAPI_ENDPOINT` or `--endpoint`), and
+gets back Markdown prose with section headings when the talk needs them.
+Paragraphs are hard-wrapped at 80 columns. The result goes to
+`contents/<date>_<slug>.<lang>.md`. A second run reports `already generated` and
+writes nothing.
+
+The command runs in two sequential phases. `Download transcripts` reports how many
+videos have a transcript (`available/total`) and fetches the missing ones, at
+most `--ytdlp-concurrency` at a time (default 2); only downloads, skips and errors
+are listed. The `Generate markdown` phase then converts the transcripts, at most
+`--concurrency` at a time (default 6), reporting the number of up-to-date
+Markdown files over the whole feed (`ready/total, N already`). Each phase keeps
+only the last 20 events on screen, so it stays bounded even with a large feed.
+
+Retryable API errors (HTTP 408, 409, 429 or >= 500) are retried with growing
+delays — 10 s, 30 s, 60 s by default, configurable with `--retry-delays`
+(comma-separated seconds, or `YT_TO_MD_RETRY_DELAYS`). A `Retry-After` header
+from the server is honored when longer than the schedule. Each retry is shown in
+the phase output; after the last attempt the file is reported as an error and is
+retried on the next run.
+
+### Frontmatter
+
+Each generated file starts with a YAML frontmatter describing the run:
+
+```yaml
+---
+source_url: https://www.youtube.com/watch?v=PxRPpdzmbUQ
+video_title: Pourquoi π est-il si fou ? Relativité 1
+generated_at: 2026-09-19T10:12:33.456Z
+llm:
+  model: mimo-v2.5
+  duration_seconds: 87.1
+  input_tokens: 4428
+  cached_input_tokens: 0
+  output_tokens: 3262
+  estimated_cost_usd: 0.000927
+  finish_reason: stop
+---
+```
+
+`video_title` is the original video title in the file's language and
+`generated_at` is the UTC time of the generation. `estimated_cost_usd` is `null`
+when the model is not in the price table. Existing files are left untouched: run
+`youtube-to-markdown generate-markdown --force` to regenerate every file, which
+is also the way to add the frontmatter to files generated before it existed.
+
+## Folders
+
+A channel may set `folder_slug` to keep its files in their own subdirectory of
+`contents_path`. When you add, change or remove that setting, `reorganize` moves
+the already downloaded transcripts and generated Markdown to match. It runs for
+real by default; add `--dry-run` to preview:
+
+```sh
+$ youtube-to-markdown reorganize --dry-run
+[ 1/38] moved     science-4-all/2016-07-14_le-sol-accelere….fr.vtt (dry run)
+…
+dry run: no file was moved
+summary: 38 to move, 0 to orphan, 112 already in place, 0 conflict
+```
+
+A file that no longer matches any video of the feed is an orphan: the command
+moves it under `contents_path/_orphans/`, keeping its relative path. A file whose
+target already exists, or a filename shared by two channels with different
+`folder_slug`, is reported as a conflict and left untouched.
+
+## Stats
+
+`markdown-stats` reads the frontmatter of every `*.md` under `contents_path`
+(recursively) and prints global totals — tokens, estimated cost, processing
+time — with a breakdown per model, per language and per folder (`folder_slug`,
+or `(root)` when the file sits at the root of `contents_path`):
+
+```sh
+$ youtube-to-markdown markdown-stats
+contents/ — 43 file(s)
+  with frontmatter     3
+  without frontmatter  40
+
+tokens
+  input                16,134
+  cached input         16,000
+  output               14,486
+  total                30,620
+
+cost
+  estimated            $0.004120
+  unknown prices       0 file(s)
+
+processing time
+  total                395.6s
+  mean                 131.9s
+  min / max            111.8s / 165.3s
+
+generated_at
+  first                2026-09-19T08:56:54.497Z
+  last                 2026-09-19T08:57:47.976Z
+
+models
+  mimo-v2.5           3 file(s)     16,134 in /    14,486 out  $0.004120
+
+languages
+  fr                  3 file(s)     16,134 in /    14,486 out  $0.004120
+
+folders
+  science-4-all       2 file(s)     10,134 in /     9,486 out  $0.002120
+  (root)              1 file(s)      6,000 in /     5,000 out  $0.002000
+```
+
+Files without frontmatter (generated before the frontmatter existed) are counted
+but left out of the totals.
+
+## Development
+
+To work on this project, [mise](https://mise.jdx.dev/) installs both yt-dlp and
+Node.js and defines the `youtube-to-markdown` shell alias; `npm install` fetches
+the JavaScript dependencies:
+
+```sh
+$ mise install
+$ npm install
+```
+
+The shell alias maps `youtube-to-markdown` to `node src/cli.js`. It is set when
+you enter the project from an interactive bash, zsh or fish shell, so it is only
+available in interactive shells; in scripts and CI, call
+`node src/cli.js <command>` instead.
+
+To generate Markdown without exporting `OPENAI_API_KEY` in your shell, copy
+`.secret.sh.example` to `.secret.sh` and put your API key in it; mise sources
+that file automatically:
+
+```sh
+$ cp .secret.sh.example .secret.sh
+$ $EDITOR .secret.sh
+```
+
+## Publish
+
+I publish the package to npmjs from this repository with mise:
+
+```sh
+$ npm login
+$ npm version patch
+$ mise run publish-to-npmjs
+```
+
+`npm login` authenticates against npmjs, once per machine. `npm version patch`
+bumps the version and creates the matching git tag (use `minor` or `major` for
+larger changes). `mise run publish-to-npmjs` first checks the authentication with
+`npm whoami`, then runs `npm publish`; it publishes the version written in
+`package.json`, so bump it first. The package is scoped, so `publishConfig` in
+`package.json` forces public access.
+

package/package.json

@@ -0,0 +1,34 @@
+{
+  "name": "@stephane-klein/youtube-to-markdown",
+  "version": "0.1.0",
+  "description": "Collect YouTube channel videos into a YAML config with yt-dlp, download transcripts and turn them into Markdown prose with an LLM.",
+  "type": "module",
+  "license": "MIT",
+  "author": "Stéphane Klein <contact@stephane-klein.info>",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/stephane-klein/youtube-feed-to-markdown.git"
+  },
+  "engines": {
+    "node": ">=22"
+  },
+  "bin": {
+    "youtube-to-markdown": "src/cli.js"
+  },
+  "files": [
+    "src"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "youtube-to-markdown": "node src/cli.js"
+  },
+  "dependencies": {
+    "@ai-sdk/openai-compatible": "~3.0.51",
+    "ai": "~7.0.105",
+    "listr2": "~11.1.0",
+    "yaml": "~2.9.1",
+    "yargs": "~18.1.0"
+  }
+}

package/src/cli.js

@@ -0,0 +1,155 @@
+#!/usr/bin/env node
+
+import { createRequire } from "node:module";
+import { dirname, resolve } from "node:path";
+import yargs from "yargs";
+import { hideBin } from "yargs/helpers";
+import { loadConfig, loadEnv, settingsFromConfig } from "./config.js";
+import { runExtract } from "./extract-video-metadata.js";
+import { runDownloadVtt } from "./download-vtt.js";
+import { runGenerateMarkdown } from "./generate-markdown.js";
+import { runMarkdownStats } from "./markdown-stats.js";
+import { runReorganize } from "./organize-files.js";
+
+const { version } = createRequire(import.meta.url)("../package.json");
+
+const SETTING_KEYS = [
+  "model",
+  "endpoint",
+  "apiKey",
+  "session",
+  "concurrency",
+  "ytdlpConcurrency",
+  "retryDelays",
+  "maxOutputTokens",
+];
+
+const handle = (run) => async (argv) => {
+  try {
+    await run(argv);
+  } catch (error) {
+    console.error(`error: ${error.message}`);
+    process.exitCode = 1;
+  }
+};
+
+const config = {
+  describe: "Path to the YAML configuration file",
+  type: "string",
+  default: "youtube_to_markdown.yaml",
+};
+
+const project = (argv) => {
+  const yaml = loadConfig(argv.config);
+  const settings = { ...settingsFromConfig(yaml) };
+  for (const key of SETTING_KEYS) {
+    if (argv[key] !== undefined) settings[key] = argv[key];
+  }
+  const dir = resolve(dirname(argv.config), yaml.contents_path ?? "contents");
+  return { path: argv.config, config: yaml, feed: yaml.feed ?? [], settings, dir };
+};
+
+yargs(hideBin(process.argv))
+  .scriptName("youtube-to-markdown")
+  .config(loadEnv())
+  .option("config", config)
+  .command(
+    "extract-video-metadata",
+    "Fetch the channel videos from YouTube into the configuration file",
+    (yargs) =>
+      yargs
+        .option("force-dates", {
+          describe: "Fetch every upload date again, ignoring the known ones",
+          type: "boolean",
+          default: false,
+        }),
+    handle(async (argv) => {
+      const { path, config: yaml } = project(argv);
+      await runExtract({ path, config: yaml, forceDates: argv.forceDates });
+    }),
+  )
+  .command(
+    "download-vtt",
+    "Download the VTT transcripts of the marked videos",
+    () => {},
+    handle(async (argv) => {
+      const { feed, dir } = project(argv);
+      await runDownloadVtt({ feed, dir });
+    }),
+  )
+  .command(
+    "generate-markdown",
+    "Turn the VTT transcripts into Markdown prose with an LLM",
+    (yargs) =>
+      yargs
+        .option("model", {
+          describe: "LLM model id (default: model_id in the configuration)",
+          type: "string",
+        })
+        .option("endpoint", {
+          describe: "OpenAI-compatible chat completions endpoint",
+          type: "string",
+        })
+        .option("api-key", {
+          describe: "API key (defaults to OPENAI_API_KEY)",
+          type: "string",
+        })
+        .option("session", {
+          describe: "Value of the X-OpenCode-Session header",
+          type: "string",
+        })
+        .option("concurrency", {
+          describe: "Maximum number of concurrent LLM calls (default: 6)",
+          type: "number",
+        })
+        .option("ytdlp-concurrency", {
+          describe: "Maximum number of concurrent transcript downloads (default: 2)",
+          type: "number",
+        })
+        .option("retry-delays", {
+          describe: "Comma-separated retry delays in seconds (default: 10,30,60)",
+          type: "string",
+        })
+        .option("max-output-tokens", {
+          describe: "Override the computed output token budget",
+          type: "number",
+        })
+        .option("force", {
+          describe: "Regenerate every Markdown file",
+          type: "boolean",
+          default: false,
+        }),
+    handle(async (argv) => {
+      const { feed, settings, dir } = project(argv);
+      await runGenerateMarkdown({ feed, ...settings, dir, force: argv.force });
+    }),
+  )
+  .command(
+    "markdown-stats",
+    "Report global tokens, cost and time from the Markdown frontmatter",
+    () => {},
+    handle(async (argv) => {
+      const { dir } = project(argv);
+      await runMarkdownStats({ dir });
+    }),
+  )
+  .command(
+    "reorganize",
+    "Move the transcripts and Markdown into their folder_slug folders",
+    (yargs) =>
+      yargs.option("dry-run", {
+        describe: "Show the moves without doing them",
+        type: "boolean",
+        default: false,
+      }),
+    handle(async (argv) => {
+      const { feed, dir } = project(argv);
+      await runReorganize({ feed, dir, dryRun: argv.dryRun });
+    }),
+  )
+  .demandCommand(1, "Use one of the available commands")
+  .strictCommands()
+  .recommendCommands()
+  .version(version)
+  .help()
+  .parse();

package/src/config.js

@@ -0,0 +1,45 @@
+import { readFileSync } from "node:fs";
+import { parse } from "yaml";
+
+export function loadConfig(path) {
+  return parse(readFileSync(path, "utf8")) ?? {};
+}
+
+const YAML_KEYS = {
+  model: "model_id",
+  endpoint: "openaiapi_endpoint",
+  apiKey: "api_key",
+  session: "x_opencode_session",
+  concurrency: "concurrency",
+  ytdlpConcurrency: "ytdlp_concurrency",
+  retryDelays: "retry_delays",
+  maxOutputTokens: "max_output_tokens",
+};
+
+export function settingsFromConfig(config) {
+  return Object.fromEntries(
+    Object.entries(YAML_KEYS)
+      .map(([key, yamlKey]) => [key, config[yamlKey]])
+      .filter(([, value]) => value !== undefined),
+  );
+}
+
+const ENV = {
+  config: "YT_TO_MD_CONFIG",
+  model: "YT_TO_MD_MODEL_ID",
+  endpoint: "YT_TO_MD_OPENAIAPI_ENDPOINT",
+  apiKey: "YT_TO_MD_API_KEY",
+  session: "YT_TO_MD_X_OPENCODE_SESSION",
+  concurrency: "YT_TO_MD_CONCURRENCY",
+  ytdlpConcurrency: "YT_TO_MD_YTDLP_CONCURRENCY",
+  retryDelays: "YT_TO_MD_RETRY_DELAYS",
+  maxOutputTokens: "YT_TO_MD_MAX_OUTPUT_TOKENS",
+};
+
+export function loadEnv(env = process.env) {
+  return Object.fromEntries(
+    Object.entries(ENV)
+      .map(([key, name]) => [key, env[name]])
+      .filter(([, value]) => value !== undefined),
+  );
+}

package/src/download-vtt.js

@@ -0,0 +1,59 @@
+import { mkdir } from "node:fs/promises";
+import { dirname, relative } from "node:path";
+import { DEFAULT_DIR, downloadVtt, exists, videoJobs, writeMarker } from "./vtt.js";
+
+export async function runDownloadVtt({ feed = [], dir = DEFAULT_DIR } = {}) {
+  await mkdir(dir, { recursive: true });
+
+  const jobs = feed.flatMap((channel) =>
+    (channel.videos ?? []).flatMap((video) =>
+      videoJobs(video, dir, channel.folder_slug),
+    ),
+  );
+
+  console.error(`${jobs.length} transcript(s) to check`);
+
+  const width = String(jobs.length).length;
+  let downloaded = 0;
+  let present = 0;
+  let missing = 0;
+  let errors = 0;
+
+  for (const [index, job] of jobs.entries()) {
+    const prefix = `[${String(index + 1).padStart(width)}/${jobs.length}]`;
+    const name = relative(dir, job.vtt);
+
+    if (await exists(job.vtt)) {
+      present++;
+      console.error(`${prefix} ${"already downloaded".padEnd(18)} ${name}`);
+      continue;
+    }
+
+    if (await exists(job.marker)) {
+      missing++;
+      console.error(`${prefix} ${"missing".padEnd(18)} ${name}`);
+      continue;
+    }
+
+    await mkdir(dirname(job.vtt), { recursive: true });
+
+    const status = await downloadVtt(job.url, job.lang, job.vtt);
+    if (status === "downloaded") {
+      downloaded++;
+      console.error(`${prefix} ${"downloaded".padEnd(18)} ${name}`);
+    } else if (status === "missing") {
+      await writeMarker(job.marker, job.url);
+      missing++;
+      console.error(`${prefix} ${"missing".padEnd(18)} ${name}`);
+    } else {
+      errors++;
+      console.error(
+        `${prefix} ${"error".padEnd(18)} ${name} (will retry on next run)`,
+      );
+    }
+  }
+
+  console.error(
+    `summary: ${downloaded} downloaded, ${present} already downloaded, ${missing} missing, ${errors} error`,
+  );
+}