@stephane-klein/youtube-to-markdown 0.4.0 → 0.9.0

package/README.md

@@ -13,6 +13,37 @@ channel or playlist and then to download its transcripts.
 In the final step, I use an LLM through a standard OpenAI-compatible API to
 reconstruct a complete text from the video transcripts.
 
+## AI usage in this project
+
+This project was developed using:
+
+- Harness: [OpenCode](https://opencode.ai)
+- Models: [DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) via [OpenCode Go](https://opencode.ai/go)
+- I mostly vibe coded this project: I made the implementation, library and architecture choices, but I reviewed the code very little.
+
+The figures below cover the period from the start of the project to version 0.4.0
+([de39801](https://github.com/stephane-klein/youtube-to-markdown/commit/de39801f123fa1cfaeaa3b9cfed5071968977474)).
+
+Metrics:
+
+- 16 OpenCode sessions, 1426 interactions;
+- 11 h of human time on this project, against 4 h of AI generation time over the same period, so a human/AI time ratio of 2.75 (11 h / 4 h);
+- Cost: $2.4958;
+- Tokens:
+  - input: 7,047,916
+  - output: 421,770
+  - cache read: 228,841,600
+  - total: 236,311,286
+
+Per agent:
+
+| agent   | interactions | share | output tokens | active hours | active share | generation hours | cost   |
+|:--------|-------------:|------:|--------------:|-------------:|-------------:|-----------------:|-------:|
+| build   |          962 |   68% |       259,706 |         3.71 |          38% |             1.93 | $1.385 |
+| plan    |          450 |   32% |       155,136 |         5.91 |          61% |             2.10 | $1.098 |
+| explore |           14 |    1% |         6,928 |         0.02 |           0% |             0.02 | $0.013 |
+
+
 ## Requirements
 
 - Node.js 22 or later;
@@ -129,7 +160,13 @@ can gate a CI job.
 ```sh
 $ youtube-to-markdown extract-video-metadata
 → https://www.youtube.com/@le_science4all
+  exact dates 50/2217  2.4s elapsed  ~1m 44s left
+  exact dates 100/2217  4.8s elapsed  ~1m 42s left
+  exact dates 2217/2217  1m 46s elapsed
+  done in 1m 46s
 → https://www.youtube.com/@MonsieurPhi
+  done in 12.3s
+summary: 2 sources, 2678 videos, 2m 05s elapsed
 ```
 
 The command is idempotent: a new run only adds the missing videos.
@@ -147,8 +184,9 @@ variables, then the settings written in the YAML file, then the built-in
 defaults. The API key stays a secret: `--api-key` or `OPENAI_API_KEY`. The
 settings are `x_opencode_session`, `model_id`, `openaiapi_endpoint`,
 `concurrency` and `ytdlp_concurrency` (plus the optional `api_key`,
-`retry_delays`, `max_output_tokens` and `chunk_target_tokens`). I only set the
-source URLs, and the `extract-video-metadata` command fills the rest:
+`retry_delays`, `max_output_tokens`, `chunk_target_tokens` and `reasoning`). I
+only set the source URLs, and the `extract-video-metadata` command fills the
+rest:
 
 `x_opencode_session` is the base of the `X-OpenCode-Session` header; it is
 optional and specific to OpenCode. A stable per-video suffix is appended to it,
@@ -298,9 +336,49 @@ only the last 20 events on screen, so it stays bounded even with a large feed.
 Retryable API errors (HTTP 408, 409, 429 or >= 500) are retried with growing
 delays — 10 s, 30 s, 60 s by default, configurable with `--retry-delays`
 (comma-separated seconds, or `YT_TO_MD_RETRY_DELAYS`). A `Retry-After` header
-from the server is honored when longer than the schedule. Each retry is shown in
-the phase output; after the last attempt the file is reported as an error and is
-retried on the next run.
+from the server is honored when longer than the schedule. Each retry line shows
+the reason reported by the API (status and response body), for example:
+
+```
+retrying in 30s (attempt 3/4) HTTP 429 rate limited  david-cayla/....fr.md
+```
+
+After the last attempt the file is reported as an error and is retried on the
+next run. At the end of the run, a `failures:` block lists each failed file with
+its reason, so the cause stays visible even after the phase output has scrolled.
+The command exits with a non-zero status when at least one file failed.
+
+### Empty output
+
+A model can finish with `finish_reason: stop` but return no text at all, for
+example when it spends its whole response on internal reasoning. Such a
+generation is reported as an error and no file is written:
+
+```
+error (empty output: part 1/1, finish=stop, text_tokens=1, reasoning_tokens=4001)  ....fr.md
+```
+
+The `failures:` block repeats the reason at the end of the run. Add
+`--debug-llm` (or `YT_TO_MD_DEBUG_LLM=1`) to log the request's `max_tokens` and
+`reasoning_effort`, plus the size of the provider's `content` and
+`reasoning_content`, which helps diagnose provider-side empty answers.
+
+### Reasoning effort
+
+The command asks the model for a reasoning effort of `low` by default. Set
+`reasoning` in the YAML file, `YT_TO_MD_REASONING`, or `--reasoning` to change
+it. The accepted values are `provider-default`, `none`, `minimal`, `low`,
+`medium`, `high` and `xhigh`, as exposed by ai-sdk. `none` disables reasoning:
+on a DeepSeek endpoint it sends the provider-specific `thinking: disabled`
+option, elsewhere it sends `reasoning: none`. `provider-default` leaves the
+choice to the provider.
+
+```yaml
+reasoning: "none"
+```
+
+The `doctor` command shows the effective value and applies it to its test
+request, so it reflects the real configuration.
 
 ### Output budget and long transcripts
 
@@ -319,7 +397,8 @@ LLM call that rewrites the beginning of the next section so it flows from the
 previous one. `--chunk-target-tokens` (or `YT_TO_MD_CHUNK_TARGET_TOKENS`,
 default 8000) sets the target input size of each chunk. The generation of a
 chunk is guaranteed to stop with `stop`; otherwise the file is reported as an
-error and nothing is written. The frontmatter records how many chunks were used.
+error and nothing is written. The same happens when a part comes back with an
+empty body. The frontmatter records how many chunks were used.
 
 ### Generated frontmatter
 
@@ -337,8 +416,11 @@ llm:
   input_tokens: 4428
   cached_input_tokens: 0
   output_tokens: 3262
+  reasoning_tokens: 0
+  text_tokens: 3262
   estimated_cost_usd: 0.000927
   finish_reason: stop
+  raw_finish_reason: stop
   pricing:
     source: models.dev
     peak: null
@@ -348,10 +430,10 @@ llm:
 `video_title` is the original video title in the file's language and
 `generated_at` is the UTC time of the generation. `pricing.source` names the
 catalog the rates came from and `pricing.peak` tells whether the peak rate
-applied (`null` when the source has no peak/off-peak pricing). Existing files
-are left untouched: run `youtube-to-markdown generate-markdown --force` to
-regenerate every file, which is also the way to add the frontmatter to files
-generated before it existed.
+applied to the first call (`null` when the source has no peak/off-peak pricing).
+Existing files are left untouched: run `youtube-to-markdown generate-markdown
+--force` to regenerate every file, which is also the way to add the frontmatter
+to files generated before it existed.
 
 ### Pricing
 
@@ -364,6 +446,10 @@ catalogs are cached for one day under `$XDG_CACHE_HOME/youtube-to-markdown`
 (`~/.cache/youtube-to-markdown` by default). When a catalog or the model is
 unavailable, `estimated_cost_usd` is `null` and the run continues.
 
+The rate is sampled at the start of every LLM call, so a run that straddles a
+peak/off-peak boundary is priced with the right rate on each side. The
+`pricing` block in the frontmatter describes the first call of the file.
+
 ## Organize the files into folders
 
 A channel may set `folder_slug` to keep its files in their own subdirectory of

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stephane-klein/youtube-to-markdown",
-  "version": "0.4.0",
+  "version": "0.9.0",
   "description": "Collect YouTube channel videos into a YAML config with yt-dlp, download transcripts and turn them into Markdown prose with an LLM.",
   "type": "module",
   "license": "MIT",

package/src/cli.js

@@ -24,6 +24,7 @@ const SETTING_KEYS = [
   "retryDelays",
   "maxOutputTokens",
   "chunkTargetTokens",
+  "reasoning",
 ];
 
 const handle = (run) => async (argv) => {
@@ -121,6 +122,15 @@ const cli = yargs(hideBin(process.argv))
           describe: "Target input tokens per chunk when a transcript is split (default: 8000)",
           type: "number",
         })
+        .option("reasoning", {
+          describe: "Reasoning effort: provider-default, none, minimal, low, medium, high or xhigh (default: low)",
+          type: "string",
+        })
+        .option("debug-llm", {
+          describe: "Print LLM request and response diagnostics",
+          type: "boolean",
+          default: false,
+        })
         .option("force", {
           describe: "Regenerate every Markdown file",
           type: "boolean",
@@ -128,7 +138,13 @@ const cli = yargs(hideBin(process.argv))
         }),
     handle(async (argv) => {
       const { feed, settings, dir } = project(argv);
-      await runGenerateMarkdown({ feed, ...settings, dir, force: argv.force });
+      await runGenerateMarkdown({
+        feed,
+        ...settings,
+        dir,
+        force: argv.force,
+        debugLlm: argv.debugLlm,
+      });
     }),
   )
   .command(
@@ -175,6 +191,10 @@ const cli = yargs(hideBin(process.argv))
           describe: "Value of the X-OpenCode-Session header",
           type: "string",
         })
+        .option("reasoning", {
+          describe: "Reasoning effort to test (default: low)",
+          type: "string",
+        })
         .option("llm", {
           describe: "Run a minimal LLM request to test the API access",
           type: "boolean",
@@ -193,6 +213,7 @@ const cli = yargs(hideBin(process.argv))
           endpoint: argv.endpoint,
           apiKey: argv.apiKey,
           session: argv.session,
+          reasoning: argv.reasoning,
         },
         llm: argv.llm,
         timeoutMs: argv.timeout * 1000,

package/src/config.js

@@ -15,6 +15,7 @@ const YAML_KEYS = {
   retryDelays: "retry_delays",
   maxOutputTokens: "max_output_tokens",
   chunkTargetTokens: "chunk_target_tokens",
+  reasoning: "reasoning",
 };
 
 export function settingsFromConfig(config) {
@@ -36,6 +37,8 @@ const ENV = {
   retryDelays: "YT_TO_MD_RETRY_DELAYS",
   maxOutputTokens: "YT_TO_MD_MAX_OUTPUT_TOKENS",
   chunkTargetTokens: "YT_TO_MD_CHUNK_TARGET_TOKENS",
+  reasoning: "YT_TO_MD_REASONING",
+  debugLlm: "YT_TO_MD_DEBUG_LLM",
 };
 
 export function loadEnv(env = process.env) {

package/src/doctor.js

@@ -2,7 +2,12 @@ import { access, constants } from "node:fs/promises";
 import { dirname, resolve } from "node:path";
 import { generateText } from "ai";
 import { loadConfig, settingsFromConfig } from "./config.js";
-import { createModel, describeError } from "./generate-markdown.js";
+import {
+  createModel,
+  DEFAULT_REASONING,
+  describeError,
+  reasoningOptions,
+} from "./generate-markdown.js";
 import { loadPricing } from "./pricing.js";
 import { exists } from "./vtt.js";
 import { runYtDlp } from "./ytdlp.js";
@@ -124,11 +129,13 @@ async function checkLlm({ llm, settings, key, timeoutMs }) {
       session: settings.session,
     });
     const started = performance.now();
+    const reasoning = settings.reasoning ?? DEFAULT_REASONING;
     const { usage } = await generateText({
       model,
       prompt: "ping",
       maxOutputTokens: 8,
       maxRetries: 0,
+      ...reasoningOptions(reasoning, settings.endpoint),
       abortSignal: AbortSignal.timeout(timeoutMs),
     });
     const elapsed = (performance.now() - started) / 1000;
@@ -218,6 +225,12 @@ export async function runDoctor({
   if (settings.session) {
     rows.push({ label: "session", value: "set", status: "info", note: null });
   }
+  rows.push({
+    label: "reasoning",
+    value: settings.reasoning ?? DEFAULT_REASONING,
+    status: "info",
+    note: null,
+  });
   rows.push({ label: "pricing", ...(await checkPricing({ settings })) });
   const llmRow = await checkLlm({ llm, settings, key, timeoutMs });
   rows.push({ label: "llm", ...llmRow });

package/src/extract-video-metadata.js

@@ -1,9 +1,12 @@
 import { writeFile } from "node:fs/promises";
 import { stringify } from "yaml";
+import { formatDuration, progressLine } from "./progress.js";
 import { runYtDlp } from "./ytdlp.js";
 
 const DATE_CHUNK = 50;
 
+const count = (value, word) => `${value} ${word}${value === 1 ? "" : "s"}`;
+
 const toIso = (raw) =>
   /^\d{8}$/.test(raw)
     ? `${raw.slice(0, 4)}-${raw.slice(4, 6)}-${raw.slice(6, 8)}`
@@ -61,12 +64,10 @@ const videoId = (url) => /[?&]v=([\w-]+)/.exec(url)?.[1] ?? url;
 
 async function fetchExactDates(urls) {
   const dates = new Map();
+  const startedAt = performance.now();
 
   for (let i = 0; i < urls.length; i += DATE_CHUNK) {
     const chunk = urls.slice(i, i + DATE_CHUNK);
-    console.error(
-      `  exact dates ${Math.min(i + DATE_CHUNK, urls.length)}/${urls.length}`,
-    );
     const out = await ytDlp(
       [
         "--skip-download",
@@ -84,6 +85,15 @@ async function fetchExactDates(urls) {
       const iso = toIso(raw ?? "");
       if (id && iso) dates.set(id, iso);
     }
+
+    console.error(
+      progressLine({
+        label: "exact dates",
+        done: Math.min(i + DATE_CHUNK, urls.length),
+        total: urls.length,
+        startedAt,
+      }),
+    );
   }
 
   return dates;
@@ -180,9 +190,14 @@ export async function runExtract({
   config = {},
   forceDates = false,
 } = {}) {
+  const startedAt = performance.now();
+  let sourceCount = 0;
+  let videoCount = 0;
+
   for (const channel of config.feed ?? []) {
     if (typeof channel.url !== "string" || !channel.url) continue;
 
+    const channelStartedAt = performance.now();
     const videos = channel.videos ?? [];
     const knownDates = forceDates
       ? new Map()
@@ -197,7 +212,17 @@ export async function runExtract({
     channel.videos = mergeVideos(videos, fetched.videos, {
       sort: !fetched.playlist,
     });
+
+    sourceCount += 1;
+    videoCount += channel.videos.length;
+    console.error(
+      `  done in ${formatDuration(performance.now() - channelStartedAt)}`,
+    );
   }
 
   await writeFile(path, stringify(config, { lineWidth: 0 }));
+
+  console.error(
+    `summary: ${count(sourceCount, "source")}, ${count(videoCount, "video")}, ${formatDuration(performance.now() - startedAt)} elapsed`,
+  );
 }

package/src/generate-markdown.js

@@ -68,6 +68,7 @@ function estimateCost(price, usage) {
 const OUTPUT_FACTOR = 2;
 const OUTPUT_FLOOR = 2048;
 const DEFAULT_CEILING = 32768;
+const DEFAULT_REASONING = "low";
 const DEFAULT_CHUNK_TARGET_TOKENS = 8000;
 
 function isSet(value) {
@@ -88,19 +89,33 @@ function budgetForPart(text, { maxOutputTokens, ceiling }) {
 }
 
 function metricsLine(modelId, usage, elapsed, cost) {
+  const reasoning = usage?.outputTokenDetails?.reasoningTokens;
   return [
     modelId,
     `llm ${elapsed.toFixed(1)}s`,
     usage
       ? `${usage.inputTokens ?? 0} in / ${usage.outputTokens ?? 0} out`
       : "usage ?",
+    reasoning ? `${reasoning} reasoning` : null,
     cost === null ? null : `~$${cost.toFixed(6)}`,
   ]
     .filter(Boolean)
     .join("  ");
 }
 
-function frontmatter(job, { modelId, usage, elapsed, cost, finishReason, price, chunks }) {
+function frontmatter(
+  job,
+  {
+    modelId,
+    usage,
+    elapsed,
+    cost,
+    finishReason,
+    rawFinishReason,
+    price,
+    chunks,
+  },
+) {
   const data = {
     source_url: job.url,
     video_title: titleForLang(job.titles, job.lang),
@@ -112,8 +127,11 @@ function frontmatter(job, { modelId, usage, elapsed, cost, finishReason, price,
       input_tokens: usage?.inputTokens ?? null,
       cached_input_tokens: usage ? cachedInputTokens(usage) : null,
       output_tokens: usage?.outputTokens ?? null,
+      reasoning_tokens: usage?.outputTokenDetails?.reasoningTokens ?? null,
+      text_tokens: usage?.outputTokenDetails?.textTokens ?? null,
       estimated_cost_usd: cost === null ? null : Number(cost.toFixed(6)),
       finish_reason: finishReason,
+      raw_finish_reason: rawFinishReason ?? null,
       pricing: price ? { source: price.source, peak: price.peak } : null,
     },
   };
@@ -121,6 +139,26 @@ function frontmatter(job, { modelId, usage, elapsed, cost, finishReason, price,
   return `---\n${stringify(data, { lineWidth: 0 }).trimEnd()}\n---\n\n`;
 }
 
+export function emptyOutputReason({
+  emptyParts,
+  chunks,
+  finishReason,
+  rawFinishReason,
+  usage,
+}) {
+  const raw =
+    rawFinishReason && rawFinishReason !== finishReason
+      ? `/${rawFinishReason}`
+      : "";
+  const text = usage?.outputTokenDetails?.textTokens ?? null;
+  const reasoning = usage?.outputTokenDetails?.reasoningTokens ?? null;
+  return (
+    `empty output: part ${emptyParts.join(",")}/${chunks}, ` +
+    `finish=${finishReason}${raw}, text_tokens=${text}, ` +
+    `reasoning_tokens=${reasoning}`
+  );
+}
+
 const isHeading = (text) => /^#{1,6}\s/.test(text);
 
 const isStructured = (text) =>
@@ -205,13 +243,16 @@ function retryDelayMs(error, fallbackMs) {
   return Math.max(headerMs, fallbackMs);
 }
 
-export function describeError(error, attempts) {
-  const suffix = ` after ${attempts} attempt(s)`;
-  if (!APICallError.isInstance(error)) return `${error.message}${suffix}`;
+export function describeCause(error) {
+  if (!APICallError.isInstance(error)) return `${error.message}`;
 
   const status = error.statusCode ? `HTTP ${error.statusCode}` : "API error";
   const body = error.responseBody?.trim().replace(/\s+/g, " ").slice(0, 200);
-  return `${status} ${error.message}${suffix}${body ? `: ${body}` : ""}`;
+  return `${status} ${error.message}${body ? `: ${body}` : ""}`;
+}
+
+export function describeError(error, attempts) {
+  return `${describeCause(error)} after ${attempts} attempt(s)`;
 }
 
 function parseRetryDelays(value) {
@@ -258,6 +299,44 @@ export function createModel({ modelId, endpoint, apiKey, session }) {
   return provider(modelId);
 }
 
+export function reasoningOptions(reasoning, endpoint = "") {
+  if (reasoning === "none") {
+    return /deepseek/i.test(endpoint)
+      ? {
+          providerOptions: {
+            opencode: { thinking: { type: "disabled" } },
+          },
+        }
+      : { reasoning: "none" };
+  }
+  return reasoning ? { reasoning } : {};
+}
+
+function logLlmDebug({
+  maxOutputTokens,
+  reasoning,
+  finishReason,
+  rawFinishReason,
+  text,
+  reasoningText,
+  warnings,
+}) {
+  console.error(
+    `llm debug: max_tokens=${maxOutputTokens ?? "-"} ` +
+      `reasoning=${reasoning ?? "provider-default"} ` +
+      `finish=${finishReason} raw=${rawFinishReason ?? "-"} ` +
+      `text_chars=${text?.length ?? 0} ` +
+      `reasoning_chars=${reasoningText?.length ?? 0}`,
+  );
+  if (warnings?.length) {
+    console.error(`llm debug: warnings=${JSON.stringify(warnings)}`);
+  }
+  if (!`${text ?? ""}`.trim() && reasoningText) {
+    const preview = reasoningText.replace(/\s+/g, " ").slice(0, 300);
+    console.error(`llm debug: empty text, reasoning preview: ${preview}`);
+  }
+}
+
 function videoSession(base, job) {
   const id = createHash("sha256")
     .update(`${job.url}#${job.lang}`)
@@ -275,7 +354,16 @@ function cleanBody(text) {
 }
 
 async function generateOnce(
-  { model, retryDelays, session, limiter },
+  {
+    model,
+    retryDelays,
+    session,
+    limiter,
+    priceAt,
+    reasoning,
+    endpoint,
+    debug,
+  },
   { system, prompt, maxOutputTokens, onRetry },
 ) {
   const totalAttempts = retryDelays.length + 1;
@@ -283,29 +371,60 @@ async function generateOnce(
   for (let attempt = 1;; attempt++) {
     const started = performance.now();
     try {
-      const call = () =>
-        generateText({
+      let at;
+      const call = () => {
+        at = new Date();
+        return generateText({
           model,
           system,
           prompt,
           temperature: 0.2,
-          reasoning: "low",
+          ...reasoningOptions(reasoning, endpoint),
           maxOutputTokens,
           maxRetries: 0,
           headers: { "X-OpenCode-Session": session },
         });
-      const { text, usage, finishReason } = await (
-        limiter ? limiter(call) : call()
-      );
+      };
+      const result = await (limiter ? limiter(call) : call());
+      const {
+        text,
+        usage,
+        finishReason,
+        rawFinishReason,
+        reasoningText,
+        warnings,
+      } = result;
       const elapsed = (performance.now() - started) / 1000;
+      const price = priceAt ? priceAt(at) : null;
+      const cost = estimateCost(price, usage);
+
+      if (debug) {
+        logLlmDebug({
+          maxOutputTokens,
+          reasoning,
+          finishReason,
+          rawFinishReason,
+          text,
+          reasoningText,
+          warnings,
+        });
+      }
 
-      return { text, usage, finishReason, elapsed };
+      return {
+        text,
+        usage,
+        finishReason,
+        rawFinishReason,
+        elapsed,
+        price,
+        cost,
+      };
     } catch (error) {
       if (!isRetryable(error) || attempt > retryDelays.length) {
         throw new Error(describeError(error, attempt), { cause: error });
       }
       const waitMs = retryDelayMs(error, retryDelays[attempt - 1]);
-      onRetry?.(attempt + 1, totalAttempts, waitMs);
+      onRetry?.(attempt + 1, totalAttempts, waitMs, error);
       await delay(waitMs);
     }
   }
@@ -400,6 +519,10 @@ async function toMarkdown(
     chunkTargetTokens,
     session,
     limiter,
+    priceAt,
+    reasoning,
+    endpoint,
+    debug,
   },
   title,
   transcript,
@@ -423,12 +546,25 @@ async function toMarkdown(
   const usages = [];
   let elapsed = 0;
   let finishReason = "stop";
+  let rawFinishReason;
+  let cost = 0;
+  let costKnown = true;
+  let price = null;
 
   await Promise.all(
     parts.map(async (part, index) => {
       onProgress?.(index + 1, total);
       const result = await generateOnce(
-        { model, retryDelays, session, limiter },
+        {
+          model,
+          retryDelays,
+          session,
+          limiter,
+          priceAt,
+          reasoning,
+          endpoint,
+          debug,
+        },
         {
           system: SYSTEM,
           prompt: chunkPrompt(title, part, index, total),
@@ -439,8 +575,15 @@ async function toMarkdown(
       bodies[index] = cleanBody(result.text);
       usages.push(result.usage);
       elapsed += result.elapsed;
+      if (result.cost === null) costKnown = false;
+      else cost += result.cost;
+      if (price === null && result.price) price = result.price;
+      if (rawFinishReason === undefined) {
+        rawFinishReason = result.rawFinishReason;
+      }
       if (result.finishReason !== "stop" && finishReason === "stop") {
         finishReason = result.finishReason;
+        rawFinishReason = result.rawFinishReason;
       }
     }),
   );
@@ -452,7 +595,16 @@ async function toMarkdown(
       if (!previousTail || !nextHead) continue;
 
       const smoothing = await generateOnce(
-        { model, retryDelays, session, limiter },
+        {
+          model,
+          retryDelays,
+          session,
+          limiter,
+          priceAt,
+          reasoning,
+          endpoint,
+          debug,
+        },
         {
           system: SMOOTH_SYSTEM,
           prompt: smoothPrompt(previousTail, nextHead),
@@ -462,6 +614,9 @@ async function toMarkdown(
       );
       usages.push(smoothing.usage);
       elapsed += smoothing.elapsed;
+      if (smoothing.cost === null) costKnown = false;
+      else cost += smoothing.cost;
+      if (price === null && smoothing.price) price = smoothing.price;
       if (smoothing.finishReason === "stop") {
         const rewritten = cleanBody(smoothing.text);
         if (rewritten) {
@@ -477,13 +632,20 @@ async function toMarkdown(
       .replace(/\n{3,}/g, "\n\n")
       .trim(),
   );
+  const emptyParts = bodies.flatMap((part, index) =>
+    part ? [] : [index + 1],
+  );
 
   return {
     body,
     usage: sumUsage(usages),
     finishReason,
+    rawFinishReason,
+    emptyParts,
     elapsed,
     chunks: total,
+    cost: costKnown ? cost : null,
+    price,
   };
 }
 
@@ -510,7 +672,9 @@ export async function runGenerateMarkdown({
   retryDelays = "10,30,60",
   maxOutputTokens,
   chunkTargetTokens = DEFAULT_CHUNK_TARGET_TOKENS,
+  reasoning = DEFAULT_REASONING,
   force = false,
+  debugLlm = false,
   dir = DEFAULT_DIR,
 } = {}) {
   const key = apiKey ?? process.env.OPENAI_API_KEY;
@@ -518,6 +682,7 @@ export async function runGenerateMarkdown({
   if (!modelId) throw new Error("model is not set (--model)");
   if (!endpoint) throw new Error("endpoint is not set (--endpoint)");
 
+  const debug = debugLlm === true || /^(1|true|yes|on)$/i.test(`${debugLlm}`);
   const RETRY_DELAYS = parseRetryDelays(retryDelays);
   const llmConcurrency = Number(concurrency);
   const vttConcurrency = Number(ytdlpConcurrency);
@@ -540,6 +705,7 @@ export async function runGenerateMarkdown({
     already: 0,
     skipped: 0,
     errors: 0,
+    failures: [],
     totalSeconds: 0,
     totalInput: 0,
     totalOutput: 0,
@@ -581,6 +747,7 @@ export async function runGenerateMarkdown({
     : isSet(modelOutputLimit)
       ? Number(modelOutputLimit)
       : DEFAULT_CEILING;
+  const priceAt = prices ? (at) => prices.priceAt(modelId, at) : null;
   console.error(
     `output budget: up to ${ceiling} tokens per call, chunk target ${chunkTargetTokens} tokens`,
   );
@@ -660,8 +827,17 @@ export async function runGenerateMarkdown({
         const name = relative(dir, md);
         try {
           const transcript = vttToText(await readFile(job.vtt, "utf8"));
-          const price = prices ? prices.priceAt(modelId) : null;
-          const { body, usage, finishReason, elapsed, chunks } = await toMarkdown(
+          const {
+            body,
+            usage,
+            finishReason,
+            rawFinishReason,
+            emptyParts,
+            elapsed,
+            chunks,
+            cost,
+            price,
+          } = await toMarkdown(
             {
               model,
               retryDelays: RETRY_DELAYS,
@@ -670,40 +846,63 @@ export async function runGenerateMarkdown({
               chunkTargetTokens,
               session: videoSession(sessionBase, job),
               limiter,
+              priceAt,
+              reasoning,
+              endpoint,
+              debug,
             },
             job.title,
             transcript,
-            (nextAttempt, totalAttempts, waitMs) => {
+            (nextAttempt, totalAttempts, waitMs, error) => {
               task.output = `retrying in ${Math.round(waitMs / 1000)}s ` +
-                `(attempt ${nextAttempt}/${totalAttempts})  ${name}`;
+                `(attempt ${nextAttempt}/${totalAttempts}) ` +
+                `${describeCause(error)}  ${name}`;
             },
             (part, totalParts) => {
-              if (totalParts > 1) {
-                task.output = `generating part ${part}/${totalParts}  ${name}`;
-              }
+              task.output = totalParts > 1
+                ? `generating part ${part}/${totalParts}  ${name}`
+                : `generating  ${name}`;
             },
           );
 
           stats.totalSeconds += elapsed;
           stats.totalInput += usage?.inputTokens ?? 0;
           stats.totalOutput += usage?.outputTokens ?? 0;
-          const cost = estimateCost(price, usage);
           if (cost === null) stats.costKnown = false;
           else stats.totalCost += cost;
           const metrics = metricsLine(modelId, usage, elapsed, cost);
 
           if (finishReason !== "stop") {
             stats.errors++;
+            stats.failures.push({
+              name,
+              reason: `finishReason=${finishReason}`,
+            });
             task.output = `error (finishReason=${finishReason})  ${name}`;
             return;
           }
 
+          if (emptyParts.length > 0) {
+            const reason = emptyOutputReason({
+              emptyParts,
+              chunks,
+              finishReason,
+              rawFinishReason,
+              usage,
+            });
+            stats.errors++;
+            stats.failures.push({ name, reason });
+            task.output = `error (${reason})  ${name}`;
+            return;
+          }
+
           const front = frontmatter(job, {
             modelId,
             usage,
             elapsed,
             cost,
             finishReason,
+            rawFinishReason,
             price,
             chunks,
           });
@@ -716,6 +915,7 @@ export async function runGenerateMarkdown({
           task.output = `generated  ${name}  ${metrics}`;
         } catch (error) {
           stats.errors++;
+          stats.failures.push({ name, reason: error.message });
           task.output = `error (${error.message})  ${name}`;
         } finally {
           const upToDate = stats.already + stats.generated;
@@ -741,6 +941,15 @@ export async function runGenerateMarkdown({
       stats.costKnown ? `, ~$${stats.totalCost.toFixed(6)}` : ""
     }`,
   );
+
+  if (stats.failures.length > 0) {
+    console.error("\nfailures:");
+    for (const failure of stats.failures) {
+      console.error(`  - ${failure.name}: ${failure.reason}`);
+    }
+  }
+
+  if (stats.errors > 0) process.exitCode = 1;
 }
 
 export {
@@ -748,6 +957,9 @@ export {
   chunkPrompt,
   cleanBody,
   createLimiter,
+  DEFAULT_REASONING,
+  estimateCost,
+  metricsLine,
   plannedBudget,
   smoothPrompt,
   sumUsage,

package/src/progress.js

@@ -0,0 +1,32 @@
+export function formatDuration(ms) {
+  const value = Number.isFinite(ms) && ms > 0 ? ms : 0;
+  if (value < 1000) return `${Math.round(value)}ms`;
+
+  const seconds = value / 1000;
+  if (seconds < 59.95) return `${seconds.toFixed(1)}s`;
+
+  const rounded = Math.round(seconds);
+  const minutes = Math.floor(rounded / 60);
+  if (minutes < 60) return `${minutes}m ${String(rounded % 60).padStart(2, "0")}s`;
+
+  const hours = Math.floor(minutes / 60);
+  return `${hours}h ${String(minutes % 60).padStart(2, "0")}m`;
+}
+
+export function progressStats({ done, total, startedAt, now = performance.now() }) {
+  const elapsed = Math.max(0, now - startedAt);
+  const completed = Math.min(Math.max(done, 0), Math.max(total, 0));
+
+  if (completed === 0 || total <= 0) return { elapsed, eta: null };
+
+  return { elapsed, eta: (elapsed * (total - completed)) / completed };
+}
+
+export function progressLine({ label, done, total, startedAt, now }) {
+  const { elapsed, eta } = progressStats({ done, total, startedAt, now });
+  const parts = [`  ${label} ${done}/${total}`, `${formatDuration(elapsed)} elapsed`];
+
+  if (eta !== null && eta > 0) parts.push(`~${formatDuration(eta)} left`);
+
+  return parts.join("  ");
+}