@steipete/summarize 0.2.0 → 0.4.0

package/docs/site/docs/extract-only.html

@@ -4,7 +4,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width,initial-scale=1" />
     <meta name="color-scheme" content="dark light" />
-    <title>Extract-only — summarize</title>
+    <title>Extract — summarize</title>
     <link rel="canonical" href="https://summarize.sh/docs/extract-only" />
     <link rel="preconnect" href="https://fonts.googleapis.com" />
     <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -40,7 +40,7 @@
 
         <article class="doc reveal">
           <p class="kicker">mode</p>
-          <h1>Extract-only</h1>
+          <h1>Extract</h1>
           <p>Print the extracted content and stop. No summary call.</p>
 
           <h2>Usage</h2>
@@ -50,7 +50,7 @@
               <span class="terminal__dot terminal__dot--b"></span>
               <span class="terminal__dot terminal__dot--c"></span>
             </div>
-            <pre><code id="ex-only">summarize --extract-only "https://example.com/article"</code></pre>
+            <pre><code id="ex-only">summarize --extract "https://example.com/article"</code></pre>
           </div>
           <div class="copyRow">
             <span class="hint">Good for piping into your own tooling.</span>
@@ -63,7 +63,9 @@
             <li><code>--verbose</code> — show which extractor ran and why.</li>
             <li><code>--timeout</code> — tune crawling budget (<code>2m</code> default).</li>
             <li><code>--firecrawl off|auto|always</code> — choose the fallback strategy.</li>
-            <li><code>--markdown off|auto|llm</code> — for non-YouTube URLs, control HTML→Markdown conversion.</li>
+            <li><code>--format md|text</code> — choose extracted output format.</li>
+            <li><code>--markdown-mode off|auto|llm</code> — for non-YouTube URLs with <code>--format md</code>, control HTML→Markdown conversion (LLM) + fallbacks.</li>
+            <li><code>--preprocess off|auto|always</code> — controls markitdown usage (default <code>auto</code>).</li>
           </ul>
         </article>
       </section>

package/docs/site/docs/firecrawl.html

@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -41,7 +41,7 @@
         <article class="doc reveal">
           <p class="kicker">extractor</p>
           <h1>Firecrawl</h1>
-          <p>Used as a fallback when HTML extraction looks blocked or too thin (or when forced via <code>--firecrawl always</code>).</p>
+          <p>Used as a fallback when HTML extraction looks blocked or too thin — and as a preferred Markdown source in extract mode (when configured).</p>
 
           <h2>Key</h2>
           <ul>
@@ -49,14 +49,14 @@
             <li>Control behavior with <code>--firecrawl off|auto|always</code>.</li>
           </ul>
 
-          <h2>Extract-only + Markdown</h2>
+          <h2>Extract + Markdown</h2>
           <ul>
-            <li><code>--extract-only</code> prints the extracted content.</li>
-            <li><code>--markdown auto|llm</code> can output Markdown for non-YouTube URLs.</li>
+            <li><code>--extract</code> prints the extracted content.</li>
+            <li><code>--extract --format md</code> outputs Markdown for non-YouTube URLs.</li>
           </ul>
 
           <div class="note">
-            If you only want plain text: use <code>--firecrawl off --markdown off</code>.
+            If you only want plain text: use <code>--extract --format text</code>.
           </div>
         </article>
       </section>

package/docs/site/docs/index.html

@@ -34,7 +34,7 @@
             <div class="pillRow">
               <span class="pill"><span class="pill__dot" aria-hidden="true"></span> Website</span>
               <span class="pill"><span class="pill__dot" aria-hidden="true" style="background: var(--accent2)"></span> YouTube</span>
-              <span class="pill"><span class="pill__dot" aria-hidden="true" style="background: var(--accent3)"></span> Extract-only</span>
+              <span class="pill"><span class="pill__dot" aria-hidden="true" style="background: var(--accent3)"></span> Extract</span>
             </div>
           </div>
         </div>
@@ -52,7 +52,7 @@
           <div class="small">docs/youtube.md</div>
         </a>
         <a class="card reveal" href="./extract-only.html">
-          <h2>Extract-only</h2>
+          <h2>Extract</h2>
           <p>Get the cleaned content and stop; perfect for piping.</p>
           <div class="small">docs/extract-only.md</div>
         </a>

package/docs/site/docs/llm.html

@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -53,7 +53,7 @@
           <h2>Practical advice</h2>
           <ul>
             <li>Pin <code>--model</code> for stable output.</li>
-            <li>When using <code>--markdown llm</code>, provider fallback is disabled by design.</li>
+            <li>When using <code>--markdown-mode llm</code>, provider fallback is disabled by design.</li>
             <li>For audits / tooling, prefer <code>--json</code> + fixed model.</li>
           </ul>
         </article>

package/docs/site/docs/openai.html

@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -45,7 +45,7 @@
 
           <h2>Notes</h2>
           <ul>
-            <li>Some modes (like <code>--extract-only</code>) don’t need an LLM at all.</li>
+            <li>Some modes (like <code>--extract</code>) don’t need an LLM at all.</li>
             <li>When output is used downstream, prefer <code>--json</code> and pin <code>--model</code>.</li>
           </ul>
 

package/docs/site/docs/website.html

@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -41,20 +41,23 @@
         <article class="doc reveal">
           <p class="kicker">mode</p>
           <h1>Website mode</h1>
-          <p>Fetch HTML → extract “article-ish” content → normalize to clean text. If extraction looks blocked or too thin, retry via Firecrawl Markdown (optional).</p>
+          <p>Fetch HTML → extract “article-ish” content → normalize to clean text. If extraction looks blocked or too thin, retry via Firecrawl Markdown (optional). With <code>--format md</code>, the CLI prefers Firecrawl Markdown when configured and can also convert HTML → Markdown via <code>--markdown-mode</code> (LLM) or <code>uvx markitdown</code>.</p>
 
           <h2>Flags</h2>
           <ul>
             <li><code>--firecrawl off|auto|always</code></li>
             <li><code>--timeout 30s|2m|5000ms</code> (default <code>2m</code>)</li>
-            <li><code>--extract-only</code> (print extracted content; no summary call)</li>
+            <li><code>--extract</code> (print extracted content; no summary call)</li>
+            <li><code>--format md|text</code> (default <code>text</code>)</li>
+            <li><code>--markdown-mode off|auto|llm</code> (only with <code>--format md</code>)</li>
+            <li><code>--preprocess off|auto|always</code> (default <code>auto</code>; controls markitdown usage)</li>
             <li><code>--json</code> (emit a single JSON object)</li>
             <li><code>--verbose</code> (progress + which extractor was used)</li>
             <li><code>--metrics off|on|detailed</code></li>
           </ul>
 
           <div class="note">
-            Plain-text mode: <code>--firecrawl off --markdown off</code>.
+            Plain-text mode: <code>--extract --format text</code>.
           </div>
         </article>
       </section>

package/docs/site/docs/youtube.html

@@ -31,7 +31,7 @@
           <a href="./index.html">Overview</a>
           <a href="./website.html">Website mode</a>
           <a href="./youtube.html">YouTube mode</a>
-          <a href="./extract-only.html">Extract-only</a>
+          <a href="./extract-only.html">Extract</a>
           <a href="./llm.html">LLM</a>
           <a href="./openai.html">OpenAI</a>
           <a href="./firecrawl.html">Firecrawl</a>
@@ -45,7 +45,7 @@
 
           <h2>Tip</h2>
           <ul>
-            <li>If you only want the transcript: use <code>--extract-only</code>.</li>
+            <li>If you only want the transcript: use <code>--extract</code>.</li>
             <li>For pipelines: add <code>--json</code>.</li>
           </ul>
         </article>

package/docs/site/index.html

@@ -104,7 +104,7 @@
         </div>
         <div class="card reveal">
           <h2>Built for pipelines</h2>
-          <p><code>--extract-only</code>, <code>--json</code>, and <code>--metrics</code> make it scriptable.</p>
+          <p><code>--extract</code>, <code>--json</code>, and <code>--metrics</code> make it scriptable.</p>
           <div class="small">Compose it with your own tools</div>
         </div>
         <div class="card reveal">

package/docs/website.md

@@ -7,21 +7,24 @@ Use this for non-YouTube URLs.
 - Fetches the page HTML.
 - Extracts “article-ish” content and normalizes it into clean text.
 - If extraction looks blocked or too thin, it can retry via Firecrawl (Markdown).
-- In `--extract-only` mode, Firecrawl is only used when HTML extraction looks blocked/thin (or when forced via `--firecrawl always`).
-- In `--extract-only` mode, `--markdown auto|llm` can convert HTML → Markdown via an LLM using the configured `--model` (no provider fallback).
+- With `--format md`, the CLI prefers Firecrawl Markdown by default when `FIRECRAWL_API_KEY` is configured.
+- With `--format md`, `--markdown-mode auto|llm` can also convert HTML → Markdown via an LLM using the configured `--model` (no provider fallback).
+- With `--format md`, `--markdown-mode auto` may fall back to `uvx markitdown` when available (disable with `--preprocess off`).
 
 ## Flags
 
 - `--firecrawl off|auto|always`
-- `--markdown off|auto|llm` (default: `auto`; only affects `--extract-only` for non-YouTube URLs)
-- Plain-text mode: use `--firecrawl off --markdown off`.
+- `--format md|text` (default: `text`)
+- `--markdown-mode off|auto|llm` (default: `auto`; only affects `--format md` for non-YouTube URLs)
+- `--preprocess off|auto|always` (default: `auto`; controls markitdown usage; `always` only affects file inputs)
+- Plain-text mode: use `--format text`.
 - `--timeout 30s|30|2m|5000ms` (default: `2m`)
-- `--extract-only` (print extracted content; no summary LLM call)
+- `--extract` (print extracted content; no summary LLM call)
 - `--json` (emit a single JSON object)
 - `--verbose` (progress + which extractor was used)
 - `--metrics off|on|detailed` (default: `on`; `detailed` prints a breakdown to stderr)
 
 ## API keys
 
-- Optional: `FIRECRAWL_API_KEY` (for the Firecrawl fallback)
-- Optional: `XAI_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY` (also accepts `GOOGLE_GENERATIVE_AI_API_KEY` / `GOOGLE_API_KEY`) (required only when `--markdown llm` is used, or when `--markdown auto` falls back to LLM conversion)
+- Optional: `FIRECRAWL_API_KEY` (for the Firecrawl fallback / preferred Markdown output)
+- Optional: `XAI_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY` (also accepts `GOOGLE_GENERATIVE_AI_API_KEY` / `GOOGLE_API_KEY`) (required only when `--markdown-mode llm` is used, or when `--markdown-mode auto` falls back to LLM conversion)

package/docs/youtube.md

@@ -2,11 +2,12 @@
 
 YouTube URLs use transcript-first extraction.
 
-## `--youtube auto|web|apify`
+## `--youtube auto|web|apify|yt-dlp`
 
-- `auto` (default): try `youtubei` → `captionTracks` → Apify (if token exists)
+- `auto` (default): try `youtubei` → `captionTracks` → Apify (if token exists) → `yt-dlp` (if configured)
 - `web`: try `youtubei` → `captionTracks` only
 - `apify`: Apify only
+- `yt-dlp`: download audio + transcribe (OpenAI Whisper preferred; FAL fallback)
 
 ## `youtubei` vs `captionTracks`
 
@@ -24,9 +25,11 @@ YouTube URLs use transcript-first extraction.
 - If no transcript is available, we still extract `ytInitialPlayerResponse.videoDetails.shortDescription` so YouTube links can still summarize meaningfully.
 - Apify is an optional fallback (needs `APIFY_API_TOKEN`).
   - By default, we use the actor id `faVsWy9VTSNVIhWpR` (Pinto Studio’s “Youtube Transcript Scraper”).
+- `yt-dlp` requires `YT_DLP_PATH` and either `OPENAI_API_KEY` (preferred) or `FAL_KEY`.
+  - If OpenAI transcription fails and `FAL_KEY` is set, we fall back to FAL automatically.
 
 ## Example
 
 ```bash
-pnpm summarize -- --extract-only "https://www.youtube.com/watch?v=I845O57ZSy4&t=11s"
+pnpm summarize -- --extract "https://www.youtube.com/watch?v=I845O57ZSy4&t=11s"
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@steipete/summarize",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "description": "Link → clean text → summary.",
   "type": "module",
   "bin": {
@@ -30,7 +30,11 @@
     "README.md",
     "LICENSE"
   ],
+  "engines": {
+    "node": ">=22"
+  },
   "dependencies": {
+    "@fal-ai/client": "^1.2.1",
     "cheerio": "^1.1.2",
     "es-toolkit": "^1.43.0",
     "gpt-tokenizer": "^3.4.0",