@steipete/summarize 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -1,10 +1,40 @@
 # Changelog
 
+## 0.4.0 - 2025-12-21
+
+### Changes
+
+- Add URL extraction mode via `--extract` (deprecated alias: `--extract-only`) with `--format md|text`.
+- Rename HTML→Markdown conversion flag to `--markdown-mode` (deprecated alias: `--markdown`).
+- Add `--preprocess off|auto|always` and a `uvx markitdown` fallback for Markdown extraction and unsupported file attachments (when `--format md` is used).
+
+## 0.3.0 - 2025-12-20
+### Changes
+
+- Add yt-dlp audio transcription fallback for YouTube; prefer OpenAI Whisper with FAL fallback. Thanks @dougvk.
+- Add `--no-playlist` to yt-dlp downloads to avoid transcript mismatches.
+- Run yt-dlp after web + Apify in `--youtube auto`, and error early for missing keys in `--youtube yt-dlp`.
+- Require Node 22+.
+- Respect `OPENAI_BASE_URL` when set, even with OpenRouter keys.
+- Apply OpenRouter provider ordering headers to HTML→Markdown conversion.
+- Add OpenRouter configuration tests. Thanks @dougvk for the initial OpenRouter support.
+- Build and ship a Bun bytecode arm64 binary for Homebrew.
+
+### Tests
+
+- Add coverage for yt-dlp ordering, missing-key errors, and helper paths.
+- Add live coverage for yt-dlp transcript mode and missing-caption YouTube pages.
+
+### Dev
+
+- Add `Dockerfile.test` for containerized yt-dlp testing.
+
 ## 0.2.0 - 2025-12-20
 
 ### Changes
 
-- Remove map-reduce summarization; reject inputs that exceed the model’s context window.
+- Add native OpenRouter support via `OPENROUTER_API_KEY` with optional provider ordering (`OPENROUTER_PROVIDERS`).
+- Remove map-reduce summarization; reject inputs that exceed the model's context window.
 - Preflight text prompts with the GPT tokenizer and the model’s max input tokens.
 - Reject text files over 10 MB before tokenization.
 - Reject too-small numeric `--length` and `--max-output-tokens` values.
@@ -71,7 +101,7 @@ First public release.
   - `--max-output-tokens <count>` (optional hard cap)
   - `--timeout <duration>` (default `2m`)
   - `--stream auto|on|off`, `--render auto|md-live|md|plain`
-  - `--extract-only` (URLs only; no summary)
+  - `--extract` (URLs only; no summary; deprecated alias: `--extract-only`)
   - `--json` (structured output incl. input config, prompt, extracted content, LLM metadata, and metrics)
   - `--metrics off|on|detailed` (default `on`)
   - `--verbose`
@@ -80,7 +110,7 @@ First public release.
 
 - Websites: fetch + extract “article-ish” content + normalization for prompts.
 - Firecrawl fallback for blocked/thin sites (`--firecrawl off|auto|always`, via `FIRECRAWL_API_KEY`).
-- Markdown extraction for websites in `--extract-only` mode (`--markdown off|auto|llm`).
+- Markdown extraction for websites in `--extract` mode (`--format md|text`, `--markdown-mode off|auto|llm`).
 - YouTube (`--youtube auto|web|apify`):
   - best-effort transcript endpoints
   - optional Apify fallback (requires `APIFY_API_TOKEN`; single actor `faVsWy9VTSNVIhWpR`)

package/README.md

@@ -11,6 +11,8 @@ It streams output by default on TTY and renders Markdown to ANSI (via `markdansi
 
 ## Install
 
+Requires Node 22+.
+
 - npx (no install):
 
 ```bash
@@ -23,6 +25,8 @@ npx -y @steipete/summarize "https://example.com" --model google/gemini-3-flash-p
 brew install steipete/tap/summarize
 ```
 
+Apple Silicon only (arm64).
+
 ## Quickstart
 
 ```bash
@@ -108,7 +112,10 @@ npx -y @steipete/summarize <input> [flags]
 - `--max-output-tokens <count>`: hard cap for LLM output tokens (optional)
 - `--stream auto|on|off`: stream LLM output (`auto` = TTY only; disabled in `--json` mode)
 - `--render auto|md-live|md|plain`: Markdown rendering (`auto` = best default for TTY)
-- `--extract-only`: print extracted content and exit (no summary) — only for URLs
+- `--format md|text`: website/file content format (default `text`)
+- `--preprocess off|auto|always`: preprocess files (only with `--format md`) for model compatibility (default `auto`)
+- `--extract`: print extracted content and exit (no summary) — only for URLs
+  - Deprecated alias: `--extract-only`
 - `--json`: machine-readable output with diagnostics, prompt, `metrics`, and optional summary
 - `--verbose`: debug/diagnostics on stderr
 - `--metrics off|on|detailed`: metrics output (default `on`; `detailed` prints a breakdown to stderr)
@@ -118,14 +125,23 @@ npx -y @steipete/summarize <input> [flags]
 Non-YouTube URLs go through a “fetch → extract” pipeline. When the direct fetch/extraction is blocked or too thin, `--firecrawl auto` can fall back to Firecrawl (if configured).
 
 - `--firecrawl off|auto|always` (default `auto`)
-- `--markdown off|auto|llm` (default `auto`; only affects `--extract-only` for non-YouTube URLs)
-- Plain-text mode: use `--firecrawl off --markdown off`.
+- `--extract --format md|text` (default `text`)
+- `--markdown-mode off|auto|llm` (default `auto`; only affects `--format md` for non-YouTube URLs)
+- Plain-text mode: use `--format text`.
+
+## YouTube transcripts
+
+`--youtube auto` tries best-effort web transcript endpoints first. When captions aren't available, it falls back to:
 
-## YouTube transcripts (Apify fallback)
+1. **Apify** (if `APIFY_API_TOKEN` is set): Uses a scraping actor (`faVsWy9VTSNVIhWpR`)
+2. **yt-dlp + Whisper** (if `YT_DLP_PATH` is set): Downloads audio via yt-dlp, transcribes with OpenAI Whisper if `OPENAI_API_KEY` is set, otherwise falls back to FAL (`FAL_KEY`)
 
-`--youtube auto` tries best-effort web transcript endpoints first, then falls back to Apify *only if* `APIFY_API_TOKEN` is set.
+Environment variables for yt-dlp mode:
+- `YT_DLP_PATH` - path to yt-dlp binary
+- `OPENAI_API_KEY` - OpenAI Whisper transcription (preferred)
+- `FAL_KEY` - FAL AI Whisper fallback
 
-Apify uses a single actor (`faVsWy9VTSNVIhWpR`). It costs money but tends to be more reliable.
+Apify costs money but tends to be more reliable when captions exist.
 
 ## Configuration
 
@@ -160,13 +176,29 @@ Set the key matching your chosen `--model`:
 
 OpenRouter (OpenAI-compatible):
 
-- Set `OPENAI_BASE_URL=https://openrouter.ai/api/v1`
-- Prefer `OPENROUTER_API_KEY=...` (instead of reusing `OPENAI_API_KEY`)
-- Use OpenRouter models via the `openai/...` prefix, e.g. `--model openai/xiaomi/mimo-v2-flash:free`
+- Set `OPENROUTER_API_KEY=...` to route `openai/...` models through OpenRouter
+- Use OpenRouter models via the `openai/...` prefix, e.g. `--model openai/openai/gpt-oss-20b`
+- Optional: `OPENROUTER_PROVIDERS=...` to specify provider fallback order (e.g. `groq,google-vertex`)
+
+Example:
+
+```bash
+OPENROUTER_API_KEY=sk-or-... summarize "https://example.com" --model openai/openai/gpt-oss-20b
+```
+
+With provider ordering (falls back through providers in order):
+
+```bash
+OPENROUTER_API_KEY=sk-or-... OPENROUTER_PROVIDERS="groq,google-vertex" summarize "https://example.com"
+```
+
+Legacy: `OPENAI_BASE_URL=https://openrouter.ai/api/v1` with `OPENAI_API_KEY` also works.
 
 Optional services:
 
 - `FIRECRAWL_API_KEY` (website extraction fallback)
+- `YT_DLP_PATH` (path to yt-dlp binary for audio extraction)
+- `FAL_KEY` (FAL AI API key for audio transcription via Whisper)
 - `APIFY_API_TOKEN` (YouTube transcript fallback)
 
 ## Model limits