@steipete/summarize 0.1.2 → 0.3.0

package/CHANGELOG.md

@@ -1,12 +1,75 @@
 # Changelog
 
-All notable changes to this project are documented here.
+## 0.3.0 - 2025-12-20
 
-## 0.1.2 - 2025-12-20
+### Changes
+
+- Add yt-dlp audio transcription fallback for YouTube; prefer OpenAI Whisper with FAL fallback. Thanks @dougvk.
+- Add `--no-playlist` to yt-dlp downloads to avoid transcript mismatches.
+- Run yt-dlp after web + Apify in `--youtube auto`, and error early for missing keys in `--youtube yt-dlp`.
+- Require Node 22+.
+- Respect `OPENAI_BASE_URL` when set, even with OpenRouter keys.
+- Apply OpenRouter provider ordering headers to HTML→Markdown conversion.
+- Add OpenRouter configuration tests. Thanks @dougvk for the initial OpenRouter support.
+- Build and ship a Bun bytecode arm64 binary for Homebrew.
+
+### Tests
+
+- Add coverage for yt-dlp ordering, missing-key errors, and helper paths.
+- Add live coverage for yt-dlp transcript mode and missing-caption YouTube pages.
+
+### Dev
+
+- Add `Dockerfile.test` for containerized yt-dlp testing.
+
+## 0.2.0 - 2025-12-20
+
+### Changes
+
+- Add native OpenRouter support via `OPENROUTER_API_KEY` with optional provider ordering (`OPENROUTER_PROVIDERS`).
+- Remove map-reduce summarization; reject inputs that exceed the model's context window.
+- Preflight text prompts with the GPT tokenizer and the model’s max input tokens.
+- Reject text files over 10 MB before tokenization.
+- Reject too-small numeric `--length` and `--max-output-tokens` values.
+- Cap summaries to the extracted content length when a requested size is larger.
+- Skip summarization for tweets when extracted content is already below the requested length.
+- Use bird CLI for tweet extraction when available and surface it in the status line.
+- Fall back to Nitter for tweet extraction when bird fails; report a clear error when tweet data is unavailable.
+- Compute cost totals via tokentally’s tally helpers.
+- Improve fetch spinner with elapsed time and throughput updates.
+- Show Firecrawl fallback status and reason when scraping kicks in.
+- Enforce a hard deadline for stalled streaming LLM responses.
+- Merge cumulative streaming chunks correctly and keep stream-merge for streaming output.
+- Fall back to non-streaming when streaming requests time out.
+- Preserve parentheses in URL paths when resolving inputs.
+- Stop forcing Firecrawl for --extract-only; only use it as a fallback.
+- Avoid Firecrawl fallback when block keywords only appear in scripts/styles.
+
+### Tests
+
+- Add CLI + live coverage for prompt length capping.
+- Add coverage for cumulative stream merge handling.
+- Add coverage for streaming timeout fallback.
+- Add live coverage for Wikipedia URLs with parentheses.
+- Add coverage for tweet summaries that bypass the LLM when short.
+- Add coverage for content budget paths and TOKENTALLY cache dir overrides.
+
+### Docs
+
+- Update release checklist to all-in-one flow.
+- Fix release script quoting.
+- Document input limits and minimum length/token values.
+
+### Dev
+
+- Add a tokenization benchmark script.
 
 ### Fixes
 
-- Avoid duplicate streamed output when providers emit cumulative chunks instead of deltas.
+- Preserve balanced parentheses/brackets in URL paths (e.g. Wikipedia titles).
+- Avoid Firecrawl fallback when block keywords only appear in scripts/styles.
+- Add a Bird install tip when Twitter/X fetch fails without bird installed.
+- Graceful error when tweet extraction fails after bird + Nitter fallback.
 
 ## 0.1.1 - 2025-12-19
 

package/README.md

@@ -11,6 +11,8 @@ It streams output by default on TTY and renders Markdown to ANSI (via `markdansi
 
 ## Install
 
+Requires Node 22+.
+
 - npx (no install):
 
 ```bash
@@ -23,6 +25,8 @@ npx -y @steipete/summarize "https://example.com" --model google/gemini-3-flash-p
 brew install steipete/tap/summarize
 ```
 
+Apple Silicon only (arm64).
+
 ## Quickstart
 
 ```bash
@@ -89,6 +93,12 @@ npx -y @steipete/summarize "https://example.com" --length 20k
 - Character targets: `1500`, `20k`, `20000`
 - Optional hard cap: `--max-output-tokens <count>` (e.g. `2000`, `2k`)
   - Provider/model APIs still enforce their own maximum output limits.
+- Minimums: `--length` numeric values must be ≥ 50 chars; `--max-output-tokens` must be ≥ 16.
+
+## Limits
+
+- Text inputs over 10 MB are rejected before tokenization.
+- Text prompts are preflighted against the model’s input limit (LiteLLM catalog), using a GPT tokenizer.
 
 ## Common flags
 
@@ -115,11 +125,19 @@ Non-YouTube URLs go through a “fetch → extract” pipeline. When the direct
 - `--markdown off|auto|llm` (default `auto`; only affects `--extract-only` for non-YouTube URLs)
 - Plain-text mode: use `--firecrawl off --markdown off`.
 
-## YouTube transcripts (Apify fallback)
+## YouTube transcripts
 
-`--youtube auto` tries best-effort web transcript endpoints first, then falls back to Apify *only if* `APIFY_API_TOKEN` is set.
+`--youtube auto` tries best-effort web transcript endpoints first. When captions aren't available, it falls back to:
 
-Apify uses a single actor (`faVsWy9VTSNVIhWpR`). It costs money but tends to be more reliable.
+1. **Apify** (if `APIFY_API_TOKEN` is set): Uses a scraping actor (`faVsWy9VTSNVIhWpR`)
+2. **yt-dlp + Whisper** (if `YT_DLP_PATH` is set): Downloads audio via yt-dlp, transcribes with OpenAI Whisper if `OPENAI_API_KEY` is set, otherwise falls back to FAL (`FAL_KEY`)
+
+Environment variables for yt-dlp mode:
+- `YT_DLP_PATH` - path to yt-dlp binary
+- `OPENAI_API_KEY` - OpenAI Whisper transcription (preferred)
+- `FAL_KEY` - FAL AI Whisper fallback
+
+Apify costs money but tends to be more reliable when captions exist.
 
 ## Configuration
 
@@ -154,13 +172,29 @@ Set the key matching your chosen `--model`:
 
 OpenRouter (OpenAI-compatible):
 
-- Set `OPENAI_BASE_URL=https://openrouter.ai/api/v1`
-- Prefer `OPENROUTER_API_KEY=...` (instead of reusing `OPENAI_API_KEY`)
-- Use OpenRouter models via the `openai/...` prefix, e.g. `--model openai/xiaomi/mimo-v2-flash:free`
+- Set `OPENROUTER_API_KEY=...` to route `openai/...` models through OpenRouter
+- Use OpenRouter models via the `openai/...` prefix, e.g. `--model openai/openai/gpt-oss-20b`
+- Optional: `OPENROUTER_PROVIDERS=...` to specify provider fallback order (e.g. `groq,google-vertex`)
+
+Example:
+
+```bash
+OPENROUTER_API_KEY=sk-or-... summarize "https://example.com" --model openai/openai/gpt-oss-20b
+```
+
+With provider ordering (falls back through providers in order):
+
+```bash
+OPENROUTER_API_KEY=sk-or-... OPENROUTER_PROVIDERS="groq,google-vertex" summarize "https://example.com"
+```
+
+Legacy: `OPENAI_BASE_URL=https://openrouter.ai/api/v1` with `OPENAI_API_KEY` also works.
 
 Optional services:
 
 - `FIRECRAWL_API_KEY` (website extraction fallback)
+- `YT_DLP_PATH` (path to yt-dlp binary for audio extraction)
+- `FAL_KEY` (FAL AI API key for audio transcription via Whisper)
 - `APIFY_API_TOKEN` (YouTube transcript fallback)
 
 ## Model limits