@steipete/summarize 0.11.1 → 0.13.0

package/docs/llm.md

@@ -24,7 +24,10 @@ installed, auto mode can use local CLI models via `cli.enabled` or implicit auto
 - `OPENAI_API_KEY` (required for `openai/...` models)
 - `OPENAI_BASE_URL` (optional; OpenAI-compatible API endpoint, e.g. OpenRouter)
 - `OPENAI_USE_CHAT_COMPLETIONS` (optional; force OpenAI chat completions)
+- `NVIDIA_API_KEY` (required for `nvidia/...` models; alias: `NGC_API_KEY`)
+- `NVIDIA_BASE_URL` (optional; override NVIDIA OpenAI-compatible API endpoint; default: `https://integrate.api.nvidia.com/v1`)
 - `OPENROUTER_API_KEY` (optional; required for `openrouter/...` models; also used when `OPENAI_BASE_URL` points to OpenRouter)
+- `GITHUB_TOKEN` / `GH_TOKEN` (required for `github-copilot/...` models via GitHub Models)
 - `Z_AI_API_KEY` (required for `zai/...` models; supports `ZAI_API_KEY` alias)
 - `Z_AI_BASE_URL` (optional; override default Z.AI base URL)
 - `GEMINI_API_KEY` (required for `google/...` models; also accepts `GOOGLE_GENERATIVE_AI_API_KEY` / `GOOGLE_API_KEY`)
@@ -32,7 +35,7 @@ installed, auto mode can use local CLI models via `cli.enabled` or implicit auto
 - `ANTHROPIC_API_KEY` (required for `anthropic/...` models)
 - `ANTHROPIC_BASE_URL` (optional; override Anthropic API endpoint)
 - `SUMMARIZE_MODEL` (optional; overrides default model selection)
-- `CLAUDE_PATH` / `CODEX_PATH` / `GEMINI_PATH` / `AGENT_PATH` (optional; override CLI binary paths)
+- `CLAUDE_PATH` / `CODEX_PATH` / `GEMINI_PATH` / `AGENT_PATH` / `OPENCLAW_PATH` / `OPENCODE_PATH` (optional; override CLI binary paths)
 
 ## Flags
 
@@ -40,17 +43,25 @@ installed, auto mode can use local CLI models via `cli.enabled` or implicit auto
   - Examples:
     - `cli/codex/gpt-5.2`
     - `cli/claude/sonnet`
-    - `cli/gemini/gemini-3-flash-preview`
+    - `cli/gemini/gemini-3-flash`
     - `cli/agent/gpt-5.2`
-    - `google/gemini-3-flash-preview`
+    - `cli/openclaw/main`
+    - `cli/opencode/openai/gpt-5.4`
+    - `openai/gpt-5.4`
+    - `openai/gpt-5.4-mini`
+    - `openai/gpt-5.4-nano`
+    - `google/gemini-3-flash`
     - `openai/gpt-5-mini`
+    - `openai/gpt-5-nano`
+    - `github-copilot/gpt-5.4`
+    - `nvidia/z-ai/glm5`
     - `zai/glm-4.7`
     - `xai/grok-4-fast-non-reasoning`
     - `google/gemini-2.0-flash`
     - `anthropic/claude-sonnet-4-5`
     - `openrouter/meta-llama/llama-3.3-70b-instruct:free` (force OpenRouter)
 - `--cli [provider]`
-  - Examples: `--cli claude`, `--cli Gemini`, `--cli codex`, `--cli agent` (equivalent to `--model cli/<provider>`); `--cli` alone uses auto selection with CLI enabled.
+  - Examples: `--cli claude`, `--cli Gemini`, `--cli codex`, `--cli agent`, `--cli openclaw`, `--cli opencode` (equivalent to `--model cli/<provider>`); `--cli` alone uses auto selection with CLI enabled.
 - `--model auto`
   - See `docs/model-auto.md`
 - `--model <preset>`
@@ -70,7 +81,8 @@ installed, auto mode can use local CLI models via `cli.enabled` or implicit auto
 - `--length short|medium|long|xl|xxl|<chars>`
   - This is _soft guidance_ to the model (no hard truncation).
   - Minimum numeric value: 50 chars.
-  - Default: `long`.
+  - Built-in default: `xl`.
+  - Config default: `output.length` in `~/.summarize/config.json`.
   - Output format is Markdown; use short paragraphs and only add bullets when they improve scanability.
 - `--force-summary`
   - Always run the LLM even when extracted content is shorter than the requested length.
@@ -95,6 +107,23 @@ installed, auto mode can use local CLI models via `cli.enabled` or implicit auto
 
 Use `--model zai/<model>` (e.g. `zai/glm-4.7`). Defaults to Z.AI’s base URL and uses chat completions.
 
+## GitHub Copilot / GitHub Models
+
+Use `--model github-copilot/<model>` for explicit GitHub-hosted model calls.
+
+- Examples:
+  - `github-copilot/gpt-5.4`
+  - `github-copilot/gpt-5.4-mini`
+  - `github-copilot/gpt-5.4-nano`
+  - `github-copilot/gpt-5-mini`
+  - `github-copilot/gpt-5-nano`
+  - `github-copilot/anthropic/claude-haiku-4.5`
+- Auth: `GITHUB_TOKEN` or `GH_TOKEN`
+- Transport: GitHub Models chat completions (`https://models.github.ai/inference`)
+- Notes:
+  - bare shorthand like `github-copilot/gpt-5.4` or `github-copilot/claude-opus-4.6` auto-expands to the provider-qualified backend id
+  - document attachments stay unsupported in this mode
+
 ## Input limits
 
 - Text prompts are checked against the model’s max input tokens (LiteLLM catalog) using a GPT tokenizer.

package/docs/manual-tests.md

@@ -11,6 +11,7 @@ Goal: sanity-check auto selection + presets end-to-end.
 ## Setup
 
 - `OPENAI_API_KEY=...` (optional)
+- `ASSEMBLYAI_API_KEY=...` (optional)
 - `GEMINI_API_KEY=...` (optional)
 - `ANTHROPIC_API_KEY=...` (optional)
 - `XAI_API_KEY=...` (optional)
@@ -27,6 +28,8 @@ Tip: use `--verbose` to see model attempts + the chosen model.
   - `summarize --max-output-tokens 99999 https://example.com`
 - Missing-key skip (configure only one key; should skip other providers, still succeed):
   - Set only `OPENAI_API_KEY`, then run a website summary; should not try Gemini/Anthropic/XAI.
+- AssemblyAI transcript path:
+  - Set only `ASSEMBLYAI_API_KEY`, then run a podcast URL or `--youtube yt-dlp` flow; `transcriptionProvider` should report `assemblyai`.
 - Podcast URL (Whisper): should show “Downloading audio …” then “Transcribing …” with duration when known.
   - `summarize https://podcasts.apple.com/us/podcast/2424-jelly-roll/id360084272?i=1000740717432 --metrics detailed`
 

package/docs/media.md

@@ -14,7 +14,7 @@ read_when:
 ## Transcript resolution order
 
 1. Embedded captions (VTT/JSON) when available.
-2. yt-dlp download + Whisper transcription (Groq first; then ONNX/local/OpenAI/FAL fallback).
+2. yt-dlp download + transcription (Groq first; then ONNX/local whisper.cpp; then AssemblyAI/Gemini/OpenAI/FAL fallback).
 
 ## CLI behavior
 
@@ -25,6 +25,14 @@ read_when:
 - X/Twitter status URLs with detected video auto-switch to transcript-first (yt-dlp), even in auto mode.
 - X broadcasts (`/i/broadcasts/...`) are treated as media-only and go transcript-first by default.
 - Local media files are capped at 2 GB; remote media URLs are best-effort via yt-dlp (no explicit size limit).
+- Remote transcription providers: `ASSEMBLYAI_API_KEY`, `GEMINI_API_KEY` / `GOOGLE_GENERATIVE_AI_API_KEY` / `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `FAL_KEY` (plus `GROQ_API_KEY` before local/remote fallback).
+- Gemini uses the Files API automatically for larger uploads.
+
+## Shared helpers
+
+- Direct media classification lives in `packages/core/src/content/direct-media.ts`.
+- Local path/`file://` normalization + mtime lookup lives in `packages/core/src/content/local-file.ts`.
+- Slides, URL extraction, and transcription should reuse those helpers instead of re-parsing extensions separately.
 
 ## Chrome extension behavior
 

package/docs/model-auto.md

@@ -24,7 +24,7 @@ This is also the built-in default when you don’t specify a model.
 
 Model ids:
 
-- Native: `<provider>/<model>` (e.g. `openai/gpt-5-mini`, `google/gemini-3-flash-preview`)
+- Native: `<provider>/<model>` (e.g. `openai/gpt-5-mini`, `google/gemini-3-flash`)
 - Forced OpenRouter: `openrouter/<author>/<slug>` (e.g. `openrouter/meta-llama/llama-3.3-70b-instruct:free`)
 
 Behavior:
@@ -68,7 +68,7 @@ Example:
     "rules": [
       {
         "when": ["video"],
-        "candidates": ["google/gemini-3-flash-preview"]
+        "candidates": ["google/gemini-3-flash"]
       },
       {
         "when": ["website", "youtube"],

package/docs/model-provider-resolution.md

@@ -0,0 +1,57 @@
+---
+summary: "Map of model auto-selection, provider capabilities, and LLM execution paths."
+read_when:
+  - "When changing auto model order or CLI fallback behavior."
+  - "When adding provider capabilities or changing document/streaming support."
+---
+
+# Model / Provider Resolution
+
+Goal: reduce implicit provider knowledge.
+
+## Shared capability registry
+
+- `src/llm/provider-capabilities.ts`
+  Source of truth for:
+  - required env per provider
+  - CLI default models
+  - auto CLI order
+  - document support
+  - streaming support
+
+If a provider rule changes, update this file first.
+
+## Auto model selection
+
+- `src/model-auto.ts`
+  Responsibilities:
+  - resolve auto rules
+  - prepend CLI candidates
+  - map native candidates to OpenRouter when safe
+  - emit attempts with required env + transport
+
+Keep it selection-focused.
+Do not add provider-specific capability branches there unless the registry cannot express them.
+
+## Execution
+
+- `src/llm/generate-text.ts`
+  Responsibilities:
+  - parse requested model id
+  - validate input shape
+  - route to provider transport
+  - normalize retries / fallbacks
+
+Provider-specific SDK/http details belong under `src/llm/providers/*`.
+
+## Error shaping
+
+- access / model availability normalization stays provider-local when truly provider-specific
+- generic capability errors should come from the shared registry
+- unsupported functionality errors should be thrown before transport setup
+
+## Rules
+
+- add capability once; consume it in `model-auto` and `generate-text`
+- keep provider env alias handling centralized
+- keep default CLI model changes in the registry, not scattered tests/constants

package/docs/releasing.md

@@ -1,5 +1,5 @@
 ---
-summary: "Release checklist + Homebrew tap update."
+summary: "Release checklist + Homebrew/core verify step."
 ---
 
 # Releasing
@@ -8,22 +8,19 @@ summary: "Release checklist + Homebrew tap update."
 
 - Ship npm packages (core first, then CLI).
 - Tag + GitHub release.
-- Update Homebrew tap so `brew install steipete/tap/summarize` matches latest tag.
+- Verify the Homebrew/core formula so `brew install summarize` matches the latest tag.
 
 ## Checklist
 
-1. `scripts/release.sh all` (gates → build → verify → publish → smoke → tag → tap).
+1. `scripts/release.sh all` (gates → build → verify → publish → smoke → tag).
 2. Create GitHub release for the new tag (match version, attach notes/assets as needed).
-3. If you didn’t run `tap` in the script, update the Homebrew tap formula for `summarize`:
-   - Bump version to the new tag.
-   - Update tarball URL + SHA256 for the new release.
-4. Verify Homebrew install reflects the new version:
-   - `brew install steipete/tap/summarize`
+3. Verify the Homebrew/core formula reflects the new version:
+   - `brew install summarize`
    - `summarize --version` matches tag.
-   - Run a feature added in the release (e.g. `summarize daemon install` for v0.8.2).
-5. If anything fails, fix and re-cut the release (no partials).
+   - Run a feature added in the release (for example `summarize daemon install`).
+4. If anything fails, fix and re-cut the release (no partials).
 
 ## Common failure
 
-- NPM/GitHub release updated, tap not updated → users stuck on old version.
-  Fix: always do step 3–4 before announcing.
+- NPM/GitHub release updated, but Homebrew/core still serves the old version.
+  Fix: always do step 3 before announcing.

package/docs/site/docs/chrome-extension.html

@@ -54,7 +54,7 @@
           <ul>
             <li>
               <code>npm i -g @steipete/summarize</code> or
-              <code>brew install steipete/tap/summarize</code>
+              <code>brew install summarize</code>
             </li>
             <li>Build extension: <code>pnpm -C apps/chrome-extension build</code></li>
             <li>Load unpacked: <code>apps/chrome-extension/.output/chrome-mv3</code></li>

package/docs/site/index.html

@@ -137,6 +137,11 @@ summarize "/path/report.pdf" --model google/gemini-3-flash-preview</code></pre>
           </div>
           <div class="ctaRow">
             <a class="btn btn--ghost" href="./docs/chrome-extension.html">Extension docs</a>
+            <a
+              class="btn"
+              href="https://chromewebstore.google.com/detail/summarize/cejgnmmhbbpdmjnfppjdfkocebngehfg"
+              >Chrome Web Store</a
+            >
             <a class="btn" href="https://github.com/steipete/summarize/releases/latest"
               >Latest release</a
             >

package/docs/slides-rendering-flow.md

@@ -0,0 +1,46 @@
+---
+summary: "Map of slide extraction terminal output and extension streaming flow."
+read_when:
+  - "When changing slide terminal output, streaming, or sidepanel state."
+  - "When debugging slide rendering regressions in CLI or extension."
+---
+
+# Slides Rendering Flow
+
+Two main paths.
+
+## CLI / terminal
+
+- `src/run/flows/url/slides-output.ts`
+  Public construction + orchestration only.
+- `slides-output-state.ts`
+  Slide timeline state.
+  Waiters.
+  Finalization.
+- `slides-output-render.ts`
+  Terminal rendering.
+  Inline-image policy.
+  Debug path.
+- `slides-output-stream.ts`
+  Summary-stream parsing glue.
+
+Rule: keep terminal I/O in render helpers; keep state mutations in the state store.
+
+## Chrome extension
+
+- `apps/chrome-extension/src/entrypoints/sidepanel/stream-controller.ts`
+  Transport lifecycle.
+- `stream-controller-policy.ts`
+  Chunk/status terminal state rules.
+- `apps/chrome-extension/src/lib/extension-logs.ts`
+  Storage queue + flush only.
+- `extension-log-format.ts`
+  Pure log serialization/truncation.
+
+Rule: push pure status/log logic out of adapters first.
+
+## When debugging
+
+1. Check state transitions before DOM issues.
+2. Check stream policy before transport retry logic.
+3. Check cache/hydration helpers before blaming rendering.

package/docs/slides.md

@@ -44,11 +44,11 @@ read_when:
 
 ## UI behavior
 
-- Default summary stays unchanged (no slide text).
-- Slide strip (compact) stays horizontal; no modal required.
-- Expand toggle switches to vertical full-width list:
-  - cards: thumbnail, timestamp, text.
-  - cards appear above summary.
+- Side panel slide mode is slide-first:
+  - vertical full-width cards by default
+  - thumbnail + timestamp + text
+  - transcript/OCR text appears before slide images finish extracting
+- No giant summary block under active slide cards.
 - Slide click: seek only (no modal).
 - OCR toggle appears near summarize control only when OCR is significant
   (enough slides + total OCR chars); otherwise hide it.

package/docs/smoketest.md

@@ -16,7 +16,7 @@ Goal: exercise URL + file inputs, extraction + LLM summary paths, multiple model
 ## Models (cheap/fast)
 
 - `openai/gpt-5-mini`
-- `google/gemini-3-flash-preview`
+- `google/gemini-3-flash`
 
 ## Matrix (20 cases)
 

package/docs/transcript-provider-flow.md

@@ -0,0 +1,73 @@
+---
+summary: "Map of transcript provider selection and transcription fallback flow."
+read_when:
+  - "When changing podcast, YouTube, or generic transcript provider order."
+  - "When changing remote transcription fallbacks or provider setup errors."
+---
+
+# Transcript Provider Flow
+
+Goal: keep provider entrypoints thin; keep provider policy explicit.
+
+## Provider entrypoints
+
+- `packages/core/src/content/transcript/providers/youtube.ts`
+  YouTube orchestration only.
+  Web captions first.
+  `yt-dlp` or Apify fallback next.
+- `packages/core/src/content/transcript/providers/podcast.ts`
+  Podcast orchestration only.
+  Feed/Spotify/Apple/enclosure/`yt-dlp` chain.
+- `packages/core/src/content/transcript/providers/generic.ts`
+  Thin orchestration only.
+  Embedded tracks first.
+  Direct-media / X media fallback next.
+- `packages/core/src/content/transcript/providers/generic-embedded.ts`
+  Embedded media detection + caption-track parsing only.
+- `packages/core/src/content/transcript/providers/generic-direct-media.ts`
+  Direct-media yt-dlp/transcription fallback only.
+- `packages/core/src/content/transcript/providers/generic-twitter.ts`
+  X/Twitter cookies + yt-dlp orchestration only.
+
+## Shared policy
+
+- `transcription-capability.ts`
+  One place for:
+  - `resolveTranscriptProviderCapabilities`
+  - `canTranscribe`
+  - `canRunYtDlp`
+  - missing-provider note/result shaping
+- `transcription-start.ts`
+  Runtime availability only.
+  Local whisper, ONNX, cloud presence, display hints.
+
+## Remote fallback
+
+- `packages/core/src/transcription/whisper/cloud-providers.ts`
+  Provider order + labels + model-id chain.
+- `packages/core/src/transcription/whisper/remote-provider-attempts.ts`
+  Per-provider byte/file attempts.
+- `packages/core/src/transcription/whisper/remote.ts`
+  Order loop only.
+  Fallback notes.
+  OpenAI chunk/delegate policy.
+
+## Current order
+
+- local ONNX / whisper.cpp before cloud
+- cloud bytes/file order:
+  - AssemblyAI
+  - Gemini
+  - OpenAI
+  - FAL
+
+## Rules
+
+- keep entrypoints thin
+- add provider notes in shared helpers, not scattered strings
+- prefer pure parser helpers before touching orchestration
+- if adding a new provider:
+  - register cloud metadata
+  - add remote attempt handler
+  - widen shared capability helper
+  - add focused provider tests before live tests

package/docs/website.md

@@ -13,10 +13,11 @@ Use this for non-YouTube URLs.
 - Fetches the page HTML.
 - Extracts “article-ish” content and normalizes it into clean text.
 - If extraction looks blocked or too thin, it can retry via Firecrawl (Markdown).
+  - Firecrawl applies to non-YouTube URLs only; YouTube URLs use the `--youtube` transcript path instead.
 - If a page is effectively “video-only”, it may treat it as a video input (see `--video-mode`).
 - `--video-mode transcript` prefers embedded media transcripts on pages with audio/video (captions → yt-dlp/Whisper fallback).
 - With `--format md`, the CLI defaults to `--markdown-mode readability` (Readability article HTML as the Markdown input).
-  - Use `--firecrawl always` to try Firecrawl first.
+  - Use `--firecrawl always` to try Firecrawl first for non-YouTube URLs.
 - With `--format md`, `--markdown-mode auto|llm|readability` can also convert HTML → Markdown via an LLM using the configured `--model` (no provider fallback).
 - With `--format md`, `--markdown-mode auto` may fall back to `uvx markitdown` when available (disable with `--preprocess off`).
 - For podcast URLs (Apple Podcasts, RSS, Spotify episodes), it downloads the episode audio and transcribes via Whisper (prefers local `whisper.cpp` when installed + model available); progress shows “Downloading audio …” then “Transcribing …” (duration uses RSS hints or `ffprobe` when available).
@@ -33,6 +34,7 @@ Daemon note:
 ## Twitter/X
 
 - Tweet audio transcription runs only with `--video-mode transcript` (auto mode skips yt-dlp for tweets).
+- Tweet text extraction prefers `xurl` when installed, falls back to `bird`, then Nitter.
 
 ## Flags
 

package/docs/youtube.md

@@ -14,7 +14,7 @@ YouTube URLs use transcript-first extraction.
 - `web`: try `youtubei` → `captionTracks` only
 - `no-auto`: try creator captions only (skip auto-generated/ASR) → `yt-dlp` (if configured)
 - `apify`: Apify only
-- `yt-dlp`: download audio + transcribe (local `whisper.cpp` preferred; OpenAI/FAL fallback)
+- `yt-dlp`: download audio + transcribe (Groq first; then local `whisper.cpp`; then AssemblyAI/Gemini/OpenAI/FAL fallback)
 
 ## `youtubei` vs `captionTracks`
 
@@ -32,7 +32,9 @@ YouTube URLs use transcript-first extraction.
 - If no transcript is available, we still extract `ytInitialPlayerResponse.videoDetails.shortDescription` so YouTube links can still summarize meaningfully.
 - Apify is an optional fallback (needs `APIFY_API_TOKEN`).
   - By default, we use the actor id `faVsWy9VTSNVIhWpR` (Pinto Studio’s “Youtube Transcript Scraper”).
-- `yt-dlp` requires the `yt-dlp` binary (either set `YT_DLP_PATH` or have it on `PATH`) and either local `whisper.cpp` (preferred) or `OPENAI_API_KEY` / `FAL_KEY`.
+- `yt-dlp` requires the `yt-dlp` binary (either set `YT_DLP_PATH` or have it on `PATH`) and either local `whisper.cpp` or one of `GROQ_API_KEY`, `ASSEMBLYAI_API_KEY`, `GEMINI_API_KEY`, `OPENAI_API_KEY`, or `FAL_KEY`.
+  - AssemblyAI is supported as a dedicated remote transcription provider in the fallback chain.
+  - Gemini is used automatically when available after AssemblyAI/local providers, and handles larger uploads via the Files API.
   - If OpenAI transcription fails and `FAL_KEY` is set, we fall back to FAL automatically.
 
 ## Example

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@steipete/summarize",
-  "version": "0.11.1",
+  "version": "0.13.0",
   "description": "Link → clean text → summary.",
   "bin": {
     "summarize": "./dist/cli.js",
@@ -32,9 +32,9 @@
     }
   },
   "dependencies": {
-    "@mariozechner/pi-ai": "^0.52.12",
+    "@mariozechner/pi-ai": "^0.65.2",
     "commander": "^14.0.3",
-    "file-type": "^21.3.0",
+    "file-type": "^22.0.0",
     "gpt-tokenizer": "^3.4.0",
     "json5": "^2.2.3",
     "markdansi": "^0.2.1",
@@ -43,22 +43,22 @@
     "osc-progress": "^0.3.0",
     "tokentally": "^0.1.1",
     "tslog": "^4.10.2",
-    "@steipete/summarize-core": "0.11.1"
+    "@steipete/summarize-core": "0.13.0"
   },
   "devDependencies": {
-    "@fal-ai/client": "^1.9.1",
-    "@types/jsdom": "^27.0.0",
-    "@types/node": "^25.2.3",
-    "@types/sanitize-html": "^2.16.0",
-    "@vitest/coverage-v8": "^4.0.18",
-    "esbuild": "^0.27.3",
-    "oxfmt": "0.32.0",
-    "oxlint": "^1.47.0",
-    "oxlint-tsgolint": "^0.12.2",
-    "rimraf": "^6.1.2",
+    "@fal-ai/client": "^1.9.5",
+    "@types/jsdom": "^28.0.1",
+    "@types/node": "^25.5.2",
+    "@types/sanitize-html": "^2.16.1",
+    "@vitest/coverage-v8": "^4.1.2",
+    "esbuild": "^0.28.0",
+    "oxfmt": "0.44.0",
+    "oxlint": "^1.59.0",
+    "oxlint-tsgolint": "^0.20.0",
+    "rimraf": "^6.1.3",
     "tsx": "^4.21.0",
-    "typescript": "^5.9.3",
-    "vitest": "^4.0.18"
+    "typescript": "^6.0.2",
+    "vitest": "^4.1.2"
   },
   "engines": {
     "node": ">=22"
@@ -71,6 +71,7 @@
     "build:lib": "tsc -p tsconfig.build.json",
     "check": "pnpm format:check && pnpm lint && pnpm test:coverage",
     "clean": "rimraf dist packages/core/dist",
+    "dev:cli": "pnpm -C packages/core build && tsx src/cli.ts",
     "docs:list": "tsx scripts/docs-list.ts",
     "format": "oxfmt --write",
     "format:check": "oxfmt --check",