@steipete/summarize 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+## 0.1.0 - 2025-12-19
+
+First public release.
+
+### CLI
+
+- `summarize` CLI shipped via `@steipete/summarize` (plus optional library exports).
+- Inputs: URL, local file path, or remote file URL (PDFs/images/audio/video/text).
+- Automatic map-reduce for large inputs.
+- Streaming output by default on TTY, with Markdown → ANSI rendering (via `markdansi`).
+- Final “Finished in …” line: timing, token usage, cost estimate (when pricing is available), and service counts.
+- Flags:
+  - `--model <provider/model>` (default `google/gemini-3-flash-preview`)
+  - `--length short|medium|long|xl|xxl|<chars>` (guideline; no hard truncation)
+  - `--max-output-tokens <count>` (optional hard cap)
+  - `--timeout <duration>` (default `2m`)
+  - `--stream auto|on|off`, `--render auto|md-live|md|plain`
+  - `--extract-only` (URLs only; no summary)
+  - `--json` (structured output incl. input config, prompt, extracted content, LLM metadata, and metrics)
+  - `--metrics off|on|detailed` (default `on`)
+  - `--verbose`
+
+### Sources
+
+- Websites: fetch + extract “article-ish” content + normalization for prompts.
+- Firecrawl fallback for blocked/thin sites (`--firecrawl off|auto|always`, via `FIRECRAWL_API_KEY`).
+- Markdown extraction for websites in `--extract-only` mode (`--markdown off|auto|llm`).
+- YouTube (`--youtube auto|web|apify`):
+  - best-effort transcript endpoints
+  - optional Apify fallback (requires `APIFY_API_TOKEN`; single actor `faVsWy9VTSNVIhWpR`)
+- Files (remote or local): MIME sniffing + best-effort forwarding to the model.
+  - text-like inputs are inlined for provider compatibility
+
+### LLM providers
+
+- Direct-provider API keys (no gateway).
+- OpenAI-compatible base URL support (`OPENAI_BASE_URL`, `OPENROUTER_API_KEY`).
+- Model ids: `openai/...`, `anthropic/...`, `xai/...`, `google/...`.
+- Auto-handling of provider/model limitations (e.g. no streaming support → non-streaming call; unsupported media types → friendly error).
+
+### Pricing + limits
+
+- Token/cost estimates and model limits derived from LiteLLM’s model catalog, downloaded + cached under `~/.summarize/cache/`.
+
+### Quality
+
+- CI: lint, tests (coverage), and pack.
+- Tooling: Biome (lint/format) + Vitest (tests + coverage gate).

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Peter Steinberger
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,185 @@
+# Summarize 👉 Point at any URL or file. Get the gist.
+
+Fast CLI for summarizing *anything you can point at*:
+
+- Web pages (article extraction; Firecrawl fallback if sites block agents)
+- YouTube links (best-effort transcripts, optional Apify fallback)
+- Remote files (PDFs/images/audio/video via URL — downloaded and forwarded to the model)
+- Local files (PDFs/images/audio/video/text — forwarded or inlined; support depends on provider/model)
+
+It streams output by default on TTY and renders Markdown to ANSI (via `markdansi`). At the end it prints a single “Finished in …” line with timing, token usage, and a best-effort cost estimate (when pricing is available).
+
+## Install
+
+- npx (no install):
+
+```bash
+npx -y @steipete/summarize "https://example.com" --model google/gemini-3-flash-preview
+```
+
+- Homebrew (custom tap):
+
+```bash
+brew install steipete/tap/summarize
+```
+
+## Quickstart
+
+```bash
+summarize "https://example.com" --model google/gemini-3-flash-preview
+```
+
+Input can be a URL or a local file path:
+
+```bash
+npx -y @steipete/summarize "/path/to/file.pdf" --model google/gemini-3-flash-preview
+npx -y @steipete/summarize "/path/to/image.jpeg" --model google/gemini-3-flash-preview
+```
+
+Remote file URLs work the same (best-effort; the file is downloaded and passed to the model):
+
+```bash
+npx -y @steipete/summarize "https://example.com/report.pdf" --model google/gemini-3-flash-preview
+```
+
+YouTube (supports `youtube.com` and `youtu.be`):
+
+```bash
+npx -y @steipete/summarize "https://youtu.be/dQw4w9WgXcQ" --youtube auto
+```
+
+## What file types work?
+
+This is “best effort” and depends on what your selected model/provider accepts. In practice these usually work well:
+
+- `text/*` and common structured text (`.txt`, `.md`, `.json`, `.yaml`, `.xml`, …)  
+  - text-like files are **inlined into the prompt** (instead of attached as a file part) for better provider compatibility
+- PDFs: `application/pdf` (provider support varies; Google is the most reliable in this CLI right now)
+- Images: `image/jpeg`, `image/png`, `image/webp`, `image/gif`
+- Audio/Video: `audio/*`, `video/*` (when supported by the model)
+
+Notes:
+
+- If a provider rejects a media type, the CLI fails fast with a friendly message (no “mystery stack traces”).
+- xAI models currently don’t support attaching generic files (like PDFs) via the AI SDK; use a Google/OpenAI/Anthropic model for those.
+
+## Model ids
+
+Use “gateway-style” ids: `<provider>/<model>`.
+
+Examples:
+
+- `openai/gpt-5.2`
+- `anthropic/claude-opus-4-5`
+- `xai/grok-4-fast-non-reasoning`
+- `google/gemini-3-flash-preview`
+
+Note: some models/providers don’t support streaming or certain file media types. When that happens, the CLI prints a friendly error (or auto-disables streaming for that model when supported by the provider).
+
+## Output length
+
+`--length` controls *how much output we ask for* (guideline), not a hard truncation.
+
+```bash
+npx -y @steipete/summarize "https://example.com" --length long
+npx -y @steipete/summarize "https://example.com" --length 20k
+```
+
+- Presets: `short|medium|long|xl|xxl`
+- Character targets: `1500`, `20k`, `20000`
+- Optional hard cap: `--max-output-tokens <count>` (e.g. `2000`, `2k`)
+  - Provider/model APIs still enforce their own maximum output limits.
+
+## Common flags
+
+```bash
+npx -y @steipete/summarize <input> [flags]
+```
+
+- `--model <provider/model>`: which model to use (defaults to `google/gemini-3-flash-preview`)
+- `--timeout <duration>`: `30s`, `2m`, `5000ms` (default `2m`)
+- `--length short|medium|long|xl|xxl|<chars>`
+- `--max-output-tokens <count>`: hard cap for LLM output tokens (optional)
+- `--stream auto|on|off`: stream LLM output (`auto` = TTY only; disabled in `--json` mode)
+- `--render auto|md-live|md|plain`: Markdown rendering (`auto` = best default for TTY)
+- `--extract-only`: print extracted content and exit (no summary) — only for URLs
+- `--json`: machine-readable output with diagnostics, prompt, `metrics`, and optional summary
+- `--verbose`: debug/diagnostics on stderr
+- `--metrics off|on|detailed`: metrics output (default `on`; `detailed` prints a breakdown to stderr)
+
+## Website extraction (Firecrawl + Markdown)
+
+Non-YouTube URLs go through a “fetch → extract” pipeline. When the direct fetch/extraction is blocked or too thin, `--firecrawl auto` can fall back to Firecrawl (if configured).
+
+- `--firecrawl off|auto|always` (default `auto`)
+- `--markdown off|auto|llm` (default `auto`; only affects `--extract-only` for non-YouTube URLs)
+- Plain-text mode: use `--firecrawl off --markdown off`.
+
+## YouTube transcripts (Apify fallback)
+
+`--youtube auto` tries best-effort web transcript endpoints first, then falls back to Apify *only if* `APIFY_API_TOKEN` is set.
+
+Apify uses a single actor (`faVsWy9VTSNVIhWpR`). It costs money but tends to be more reliable.
+
+## Configuration
+
+Single config location:
+
+- `~/.summarize/config.json`
+
+Supported keys today:
+
+```json
+{
+  "model": "openai/gpt-5.2"
+}
+```
+
+Precedence:
+
+1) `--model`
+2) `SUMMARIZE_MODEL`
+3) `~/.summarize/config.json`
+4) default
+
+## Environment variables
+
+Set the key matching your chosen `--model`:
+
+- `OPENAI_API_KEY` (for `openai/...`)
+- `ANTHROPIC_API_KEY` (for `anthropic/...`)
+- `XAI_API_KEY` (for `xai/...`)
+- `GEMINI_API_KEY` (for `google/...`)  
+  - also accepts `GOOGLE_GENERATIVE_AI_API_KEY` and `GOOGLE_API_KEY` as aliases
+
+OpenRouter (OpenAI-compatible):
+
+- Set `OPENAI_BASE_URL=https://openrouter.ai/api/v1`
+- Prefer `OPENROUTER_API_KEY=...` (instead of reusing `OPENAI_API_KEY`)
+- Use OpenRouter models via the `openai/...` prefix, e.g. `--model openai/xiaomi/mimo-v2-flash:free`
+
+Optional services:
+
+- `FIRECRAWL_API_KEY` (website extraction fallback)
+- `APIFY_API_TOKEN` (YouTube transcript fallback)
+
+## Model limits
+
+The CLI uses the LiteLLM model catalog for model limits (like max output tokens):
+
+- Downloaded from: `https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json`
+- Cached at: `~/.summarize/cache/`
+
+## Library usage (optional)
+
+This package also exports a small library:
+
+- `@steipete/summarize/content`
+- `@steipete/summarize/prompts`
+
+## Development
+
+```bash
+pnpm install
+pnpm check
+```