smartpipe-cli 1.4.0rc1__tar.gz → 1.4.0rc2__tar.gz

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/CHANGELOG.md

@@ -9,7 +9,68 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) · Versioning:
 
 The identity release.
 
+### Knowledge graphs (the highlight)
+- **`smartpipe graph --fast` turns a corpus into a knowledge graph for
+  exactly $0.** A local zero-shot NER model (GLiNER-small over ONNX -
+  no new dependencies, no torch, a one-time ~190 MB download like the
+  local embedder) finds YOUR entity types (`--entities "person, vessel,
+  account"`), entities co-occurring in a window become weighted edges,
+  near-duplicate names fold into one node (disclosed), and every edge
+  carries its `sources` - the files, pages, and minutes where it was
+  seen. Zero model calls, pinned by test. Free transcripts (local
+  whisper) mean audio and video speech join the graph; scans and
+  images wait for the paid mode, counted out loud.
+- **Six ways out, all deterministic**: `.graphml` (Gephi), `.dot`
+  (Graphviz), `.mmd` (Mermaid), `nodes.csv`+`edges.csv` (Neo4j/Kuzu),
+  a self-contained interactive `.html` (search, weight slider, hover
+  cards with clickable `file://` evidence links), or a trailing `/`
+  for an Obsidian vault of wikilinked entity notes.
+- **A focus prompt buys named relations.** `smartpipe graph "people,
+  orgs, and money flows" case/**/*` extracts typed triples per chunk
+  through the full multimodal ladders - vision, OCR, native video -
+  with `--entities`/`--relations` compiling to enums when you want a
+  strict ontology. The cost plan prints BEFORE any spend (`~810
+  extraction calls; belt is 500 - the graph will be partial`), a
+  terminal decline exits clean at zero calls, and a belted run drains
+  to a valid partial graph at exit 1, never a fake success.
+- **The hybrid ladder: `--name-top 200`.** The free pass finds the
+  candidates; the model names only the strongest links - one call per
+  edge, and if the belt runs out the remainder keeps its honest
+  `co-occurs` label, disclosed. Free graph in seconds; name the top
+  edges for cents.
+- **Bring your own edges.** Records with `{source, target}` or
+  `{subject, relation, object}` on stdin skip extraction entirely -
+  canonicalize, fold, serialize. Your own `extend` pipeline is the
+  power path.
+
 ### The item — one law for everything in a pipe
+- **Dates are a type, and they come back ISO.** `{due date}` and
+  `{ts datetime}` join the brace grammar. However the model phrases it
+  ("Jan 15, 2026", "15/01/2026"), the field canonicalizes to
+  `YYYY-MM-DD` / full ISO-8601 - offsets preserved, never invented;
+  genuinely ambiguous slash-dates read month-first and say so on
+  stderr. Downstream just works: `where 'due >= "2026-01-01"'`
+  compares temporally, `sort --by due` orders by instant, and
+  `summarize 'count() by bin(due, 1d)'` buckets date-only values.
+- **The model sees your data framed, not dumped.** Every item payload
+  now arrives inside an `<input>` fence - records rendered as clean
+  YAML-ish (the `__` plumbing excluded), text as itself. The
+  instruction can no longer blur into the data, and the fence is the
+  foundation the planned request-batching builds on.
+- **CSV is a first-class row format.** `--as csv` (and `.csv`/`.tsv`
+  extension defaults - tab dialect for `.tsv`) reads the header row as
+  field names and streams every data row as a record, with per-cell
+  int → float → string coercion and multi-line quoted cells handled. A
+  ragged row is a loud error naming the file and physical line; rows
+  carry `__source {"as": "csv", "line": N}`. Works on stdin too, and
+  `.sem` files gain the `as` key.
+- **Malformed model replies repair for free before they cost money.**
+  A structured reply that fails to parse now passes through a
+  deterministic fixer first - code fences, JSON-island extraction,
+  Python-repr literals, trailing commas, quote fixes, all validated by
+  an actual parse - and only escalates to the paid model-repair round
+  trip when that fails. Disclosed once per run:
+  `note: N replies repaired deterministically (fences/commas/quotes)`.
 - **The `__` metadata spine.** Tool metadata lives in a reserved
   double-underscore namespace that rides every record: `__source` (path plus
   HOW the item was cut - file, lines, jsonl, pages, minutes, segment index),
@@ -70,12 +131,182 @@ The identity release.
   model calls. Verb tables mark the free rungs.
 
 ### The terminal behaves
+- **A real progress bar.** When the total is knowable (file lists), the
+  status line becomes `[██████░░░░░░░░░] 41% · 205/500 · 12/s · ~25s
+  left` - fill and percent truncate so 100% is earned, the ETA appears
+  once the first item completes, ASCII terminals get `[=====>....]`,
+  and multi-stage runs prefix the stage name. Streams keep the
+  count-and-rate line: a `tail -f` has no percent.
+- **Braces extract lists of objects.** `{events {name string, when
+  date, severity enum(low, high)}[]}` pulls every event out of an item
+  in one call, typed and validated - inner dates still canonicalize to
+  ISO, `--explode events` turns the list into one row each. One level
+  deep by design; deeper nesting faults free at parse time with the
+  two-pass hint.
 - Result writes pause/erase/redraw the status line under an arbiter - the
   spinner can never interleave with output again. Only the final pipe stage
   (stdout a TTY) animates; mid-pipe stages stay line-atomic.
 - Records render at the TTY as YAML-ish blocks. The welcome screen links
   docs, cookbook, and issues. The wizard offers to install shell
   completions (consented, idempotent).
+- **`chart` re-platformed: plotext draws the terminal, matplotlib renders
+  `--save`.** TTY bars get color (cyan bars, green time-series); piped or
+  `NO_COLOR` output keeps the plain-ASCII contract byte-for-byte. `--save`
+  now writes SVG or PNG by extension (identity-styled, deterministic
+  output); svgwrite - unmaintained since 2022 - leaves the tree. Both
+  libraries ship in core per the no-extras rule; imports stay
+  function-local so startup holds. The run receipt now reads
+  `run: ↑40 ↓25 tok` (arrows for in/out). The wizard validates model names
+  before saving (two strikes, then a typed fault), prints completions
+  before the paste-bait "Try it" line, prefers a sensible local default
+  over alphabetical luck (cloud tags compete equally), and speaks in the
+  same color voice as the other commands with a menu of detected local
+  tags and provider-prefixed cloud examples.
+- **Docs bug, live-caught: `cluster | chart` drew flat bars.** The summary
+  rows are one-per-cluster, so charting them counts every theme once. The
+  README quick-tour step, the `cluster` help epilog, and the `.sem`
+  pipeline example now route `cluster --explode members` into
+  `chart cluster --top 8`, which draws real sizes.
+
+### Every example, run for real
+- **`map` and `extend` records finally carry their provenance.** The
+  `__source` spine - the law said it rides every record - was being
+  dropped by braces extraction: a reader-stamped row went in with
+  `{"path", "as", "line"}` and came out with only the extracted fields.
+  Every record leaving `map`/`extend` now carries `__source` (incoming
+  spines adopted verbatim, fresh ones stamped from the item's
+  provenance), which also means the terminal preview now shows the dim
+  file-and-line origin under each block. `--bare` still strips it.
+- **Docs truth sweep** (owner-reported): four `echo "... $1250"`
+  examples silently shell-expanded to "250" - single-quoted; text files
+  are declared one-item-per-line at first use; "everything outside the
+  braces is the instruction" stated wherever braces are taught; typed
+  braces are the norm across the examples; the playground corpus is
+  linked from README, docs home, and the cookbook; the demo video plays
+  on the docs front page.
+- **Row cuts keep their rows through `filter` and `top_k`.** The grep-l
+  contract (match whole files, get paths back) leaked onto `--as jsonl` /
+  `--as lines` rows read from a positional file: every matching ROW came
+  out as the filename. Both verbs now key path-back on the whole-file cut;
+  rows pass through as records. Found by a zero-context agent driving
+  smartpipe purely from SKILL.md; pinned in tests both ways.
+- **The docs were audited by executing them**: all 236 example blocks
+  (README, docs, SKILL files, CLI help) inventoried, statically checked,
+  and ~105 run verbatim against a generated corpus of real-format files.
+  Three code bugs surfaced and are fixed with pinning tests:
+  - **Quoted globs reach the reader.** `smartpipe 'logs/*.jsonl'` - a
+    documented form - exited 64 ("no verb"); unexpanded glob tokens now
+    route to reader mode, and an unmatched glob errs loudly.
+  - **Video is refused pre-send on wires that can't watch.** The
+    ChatGPT-login (codex) and anthropic adapters silently dropped video
+    parts and sent prompt-only requests - the model answered "no video
+    provided", no degraded note fired, and video-RAG examples embedded
+    that refusal prose as vectors. Both wires now raise before sending,
+    so the documented ladder (`degraded: … video → frames+audio`)
+    actually engages.
+  - 7 broken and 12 misleading example sites corrected against observed
+    behavior (SKILL files now run verbatim; the `.sem` key table is
+    re-derived from the runtime's own validation errors; `__score`
+    spelling; stale "oversized items are refused" claims rewritten for
+    auto-chunking).
+
+### The terminal shows your media
+- **Every block at the terminal gets a number.** The human view opens
+  each record block with a cyan `#N` - so "look at object 5" is a thing
+  you can say. The ordinal is the one handle every output is guaranteed
+  to have (provenance can repeat across rows - fifty pages of one PDF
+  share a path - or be absent on plain text). Display only: piped
+  output is byte-identical.
+- **Media previews at the human view.** At a terminal (and through
+  `readable`), the first media part of an item now renders under its
+  summary line: images as color thumbnails, video as a three-frame strip
+  sampled at 10/50/90% of its duration (never frame zero - intros are
+  black), audio as a peak-envelope waveform. Audio and video that still
+  live on disk get a clickable `▶ play (0:42, 2.1 MB)` link (OSC 8 -
+  opens your system player in iTerm2/kitty/WezTerm/Ghostty/Windows
+  Terminal). Piped output, JSONL, NO_COLOR, and `--bare` are
+  byte-identical to before - previews exist only where a human is
+  looking. `smartpipe config media-previews off` turns them off.
+
+### Documents parse where they enter, vectors stay honest
+- **The `ocr-model` role: a document parser at ingestion.** Set
+  `smartpipe config ocr-model mistral/mistral-ocr-latest` (or
+  `SMARTPIPE_OCR_MODEL`, or `--ocr-model` per run) and ingested PDFs and
+  images parse through it - one item per page on the `pages` cut,
+  markdown as the text, disclosed per row and metered as a paid
+  conversion. Mistral refs ride the dedicated `/v1/ocr` wire
+  (live-verified, whole PDFs in one call); any other model ref works as
+  chat-vision with extract-the-text framing, so `ollama/llava` is a free
+  local OCR. Parse failures fall back to the local extraction ladder,
+  disclosed, never fatal. Unset = exactly the old behavior.
+- **The `media-embed-model` role: one joint space for pixels and prose.**
+  Text keeps `embed-model`; media routes to the joint-space model
+  (jina-clip-v2 and friends). If a run would mix two vector spaces, the
+  geometry fence stops it with exit 64 BEFORE any spend, naming both
+  models and both fixes. A text query now ranks an image corpus in the
+  joint space - that is the point.
+- **Embed rows carry their provenance.** Output rows are now
+  `{"text", "vector", "__embedder", "__source"}` - the spine like every
+  other verb, plus the model that made the vector. `top_k` refuses a
+  corpus whose stamp disagrees with the query's embedder (the
+  same-dimensions-different-model trap), and reads old-style rows for
+  one release. Two bugs died en route: records piped from the reader
+  embedded their serialized JSON instead of their content text, and
+  `--max-calls` silently demoted pixels to captions by stripping the
+  media-embedding capability.
+
+### Setup that reads the room
+- **The schema workshop.** `smartpipe schema` with no arguments at a
+  terminal opens a small interactive draft loop - your schema pinned up
+  top as colored braces, commands scrolling below: `/add name type`,
+  `/drop`, `/test data.jsonl` (per-field coverage bars against your real
+  rows), `/example` (a synthetic instance), `/save` (writes schema.json
+  and prints the paste-ready braces and `--schema` lines). Zero model
+  calls by construction - iterate freely, commit when the bars are
+  green. Paste any `{braces}` string to replace the draft. Piped and
+  argument invocations are unchanged.
+- **`smartpipe config` is now a provider-first picker.** Detect what's
+  already connected (API keys in the environment, a ChatGPT login, local
+  Ollama), pick a provider, pick from its LIVE model catalog - arrow keys
+  on a real terminal, a numbered prompt everywhere else. Undetected
+  providers show the exact `export ..._API_KEY=` line (smartpipe never
+  prompts for or stores a key). Catalogs cache for a day; a failed fetch
+  degrades to typed input with provider-prefixed examples. Picking a
+  provider auto-pairs a coherent embedding model (announced, and never
+  overwrites one you chose yourself), and one y/N question adds a backup
+  `fallback-model` for provider outages. After a `doctor --probe`,
+  catalog entries carry capability chips (`sees, hears - probed 2d ago`)
+  - probed facts, never claims. All the setter subcommands still work.
+
+### Getting it and keeping it current
+- **One-line install on every platform.**
+  `curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh`
+  (macOS/Linux) and the `install.ps1` analog for Windows PowerShell, both
+  published at the docs-site root. The script uses Homebrew when present,
+  otherwise bootstraps uv and runs `uv tool install smartpipe-cli`;
+  `SMARTPIPE_VERSION` pins a version; it ends with a loud
+  `smartpipe --version` verify. brew/uv/pipx/pip remain as explicit
+  alternatives.
+- **The installers survive the real world.** Rerunning the one-liner
+  upgrades in place instead of erroring (brew-, uv-, and
+  otherwise-managed installs each get the right move); the uv bootstrap
+  falls back to `wget` when `curl` is missing; Alpine/musl systems get
+  a wheels warning before anything installs; Rosetta-translated macOS
+  shells get an arch hint; GitHub Actions runners get `~/.local/bin`
+  appended to `GITHUB_PATH`. Footguns adopted from studying opencode's
+  installer. `install.sh` stays readable POSIX sh at 96 lines.
+- **`smartpipe update`.** Detects how smartpipe was installed (homebrew,
+  uv tool, pipx, or pip - fingerprinted from the executable path), shows
+  the exact upgrade command, asks consent (`--yes` skips), runs it, and
+  reports honestly. Unrecognized installs get the per-channel commands
+  instead of a guess.
+- **A calm update notice.** At most once a day smartpipe checks PyPI in a
+  background thread (2 s cap, silent on any failure) and - on a later
+  run, at a terminal, never in CI, never on stdout - prints one dim line:
+  `note: smartpipe X.Y.Z is available (you have A.B.C) - run: smartpipe
+  update`. Rc users aren't nagged about older stables. Off switches:
+  `SMARTPIPE_NO_UPDATE_CHECK=1` or `smartpipe config update-check off`.
+  The welcome screen's utility list now includes `update`.
 
 ### Docs: a shape, not a pile
 - New Learn track (`docs/learn/1…6`), `concepts/the-item.md` (the five
@@ -435,6 +666,10 @@ estimated dollars.
   live-caught). `doctor` now closes with a bold yellow pointer at `--probe`.
   Also fixed live: streaming `embed` emitted the pre-conversion item text
   (empty for media), so records now carry the text the vector actually means.
+  And the streaming path (piped stdin) skipped the native media route
+  entirely - image-only items caption-pivoted even on jina-clip while the
+  finite-corpus branch embedded pixels. Both branches now share one router
+  (live-verified: `dims: 1024` from jina-clip-v2 on a piped image).
 - **Gemini watches video natively (D34).** Gemini chat moved to Google's
   native `:generateContent` wire — the only wired endpoint with video input.
   `map "what happens?" --in demo.mp4 --model gemini-2.5-flash` sends the

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/CITATION.cff

@@ -7,5 +7,5 @@ authors:
     given-names: Prabal
 repository-code: "https://github.com/prabal-rje/smartpipe"
 license: Apache-2.0
-version: 1.4.0rc1
+version: 1.4.0rc2
 date-released: '2026-07-08'

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/CONTRIBUTING.md

@@ -44,6 +44,24 @@ Optional but recommended: `uv run pre-commit install` wires ruff into your commi
 
 Release ritual: [RELEASING.md](RELEASING.md) · upkeep: [MAINTENANCE.md](MAINTENANCE.md).
 
+## Docs search (Algolia DocSearch)
+
+The site is pre-wired for [DocSearch](https://docsearch.algolia.com/) but ships
+with it OFF: `extra.algolia` in `mkdocs.yml` holds three empty strings, and
+while any of them is empty `overrides/main.html` renders nothing extra -
+mkdocs-material's built-in search keeps working. To activate once the
+DocSearch application is approved:
+
+1. Put the credentials Algolia sends into `mkdocs.yml` → `extra.algolia`
+   (`app_id`, `index_name`, and the **search-only** public `api_key` - that
+   key is safe to commit; never the admin key).
+2. `make docs-check` - the strict build must stay green.
+3. Open the built site and confirm the DocSearch button replaced the
+   built-in search box (the override hides `.md-search` and mounts
+   docsearch.js v3 in the header).
+
+Reverting to built-in search is emptying the three values again.
+
 ## New dependencies
 
 Core install weight is a feature: the runtime dependency list is `click`, `httpx`,

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: smartpipe-cli
-Version: 1.4.0rc1
+Version: 1.4.0rc2
 Summary: Semantic pipes and queries for your terminal - PDFs, images, audio, video, and text through Unix verbs that understand.
 Project-URL: Homepage, https://github.com/prabal-rje/smartpipe
 Project-URL: Changelog, https://github.com/prabal-rje/smartpipe/blob/main/CHANGELOG.md
@@ -28,9 +28,10 @@ Requires-Dist: httpx>=0.27
 Requires-Dist: imageio-ffmpeg>=0.5
 Requires-Dist: jsonschema>=4.23
 Requires-Dist: markitdown[docx,pdf,pptx,xlsx]>=0.1.6; python_version < '3.14'
+Requires-Dist: matplotlib>=3.8
 Requires-Dist: pdfminer-six>=20231228
+Requires-Dist: plotext>=5
 Requires-Dist: pypdf>=5.0
-Requires-Dist: svgwrite>=1.4
 Requires-Dist: tomli-w>=1.0
 Description-Content-Type: text/markdown
 
@@ -47,17 +48,43 @@ Description-Content-Type: text/markdown
 Run PDFs, images, audio, video, and text through Unix verbs that understand
 their input.
 
-[![smartpipe demo - click to play with sound and controls](https://github.com/prabal-rje/smartpipe/releases/download/demo-assets/smartpipe-teaser-v2.gif)](https://prabal-rje.github.io/smartpipe/demo/)
+[![smartpipe demo - click to play with sound and controls](https://github.com/prabal-rje/smartpipe/releases/download/demo-assets/smartpipe-teaser-v3.gif)](https://prabal-rje.github.io/smartpipe/demo/)
 
 *[▶ Watch the full 70-second demo](https://prabal-rje.github.io/smartpipe/demo/) - sound, controls, no download.*
 
 ## Install
 
+**macOS** - Homebrew is the happy path:
+
+```bash
+brew install prabal-rje/tap/smartpipe
+```
+
+**Linux** - one line (uses Homebrew if you have it, bootstraps uv otherwise;
+works on macOS too):
+
+```bash
+curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh
+```
+
+**Windows**:
+
+```powershell
+powershell -ExecutionPolicy Bypass -c "irm https://prabal-rje.github.io/smartpipe/install.ps1 | iex"
+```
+
+Prefer your own Python tooling? Any of these works on every platform:
+
 ```bash
-# zero-install trial (or: pip install smartpipe-cli)
-uvx --from smartpipe-cli smartpipe
+uv tool install smartpipe-cli           # uv
+pipx install smartpipe-cli              # pipx
+pip install smartpipe-cli               # pip
+uvx --from smartpipe-cli smartpipe      # zero-install trial
 ```
 
+Stay current with `smartpipe update` - it detects how smartpipe was installed
+and runs that tool's own upgrade.
+
 ## Point it at a model
 
 ```bash
@@ -146,8 +173,8 @@ cat tickets.jsonl \
 
 # 4. group feedback by meaning, label each theme, chart it
 cat feedback.txt \
-| smartpipe cluster --top 8 \
-| smartpipe chart cluster --save themes.svg
+| smartpipe cluster --explode members \
+| smartpipe chart cluster --top 8 --save themes.svg
 
 # 5. cut for free with `where`, then let the model judge only what is left
 cat app.log \
@@ -167,6 +194,10 @@ smartpipe embed 'sessions/**/*.mp4' > sessions.embeddings
 smartpipe top_k 3 --near "user gives up after the coupon fails" < sessions.embeddings
 ```
 
+A `.txt` on a pipe arrives one item per line; `--as file` treats the whole
+file as one item ([feeding smartpipe](docs/concepts/feeding-smartpipe.md) has
+the full ingestion table).
+
 Numbers 7 and 8 are full recipes -
 [invoice reconciliation](docs/cookbook/invoice-reconciliation.md) and
 [video RAG](docs/cookbook/video-qa.md) - two of a dozen in the
@@ -175,6 +206,21 @@ Numbers 7 and 8 are full recipes -
 New to this? The [Learn track][quickstart] starts at zero and assumes nothing, including
 what a "model" is.
 
+## Try it on real files
+
+No corpus handy? [smartpipe-playground][playground] ships 26 MB of CC0 /
+public-domain practice files - invoices, reports, photos, recordings, screen
+sessions, and JSONL data:
+
+```bash
+curl -L https://github.com/prabal-rje/smartpipe-playground/archive/refs/tags/v1.tar.gz | tar xz
+cd smartpipe-playground-1
+
+smartpipe map "Extract {vendor, invoice_number, total number}" 'invoices/*.pdf'
+smartpipe filter "the customer sounds frustrated" 'recordings/*.mp3'
+smartpipe top_k 3 --near "supply chain risk" 'reports/*.pdf'
+```
+
 ## For AI agents
 
 Driving smartpipe from an agent? [SKILL.md](SKILL.md) is the operating
@@ -245,6 +291,7 @@ lives in [`qa/`](qa/README.md). The CLI surface is a SemVer contract.
 [models]: https://prabal-rje.github.io/smartpipe/concepts/models-and-providers/
 [privacy]: https://prabal-rje.github.io/smartpipe/privacy/
 [ollama]: https://ollama.com
+[playground]: https://github.com/prabal-rje/smartpipe-playground
 [map]: https://prabal-rje.github.io/smartpipe/verbs/map/
 [extend]: https://prabal-rje.github.io/smartpipe/verbs/extend/
 [filter]: https://prabal-rje.github.io/smartpipe/verbs/filter/

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/README.md

@@ -11,17 +11,43 @@
 Run PDFs, images, audio, video, and text through Unix verbs that understand
 their input.
 
-[![smartpipe demo - click to play with sound and controls](https://github.com/prabal-rje/smartpipe/releases/download/demo-assets/smartpipe-teaser-v2.gif)](https://prabal-rje.github.io/smartpipe/demo/)
+[![smartpipe demo - click to play with sound and controls](https://github.com/prabal-rje/smartpipe/releases/download/demo-assets/smartpipe-teaser-v3.gif)](https://prabal-rje.github.io/smartpipe/demo/)
 
 *[▶ Watch the full 70-second demo](https://prabal-rje.github.io/smartpipe/demo/) - sound, controls, no download.*
 
 ## Install
 
+**macOS** - Homebrew is the happy path:
+
+```bash
+brew install prabal-rje/tap/smartpipe
+```
+
+**Linux** - one line (uses Homebrew if you have it, bootstraps uv otherwise;
+works on macOS too):
+
+```bash
+curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh
+```
+
+**Windows**:
+
+```powershell
+powershell -ExecutionPolicy Bypass -c "irm https://prabal-rje.github.io/smartpipe/install.ps1 | iex"
+```
+
+Prefer your own Python tooling? Any of these works on every platform:
+
 ```bash
-# zero-install trial (or: pip install smartpipe-cli)
-uvx --from smartpipe-cli smartpipe
+uv tool install smartpipe-cli           # uv
+pipx install smartpipe-cli              # pipx
+pip install smartpipe-cli               # pip
+uvx --from smartpipe-cli smartpipe      # zero-install trial
 ```
 
+Stay current with `smartpipe update` - it detects how smartpipe was installed
+and runs that tool's own upgrade.
+
 ## Point it at a model
 
 ```bash
@@ -110,8 +136,8 @@ cat tickets.jsonl \
 
 # 4. group feedback by meaning, label each theme, chart it
 cat feedback.txt \
-| smartpipe cluster --top 8 \
-| smartpipe chart cluster --save themes.svg
+| smartpipe cluster --explode members \
+| smartpipe chart cluster --top 8 --save themes.svg
 
 # 5. cut for free with `where`, then let the model judge only what is left
 cat app.log \
@@ -131,6 +157,10 @@ smartpipe embed 'sessions/**/*.mp4' > sessions.embeddings
 smartpipe top_k 3 --near "user gives up after the coupon fails" < sessions.embeddings
 ```
 
+A `.txt` on a pipe arrives one item per line; `--as file` treats the whole
+file as one item ([feeding smartpipe](docs/concepts/feeding-smartpipe.md) has
+the full ingestion table).
+
 Numbers 7 and 8 are full recipes -
 [invoice reconciliation](docs/cookbook/invoice-reconciliation.md) and
 [video RAG](docs/cookbook/video-qa.md) - two of a dozen in the
@@ -139,6 +169,21 @@ Numbers 7 and 8 are full recipes -
 New to this? The [Learn track][quickstart] starts at zero and assumes nothing, including
 what a "model" is.
 
+## Try it on real files
+
+No corpus handy? [smartpipe-playground][playground] ships 26 MB of CC0 /
+public-domain practice files - invoices, reports, photos, recordings, screen
+sessions, and JSONL data:
+
+```bash
+curl -L https://github.com/prabal-rje/smartpipe-playground/archive/refs/tags/v1.tar.gz | tar xz
+cd smartpipe-playground-1
+
+smartpipe map "Extract {vendor, invoice_number, total number}" 'invoices/*.pdf'
+smartpipe filter "the customer sounds frustrated" 'recordings/*.mp3'
+smartpipe top_k 3 --near "supply chain risk" 'reports/*.pdf'
+```
+
 ## For AI agents
 
 Driving smartpipe from an agent? [SKILL.md](SKILL.md) is the operating
@@ -209,6 +254,7 @@ lives in [`qa/`](qa/README.md). The CLI surface is a SemVer contract.
 [models]: https://prabal-rje.github.io/smartpipe/concepts/models-and-providers/
 [privacy]: https://prabal-rje.github.io/smartpipe/privacy/
 [ollama]: https://ollama.com
+[playground]: https://github.com/prabal-rje/smartpipe-playground
 [map]: https://prabal-rje.github.io/smartpipe/verbs/map/
 [extend]: https://prabal-rje.github.io/smartpipe/verbs/extend/
 [filter]: https://prabal-rje.github.io/smartpipe/verbs/filter/

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/docs/concepts/feeding-smartpipe.md

@@ -22,6 +22,8 @@ Files go after the prompt (quote globs). Each path's default:
 
 - `.jsonl` / `.ndjson` → strict records, one per line; a bad row is a loud
   error naming file and line.
+- `.csv` / `.tsv` → header-named records, one per row (`.tsv` cuts on tabs);
+  see [csv rows](#csv-rows) below.
 - everything else → one whole-file item: documents extract their text (and
   carry embedded figures to vision models), media carry their bytes.
 
@@ -30,19 +32,50 @@ Files go after the prompt (quote globs). Each path's default:
 
 ## The `--as` dial
 
-`--as file|lines|jsonl` overrides every default, stdin included:
+`--as file|lines|jsonl|csv` overrides every default, stdin included:
 
 ```bash
 cat poem.txt | smartpipe map "translate, keep the shape" --as file
 smartpipe map "translate" 'notes/*.txt' --as lines
 smartpipe map "classify {label}" export.txt --as jsonl
+cat export.txt | smartpipe map "classify {label}" --as csv
 ```
 
-An explicit `--as lines`/`jsonl` must hold for EVERY matched file: images
-refuse (no finer granularity), audio/video point at `split --by
+An explicit `--as lines`/`jsonl`/`csv` must hold for EVERY matched file:
+images refuse (no finer granularity), audio/video point at `split --by
 minutes/seconds`, documents point at `split --by pages` - with offender
 counts, never silent partial application.
 
+## csv rows
+
+`--as csv` (the default for `.csv`/`.tsv` paths) reads the header row as the
+field names and every later row as one record - `name,age` + `alice,31`
+becomes `{"name": "alice", "age": 31}`. On stdin the header is simply the
+first line. Rows stream one at a time, exactly like jsonl - a 10 GB export
+never materializes in memory.
+
+The details, all deliberate:
+
+- **Cell coercion is int → float → string.** A cell that is a whole number
+  becomes an int (`31`), a decimal or scientific number becomes a float
+  (`2.5`, `1e3`), and anything else - empty cells included - stays the string
+  it was (`007x` keeps its leading zeros). `NaN`/`Infinity` spellings stay
+  strings; they have no JSON form.
+- **`__source` carries the PHYSICAL line number.** The header is line 1, the
+  first data row line 2 - so `grep -n`/`sed -n` line references match the
+  file exactly. A quoted cell may span lines; such a row carries its FIRST
+  physical line.
+- **Dialect by extension.** `.tsv` cuts on tabs, with or without an explicit
+  `--as csv`; everything else - stdin included - cuts on commas. There is no
+  delimiter sniffing beyond that: extensions are predictable, sniffers guess.
+- **Errors are loud, jsonl-style.** A ragged row names the file, the physical
+  line, and both column counts; an empty file (or blank header cell, or
+  duplicate column name) refuses with the fix. `--as csv` on media or
+  documents refuses with the same signposts as `lines`/`jsonl`.
+
+Egress is the mirror you already have: `--output csv` on the verbs that emit
+records, so `csv in → csv out` round-trips.
+
 ## Reader mode
 
 `smartpipe PATH…` with no verb emits the crate's items as JSONL records -

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/docs/concepts/models-and-providers.md

@@ -217,6 +217,53 @@ account. Only the `openai` wire exists today; the key accepts
 `provider/model` so more can land behind the same seam.
 
 
+## The ocr-model role
+
+`smartpipe config ocr-model mistral-ocr-latest` names a dedicated document
+parser. When set, ingested PDFs and images run through it instead of the
+local extraction ladder - page markdown becomes the item text, and a
+multi-page PDF becomes one item per page (the `__source` spine carries the
+page cut, exactly like `split --by pages`). Setting the role is the consent;
+every parsed page is disclosed per row:
+
+```text
+⚠ degraded: report.pdf p.3 document → markdown (parsed by mistral/mistral-ocr-latest)
+```
+
+The role is provider-agnostic:
+
+- A **mistral** ref rides the dedicated `/v1/ocr` wire (needs
+  `MISTRAL_API_KEY`; the endpoint charges per page).
+- **Any other ref** reads pages through the normal chat-vision wire with an
+  extract-the-text framing - `smartpipe config ocr-model ollama/llava` is a
+  free local OCR. For PDFs on this rung, pages with a healthy text layer
+  keep their locally extracted text (zero calls); only thin, image-bearing
+  pages (scans) go through the model.
+
+Inside the conversion ladder, a configured `ocr-model` also outranks the
+vision-chat rung for page images, so scanned documents read as text
+everywhere. If a parse fails, the item falls back to the local extraction
+ladder with a note - never a hard stop. `--ocr-model` (per run) >
+`SMARTPIPE_OCR_MODEL` > the config key. Unset: nothing changes.
+
+## The media-embed-model role
+
+`smartpipe config media-embed-model jina/jina-clip-v2` names a JOINT
+text+image embedder. When set, `embed` and `top_k` route media items to it
+as pixels while text items keep using `embed-model` - and a text query can
+rank an image corpus, because both live in the same space.
+
+One vector space per run is the law: if a run holds BOTH text and media
+items while the two roles name different models, smartpipe refuses loudly
+(vectors from two models can't be compared). The fix it names: set
+`embed-model` to the joint model too, or feed media-only input. Unset, media
+items keep today's behavior (native embedding when `embed-model` itself is
+media-capable, the caption pivot otherwise).
+
+`--media-embed-model` (per run) > `SMARTPIPE_MEDIA_EMBED_MODEL` > the config
+key.
+
+
 ## The usage ledger
 
 `smartpipe usage` shows what the meter observed over the past hour, day, week,

{smartpipe_cli-1.4.0rc1 → smartpipe_cli-1.4.0rc2}/docs/concepts/output-formats.md

@@ -15,6 +15,31 @@ With no `--output` flag, smartpipe looks at whether `stdout` is a terminal:
 So `smartpipe map "Extract {name, role}"` shows you a tidy view on screen, but the
 moment you pipe it into `jq` or a file, it becomes clean JSONL - no flags needed.
 
+### Media previews at the terminal
+
+In the terminal view (and in [`readable`](../verbs/readable.md)), an item
+carrying media gets more than its `image/png (48 KB)` summary line - the first
+media part renders right under it:
+
+- **Images** show a small color thumbnail (aspect preserved, about 40x12
+  cells).
+- **Video** shows a 3-frame strip sampled at 10%/50%/90% of the duration -
+  never the first frame, which is usually a black or logo intro.
+- **Audio** shows a waveform envelope (long files decode at most the first
+  10 minutes).
+- **Audio and video with a real file behind them** add a
+  `▶ play (0:42, 2.1 MB)` line - an OSC 8 hyperlink to the `file://` URL, so
+  a click opens your system player. Media that exists only as pipe bytes
+  still gets its picture, just no link.
+
+Previews exist only where the human view does: pipes, `--output json`,
+`NO_COLOR`, and `TERM=dumb` output stay byte-for-byte what they were. The
+persisted kill switch:
+
+```bash
+smartpipe config media-previews off   # back to summary lines only
+```
+
 ## Explicit formats
 
 | `--output` | What you get |