smartpipe-cli 1.3.0__tar.gz → 1.4.0__tar.gz

{smartpipe_cli-1.3.0 → smartpipe_cli-1.4.0}/.gitignore

@@ -23,3 +23,7 @@ context/
 /plan/
 /TODO.md
 /idea.md
+
+# Local-only scratch folder — never tracked, never in the sdist
+/local/
+.claude/

{smartpipe_cli-1.3.0 → smartpipe_cli-1.4.0}/CHANGELOG.md

@@ -5,6 +5,457 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) · Versioning:
 
 ## [Unreleased]
 
+
+## [1.4.0] — 2026-07-10
+
+The identity release.
+
+### Knowledge graphs (the highlight)
+- **The showcase**: a full verb page with the modality matrix and a
+  live-rendering Mermaid sample, a cookbook chapter written from real
+  runs against the downloadable playground corpus, the README highlight
+  with a screenshot of the case graph, agent SKILL recipes, and `graph`
+  on the welcome screen. Found en route: the playground download in the
+  docs pointed at the source tarball (no media inside) - all three
+  sites now fetch the actual 26 MB release asset.
+
+- **`smartpipe graph --fast` turns a corpus into a knowledge graph for
+  exactly $0.** A local zero-shot NER model (GLiNER-small over ONNX -
+  no new dependencies, no torch, a one-time ~190 MB download like the
+  local embedder) finds YOUR entity types (`--entities "person, vessel,
+  account"`), entities co-occurring in a window become weighted edges,
+  near-duplicate names fold into one node (disclosed), and every edge
+  carries its `sources` - the files, pages, and minutes where it was
+  seen. Zero model calls, pinned by test. Free transcripts (local
+  whisper) mean audio and video speech join the graph; scans and
+  images wait for the paid mode, counted out loud.
+- **Six ways out, all deterministic**: `.graphml` (Gephi), `.dot`
+  (Graphviz), `.mmd` (Mermaid), `nodes.csv`+`edges.csv` (Neo4j/Kuzu),
+  a self-contained interactive `.html` (search, weight slider, hover
+  cards with clickable `file://` evidence links), or a trailing `/`
+  for an Obsidian vault of wikilinked entity notes.
+- **A focus prompt buys named relations.** `smartpipe graph "people,
+  orgs, and money flows" case/**/*` extracts typed triples per chunk
+  through the full multimodal ladders - vision, OCR, native video -
+  with `--entities`/`--relations` compiling to enums when you want a
+  strict ontology. The cost plan prints BEFORE any spend (`~810
+  extraction calls; belt is 500 - the graph will be partial`), a
+  terminal decline exits clean at zero calls, and a belted run drains
+  to a valid partial graph at exit 1, never a fake success.
+- **The hybrid ladder: `--name-top 200`.** The free pass finds the
+  candidates; the model names only the strongest links - one call per
+  edge, and if the belt runs out the remainder keeps its honest
+  `co-occurs` label, disclosed. Free graph in seconds; name the top
+  edges for cents.
+- **Bring your own edges.** Records with `{source, target}` or
+  `{subject, relation, object}` on stdin skip extraction entirely -
+  canonicalize, fold, serialize. Your own `extend` pipeline is the
+  power path.
+
+### The item — one law for everything in a pipe
+- **Nested field paths, everywhere you read.** `where 'user.plan ==
+  "pro"'`, `sort --by metrics.score`, `chart items[0].category`,
+  `summarize 'avg(metrics.score) by user.plan'`, and `"Is {user.plan}
+  worth it?"` in prompts - one grammar (`a.b.c`, `items[0]`,
+  `a.b['weird key']`) across all five surfaces. A column literally
+  named `user.plan` always wins over the traversal, so CSV headers
+  with dots keep working. Extraction names stay flat - paths are for
+  reading, and the error says so.
+
+- **Dates are a type, and they come back ISO.** `{due date}` and
+  `{ts datetime}` join the brace grammar. However the model phrases it
+  ("Jan 15, 2026", "15/01/2026"), the field canonicalizes to
+  `YYYY-MM-DD` / full ISO-8601 - offsets preserved, never invented;
+  genuinely ambiguous slash-dates read month-first and say so on
+  stderr. Downstream just works: `where 'due >= "2026-01-01"'`
+  compares temporally, `sort --by due` orders by instant, and
+  `summarize 'count() by bin(due, 1d)'` buckets date-only values.
+- **The model sees your data framed, not dumped.** Every item payload
+  now arrives inside an `<input>` fence - records rendered as clean
+  YAML-ish (the `__` plumbing excluded), text as itself. The
+  instruction can no longer blur into the data, and the fence is the
+  foundation the planned request-batching builds on.
+- **CSV is a first-class row format.** `--as csv` (and `.csv`/`.tsv`
+  extension defaults - tab dialect for `.tsv`) reads the header row as
+  field names and streams every data row as a record, with per-cell
+  int → float → string coercion and multi-line quoted cells handled. A
+  ragged row is a loud error naming the file and physical line; rows
+  carry `__source {"as": "csv", "line": N}`. Works on stdin too, and
+  `.sem` files gain the `as` key.
+- **Malformed model replies repair for free before they cost money.**
+  A structured reply that fails to parse now passes through a
+  deterministic fixer first - code fences, JSON-island extraction,
+  Python-repr literals, trailing commas, quote fixes, all validated by
+  an actual parse - and only escalates to the paid model-repair round
+  trip when that fails. Disclosed once per run:
+  `note: N replies repaired deterministically (fences/commas/quotes)`.
+- **The `__` metadata spine.** Tool metadata lives in a reserved
+  double-underscore namespace that rides every record: `__source` (path plus
+  HOW the item was cut - file, lines, jsonl, pages, minutes, segment index),
+  `__media` (one transport object for bytes crossing a pipe), `__score`
+  (top_k and join similarity), `__rank` (top_k), `__snapshot` (top_k
+  --stream markers), `__distance` (outliers), `__invalid`/`__error`/`__raw`
+  (kept failures). Known fields round-trip through saved files; unknown `__`
+  fields warn once and carry. User data owns everything up to one leading
+  underscore - which is why the last single-underscore stragglers moved:
+  pre-1.4 `top_k`/`outliers` wrote `_score`/`_rank`/`_snapshot`/`_distance`,
+  and those spellings stay readable for one release (CSV/TSV still sort them
+  into the trailing columns) but are never written again. In the same sweep,
+  `cluster --explode` overwriting an existing `cluster` field became a
+  disclosed overwrite (one stderr warning) instead of a silent one.
+- **top_k ranks can no longer collide.** Ranking keyed on each item's
+  source index, so two page-1 records from different PDFs fought over one
+  slot and a result silently vanished. Ranks are now run-scoped ordinals in
+  arrival order, in both the embedded and precomputed-vector paths.
+- **Records in, records out.** A plain-prompt `map` over records now returns
+  a record (`{"result": …}` plus the spine) instead of bare prose - structure
+  and provenance survive transformation. Text lines still leave as text.
+- **The `--as` ingestion dial** (`file` | `lines` | `jsonl`) on every input
+  verb. Auto defaults: `.jsonl` means rows (loud per-line errors on
+  non-JSON), everything else means whole-document; stdin keeps per-line
+  sniffing; `--as file` slurps stdin whole (new). Media files refuse text
+  granularities with signposts to `split`.
+- **The binary is the reader.** `smartpipe report.pdf | …` - a first
+  argument that exists on disk streams items; verbs always win names,
+  `./name` disambiguates. Every input verb also takes positional FILE
+  arguments; `--in` survives as a hidden alias.
+- **`write` — the egress mirror.** Items route to files by template
+  (`{name}`, `{stem}`, `{index}`, any record field for content fan-out);
+  egress mirrors ingress via the spine - whole-file items become files,
+  line/row items reassemble into their sources in original order. Emits
+  written paths so pipes continue. Text-only records leave as plain text.
+- **`readable` — the human door.** The same YAML-ish renderer as the TTY
+  preview, as an explicit pipe stage: nesting indented, lists bulleted,
+  multi-line strings as block scalars, spine dimmed, long values truncated
+  with counts (`--full` to disable), media summarized (`image/png (48 KB)`)
+  - never base64. Piped machine output is untouched JSONL; `--bare` strips
+  the spine from it.
+- **Mixed streams are visible.** A kind census notes mixed inputs;
+  `where`/`summarize` report rows that had no fields; `--strict-rows` turns
+  the notes into errors. Multi-line plain output into a pipe warns that
+  line framing is ambiguous (use `--output json`).
+- **`.sem` files are strict about their rows now.** A saved pipeline is a
+  program, and a stray plain line in a record stream is a bug, not a
+  shrug: `.sem` runs default to `--strict-rows` and fail at ingestion
+  naming the exact line, before any spend. Opt out with
+  `strict-rows = false` (top-level or per-stage); ad-hoc terminal runs
+  keep the permissive census. Eleven verbs that accepted `--strict-rows`
+  without actually enforcing it now do, and a text-only `where` predicate
+  over a log no longer trips strict mode over "missing fields". A new
+  docs page explains the granularity ladder (file → split → lines/rows)
+  and when mixing happens.
+- **Synthesized records tell you what they summarize.** join pairs carry
+  `__sources` (both parents' refs); reduce windows stamp
+  `{"as": "window", "span": [1, 100], "count": 100}`, groups
+  `{"as": "group", "group": …, "count": N}`; cluster and diff summary
+  rows carry `{"as": "cluster"|"diff", "count": N}`. An audit trail for
+  every row that speaks for other rows - and `--bare` strips all of it.
+
+### Judgment you can gate, retry, and keep
+- **`--keep-invalid`** (map/extend): after the repair retry fails, keep the
+  failure as one JSON row (`__invalid`/`__error`/`__raw`) instead of a skip
+  - dim one-liner at a TTY, full row when piped.
+- **`smartpipe schema`** - the free rungs: compile braces/DSL to JSON Schema,
+  `--check FILE` validates rows (exit 0/1), `--example` prints a validating
+  instance, bare stdin mode is a REPL. **`--dry-run`** (map/extend) prints
+  the composed first request and exits before any model resolution.
+- **`schema --check` stopped failing your own pipelines.** The check is
+  open-world now: rows from the documented extend workflow (original
+  fields kept, spine aboard) pass instead of tripping "additional
+  properties are not allowed" on every row, and `?`-marked fields may be
+  absent, not just null. Extras earn one dim hint; `--strict` restores
+  the closed world for contract enforcement. The schema sent to models
+  at extraction time is unchanged.
+- **Nullability is declared.** `?` on any type (`string?`, `number[]?`, bare
+  `field?`) compiles to a null union; bare fields mean scalar-or-scalar-list
+  and never admit null (D48, shipped earlier in this cycle).
+- **Circuit breaker + `fallback-model`.** Five consecutive transport
+  failures stop the run with a provider-down screen (`SMARTPIPE_BREAKER`
+  tunes; 0 disables) - or, with a configured fallback, the run switches
+  chat models wholesale, re-runs the failed window on the successor, and
+  the receipt splits counts per model. Embedding fallback is refused
+  (one vector space per run). One fallback, no chains.
+- **Deterministic rungs.** `join --on 'left.K == right.K'` alone is a free
+  key-equality join (all kinds, `--unmatched`); with a prompt it becomes
+  blocking - equality narrows pairs, the judge reasons within blocks, the
+  receipt reports pairs avoided. `distinct --exact` folds byte/value
+  duplicates only (records canonicalized, media hashed by bytes) - zero
+  model calls. Verb tables mark the free rungs.
+
+### The terminal behaves
+- **A real progress bar.** When the total is knowable (file lists), the
+  status line becomes `[██████░░░░░░░░░] 41% · 205/500 · 12/s · ~25s
+  left` - fill and percent truncate so 100% is earned, the ETA appears
+  once the first item completes, ASCII terminals get `[=====>....]`,
+  and multi-stage runs prefix the stage name. Streams keep the
+  count-and-rate line: a `tail -f` has no percent.
+- **Braces extract lists of objects.** `{events {name string, when
+  date, severity enum(low, high)}[]}` pulls every event out of an item
+  in one call, typed and validated - inner dates still canonicalize to
+  ISO, `--explode events` turns the list into one row each. One level
+  deep by design; deeper nesting faults free at parse time with the
+  two-pass hint.
+- Result writes pause/erase/redraw the status line under an arbiter - the
+  spinner can never interleave with output again. Only the final pipe stage
+  (stdout a TTY) animates; mid-pipe stages stay line-atomic.
+- Records render at the TTY as YAML-ish blocks. The welcome screen links
+  docs, cookbook, and issues. The wizard offers to install shell
+  completions (consented, idempotent).
+- **`chart` re-platformed: plotext draws the terminal, matplotlib renders
+  `--save`.** TTY bars get color (cyan bars, green time-series); piped or
+  `NO_COLOR` output keeps the plain-ASCII contract byte-for-byte. `--save`
+  now writes SVG or PNG by extension (identity-styled, deterministic
+  output); svgwrite - unmaintained since 2022 - leaves the tree. Both
+  libraries ship in core per the no-extras rule; imports stay
+  function-local so startup holds. The run receipt now reads
+  `run: ↑40 ↓25 tok` (arrows for in/out). The wizard validates model names
+  before saving (two strikes, then a typed fault), prints completions
+  before the paste-bait "Try it" line, prefers a sensible local default
+  over alphabetical luck (cloud tags compete equally), and speaks in the
+  same color voice as the other commands with a menu of detected local
+  tags and provider-prefixed cloud examples.
+- **Exit 141 is now deliberate, and the -13 flake is dead.** smartpipe used
+  to arm the default SIGPIPE handler, so a downstream `| head` killed it by
+  raw signal - and, one run in a few hundred, a stray EPIPE on asyncio's
+  own self-pipe killed a healthy run mid-teardown, skipping the receipt
+  (the CI flake that delayed rc3). SIGPIPE now stays ignored; a closed
+  stdout surfaces as BrokenPipeError and exits a silent, deliberate 141
+  with every finally honored. (click 8.4's own EPIPE-to-exit-1 trap is
+  defused on the way.) One visible delta: `kill -PIPE` no longer
+  terminates smartpipe; pipeline behavior is unchanged.
+- **A files-then-stdin wait has a name.** Chaining positional files into a
+  stdin read used to end in a silent hang when nothing was piped. All three
+  chain paths now say once, on stderr:
+  `note: files done - now reading stdin (pipe data or close it; files-only: add < /dev/null)`.
+- **Docs bug, live-caught: `cluster | chart` drew flat bars.** The summary
+  rows are one-per-cluster, so charting them counts every theme once. The
+  README quick-tour step, the `cluster` help epilog, and the `.sem`
+  pipeline example now route `cluster --explode members` into
+  `chart cluster --top 8`, which draws real sizes.
+
+### Every example, run for real
+- **`map` and `extend` records finally carry their provenance.** The
+  `__source` spine - the law said it rides every record - was being
+  dropped by braces extraction: a reader-stamped row went in with
+  `{"path", "as", "line"}` and came out with only the extracted fields.
+  Every record leaving `map`/`extend` now carries `__source` (incoming
+  spines adopted verbatim, fresh ones stamped from the item's
+  provenance), which also means the terminal preview now shows the dim
+  file-and-line origin under each block. `--bare` still strips it.
+- **Docs truth sweep** (owner-reported): four `echo "... $1250"`
+  examples silently shell-expanded to "250" - single-quoted; text files
+  are declared one-item-per-line at first use; "everything outside the
+  braces is the instruction" stated wherever braces are taught; typed
+  braces are the norm across the examples; the playground corpus is
+  linked from README, docs home, and the cookbook; the demo video plays
+  on the docs front page.
+- **Row cuts keep their rows through `filter` and `top_k`.** The grep-l
+  contract (match whole files, get paths back) leaked onto `--as jsonl` /
+  `--as lines` rows read from a positional file: every matching ROW came
+  out as the filename. Both verbs now key path-back on the whole-file cut;
+  rows pass through as records. Found by a zero-context agent driving
+  smartpipe purely from SKILL.md; pinned in tests both ways.
+- **The docs were audited by executing them**: all 236 example blocks
+  (README, docs, SKILL files, CLI help) inventoried, statically checked,
+  and ~105 run verbatim against a generated corpus of real-format files.
+  Three code bugs surfaced and are fixed with pinning tests:
+  - **Quoted globs reach the reader.** `smartpipe 'logs/*.jsonl'` - a
+    documented form - exited 64 ("no verb"); unexpanded glob tokens now
+    route to reader mode, and an unmatched glob errs loudly.
+  - **Video is refused pre-send on wires that can't watch.** The
+    ChatGPT-login (codex) and anthropic adapters silently dropped video
+    parts and sent prompt-only requests - the model answered "no video
+    provided", no degraded note fired, and video-RAG examples embedded
+    that refusal prose as vectors. Both wires now raise before sending,
+    so the documented ladder (`degraded: … video → frames+audio`)
+    actually engages.
+  - 7 broken and 12 misleading example sites corrected against observed
+    behavior (SKILL files now run verbatim; the `.sem` key table is
+    re-derived from the runtime's own validation errors; `__score`
+    spelling; stale "oversized items are refused" claims rewritten for
+    auto-chunking).
+
+### The terminal shows your media
+- **Every block at the terminal gets a number.** The human view opens
+  each record block with a cyan `#N` - so "look at object 5" is a thing
+  you can say. The ordinal is the one handle every output is guaranteed
+  to have (provenance can repeat across rows - fifty pages of one PDF
+  share a path - or be absent on plain text). Display only: piped
+  output is byte-identical.
+- **Media previews at the human view.** At a terminal (and through
+  `readable`), the first media part of an item now renders under its
+  summary line: images as color thumbnails, video as a three-frame strip
+  sampled at 10/50/90% of its duration (never frame zero - intros are
+  black), audio as a peak-envelope waveform. Audio and video that still
+  live on disk get a clickable `▶ play (0:42, 2.1 MB)` link (OSC 8 -
+  opens your system player in iTerm2/kitty/WezTerm/Ghostty/Windows
+  Terminal). Piped output, JSONL, NO_COLOR, and `--bare` are
+  byte-identical to before - previews exist only where a human is
+  looking. `smartpipe config media-previews off` turns them off.
+
+### Documents parse where they enter, vectors stay honest
+- **The `ocr-model` role: a document parser at ingestion.** Set
+  `ocr-model = "mistral/mistral-ocr-latest"` in config.toml (the setup
+  flow's OCR stage stamps it; `SMARTPIPE_OCR_MODEL` and `--ocr-model`
+  override per run) and ingested PDFs and images parse through it - one
+  item per page on the `pages` cut, markdown as the text, disclosed per
+  row and metered as a paid conversion. Mistral refs ride the dedicated
+  `/v1/ocr` wire (live-verified, whole PDFs in one call); any other
+  model ref works as chat-vision with extract-the-text framing, so
+  `ollama/llava` is a free local OCR. Parse failures fall back to the
+  local extraction ladder, disclosed, never fatal. Unset = exactly the
+  old behavior.
+- **Reader mode does what you configured.** `smartpipe scan.pdf` used to
+  be pinned "zero model calls" even when you had set an ocr-model - the
+  reader now honors the role with the same per-row disclosure, local
+  fallback, and `--max-calls` belt as the verbs (plus a preflight note
+  when more than 20 files will parse). No ocr-model configured = the
+  same free path as always, pinned by a zero-HTTP-calls test. Found en
+  route: the mistral OCR wire was never counted by `--max-calls`
+  anywhere - it is belted now.
+- **The `media-embed-model` role: one joint space for pixels and prose.**
+  Text keeps `embed-model`; media routes to the joint-space model
+  (jina-clip-v2 and friends). If a run would mix two vector spaces, the
+  geometry fence stops it with exit 64 BEFORE any spend, naming both
+  models and both fixes. A text query now ranks an image corpus in the
+  joint space - that is the point.
+- **Embed rows carry their provenance.** Output rows are now
+  `{"text", "vector", "__embedder", "__source"}` - the spine like every
+  other verb, plus the model that made the vector. `top_k` refuses a
+  corpus whose stamp disagrees with the query's embedder (the
+  same-dimensions-different-model trap), and reads old-style rows for
+  one release. Two bugs died en route: records piped from the reader
+  embedded their serialized JSON instead of their content text, and
+  `--max-calls` silently demoted pixels to captions by stripping the
+  media-embedding capability.
+
+### Setup that reads the room
+- **Rich now backs `config show` and `doctor`.** It is a deliberate core
+  dependency under the no-extras policy, imported only while those human-facing
+  screens render so `--help`, pipes, and `NO_COLOR` keep their existing contracts.
+- **The interactive config wizard can go backward.** Model, embedding, OCR, and
+  final review screens expose explicit Back rows; returning to a stage restores
+  its pre-stage checkpoint, and typed model prompts accept `back`, `b`, `no`, or
+  `n` instead of accidentally saving `ollama/n`.
+
+
+- **Discarding config is terminal and interruption is ordinary.** Choosing
+  discard no longer opens the shell-completion prompt after saying "Not saved";
+  Ctrl-C or EOF at any Click prompt exits 130 instead of showing the internal-bug
+  screen.
+
+- **`auth login` now connects every provider, not just ChatGPT.** Pick
+  from the full list - OpenAI appears twice because it really is two
+  wires (API key vs ChatGPT login, and the login wire has no
+  embeddings) - paste your key at a masked prompt after the create-a-key
+  URL, and it validates live before storing (with an honest
+  "store anyway - the provider may be having a bad minute" escape).
+  Keys live at `~/.local/share/smartpipe/auth.json`, owner-only
+  permissions from the first byte, masked everywhere they surface, and
+  the environment variable always wins. `auth list` shows what's
+  connected and which source is live; `auth logout` removes.
+- **Setup walks three questions in order: text model, embeddings,
+  document OCR.** Every stage shows every provider - connected ones
+  badged, unconnected ones connectable inline without leaving the flow.
+  The embedding stage only lists wires that embed; OCR is optional and
+  skippable in one keypress. At the end, one consented verification
+  pass (~5 tiny requests) probes what your models can actually do -
+  and if the basic text check fails, smartpipe reports a setup fault
+  instead of pretending your model lacks modalities. Results feed the
+  capability chips you see in every picker from then on, alongside a
+  public model registry and your own declarations for self-hosted
+  models.
+
+- **The schema workshop.** `smartpipe schema` with no arguments at a
+  terminal opens a small interactive draft loop - your schema pinned up
+  top as colored braces, commands scrolling below: `/add name type`,
+  `/drop`, `/test data.jsonl` (per-field coverage bars against your real
+  rows), `/example` (a synthetic instance), `/save` (writes schema.json
+  and prints the paste-ready braces and `--schema` lines). Zero model
+  calls by construction - iterate freely, commit when the bars are
+  green. Paste any `{braces}` string to replace the draft. Piped and
+  argument invocations are unchanged.
+- **`smartpipe config` is now a provider-first picker.** Detect what's
+  already connected (API keys in the environment, a ChatGPT login, local
+  Ollama), pick a provider, pick from its LIVE model catalog - arrow keys
+  on a real terminal, a numbered prompt everywhere else. Undetected
+  providers show the exact `export ..._API_KEY=` line (smartpipe never
+  prompts for or stores a key). Catalogs cache for a day; a failed fetch
+  degrades to typed input with provider-prefixed examples. Picking a
+  provider auto-pairs a coherent embedding model (announced, and never
+  overwrites one you chose yourself), and one y/N question adds a backup
+  `fallback-model` for provider outages. After a `doctor --probe`,
+  catalog entries carry capability chips (`sees, hears - probed 2d ago`)
+  - probed facts, never claims.
+- **`smartpipe use` - setup is one word now.** `use gemini` (or
+  `use claude-opus-4-8`, `use ollama/qwen3:8b`) stamps a coherent bundle
+  in one shot: the model, its paired embedder, and the captions posture,
+  each line disclosed with a ✓ and a reason. No credential for the named
+  provider = a clean refusal that names the exact `auth login` command,
+  nothing stamped. Bare `use` opens the interactive flow. `smartpipe
+  using` answers "what am I running?" with every setting, its value, and
+  WHERE it came from (flag, env, config file, default) in one aligned
+  grid. The config.toml you get is signed by its door -
+  `# stamped by: smartpipe use (2026-07-10T06:38Z)` - so a mystery
+  config explains itself. The old setter subcommands (`config model`,
+  `config embed-model`, …) and profiles are gone; an old config with
+  profile keys loads with one warning and cleans itself on the next
+  save.
+- **The no-model dead end offers a hand.** At a real terminal, the
+  no-model error now ends with `run setup now? [Y/n]` - accept, walk the
+  same setup flow, and it closes with "saved - rerun your command".
+  Scripts and CI see byte-identical output to before (the offer is
+  TTY-gated), and the exit code stays honest.
+
+### Getting it and keeping it current
+- **One-line install on every platform.**
+  `curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh`
+  (macOS/Linux) and the `install.ps1` analog for Windows PowerShell, both
+  published at the docs-site root. The script uses Homebrew when present,
+  otherwise bootstraps uv and runs `uv tool install smartpipe-cli`;
+  `SMARTPIPE_VERSION` pins a version; it ends with a loud
+  `smartpipe --version` verify. brew/uv/pipx/pip remain as explicit
+  alternatives.
+- **The installers survive the real world.** Rerunning the one-liner
+  upgrades in place instead of erroring (brew-, uv-, and
+  otherwise-managed installs each get the right move); the uv bootstrap
+  falls back to `wget` when `curl` is missing; Alpine/musl systems get
+  a wheels warning before anything installs; Rosetta-translated macOS
+  shells get an arch hint; GitHub Actions runners get `~/.local/bin`
+  appended to `GITHUB_PATH`. Footguns adopted from studying opencode's
+  installer. `install.sh` stays readable POSIX sh at 96 lines.
+- **`smartpipe update`.** Detects how smartpipe was installed (homebrew,
+  uv tool, pipx, or pip - fingerprinted from the executable path), shows
+  the exact upgrade command, asks consent (`--yes` skips), runs it, and
+  reports honestly. Unrecognized installs get the per-channel commands
+  instead of a guess.
+- **A calm update notice.** At most once a day smartpipe checks PyPI in a
+  background thread (2 s cap, silent on any failure) and - on a later
+  run, at a terminal, never in CI, never on stdout - prints one dim line:
+  `note: smartpipe X.Y.Z is available (you have A.B.C) - run: smartpipe
+  update`. Rc users aren't nagged about older stables. Off switches:
+  `SMARTPIPE_NO_UPDATE_CHECK=1` or `smartpipe config update-check off`.
+  The welcome screen's utility list now includes `update`.
+
+### Docs: a shape, not a pile
+- New Learn track (`docs/learn/1…6`), `concepts/the-item.md` (the five
+  laws), `concepts/feeding-smartpipe.md` (the ingestion chapter), nav
+  regrouped Learn/Verbs/Concepts/Cookbook/Reference, index as mode-router,
+  every example reconciled to the new syntax with outputs from tests, qa/
+  flows extended (a live walk of which caught and fixed a real write
+  round-trip bug).
+
+## [1.3.1] — 2026-07-08
+
+### Fixed
+- **PyPI no longer advertises phantom extras.** 1.3.0 kept the old extra
+  names (`[files]`, `[video]`, …) as empty compat aliases - but nothing was
+  ever published under the old name, so they protected nobody while the
+  PyPI sidebar listed six extras that contradict the everything-in-the-box
+  install story. The optional-dependencies table is gone entirely; pip
+  warns on an unknown extra and installs the complete package regardless.
+  Keywords now carry the multimodal story (pdf, audio, video, embeddings).
+
 ## [1.3.0] — 2026-07-07
 
 Installs from PyPI as **`smartpipe-cli`** (the plain name was too close to
@@ -26,9 +477,9 @@ that degrade per-item with the provider's real reason quoted.
   (PDF/DOCX/PPTX/XLSX/HTML/EPUB), video (bundled static ffmpeg), and chart
   `--save` all ship in the core install now, joining D44's whisper and local
   embeddings. One `pip install smartpipe-cli` is the entire multimodal surface.
-  The old extra names (`[files]`, `[video]`, `[charts]`, `[anthropic]`,
-  `[audio]`, `[all]`) remain as empty aliases so historical commands keep
-  working. `doctor`'s extras row now verifies the bundled components and
+  The old extra names briefly remained as empty aliases (removed for real
+  in 1.3.1 - nothing was ever published under the old name, so they only
+  advertised a false install story). `doctor`'s extras row now verifies the bundled components and
   FAILS on a broken install instead of suggesting installs.
 - **The gemini profile's default chat model is `gemini-3.1-flash-lite` (D45).**
   Verified against the live catalog (GA, not a preview) and through the
@@ -344,6 +795,10 @@ estimated dollars.
   live-caught). `doctor` now closes with a bold yellow pointer at `--probe`.
   Also fixed live: streaming `embed` emitted the pre-conversion item text
   (empty for media), so records now carry the text the vector actually means.
+  And the streaming path (piped stdin) skipped the native media route
+  entirely - image-only items caption-pivoted even on jina-clip while the
+  finite-corpus branch embedded pixels. Both branches now share one router
+  (live-verified: `dims: 1024` from jina-clip-v2 on a piped image).
 - **Gemini watches video natively (D34).** Gemini chat moved to Google's
   native `:generateContent` wire — the only wired endpoint with video input.
   `map "what happens?" --in demo.mp4 --model gemini-2.5-flash` sends the

{smartpipe_cli-1.3.0 → smartpipe_cli-1.4.0}/CITATION.cff

@@ -7,5 +7,5 @@ authors:
     given-names: Prabal
 repository-code: "https://github.com/prabal-rje/smartpipe"
 license: Apache-2.0
-version: 1.3.0
-date-released: '2026-07-07'
+version: 1.4.0
+date-released: '2026-07-08'

{smartpipe_cli-1.3.0 → smartpipe_cli-1.4.0}/CONTRIBUTING.md

@@ -9,7 +9,7 @@ Requires [uv](https://docs.astral.sh/uv/) and Python ≥ 3.11 (uv can fetch one
 
 ```console
 $ git clone https://github.com/prabal-rje/smartpipe && cd smartpipe
-$ uv sync --all-extras          # creates .venv with dev + optional deps
+$ uv sync --all-extras          # creates .venv with the project and dev tools
 $ uv run smartpipe                # welcome screen = working install
 ```
 
@@ -44,14 +44,33 @@ Optional but recommended: `uv run pre-commit install` wires ruff into your commi
 
 Release ritual: [RELEASING.md](RELEASING.md) · upkeep: [MAINTENANCE.md](MAINTENANCE.md).
 
+## Docs search (Algolia DocSearch)
+
+The site is pre-wired for [DocSearch](https://docsearch.algolia.com/) but ships
+with it OFF: `extra.algolia` in `mkdocs.yml` holds three empty strings, and
+while any of them is empty `overrides/main.html` renders nothing extra -
+mkdocs-material's built-in search keeps working. To activate once the
+DocSearch application is approved:
+
+1. Put the credentials Algolia sends into `mkdocs.yml` → `extra.algolia`
+   (`app_id`, `index_name`, and the **search-only** public `api_key` - that
+   key is safe to commit; never the admin key).
+2. `make docs-check` - the strict build must stay green.
+3. Open the built site and confirm the DocSearch button replaced the
+   built-in search box (the override hides `.md-search` and mounts
+   docsearch.js v3 in the header).
+
+Reverting to built-in search is emptying the three values again.
+
 ## New dependencies
 
-Core install weight is a feature: the runtime dependency list is `click`, `httpx`,
-`jsonschema`, `tomli-w` — a snapshot test guards it. Anything heavier goes behind an
-optional extra, and heavy imports stay function-local (startup time is budgeted).
+Core install weight is a feature, but D46 deliberately ships the complete
+multimodal surface in core: there are no optional extras. The frozen dependency
+snapshot makes every direct or transitive addition an explicit review decision.
+Heavy imports stay function-local so startup remains budgeted.
 
 **New heavy import? Function-local or justify.** `--help` must never import
-`httpx`, `jsonschema`, `anthropic`, or `markitdown` —
+`httpx`, `jsonschema`, `anthropic`, `markitdown`, or `rich` —
 `tests/test_startup_imports.py` is the enforcement (it runs `-X importtime` and
 fails on any banned module). `make startup` gives an advisory wall-clock number;
 the import test is the deterministic gate.