smartpipe-cli 1.3.1__tar.gz → 1.4.0rc2__tar.gz

{smartpipe_cli-1.3.1 → smartpipe_cli-1.4.0rc2}/.gitignore

@@ -23,3 +23,7 @@ context/
 /plan/
 /TODO.md
 /idea.md
+
+# Local-only scratch folder — never tracked, never in the sdist
+/local/
+.claude/

{smartpipe_cli-1.3.1 → smartpipe_cli-1.4.0rc2}/CHANGELOG.md

@@ -5,6 +5,317 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) · Versioning:
 
 ## [Unreleased]
 
+## [1.4.0] — 2026-07-08
+
+The identity release.
+
+### Knowledge graphs (the highlight)
+- **`smartpipe graph --fast` turns a corpus into a knowledge graph for
+  exactly $0.** A local zero-shot NER model (GLiNER-small over ONNX -
+  no new dependencies, no torch, a one-time ~190 MB download like the
+  local embedder) finds YOUR entity types (`--entities "person, vessel,
+  account"`), entities co-occurring in a window become weighted edges,
+  near-duplicate names fold into one node (disclosed), and every edge
+  carries its `sources` - the files, pages, and minutes where it was
+  seen. Zero model calls, pinned by test. Free transcripts (local
+  whisper) mean audio and video speech join the graph; scans and
+  images wait for the paid mode, counted out loud.
+- **Six ways out, all deterministic**: `.graphml` (Gephi), `.dot`
+  (Graphviz), `.mmd` (Mermaid), `nodes.csv`+`edges.csv` (Neo4j/Kuzu),
+  a self-contained interactive `.html` (search, weight slider, hover
+  cards with clickable `file://` evidence links), or a trailing `/`
+  for an Obsidian vault of wikilinked entity notes.
+- **A focus prompt buys named relations.** `smartpipe graph "people,
+  orgs, and money flows" case/**/*` extracts typed triples per chunk
+  through the full multimodal ladders - vision, OCR, native video -
+  with `--entities`/`--relations` compiling to enums when you want a
+  strict ontology. The cost plan prints BEFORE any spend (`~810
+  extraction calls; belt is 500 - the graph will be partial`), a
+  terminal decline exits clean at zero calls, and a belted run drains
+  to a valid partial graph at exit 1, never a fake success.
+- **The hybrid ladder: `--name-top 200`.** The free pass finds the
+  candidates; the model names only the strongest links - one call per
+  edge, and if the belt runs out the remainder keeps its honest
+  `co-occurs` label, disclosed. Free graph in seconds; name the top
+  edges for cents.
+- **Bring your own edges.** Records with `{source, target}` or
+  `{subject, relation, object}` on stdin skip extraction entirely -
+  canonicalize, fold, serialize. Your own `extend` pipeline is the
+  power path.
+
+### The item — one law for everything in a pipe
+- **Dates are a type, and they come back ISO.** `{due date}` and
+  `{ts datetime}` join the brace grammar. However the model phrases it
+  ("Jan 15, 2026", "15/01/2026"), the field canonicalizes to
+  `YYYY-MM-DD` / full ISO-8601 - offsets preserved, never invented;
+  genuinely ambiguous slash-dates read month-first and say so on
+  stderr. Downstream just works: `where 'due >= "2026-01-01"'`
+  compares temporally, `sort --by due` orders by instant, and
+  `summarize 'count() by bin(due, 1d)'` buckets date-only values.
+- **The model sees your data framed, not dumped.** Every item payload
+  now arrives inside an `<input>` fence - records rendered as clean
+  YAML-ish (the `__` plumbing excluded), text as itself. The
+  instruction can no longer blur into the data, and the fence is the
+  foundation the planned request-batching builds on.
+- **CSV is a first-class row format.** `--as csv` (and `.csv`/`.tsv`
+  extension defaults - tab dialect for `.tsv`) reads the header row as
+  field names and streams every data row as a record, with per-cell
+  int → float → string coercion and multi-line quoted cells handled. A
+  ragged row is a loud error naming the file and physical line; rows
+  carry `__source {"as": "csv", "line": N}`. Works on stdin too, and
+  `.sem` files gain the `as` key.
+- **Malformed model replies repair for free before they cost money.**
+  A structured reply that fails to parse now passes through a
+  deterministic fixer first - code fences, JSON-island extraction,
+  Python-repr literals, trailing commas, quote fixes, all validated by
+  an actual parse - and only escalates to the paid model-repair round
+  trip when that fails. Disclosed once per run:
+  `note: N replies repaired deterministically (fences/commas/quotes)`.
+- **The `__` metadata spine.** Tool metadata lives in a reserved
+  double-underscore namespace that rides every record: `__source` (path plus
+  HOW the item was cut - file, lines, jsonl, pages, minutes, segment index),
+  `__media` (one transport object for bytes crossing a pipe), `__score`
+  (join), `__invalid`/`__error`/`__raw` (kept failures). Known fields
+  round-trip through saved files; unknown `__` fields warn once and carry.
+  User data owns everything up to one leading underscore.
+- **Records in, records out.** A plain-prompt `map` over records now returns
+  a record (`{"result": …}` plus the spine) instead of bare prose - structure
+  and provenance survive transformation. Text lines still leave as text.
+- **The `--as` ingestion dial** (`file` | `lines` | `jsonl`) on every input
+  verb. Auto defaults: `.jsonl` means rows (loud per-line errors on
+  non-JSON), everything else means whole-document; stdin keeps per-line
+  sniffing; `--as file` slurps stdin whole (new). Media files refuse text
+  granularities with signposts to `split`.
+- **The binary is the reader.** `smartpipe report.pdf | …` - a first
+  argument that exists on disk streams items; verbs always win names,
+  `./name` disambiguates. Every input verb also takes positional FILE
+  arguments; `--in` survives as a hidden alias.
+- **`write` — the egress mirror.** Items route to files by template
+  (`{name}`, `{stem}`, `{index}`, any record field for content fan-out);
+  egress mirrors ingress via the spine - whole-file items become files,
+  line/row items reassemble into their sources in original order. Emits
+  written paths so pipes continue. Text-only records leave as plain text.
+- **`readable` — the human door.** The same YAML-ish renderer as the TTY
+  preview, as an explicit pipe stage: nesting indented, lists bulleted,
+  multi-line strings as block scalars, spine dimmed, long values truncated
+  with counts (`--full` to disable), media summarized (`image/png (48 KB)`)
+  - never base64. Piped machine output is untouched JSONL; `--bare` strips
+  the spine from it.
+- **Mixed streams are visible.** A kind census notes mixed inputs;
+  `where`/`summarize` report rows that had no fields; `--strict-rows` turns
+  the notes into errors. Multi-line plain output into a pipe warns that
+  line framing is ambiguous (use `--output json`).
+
+### Judgment you can gate, retry, and keep
+- **`--keep-invalid`** (map/extend): after the repair retry fails, keep the
+  failure as one JSON row (`__invalid`/`__error`/`__raw`) instead of a skip
+  - dim one-liner at a TTY, full row when piped.
+- **`smartpipe schema`** - the free rungs: compile braces/DSL to JSON Schema,
+  `--check FILE` validates rows (exit 0/1), `--example` prints a validating
+  instance, bare stdin mode is a REPL. **`--dry-run`** (map/extend) prints
+  the composed first request and exits before any model resolution.
+- **Nullability is declared.** `?` on any type (`string?`, `number[]?`, bare
+  `field?`) compiles to a null union; bare fields mean scalar-or-scalar-list
+  and never admit null (D48, shipped earlier in this cycle).
+- **Circuit breaker + `fallback-model`.** Five consecutive transport
+  failures stop the run with a provider-down screen (`SMARTPIPE_BREAKER`
+  tunes; 0 disables) - or, with a configured fallback, the run switches
+  chat models wholesale, re-runs the failed window on the successor, and
+  the receipt splits counts per model. Embedding fallback is refused
+  (one vector space per run). One fallback, no chains.
+- **Deterministic rungs.** `join --on 'left.K == right.K'` alone is a free
+  key-equality join (all kinds, `--unmatched`); with a prompt it becomes
+  blocking - equality narrows pairs, the judge reasons within blocks, the
+  receipt reports pairs avoided. `distinct --exact` folds byte/value
+  duplicates only (records canonicalized, media hashed by bytes) - zero
+  model calls. Verb tables mark the free rungs.
+
+### The terminal behaves
+- **A real progress bar.** When the total is knowable (file lists), the
+  status line becomes `[██████░░░░░░░░░] 41% · 205/500 · 12/s · ~25s
+  left` - fill and percent truncate so 100% is earned, the ETA appears
+  once the first item completes, ASCII terminals get `[=====>....]`,
+  and multi-stage runs prefix the stage name. Streams keep the
+  count-and-rate line: a `tail -f` has no percent.
+- **Braces extract lists of objects.** `{events {name string, when
+  date, severity enum(low, high)}[]}` pulls every event out of an item
+  in one call, typed and validated - inner dates still canonicalize to
+  ISO, `--explode events` turns the list into one row each. One level
+  deep by design; deeper nesting faults free at parse time with the
+  two-pass hint.
+- Result writes pause/erase/redraw the status line under an arbiter - the
+  spinner can never interleave with output again. Only the final pipe stage
+  (stdout a TTY) animates; mid-pipe stages stay line-atomic.
+- Records render at the TTY as YAML-ish blocks. The welcome screen links
+  docs, cookbook, and issues. The wizard offers to install shell
+  completions (consented, idempotent).
+- **`chart` re-platformed: plotext draws the terminal, matplotlib renders
+  `--save`.** TTY bars get color (cyan bars, green time-series); piped or
+  `NO_COLOR` output keeps the plain-ASCII contract byte-for-byte. `--save`
+  now writes SVG or PNG by extension (identity-styled, deterministic
+  output); svgwrite - unmaintained since 2022 - leaves the tree. Both
+  libraries ship in core per the no-extras rule; imports stay
+  function-local so startup holds. The run receipt now reads
+  `run: ↑40 ↓25 tok` (arrows for in/out). The wizard validates model names
+  before saving (two strikes, then a typed fault), prints completions
+  before the paste-bait "Try it" line, prefers a sensible local default
+  over alphabetical luck (cloud tags compete equally), and speaks in the
+  same color voice as the other commands with a menu of detected local
+  tags and provider-prefixed cloud examples.
+- **Docs bug, live-caught: `cluster | chart` drew flat bars.** The summary
+  rows are one-per-cluster, so charting them counts every theme once. The
+  README quick-tour step, the `cluster` help epilog, and the `.sem`
+  pipeline example now route `cluster --explode members` into
+  `chart cluster --top 8`, which draws real sizes.
+
+### Every example, run for real
+- **`map` and `extend` records finally carry their provenance.** The
+  `__source` spine - the law said it rides every record - was being
+  dropped by braces extraction: a reader-stamped row went in with
+  `{"path", "as", "line"}` and came out with only the extracted fields.
+  Every record leaving `map`/`extend` now carries `__source` (incoming
+  spines adopted verbatim, fresh ones stamped from the item's
+  provenance), which also means the terminal preview now shows the dim
+  file-and-line origin under each block. `--bare` still strips it.
+- **Docs truth sweep** (owner-reported): four `echo "... $1250"`
+  examples silently shell-expanded to "250" - single-quoted; text files
+  are declared one-item-per-line at first use; "everything outside the
+  braces is the instruction" stated wherever braces are taught; typed
+  braces are the norm across the examples; the playground corpus is
+  linked from README, docs home, and the cookbook; the demo video plays
+  on the docs front page.
+- **Row cuts keep their rows through `filter` and `top_k`.** The grep-l
+  contract (match whole files, get paths back) leaked onto `--as jsonl` /
+  `--as lines` rows read from a positional file: every matching ROW came
+  out as the filename. Both verbs now key path-back on the whole-file cut;
+  rows pass through as records. Found by a zero-context agent driving
+  smartpipe purely from SKILL.md; pinned in tests both ways.
+- **The docs were audited by executing them**: all 236 example blocks
+  (README, docs, SKILL files, CLI help) inventoried, statically checked,
+  and ~105 run verbatim against a generated corpus of real-format files.
+  Three code bugs surfaced and are fixed with pinning tests:
+  - **Quoted globs reach the reader.** `smartpipe 'logs/*.jsonl'` - a
+    documented form - exited 64 ("no verb"); unexpanded glob tokens now
+    route to reader mode, and an unmatched glob errs loudly.
+  - **Video is refused pre-send on wires that can't watch.** The
+    ChatGPT-login (codex) and anthropic adapters silently dropped video
+    parts and sent prompt-only requests - the model answered "no video
+    provided", no degraded note fired, and video-RAG examples embedded
+    that refusal prose as vectors. Both wires now raise before sending,
+    so the documented ladder (`degraded: … video → frames+audio`)
+    actually engages.
+  - 7 broken and 12 misleading example sites corrected against observed
+    behavior (SKILL files now run verbatim; the `.sem` key table is
+    re-derived from the runtime's own validation errors; `__score`
+    spelling; stale "oversized items are refused" claims rewritten for
+    auto-chunking).
+
+### The terminal shows your media
+- **Every block at the terminal gets a number.** The human view opens
+  each record block with a cyan `#N` - so "look at object 5" is a thing
+  you can say. The ordinal is the one handle every output is guaranteed
+  to have (provenance can repeat across rows - fifty pages of one PDF
+  share a path - or be absent on plain text). Display only: piped
+  output is byte-identical.
+- **Media previews at the human view.** At a terminal (and through
+  `readable`), the first media part of an item now renders under its
+  summary line: images as color thumbnails, video as a three-frame strip
+  sampled at 10/50/90% of its duration (never frame zero - intros are
+  black), audio as a peak-envelope waveform. Audio and video that still
+  live on disk get a clickable `▶ play (0:42, 2.1 MB)` link (OSC 8 -
+  opens your system player in iTerm2/kitty/WezTerm/Ghostty/Windows
+  Terminal). Piped output, JSONL, NO_COLOR, and `--bare` are
+  byte-identical to before - previews exist only where a human is
+  looking. `smartpipe config media-previews off` turns them off.
+
+### Documents parse where they enter, vectors stay honest
+- **The `ocr-model` role: a document parser at ingestion.** Set
+  `smartpipe config ocr-model mistral/mistral-ocr-latest` (or
+  `SMARTPIPE_OCR_MODEL`, or `--ocr-model` per run) and ingested PDFs and
+  images parse through it - one item per page on the `pages` cut,
+  markdown as the text, disclosed per row and metered as a paid
+  conversion. Mistral refs ride the dedicated `/v1/ocr` wire
+  (live-verified, whole PDFs in one call); any other model ref works as
+  chat-vision with extract-the-text framing, so `ollama/llava` is a free
+  local OCR. Parse failures fall back to the local extraction ladder,
+  disclosed, never fatal. Unset = exactly the old behavior.
+- **The `media-embed-model` role: one joint space for pixels and prose.**
+  Text keeps `embed-model`; media routes to the joint-space model
+  (jina-clip-v2 and friends). If a run would mix two vector spaces, the
+  geometry fence stops it with exit 64 BEFORE any spend, naming both
+  models and both fixes. A text query now ranks an image corpus in the
+  joint space - that is the point.
+- **Embed rows carry their provenance.** Output rows are now
+  `{"text", "vector", "__embedder", "__source"}` - the spine like every
+  other verb, plus the model that made the vector. `top_k` refuses a
+  corpus whose stamp disagrees with the query's embedder (the
+  same-dimensions-different-model trap), and reads old-style rows for
+  one release. Two bugs died en route: records piped from the reader
+  embedded their serialized JSON instead of their content text, and
+  `--max-calls` silently demoted pixels to captions by stripping the
+  media-embedding capability.
+
+### Setup that reads the room
+- **The schema workshop.** `smartpipe schema` with no arguments at a
+  terminal opens a small interactive draft loop - your schema pinned up
+  top as colored braces, commands scrolling below: `/add name type`,
+  `/drop`, `/test data.jsonl` (per-field coverage bars against your real
+  rows), `/example` (a synthetic instance), `/save` (writes schema.json
+  and prints the paste-ready braces and `--schema` lines). Zero model
+  calls by construction - iterate freely, commit when the bars are
+  green. Paste any `{braces}` string to replace the draft. Piped and
+  argument invocations are unchanged.
+- **`smartpipe config` is now a provider-first picker.** Detect what's
+  already connected (API keys in the environment, a ChatGPT login, local
+  Ollama), pick a provider, pick from its LIVE model catalog - arrow keys
+  on a real terminal, a numbered prompt everywhere else. Undetected
+  providers show the exact `export ..._API_KEY=` line (smartpipe never
+  prompts for or stores a key). Catalogs cache for a day; a failed fetch
+  degrades to typed input with provider-prefixed examples. Picking a
+  provider auto-pairs a coherent embedding model (announced, and never
+  overwrites one you chose yourself), and one y/N question adds a backup
+  `fallback-model` for provider outages. After a `doctor --probe`,
+  catalog entries carry capability chips (`sees, hears - probed 2d ago`)
+  - probed facts, never claims. All the setter subcommands still work.
+
+### Getting it and keeping it current
+- **One-line install on every platform.**
+  `curl -LsSf https://prabal-rje.github.io/smartpipe/install.sh | sh`
+  (macOS/Linux) and the `install.ps1` analog for Windows PowerShell, both
+  published at the docs-site root. The script uses Homebrew when present,
+  otherwise bootstraps uv and runs `uv tool install smartpipe-cli`;
+  `SMARTPIPE_VERSION` pins a version; it ends with a loud
+  `smartpipe --version` verify. brew/uv/pipx/pip remain as explicit
+  alternatives.
+- **The installers survive the real world.** Rerunning the one-liner
+  upgrades in place instead of erroring (brew-, uv-, and
+  otherwise-managed installs each get the right move); the uv bootstrap
+  falls back to `wget` when `curl` is missing; Alpine/musl systems get
+  a wheels warning before anything installs; Rosetta-translated macOS
+  shells get an arch hint; GitHub Actions runners get `~/.local/bin`
+  appended to `GITHUB_PATH`. Footguns adopted from studying opencode's
+  installer. `install.sh` stays readable POSIX sh at 96 lines.
+- **`smartpipe update`.** Detects how smartpipe was installed (homebrew,
+  uv tool, pipx, or pip - fingerprinted from the executable path), shows
+  the exact upgrade command, asks consent (`--yes` skips), runs it, and
+  reports honestly. Unrecognized installs get the per-channel commands
+  instead of a guess.
+- **A calm update notice.** At most once a day smartpipe checks PyPI in a
+  background thread (2 s cap, silent on any failure) and - on a later
+  run, at a terminal, never in CI, never on stdout - prints one dim line:
+  `note: smartpipe X.Y.Z is available (you have A.B.C) - run: smartpipe
+  update`. Rc users aren't nagged about older stables. Off switches:
+  `SMARTPIPE_NO_UPDATE_CHECK=1` or `smartpipe config update-check off`.
+  The welcome screen's utility list now includes `update`.
+
+### Docs: a shape, not a pile
+- New Learn track (`docs/learn/1…6`), `concepts/the-item.md` (the five
+  laws), `concepts/feeding-smartpipe.md` (the ingestion chapter), nav
+  regrouped Learn/Verbs/Concepts/Cookbook/Reference, index as mode-router,
+  every example reconciled to the new syntax with outputs from tests, qa/
+  flows extended (a live walk of which caught and fixed a real write
+  round-trip bug).
+
 ## [1.3.1] — 2026-07-08
 
 ### Fixed
@@ -355,6 +666,10 @@ estimated dollars.
   live-caught). `doctor` now closes with a bold yellow pointer at `--probe`.
   Also fixed live: streaming `embed` emitted the pre-conversion item text
   (empty for media), so records now carry the text the vector actually means.
+  And the streaming path (piped stdin) skipped the native media route
+  entirely - image-only items caption-pivoted even on jina-clip while the
+  finite-corpus branch embedded pixels. Both branches now share one router
+  (live-verified: `dims: 1024` from jina-clip-v2 on a piped image).
 - **Gemini watches video natively (D34).** Gemini chat moved to Google's
   native `:generateContent` wire — the only wired endpoint with video input.
   `map "what happens?" --in demo.mp4 --model gemini-2.5-flash` sends the

{smartpipe_cli-1.3.1 → smartpipe_cli-1.4.0rc2}/CITATION.cff

@@ -7,5 +7,5 @@ authors:
     given-names: Prabal
 repository-code: "https://github.com/prabal-rje/smartpipe"
 license: Apache-2.0
-version: 1.3.1
+version: 1.4.0rc2
 date-released: '2026-07-08'

{smartpipe_cli-1.3.1 → smartpipe_cli-1.4.0rc2}/CONTRIBUTING.md

@@ -44,6 +44,24 @@ Optional but recommended: `uv run pre-commit install` wires ruff into your commi
 
 Release ritual: [RELEASING.md](RELEASING.md) · upkeep: [MAINTENANCE.md](MAINTENANCE.md).
 
+## Docs search (Algolia DocSearch)
+
+The site is pre-wired for [DocSearch](https://docsearch.algolia.com/) but ships
+with it OFF: `extra.algolia` in `mkdocs.yml` holds three empty strings, and
+while any of them is empty `overrides/main.html` renders nothing extra -
+mkdocs-material's built-in search keeps working. To activate once the
+DocSearch application is approved:
+
+1. Put the credentials Algolia sends into `mkdocs.yml` → `extra.algolia`
+   (`app_id`, `index_name`, and the **search-only** public `api_key` - that
+   key is safe to commit; never the admin key).
+2. `make docs-check` - the strict build must stay green.
+3. Open the built site and confirm the DocSearch button replaced the
+   built-in search box (the override hides `.md-search` and mounts
+   docsearch.js v3 in the header).
+
+Reverting to built-in search is emptying the three values again.
+
 ## New dependencies
 
 Core install weight is a feature: the runtime dependency list is `click`, `httpx`,