iriq 0.30.2 → 0.35.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 598c04e3c1777787ae9e5d1be98e2bc68d441e2020ebe7743bdf8075b20fdaec
-  data.tar.gz: 396ad6b0b0acffb76b7bc2b4e31792b02ac65749c6ac769fd70a47ce5d806496
+  metadata.gz: aad1274de2d7cae934f5b940e97f498193883addffcab77e8420f47425b155fb
+  data.tar.gz: f6b0f21f92304b161e43c5ca367a0ac7f24ae087a987634d902cddd9807fad74
 SHA512:
-  metadata.gz: 16637ff46f4648a2cfc14404ed074d3cedc6fc08cabf0c46a6fa7a39553b8b78020907fde1d9024005cec3a4f2cdb4cb3e999802ec866a942c20c15be6c7af34
-  data.tar.gz: 458aa6deba73a571a07801bb3df445a4d08500a55c3d1d8c50b11df0811dbc785270fd1ef908ebbb58c6f53dfb1ecfc7013fe137e216e88bca81c9bc56d21fa4
+  metadata.gz: 443e74eb1cb032d04baa078c1114d1287fb8597258003755fcadbea0b750be074de302a97c794cadd15783c3251296bf3f40735bf96c4886a20cd036180cb93d
+  data.tar.gz: 2a93211667789f9430df9dee556ecb0aa05702f5c39fad23d3a6b803b946ba91c3af3df9d57fbd0a5c84bca40e8e7a4b3a741c920a59dbf1755e7325ab323d06

data/CHANGELOG.md

@@ -1,3 +1,128 @@
+###  0.35.0  (2026-09-15)
+
+A hardening round. Rust library consumers have migration work; CLI users mostly get fixes, plus a few rules made explicit.
+
+#### What you must do
+
+**Rust library consumers.** None of this changes the CLI.
+
+- **Import from the crate root.** The modules are private; the public API is what `lib.rs` re-exports. Gone from it: the `Storage` trait, `open_storage` and the storage backends; `Corpus::storage()`, `storage_mut()`, `new_with_classifier`, `stats_for` and `max_values_per_position`; the `Corpus.classifier` and `Corpus.host_strategy` fields (use `set_host_strategy`); `DEFAULT_CLASSIFIER`, `SegmentClassifier`, the recognizer types and `SynthesizedRecognizer`; `Clusterer`, `ClusterKey`, `cluster_key_for*` and `ExplainEntry`; `Event`, `Observation`, `Shape`, `ShapeRenderOptions` and `PathShape`; `derive_hints*`, `NormalizationEvidence`, `NullEvidence` and `normalize_identifier_with_evidence`; `explain_identifier`; the classifier helpers (`display_type`, `color_kind`, `file_kind`, `param_name_hint`, `canonical_currency`, `canonical_date`); `DEFAULT_MAX_VALUES_PER_POSITION` and the corpus tuning constants. They let one corpus change classification for every other corpus and for `normalize()`, or put a corpus out of step with its own source log.
+- **Handle `iriq::Error` with `?`.** Every `Corpus` operation returns `iriq::Result<T>`: `open`, `observe`, `observe_iri`, `save`, `close`, `reinfer`, `activate_proposal`, `activate_proposals_above`, and every read (`normalize`, `normalize_identifier`, `explain`, `params_for`, `clusters`, `size`, `host_counts`, `path_length_counts`, `raw_shape_counts`, `fingerprint_counts`, `cross_host_shapes`, `propose_recognizers`, `observed_iri_count`, `activated_recognizer_count`). It replaces `std::io::Error` and `ParseError` in corpus code. `Error` is `#[non_exhaustive]`, with `Parse`, `Io`, `Corrupt`, `Unsupported` and, with the `sqlite` feature, `Sqlite`; each storage variant carries the corpus `path`, `Display` names the corpus, and `source()` is the cause. An in-memory corpus never fails a read, but the type can't know that. `Corpus::explain` and `Corpus::params_for` now return `Error::Parse` for input that doesn't parse, instead of an empty list. The pure functions (`parse`, `normalize`, `trace`, `explain`) still fail only with `ParseError`, whose field is now private (use `.message()`) and whose `Display` no longer starts with `iriq: `.
+- **Return `iriq::Result<T>` from `batch` closures.** On SQLite a batch commits on `Ok` and rolls back on `Err` or a panic. A batch inside a batch, including an `observe` inside one, joins the outer transaction: only the outermost commits or rolls back, so an error you swallow inside keeps its writes.
+- **Update changed signatures.** `Corpus::open` and `save` take `impl AsRef<Path>`. `save` exports only JSON: pointed at another path with a SQLite extension, it returns `Error::Unsupported` and writes nothing (it used to write JSON under a `.db` name nothing could reopen). `normalize_identifier(&iri, hints)` and `trace_identifier(&iri, hints)` no longer take a classifier, and `Corpus::normalize_identifier` gains the same `hints` argument (`true` gives the old output). `cross_host_shapes(&corpus, n)` is now `corpus.cross_host_shapes(n)`. `activate_proposals_above` returns only the recognizers it newly activated. `ParamSummary`'s `value_distribution`, `subtype_distribution` and `kind_distribution` are ordered `Vec<(K, f64)>`, and `SegmentPositionStat.values` is a `Vec<(String, usize)>`, most common first (see `cluster -J` below).
+- **`.clone()` a `SegmentType`; it's no longer `Copy`.** Clone one you copied out of a borrow (`row.ty`, `hint.ty`), and match through a reference (`matches!(&t, SegmentType::Custom(c) if …)`). `SegmentType::as_str` and `CustomType::as_str` return a `&str` borrowed from the value; `.to_string()` it to keep it. Why: custom type names were interned and never freed, so every distinct name a process read stayed in memory until exit.
+- **Build custom types with `segment_type_from_name`.** `SegmentType::Custom` wraps a `CustomType` instead of a `&'static str`, so `SegmentType::Custom("integer")`, which silently differed from `SegmentType::Integer`, no longer compiles. `segment_type_from_name("ghp")` returns the built-in variant for a built-in name.
+- **Allow for `#[non_exhaustive]`.** Add a `_ =>` arm when matching `SegmentType`, `FileKind`, `Kind`, `PositionScope`, `Classification` or `HostStrategy`. The structs `Cluster`, `ParamSummary`, `SegmentPositionStat`, `PositionStats`, `Extractor`, `Identifier`, `Position`, `CorpusEntry`, `CrossHostShape`, `RecognizerProposal`, `ProposalOptions`, `SegmentHint`, `TraceResult` and `TraceRow` can't be built from literals: use `Cluster::new`, `PositionStats::new`, `Extractor::new()` or `ProposalOptions::default()`, then assign fields. `FileKind` gains `Unknown`, used only as `kind_distribution`'s bucket for unrecognized extensions.
+
+**CLI users.**
+
+- **Rebuild a `--host reg` corpus that saw hostnames ending in a dot.** The Rust CLI filed `api.foo.com.` and `api.bar.com.` together under `com.`. `iriq --corpus PATH --host reg --reinfer` splits them.
+- **Run `--reinfer` once if a `.db` may be out of step with its log.** Three ways that happened: the Rust CLI's `--reinfer` (or `--activate-above`, which reinfers) overlapped another process writing the corpus; a long-running writer (`tail -f … | iriq -n`) kept its old recognizers after one was activated elsewhere; or an activation was killed after storing its recognizer but before its reinfer finished. Rerunning `--activate-above` doesn't repair the last one, because it now sees the recognizer as already held. Once writes stop, `iriq --corpus PATH --reinfer`.
+- **Reinfer a Rust-CLI corpus with an activated recognizer** if it saw values that start with the prefix but aren't just letters and digits after it (`ghp_abc-def`, `ghp_a.b`, a bare `ghp_`). The Rust CLI filed them under the recognizer's type.
+- **Reinfer a `.json` corpus the Rust CLI wrote or opened** to get back its params' numeric ranges (`min`, `max`, `avg`).
+- **Expect status 141 from `iriq … | head`.** The Rust CLI now stops when its reader goes away, so a `set -o pipefail` pipeline ending in `| head` fails with 141, as it already did with the Ruby CLI.
+- **Expect corpus-mode `-n` output to change** at slots seen fewer than 5 times (see the evidence rule below). Script against it accordingly.
+- **Clear recognizers activated under a built-in type name.** `--propose-recognizers` could name a proposal after a built-in type (`literal_` proposed `literal`), and activating it turned every matching value into a fixed literal. `literal_` now proposes `literal_id`, but a recognizer already activated under a built-in name stays until `--reset` or a fresh corpus.
+
+#### Changed
+
+Both runtimes unless noted.
+
+- **A positional that names an existing file is read as that file,** unless it contains `://`. `iriq access.log` used to parse as the host `https://access.log/`, and `iriq --stats access.log` reported 0 observations. A path-like argument (`/x`, `./x`, `../x`) that doesn't exist, or any missing file after `cluster`, is now `iriq: no such file: PATH` (exit 1, JSON code `file_not_found`) instead of a parse error or a Ruby backtrace. A bare name that isn't a file (`nope.log`) still parses as a host.
+- **The corpus changes a shape only where it has evidence:** a position or param it has seen at least 5 times. Below that, output is exactly what `-C` gives. One sighting used to be enough to print a raw `v1` / `abc-123` / `usd` instead of `{version}` / `{post_id}` / `USD`, and corpus mode skipped param-name hints (`?phone=unknown` → `{phone}`).
+- **Dates and currencies always print canonically** (`2024-01-15`, `USD`), with or without a corpus; corpus mode no longer renders `{currency}`. Currency upcasing is ASCII-only (Ruby turned `uſd` into `USD`).
+- **`-N` / `--no-hints` works with a corpus.** It was ignored whenever a corpus was in play, which is by default. A slot that only the corpus knows is variable renders `{value}`.
+- **Piped `-n`, `-c`, `-p` and `-e` stream, rendering from the corpus as it stands.** Each extracted URL is observed, then rendered from the corpus at that moment, and output is flushed as it goes, so `tail -f app.log | iriq -n` works with the default corpus. The Rust CLI used to print nothing until stdin closed whenever a corpus was in use; Ruby rendered piped input mechanically, and piped `-e` printed only the `# URL` header. With a warm corpus, the output matches single-input mode; with a cold one, a slot stays literal until it has been seen 5 times. On SQLite each chunk of lines commits before it prints, so a killed `iriq` never printed a line its corpus lost. `--json` (one array) still prints at the end.
+- **A closed stdout stops the Rust CLI** quietly at its next write, with status 141 (the status Ruby's CLI gets from SIGPIPE); the corpus still saves everything read. `iriq -C -n big.log | head -1` used to read the whole file, and `tail -f access.log | iriq -n | head -1` never exited. Any other stdout failure exits 1 with `iriq: stdout: CAUSE` (JSON code `stdout_error`) instead of exiting 0 with the output lost.
+- **`--host` applies with `-C`.** It was ignored there, so `-C --host reg cluster` kept subdomains apart.
+- **Corpus files iriq can't safely use are refused and left untouched**, exiting 1 with `iriq: corpus PATH: REASON` (JSON code `corpus_error`): a `.json` file that isn't valid JSON or has none of a corpus's top-level keys (`{}` is still an empty corpus); a SQLite corpus written by a newer iriq (`schema version N is newer than this iriq supports (4); upgrade iriq`); a `.db` that isn't a database or can't be opened or written. Every corpus failure, read or write, now uses this form.
+- **Unreadable input is a clean error:** invalid UTF-8 is `iriq: stream did not contain valid UTF-8` (JSON code `invalid_utf8`); a file without read permission, or stdin that is a directory, is `read_error`. Both exit 1 and honor `--json`. Ruby printed backtraces.
+- **JSON corpora save through a writer-unique temp file** (`PATH.PID.N.tmp`), so concurrent saves no longer crash with `No such file or directory`. A `.json` corpus is still single-writer, and the last save wins; use `.db` for concurrent writers. `--reset` removes these temp files along with the corpus, its `-wal`/`-shm` sidecars and `PATH.tmp`, and nothing else.
+- **Processes take turns with a SQLite corpus.** A writer waits up to 10 seconds for the write lock, then exits 1 with `iriq: corpus PATH: another process held the corpus lock for over 10s`. Long jobs share it: an ingest (`cluster`, `--stats`, or piped input with no section flag) commits about once a second, and `--reinfer` and `--activate-above` rebuild without the lock and take it only to install the result (about 1.4 s for 1.2 million observations), so a `tail -f app.log | iriq -n` stream keeps going beside them. Two catches: an ingest that's killed or fails keeps what it committed, so running the same input again counts those IRIs twice; and `--reinfer` needs free space in `TMPDIR`, more than the corpus file's own size. Ruby's `--reinfer` used to fail with `corpus PATH: database is locked` while another process was writing. For the Rust CLI's `--reinfer` bug, see Fixed.
+- **`cluster -J` lists keys in one order,** the same from both runtimes and every backend: a segment's `values`, and a param's `value_distribution` and `kind_distribution`, most common first, ties by value; `subtype_distribution` `integer` before `float`. The Rust CLI's order changed from run to run, and Ruby's segment `values` depended on the backend.
+- **Library: `observe_all`** (`Corpus::observe_all` in Rust, `Corpus#observe_all` in Ruby) observes many IRIs, committing about a second at a time on SQLite so other processes get turns. Inside `batch` it joins the batch's transaction.
+- **`--help` says what the CLI does:** a file argument needs no `./`; streaming NDJSON needs a section flag (`tail -f app.log | iriq -nJ`); `--host` keys IRIs as they're observed and on `--reinfer`, and doesn't re-key an existing corpus's report; `--reset` also removes SQLite sidecars and JSON temp files; `cluster` shows every cluster in the corpus; `-e` stays mechanical even with a corpus.
+- **The "created corpus" notice prints only once a corpus has opened.** Ruby printed it before an open that then failed (a `.db` in a read-only directory, say), so the error followed a false announcement.
+- `--propose-recognizers --activate-above F` with nothing to activate says `no proposals at or above confidence F`. It said "coverage", but `F` is a confidence.
+- `--host bogus` reports `invalid argument: --host bogus (expected full|registrable|reg|none)`, without the doubled `--host`.
+- Ruby `--explain`: an already-canonical date or currency shows its value, agreeing with the normalized line, instead of `{date}` / `{currency}`; the JSON omits `host` for an IRI without one instead of emitting `null`. Rust already did both.
+- **Rust crate:** `Corpus` implements `Debug`; `Cluster`, `ParamSummary`, `PositionStats`, `SegmentPositionStat`, `Identifier` and `OrderedMap` implement `PartialEq`. The crate declares `rust-version = "1.85"`. docs.rs builds with all features and labels the SQLite-only error variant. The crate README is the crate docs, so its samples are compiled and run as doctests. The published crate no longer ships integration tests, which could only fail outside the repository. The CLI detects a terminal with `std::io::IsTerminal` instead of an `unsafe`, Unix-only `isatty` call.
+
+#### Fixed
+
+**Rust**
+
+- **Non-ASCII digits crashed the classifier** (exit 101) in a segment or query value that mixed ASCII digits with another script's (`https://x.com/1०००००००`, `?d=2024-٠١-١٥`). "Digit" and "space" now mean ASCII, as in Ruby, which also settles values the runtimes classified differently: `?v=1.٥` was `{float}`, `?v=v𝟎` was `{version}`, `?u=http://x.com/a<NBSP>b` stayed literal, and `registrable_domain("١.٢.٣.٤")` was treated as an IPv4 address.
+- **Hostnames keyed differently from Ruby.** `--host reg` keyed trailing-dot hosts (`api.foo.com.`) as `com.`; they now key as `foo.com`. A host ending a word in capital sigma (`ΑΣ-x.com`) lowercased to the final form `ας-x.com`; it is now `ασ-x.com`.
+- **IP addresses rendered as `{ipv4}` / `{ipv6}` with a corpus**, which is by default. They're `{ip}`, as with `-C` and in Ruby.
+- **SQLite failures were silent.** A read-only, full or otherwise failing `.db` accepted every observation, wrote nothing, and the CLI exited 0, so a cron job could lose data indefinitely. A failed read was just as quiet: `-n` printed a mechanical shape, `--stats` dropped the row, and some reads panicked. Both now exit 1 with `iriq: corpus PATH: CAUSE`.
+- **An observation that failed part-way stayed half-recorded** in a `.db` (a full disk, a lock timeout), leaving its views out of step with the source log until `--reinfer`. Each observation is now one transaction, as in Ruby.
+- **`--reinfer` could corrupt a `.db` other processes were writing.** It read the log, cleared the views and replayed in separate steps, so it either exited 0 with views that no longer matched the log, or died with `UNIQUE constraint failed: cluster_examples…` after the clear had committed. What to do is under What you must do.
+- **An activated recognizer matched too much.** After activating `ghp_`, the Rust CLI rendered `ghp_abc-def`, `ghp_a.b`, `ghp_x_y` and a bare `ghp_` as `{ghp}`. A recognizer now matches only its prefix followed by letters and digits, as the whole segment, as in Ruby. What to do is under What you must do.
+- **`.json` corpora lost their numeric ranges.** The Rust CLI saved a `.json` corpus without the data behind a param's `min`, `max` and `avg`, so Ruby reading one showed no ranges, and a Rust run over a Ruby-written file removed them for good. It now writes them, and an unchanged corpus saves to the same bytes every run.
+- **Exiting waited on other processes.** A run that had already printed its output waited, before exiting, for another process's read snapshot or long write on the same `.db` to clear. It now exits right away.
+- **A `Corpus` on SQLite could wedge for the rest of the process.** A panic inside `batch` left its transaction open, so later "successful" writes rolled back at close; a panic while the connection lock was held poisoned it; iterating the corpus while normalizing against it deadlocked; and a `COMMIT` that SQLite refused but kept open made every later observation fail with `cannot start a transaction within a transaction`. Each now recovers.
+- **`Corpus::save` could overwrite a live SQLite corpus** with a JSON export when given another spelling of its own path (`dir/./c.db`). It now resolves both paths and flushes in place.
+- **`Corpus::open` accepted a JSON corpus in a directory that doesn't exist** and failed only at save, with an error that didn't name the file. It now fails at open, naming the path.
+- **The human cluster view printed large or rounded numbers wrong.** A param past 2^63 (`?v=18446744073709551616`) printed `9223372036854775807..9223372036854775807  avg 9223372036854775807`; it now prints the exact value, as Ruby does. Averages and ranges round like Ruby's elsewhere too: a non-whole value that rounds to a whole number keeps its `.0` (`avg 3.0`), `-0.004` rounds to `-0.0`, `1.005` to `1.01`, and values past 15 digits use `e+15` notation. `--json` is unchanged.
+- **`avg` could differ between runs over the same corpus** in its last digits (`100000000000002.52` vs `…2.53`), because reopening a corpus summed its values in hash-map order. The sum now follows the order the corpus stores them.
+- **`kind_distribution` dropped unrecognized extensions.** A `file` param seen as `b.pdf` ×3 and `c.zzz` reported `{"document":0.75}`; it now reports `{"document":0.75,"unknown":0.25}`, as in Ruby.
+
+**Both runtimes**
+
+- **A numeric param with a 310+ digit value** overflowed to infinity: Ruby crashed `cluster` (losing the batch before a JSON corpus saved), and Rust printed `null` for `max` and `avg`. Such values still count as observations but stay out of `min`, `max` and `avg`.
+- **An interrupted activation left its recognizer stored but not applied.** Activation stored the recognizer, then reinferred in a separate step; a process killed after the first step left the recognizer stored and the views not, or only partly, rebuilt. Storing and reinferring now commit together or not at all. What to do is under What you must do.
+- **Activating a recognizer the corpus already held wasn't a no-op.** It put a second copy on the live classifier and reran `reinfer`, and Ruby also stored a duplicate in a Memory or JSON corpus. Now it changes nothing, whichever runtime or binary stored the first: an activation is identified by its prefix and type, and the Rust CLI stores a recognizer's specificity as 1.0, as Ruby does. Recognizers older Rust binaries stored at 0.3 classify the same and need nothing.
+- **A running writer ignored recognizers another process activated,** and wrote clusters under the old types beside the new ones. Each write transaction (a streamed chunk, an ingest turn, a lone observation) now picks up activations committed before it began, at the cost of one small read, and only when another process has written since.
+
+**Ruby**
+
+- **Corpus failures were backtraces:** a `.db` that isn't a database, a read-only `.db`, a stored count hand-edited to a non-integer, an unreadable or directory `.json`, a corpus directory that can't be created, a failed JSON save. They're now `iriq: corpus PATH: REASON`, with the Rust CLI's reasons (`file is not a database`, `Invalid column type Text at index: 1, name: count`, `Permission denied (os error 13)`).
+
+#### Performance
+
+No action needed, and output is unchanged. Rust timings are single runs over 20,000 URLs, v0.34.0 against now.
+
+- **Rust: corpus-informed `-n` no longer slows down as the corpus grows.** Shaping each URL read every value and example the corpus held for its route, so each line cost more than the last; it now reads only the counts classification uses. One route into a fresh corpus: `.db` 22 s → 0.4 s, `.json` 2.2 s → 0.1 s.
+- **Rust, SQLite: observing skips redundant counting.** A known route no longer scans every route to number itself, and a new value in a path slot or query param no longer re-counts the values already there (a batch remembers the count, and forgets it when another process writes). `--stats` over one route: 3.1 s → 0.3 s; over 1,000 hosts: 0.9 s → 0.4 s; with a unique `?q=` on every URL: 2.2 s → 0.2 s. `-n` with a unique `?q=` on every URL is still slow (27 s → 17 s).
+- **SQLite: a big ingest no longer shuts out other writers.** `cluster` and `--stats` held the write lock from their first observation to their last; they now hold it about a second at a time. A one-line `-n` run beside a 1.2-million-line `cluster` waited at most 1.3 s; with v0.34.0 it waited 258 s, for the whole ingest.
+- **Ruby, SQLite: `-n` with a corpus does about 50× less work.** Normalizing read every value tracked at each path position and the whole cluster, observing reloaded the cluster, and each new value re-counted its position. 10,000 URLs into a fresh `.db` went from about 150 s to 3 s of CPU.
+
+###  0.34.0  (2026-08-12)
+- **Rust: SQLite is now an optional (default-on) feature.** `cargo install iriq`, the Homebrew formula, and the CLI are unchanged — `default = ["sqlite"]`. Library consumers who only need parsing, extraction, or normalization can now take `iriq = { version = "0.34", default-features = false }` and skip the bundled C SQLite build entirely (`rusqlite` is the crate's only non-Rust dependency). Without the feature, `open_storage` rejects `.db`/`.sqlite`/`.sqlite3` paths with a clear `Unsupported` error and the CLI's auto-default corpus becomes `default.json` instead of `default.db`; Memory and JSON backends are unaffected. No behavior change with default features, so Ruby parity and the shared schema are untouched. CI now gates the no-default-features build (test + clippy) so the configuration can't rot.
+
+###  0.33.0  (2026-07-07)
+- **New: end-to-end URL calibration corpus** — 160 messy real-world-shaped inputs (tokens, i18n, encoding damage, legacy endpoints, non-http schemes, garbage) with adjudicated expected templates at `spec/fixtures/calibration/urls.json`, generated by `script/build_url_calibration.rb`, asserted by both runtimes, and CI-gated for staleness. Building it caught the three fixes below.
+- **Bugfix: opaque non-urn schemes were rewritten to `urn:` on normalize/canonical** — `mailto:support@foo.com` came back as `urn:support@foo.com`; same for `tel:`, `sms:`, `data:`, `blob:`, `magnet:`. The scheme is now preserved (both runtimes; cluster keys too).
+- New `:font` file kind (`woff`/`woff2`/`ttf`/`otf`/`eot`), plus `m3u8` (video) and `map` (web) extensions — `Inter-Bold.woff2` now normalizes to `{file}` instead of `{font_id}`.
+- **Bugfix (Rust):** out-of-range ports (`:99999`) were rejected where Ruby's deliberately lenient parser accepts them — the port field is now `Option<u64>` matching Ruby's nil-vs-integer semantics.
+- **Bugfix (Ruby, SQLite backend):** rolling numeric stats (`min`/`max`/`avg` on numeric params) were never restored when loading a corpus from disk, so numeric ranges vanished from cluster output after a reopen — diverging from the Memory/JSON backends and from Rust. Now recomputed from tracked value counts on load; locked in by a new parity scenario and storage specs.
+- Perf: trimmed hot-path allocations in both runtimes — Rust's classifier no longer clones the recognizer list (and locks once, not three times) per cache miss; Ruby builds each observation's Shape once instead of twice, passes the recognizer array without a splat, and drops a redundant `to_s`. No behavior change.
+- **The Rust CLI's `completion` subcommand now matches Ruby exactly.** It previously carried its own divergent bash/zsh scripts plus a fish script Ruby never supported; it now embeds byte-identical copies of the shared bash/zsh scripts (parity-tested), defaults the shell from `$SHELL` like Ruby, and emits Ruby's `unknown_shell` error envelope. Fish is no longer supported. The `completion` subcommand is also listed in `--help` now.
+- **Rust now supports dynamic synthesized types**, closing the last known Ruby↔Rust divergence: `--activate-above` activates a proposal under its suggested type (`activated: ghp (ghp_)`) instead of falling back to `opaque_id`, and the activated type drives `{ghp}`-style placeholders, survives corpus reopen (JSON + SQLite), and round-trips storage — parity-tested. Implementation: `SegmentType::Custom(&'static str)` backed by a leak-once interned name, keeping the enum `Copy`. Also fixes a latent Rust bug where `string`-typed param counts were dropped when reloading a JSON corpus (a hand-copied type table was missing the `string` entry).
+- **The Go port is retired.** Ruby (reference) + Rust (shipped CLI via Homebrew/crates.io) continue; the `go/` module, its CI, and the Ruby↔Go / Rust↔Go parity harnesses are removed. `script/cli_parity.sh` now diffs Ruby ↔ Rust directly. Go consumers can pin `github.com/dpep/iriq/go@v0.32.1`, the last tag with Go support. `.db` corpora written by the Go binary still open cleanly (shared schema v4).
+
+###  0.32.1  (2026-06-26)
+- Go and Rust `cluster --json` now include the per-param `values` and `value_distribution` (plus `subtype_distribution`, `kind_distribution`, and numeric `min`/`max`/`avg`) that Ruby already emitted — the cluster JSON is now identical across all three runtimes. No change to the human-readable cluster view.
+- Test coverage: new `param_summary` golden fixture exercises the const → string → enum ladder + confidence across Ruby/Go/Rust (`go test` / `cargo test`), and the parity harnesses gained a key-order-agnostic JSON comparison (`jq -S` + number canonicalization) with a `cluster --json` scenario.
+
+###  0.32.0  (2026-06-26)
+- **Query params now climb a confidence ladder: constant → string → enum.** A param with a single observed value is a constant (rendered as its value); one that varies across free-form literal values is the new `string` type (renders `{string}`); a bounded, well-supported value set is `enum` (renders `{enum}`). Previously a varying literal param just echoed whatever value you passed.
+- **Enum detection is now coverage-based and straggler-robust.** An enum is promoted when its *established* values (each seen ≥ `ENUM_MIN_VALUE_COUNT`, now 3) number 2–10 and cover ≥ 90% of observations. A single brand-new value no longer knocks an established enum back down — fixing the observe-before-normalize order dependence where normalizing `?status=<new>` could flip the type. A lone repeated value is now correctly a constant, not a one-member enum.
+- **New `confidence` score on every param** (`total / (total + 15)`): a 0–1 figure of how much evidence backs the classification, shown in the cluster view (`status  enum  conf 0.93  ...`) and the cluster JSON. The type is the guess; confidence says how sure.
+- Ruby, Go, and Rust ship this together — same thresholds, same rendering, parity preserved (68/68 Ruby↔Go, 60/60 Rust↔Go).
+
+###  0.31.1  (2026-06-26)
+- Docs: clarified the README around corpus-on-by-default — sharper IRI definition (URLs are one member of the family, alongside URNs, `mailto:`, and internationalized addresses), an accurate worked example of the corpus learning a query param is an `enum`, and a streaming example (`tail -f access.log | iriq -J`). Added the streaming example to `--help` (Ruby + Go). No behavior change.
+
+###  0.31.0  (2026-06-26)
+- **Corpus is now on by default.** Every invocation observes into a persistent corpus, so classification gets sharper the more you use the tool — the streaming/learning behavior that was the selling point is no longer hidden behind a flag. Default location: `$XDG_DATA_HOME/iriq/default.db` on Linux, `~/Library/Application Support/iriq/default.db` on macOS, `%LOCALAPPDATA%/iriq/default.db` on Windows. First-run creation prints a one-line stderr notice.
+- New flag `-C` / `--no-corpus` (and `IRIQ_NO_CORPUS=1` env) — disables the default corpus for a single invocation. Explicit `--corpus PATH` always wins, even with `--no-corpus` set in the env.
+- New env var `IRIQ_CORPUS=PATH` — overrides the default location without needing `--corpus` on every call.
+- New flag `--reset` — deletes the resolved corpus file (and SQLite `-wal` / `-shm` sidecars) and exits. Honors `--corpus` / `IRIQ_CORPUS` so you can reset a non-default file.
+- **Go build simplification:** the slim / SQLite build split is gone. The Go binary now always links `modernc.org/sqlite` (pure Go, no cgo). `make build-sqlite`, `make release-sqlite`, and `-tags sqlite` are retired. The `dpep/tools/iriq-sqlite` Homebrew formula folds into `dpep/tools/iriq`.
+- Ruby, Go, and Rust ship this together — same defaults, same flags, parity preserved.
+
 ###  0.30.2  (2026-06-23)
 - Piped stdin and `--file` now **stream** the per-IRI sections (`-n`/`-p`/`-c`/`-e`) line by line, flushing each IRI as it's processed — `tail -f access.log | iriq -n` is live and memory stays bounded on huge inputs. Output is byte-identical to before; the aggregate views (deduped URL list, clusters, `--stats`) still read the whole input. Ruby, Go, and Rust.
 

data/README.md

@@ -1,5 +1,5 @@
-Iriq
-======
+# Iriq — IRI Query
+
 [![codecov](https://codecov.io/gh/dpep/iriq/branch/main/graph/badge.svg)](https://codecov.io/gh/dpep/iriq)
 
 **Iriq finds the *shape* of a URL** — the structural template you get when you
@@ -9,17 +9,18 @@ URLs — a log file, a column of links, free-text prose — and it collapses the
 into a small set of stable, deterministic route templates. Fifty thousand
 distinct URLs become twelve shapes.
 
-(An **IRI** is just a URL — the internationalized superset of URI/URL that also
-allows non-ASCII characters. If you know URLs, you know IRIs. The name is *IRI
-Query*: iriq queries an IRI for its structure.)
+What's an IRI? Internationalized Resource Identifiers cover everyday URLs `https://…`, plus URNs like `urn:isbn:0451450523`,
+other schemes like `mailto:`, and internationalized addresses with non-ASCII
+characters like `https://例え.jp/パス`. Formally it's the Unicode superset of
+URI/URL. The name is *IRI Query*: iriq queries an IRI for its structure.
 
 Everything iriq does — parsing, normalizing, classifying path and query
 components, clustering, learning new patterns — exists to derive, render, or
 group by that shape.
 
-And it gets sharper the more you feed it. Point a *corpus* at a stream and
-classifications improve as data flows in — high-churn slots get promoted to
-placeholders, and whole types emerge that you can't see in any single URL (a
+And it gets sharper the more you feed it. A *corpus* — on by default — records
+what it sees and improves classifications as data flows in: high-churn slots get
+promoted to placeholders, and whole types emerge that no single URL can reveal (a
 position that's always 100–599 is an HTTP status; one bounded to a dozen values
 is an enum).
 
@@ -61,29 +62,93 @@ $ iriq -n https://shop.com/pricing/usd?currency=eur
 https://shop.com/pricing/USD?currency=EUR     # currency upcased
 ```
 
+Pipe in text, or name a file, and iriq extracts every URL in it:
+
+```sh
+$ cat urls.log | iriq                         # ≥ 10 IRIs → cluster view
+[6] api.example.com  /api/{version}/users/{user_id}
+    https://api.example.com/api/v1/users/123
+    https://api.example.com/api/v1/users/456
+    https://api.example.com/api/v1/users/789
+    + 3 more
+
+[3] api.example.com  /orders/{order_uuid}
+    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
+    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
+    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
+    status  string  conf 0.17  (2 distinct, 100%)
+
+[3] api.example.com  /products/{product_id}
+    https://api.example.com/products/blue-widget
+    https://api.example.com/products/red-gadget
+    https://api.example.com/products/green-gizmo
+
+$ cat urls.log | iriq --stats                 # rolling aggregates
+$ iriq urls.log -n                            # a file argument → normalize each URL
+$ iriq -nJ < urls.log                         # one JSON line per URL
+$ iriq --corpus team.db < urls.log            # use a specific corpus file
+```
+
+Reading a web server's access log? Its request lines have no host, so see
+[Access logs](#access-logs) first.
+
+Per-IRI sections (`-n`, `-c`, `-p`, `-e`) stream: each line is read, observed,
+rendered from the corpus as it stands, and flushed, so iriq works on an
+unbounded live feed:
+
 ```sh
-$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
-[190] docs.example.com  /users/{user_id}
-[186] app.example.com   /users/{user_id}
-...
-
-$ cat access.log | iriq --stats               # rolling aggregates
-$ iriq ./access.log -n                        # auto-detect file → normalize each
-$ iriq -J < access.log                        # newline-delimited JSON
-$ iriq --corpus c.db < access.log             # persist into a SQLite corpus
+$ tail -f app.log | iriq -n                   # one shape per line, as logs land
+$ tail -f app.log | iriq -nJ                  # same, as newline-delimited JSON
 ```
 
-Once a corpus has data, `-n` becomes corpus-informed — a position that only ever
-holds integers clusters to a single `{user_id}` shape, and new values normalize
-to it:
+`-J` on its own doesn't stream. Like the default view, it waits for the end of
+input, then prints the URL list (fewer than 10 IRIs) or one object per cluster.
+
+**Every invocation observes into a persistent corpus by default**, so iriq gets
+smarter the more you run it. The corpus-only types (e.g. `enum` / `http_status`)
+emerge from the *distribution* of values observed.
 
 ```sh
-$ for n in 1 2 3 4 5 6 7 8 9 10; do
-    iriq --corpus c.db https://api.foo.com/users/$n >/dev/null
+# Feed a stream where ?status only ever holds a couple of words:
+$ for n in $(seq 1 20); do
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=open"   >/dev/null
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=closed" >/dev/null
   done
 
-$ iriq -n --corpus c.db https://api.foo.com/users/999
-https://api.foo.com/users/{user_id}
+# Ask what it learned. ?status is now an enum — a verdict no single URL
+# could support, since one URL shows only one value:
+$ iriq --corpus demo.db cluster
+[40] api.foo.com  /orders/{order_id}
+    https://api.foo.com/orders/1?status=open
+    https://api.foo.com/orders/1?status=closed
+    https://api.foo.com/orders/2?status=open
+    + 37 more
+    status  enum  conf 0.73  (2 distinct, 100%)
+```
+
+`conf` is how much evidence backs the type, from 0 to 1. These learned types
+also flow into normalized output:
+
+```sh
+$ iriq --corpus demo.db -n 'https://api.foo.com/orders/99?status=open'
+https://api.foo.com/orders/{order_id}?status={enum}
+```
+
+The corpus only acts on evidence: it changes a shape only at a position or
+param it has seen at least 5 times. Until then, `-n` prints exactly what `-C`
+would. Dates and currencies always print canonically (`2024-01-15`, `USD`).
+
+The default corpus lives at `$XDG_DATA_HOME/iriq/default.db` (Linux),
+`~/Library/Application Support/iriq/default.db` (macOS), or
+`%LOCALAPPDATA%/iriq/default.db` (Windows). First-run creation prints a
+one-line stderr notice. Three knobs control it:
+
+```sh
+$ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
+$ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
+$ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
+$ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
+$ iriq --reset                                     # delete the corpus and exit
 ```
 
 ### Two ways to normalize
@@ -95,8 +160,9 @@ Pick by the question you're asking:
   lowercased, default port dropped; path and query left alone). Handy, but
   table stakes — plenty of libraries do it.
 - **`--normalize`** *(the default)* — find the URL's *shape*, erasing the
-  specifics into placeholders. `…/pull/42` → `…/pull/{id}`. This is the part
-  you came to iriq for.
+  specifics into placeholders. `…/pull/42` → `…/pull/{pull_id}`. A shape
+  ignores the `#fragment`, so `-n` drops it. This is the part you came to iriq
+  for.
 
 Same input, two questions: "what's the clean form of *this* URL?" vs "what
 *kind* of URL is this?" The second is iriq's reason to exist.
@@ -107,41 +173,13 @@ Same input, two questions: "what's the clean form of *this* URL?" vs "what
 # Homebrew (recommended)
 brew install dpep/tools/iriq
 
-# Cargo, from crates.io
+# Cargo, from crates.io (Rust 1.85 or newer)
 cargo install iriq
-
-# Cargo, from a source checkout
-cargo install --path rust/iriq
 ```
 
 One crate ships both the library and the `iriq` binary. Corpora persist to
 SQLite (bundled, WAL) out of the box — nothing to flag, install, or rebuild.
 
-## Use it as a Rust library
-
-```sh
-cargo add iriq
-```
-
-```rust
-use iriq::{parse, normalize, Corpus};
-
-let iri = parse("https://foo.com/users/123")?;
-iri.host;             // "foo.com"
-iri.path_segments;    // ["users", "123"]
-iri.canonical();      // "https://foo.com/users/123"
-
-normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"
-
-// Streaming clustering against a persistent corpus.
-let mut corpus = Corpus::open("c.db")?;
-corpus.observe("https://foo.com/users/1")?;
-corpus.save("c.db")?;
-```
-
-Full API on [docs.rs/iriq](https://docs.rs/iriq); see the
-[crate README](rust/iriq/README.md) for the library tour.
-
 ## Segment classification
 
 Iriq classifies each path/query segment into one of ~25 types — the first
@@ -180,12 +218,12 @@ what produces `{user_id}` from `/users/123` and `{order_id}` from `/orders/456`.
 Semantic types (`version`, `locale`, `currency`, `date`, `boolean`) skip the
 hint and surface as `{type}` — `/api/v1/status` renders as `/api/{version}/status`,
 not the misleading `/api/{api_id}/status`. Pass `-N` / `--no-hints` for
-mechanical placeholders (`{integer}` instead of `{user_id}`).
+mechanical placeholders (`{integer}` instead of `{user_id}`); a slot only the
+corpus knows is variable renders `{value}`.
 
 ### Types only the corpus can see
 
-Four types never come from a single URL — they emerge from the *distribution*
-of values a position has held across many observations:
+Four types emerge from the *distribution* of values across many observations:
 
 | Type | Emerges when a position… |
 | --- | --- |
@@ -195,18 +233,60 @@ of values a position has held across many observations:
 | `enum` | holds a small, bounded set of distinct values |
 
 Mechanically, `200` is just an integer. Across ten thousand URLs where that
-slot is always 100–599, it's an HTTP status. That's the corpus earning its keep.
+slot is always 100–599, it's likely an HTTP status.
 
 ## Corpus (streaming + learning)
 
-For processing many identifiers — possibly an unbounded stream — point iriq at a
-corpus. It maintains rolling aggregates and per-(host, prefix) frequency stats,
-so classification improves as more data comes in.
+The corpus maintains rolling aggregates and per-(host, prefix) frequency stats,
+so classification improves as more data comes in — handy for an unbounded stream
+of identifiers. The default corpus already persists; `--corpus PATH` points iriq
+at a specific file instead, to keep separate corpora or share one across runs.
+
+The extension picks the backend, and the two behave differently:
+
+- **`.db` / `.sqlite` / `.sqlite3` (SQLite)** — the default, and the one to
+  share. Many `iriq` processes can write at once by taking turns: a writer
+  waits up to 10 seconds for its turn. A big `cluster` commits about a second
+  at a time, and `--reinfer` rebuilds on the side and holds the corpus only to
+  swap the result in, so a `tail -f` stream keeps flowing beside either. Use
+  SQLite for streams and concurrent writers.
+- **Anything else (JSON)** — read when iriq starts and written once, when it
+  exits cleanly. It's single-writer: when two processes use one file, the last
+  to exit wins. A streaming run that's killed, Ctrl-C included, saves nothing.
+
+A few things to know:
+
+- The cluster view (`iriq cluster`, or 10+ piped IRIs) shows the whole corpus,
+  not just this input. Add `-C` to cluster one input on its own.
+- iriq keeps every IRI it observes, repeats included, so `--reinfer` can replay
+  them. That log grows without bound.
+- On SQLite, `cluster` and `--stats` commit their input about a second at a
+  time. If one is killed part-way, what it committed stays: feed it the same
+  input again and those IRIs count twice.
+- `--reinfer` (and `--activate-above`) rebuilds in temporary tables before
+  swapping the result in, so it needs free space in `TMPDIR`: plan on more than
+  the corpus file's own size.
+- `--reset` deletes the corpus file, its SQLite `-wal` / `-shm` sidecars, and any
+  temp files a JSON save left behind. Don't reset a corpus another process is
+  writing: that process carries on, exits 0, and its writes are lost with the
+  deleted file.
+- iriq refuses a corpus file it can't safely use rather than overwrite it — a
+  JSON file that isn't an iriq corpus, or a SQLite corpus written by a newer
+  iriq (upgrade to open it).
+
+### Host keying
+
+By default every hostname gets its own clusters. `--host reg` keys by
+registrable domain, so `api.foo.com` and `www.foo.com` both cluster under
+`foo.com`; `--host none` ignores the host. The mode applies when observations
+are recorded, when you `--reinfer`, and to a `-C` run's throwaway corpus. It
+doesn't re-key a report of an existing corpus: `iriq --host reg cluster` shows
+the clusters as they were recorded. To re-key a corpus, reinfer it:
 
-`--corpus PATH` makes the corpus survive across invocations. A `.db` /
-`.sqlite` / `.sqlite3` path is stored in SQLite (WAL journaling, incremental
-UPSERTs — multiple `iriq --corpus` processes can write concurrently); a
-`.json` path writes a plain JSON file instead.
+```sh
+$ iriq --corpus c.db --host reg --reinfer
+reinferred 3 observations: 3 → 1 cluster
+```
 
 ### Re-runnable inference
 
@@ -225,9 +305,7 @@ $ iriq --corpus c.db --reinfer
 Iriq doesn't just classify against a fixed list — it watches the stream and
 *proposes new recognizers* for patterns it keeps seeing. Notice `ghp_…` or
 `cus_…` recurring at a slug position and iriq will suggest a recognizer for it,
-with evidence: coverage, host count, confidence. Proposals are never
-auto-applied — you activate the ones you trust, and they persist with the
-corpus. Human-in-the-loop by design.
+with evidence: coverage, host count, confidence.
 
 ```sh
 # Print proposals (human-readable, or --json)
@@ -268,6 +346,42 @@ Known limitations (intentional):
 
 Disable scheme-less extraction with `--no-scheme-less`.
 
+### Access logs
+
+Extraction needs URLs with a host. A web server's request line
+(`"GET /api/v1/users/123 HTTP/1.1"`) has none, so on a raw access log iriq
+finds only the full URLs on each line, usually the Referer:
+
+```sh
+$ cat access.log | iriq
+[12] example.com  /referrer
+    https://example.com/referrer
+    + 11 more
+```
+
+Pull out the path and give it a host first. In the common and combined log
+formats, the path is the seventh field:
+
+```sh
+$ awk '{print "https://api.example.com" $7}' access.log | iriq
+[6] api.example.com  /api/{version}/users/{user_id}
+    https://api.example.com/api/v1/users/123
+    https://api.example.com/api/v1/users/456
+    https://api.example.com/api/v1/users/789
+    + 3 more
+
+[3] api.example.com  /orders/{order_uuid}
+    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
+    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
+    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
+    status  string  conf 0.17  (2 distinct, 100%)
+
+[3] api.example.com  /products/{product_id}
+    https://api.example.com/products/blue-widget
+    https://api.example.com/products/red-gadget
+    https://api.example.com/products/green-gizmo
+```
+
 ## How it works
 
 Under the shape sits one idea: **Position + Evidence**. A *Position* is a slot
@@ -282,21 +396,26 @@ underneath. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full model.
 **Single input** — combined parse + normalize summary; trim with section flags
 (`-p`, `-n`).
 
-**Piped stdin** — extraction runs by default. Output auto-switches: small inputs
-get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the cluster view via
-an ephemeral corpus.
+**Piped stdin, or a file argument** — extraction runs by default. With no
+section flag, iriq reads all of the input, then prints a deduplicated URL list
+(fewer than 10 IRIs) or the cluster view of the corpus (10 or more). With a
+section flag, it prints each IRI's result as the line arrives. `-n` is
+corpus-informed; `-e` is mechanical even with a corpus.
 
 | Flag                | Effect                                                  |
 | ------------------- | ------------------------------------------------------- |
 | `-p, --parse`       | Show parsed fields                                      |
 | `-n, --normalize`   | Show the shape-normalized form                          |
 | `-c, --canonical`   | Show the canonical form (no shape normalization)        |
+| `-e, --explain`     | Annotated trace — per-segment notes about why each placeholder / canonical value was chosen. Mechanical, even with a corpus |
 | `-j, --json`        | Emit JSON                                               |
-| `-J, --ndjson`      | Newline-delimited JSON (one object per line); implies `--json` |
+| `-J, --ndjson`      | Newline-delimited JSON; implies `--json`. With a section flag (`-nJ`), one line per IRI as it arrives; alone, the URL list or clusters at end of input |
 | `-N, --no-hints`    | Use `{integer}` etc. instead of `{user_id}`             |
 | `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
-| `--corpus PATH`     | Load/create a corpus at PATH (`.json` or `.db`/`.sqlite`/`.sqlite3`) |
-| `--host MODE`       | Host-keying for clustering: `full` (default), `reg` strips subdomains, `none` ignores host |
+| `--corpus PATH`     | Use a specific corpus file (`.json` or `.db`/`.sqlite`/`.sqlite3`). Overrides the default |
+| `-C, --no-corpus`   | Disable corpus persistence for this invocation (same as `IRIQ_NO_CORPUS=1`) |
+| `--reset`           | Delete the corpus file, its SQLite sidecars and JSON temp files, and exit |
+| `--host MODE`       | Host keying: `full` (default), `reg` strips subdomains, `none` ignores host. Applies when observing, to `--reinfer`, and with `-C` |
 | `--stats`           | Print rolling aggregates                                |
 | `--reinfer`         | Drop the materialized views and replay the source-IRI log through the current classifier + reducers |
 | `--propose-recognizers` | Scan observed values for shape patterns that recur enough to suggest a new recognizer. Combine with `--json` for structured output |
@@ -305,15 +424,70 @@ an ephemeral corpus.
 | `--min-coverage F`      | Proposal threshold; default 0.7                   |
 | `--min-hosts N`         | Threshold for both proposals and cross-host shapes; default 1 / 2 respectively |
 | `--activate-above F`    | With `--propose-recognizers`, auto-activate every proposal whose confidence is ≥ F |
+| `cluster [file]`        | Force the cluster view                            |
 | `completion bash\|zsh`  | Print shell completion script (Homebrew installs this automatically) |
 | `-V, --version`     | Print version                                           |
 
-A positional argument that doesn't parse as an IRI but IS an existing file is
-read and extracted from automatically — `iriq ./access.log` and
-`iriq /var/log/foo.log` Just Work. (Bare filenames like `README.md` may still
-parse as a URL; pipe with `cat` to disambiguate.)
+Environment variables:
+
+| Variable             | Effect                                                  |
+| -------------------- | ------------------------------------------------------- |
+| `IRIQ_CORPUS=PATH`   | Set the corpus path (overrides the default)             |
+| `IRIQ_NO_CORPUS=1`   | Disable the default corpus (equivalent to `-C`)         |
+
+A positional argument that names an existing file is read as a file, unless it
+contains `://` — `iriq access.log` and `iriq /var/log/foo.log` both work. A
+path-like argument (`/x`, `./x`, `../x`) that doesn't exist is an error; a bare
+name that isn't a file, like `foo.log`, parses as a host (`https://foo.log/`).
 
-Exit codes: `0` success, `1` usage error, `2` parse error.
+Errors go to stderr as `iriq: MESSAGE`, or, with `--json` / `-J`, as
+`{"error":{"code":"…","message":"…"}}`. A corpus error names the file:
+`iriq: corpus team.db: attempt to write a readonly database`.
+
+| Exit  | Meaning | JSON codes |
+| ----- | ------- | ---------- |
+| `0`   | Success | |
+| `1`   | Bad option or argument, missing or unreadable input, unusable corpus, or stdout failed | `option_error`, `unknown_shell`, `file_not_found`, `read_error`, `invalid_utf8`, `corpus_error`, `stdout_error` |
+| `2`   | The input isn't a parseable IRI | `parse_error` |
+| `141` | The reader went away (`iriq … \| head`); iriq stops quietly | |
+
+## Rust library
+
+```sh
+cargo add iriq
+```
+
+```rust
+use iriq::{normalize, parse, Corpus};
+
+fn main() -> iriq::Result<()> {
+    let iri = parse("https://foo.com/users/123")?;
+    println!("{} {:?}", iri.host, iri.path_segments); // foo.com ["users", "123"]
+    println!("{}", normalize("https://foo.com/users/123")?); // https://foo.com/users/{user_id}
+
+    // A persistent corpus: SQLite for .db, JSON otherwise.
+    let mut corpus = Corpus::open("c.db")?;
+    for n in 1..=3 {
+        corpus.observe(&format!("https://foo.com/users/{n}"))?;
+    }
+    for cluster in corpus.clusters()? {
+        println!("[{}] {} {}", cluster.count, cluster.host, cluster.shape); // [3] foo.com /users/{user_id}
+    }
+    corpus.save("c.db")?; // flushes in place; a .json corpus is written only here
+    Ok(())
+}
+```
+
+Every `Corpus` operation returns `iriq::Result`, whose `iriq::Error` names the
+corpus that failed. SQLite comes from the default-on `sqlite` feature;
+`cargo add iriq --no-default-features` drops it and keeps in-memory and JSON
+corpora. Requires Rust 1.85 or newer. A long-lived `Corpus` sees recognizers
+another process activated at its next `batch` or `observe`, not in reads
+outside one.
+
+The [crate README](rust/iriq/README.md) is the library tour: reading clusters
+and params, batches, sharing a corpus, and errors. Full API on
+[docs.rs/iriq](https://docs.rs/iriq).
 
 ## Limitations (intentional)
 
@@ -330,15 +504,3 @@ Iriq does **not**:
 
 Iriq's focus is the analysis side: classification, normalization, and clustering
 — not a complete URL implementation.
-
-----
-## Contributing
-
-Yes please  :)
-
-1. Fork it
-1. Create your feature branch (`git checkout -b my-feature`)
-1. Ensure the tests pass (`cd rust && cargo test`)
-1. Commit your changes (`git commit -am 'awesome new feature'`)
-1. Push your branch (`git push origin my-feature`)
-1. Create a Pull Request