iriq 0.33.0 → 0.35.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4290f75faee81b1292a3404fd01221299490731378ace043999e6ab7e7a0dc81
-  data.tar.gz: 0b1b074eb746ec2b50a790fb38da54c0715a420ec752929f3eb2af85bed50670
+  metadata.gz: aad1274de2d7cae934f5b940e97f498193883addffcab77e8420f47425b155fb
+  data.tar.gz: f6b0f21f92304b161e43c5ca367a0ac7f24ae087a987634d902cddd9807fad74
 SHA512:
-  metadata.gz: 81feb4d4b7c8fb0cd9f7e68dbec17fdc787642a6829f2f479849e458c14e5fa9812c251df08f57658b6cb338fcb112d8ec16393efb7aa4e4364e07b2dcf19ae1
-  data.tar.gz: 8979d705a2dd2d1db59c4b5fc794ba9307f061c2fbb7b3968c1585c8d633a7871d306c1ec372ad3dc15e11f1ec30c65a7f7833a6955dae8a3e11ca3f3a06d925
+  metadata.gz: 443e74eb1cb032d04baa078c1114d1287fb8597258003755fcadbea0b750be074de302a97c794cadd15783c3251296bf3f40735bf96c4886a20cd036180cb93d
+  data.tar.gz: 2a93211667789f9430df9dee556ecb0aa05702f5c39fad23d3a6b803b946ba91c3af3df9d57fbd0a5c84bca40e8e7a4b3a741c920a59dbf1755e7325ab323d06

data/CHANGELOG.md

@@ -1,3 +1,96 @@
+###  0.35.0  (2026-09-15)
+
+A hardening round. Rust library consumers have migration work; CLI users mostly get fixes, plus a few rules made explicit.
+
+#### What you must do
+
+**Rust library consumers.** None of this changes the CLI.
+
+- **Import from the crate root.** The modules are private; the public API is what `lib.rs` re-exports. Gone from it: the `Storage` trait, `open_storage` and the storage backends; `Corpus::storage()`, `storage_mut()`, `new_with_classifier`, `stats_for` and `max_values_per_position`; the `Corpus.classifier` and `Corpus.host_strategy` fields (use `set_host_strategy`); `DEFAULT_CLASSIFIER`, `SegmentClassifier`, the recognizer types and `SynthesizedRecognizer`; `Clusterer`, `ClusterKey`, `cluster_key_for*` and `ExplainEntry`; `Event`, `Observation`, `Shape`, `ShapeRenderOptions` and `PathShape`; `derive_hints*`, `NormalizationEvidence`, `NullEvidence` and `normalize_identifier_with_evidence`; `explain_identifier`; the classifier helpers (`display_type`, `color_kind`, `file_kind`, `param_name_hint`, `canonical_currency`, `canonical_date`); `DEFAULT_MAX_VALUES_PER_POSITION` and the corpus tuning constants. They let one corpus change classification for every other corpus and for `normalize()`, or put a corpus out of step with its own source log.
+- **Handle `iriq::Error` with `?`.** Every `Corpus` operation returns `iriq::Result<T>`: `open`, `observe`, `observe_iri`, `save`, `close`, `reinfer`, `activate_proposal`, `activate_proposals_above`, and every read (`normalize`, `normalize_identifier`, `explain`, `params_for`, `clusters`, `size`, `host_counts`, `path_length_counts`, `raw_shape_counts`, `fingerprint_counts`, `cross_host_shapes`, `propose_recognizers`, `observed_iri_count`, `activated_recognizer_count`). It replaces `std::io::Error` and `ParseError` in corpus code. `Error` is `#[non_exhaustive]`, with `Parse`, `Io`, `Corrupt`, `Unsupported` and, with the `sqlite` feature, `Sqlite`; each storage variant carries the corpus `path`, `Display` names the corpus, and `source()` is the cause. An in-memory corpus never fails a read, but the type can't know that. `Corpus::explain` and `Corpus::params_for` now return `Error::Parse` for input that doesn't parse, instead of an empty list. The pure functions (`parse`, `normalize`, `trace`, `explain`) still fail only with `ParseError`, whose field is now private (use `.message()`) and whose `Display` no longer starts with `iriq: `.
+- **Return `iriq::Result<T>` from `batch` closures.** On SQLite a batch commits on `Ok` and rolls back on `Err` or a panic. A batch inside a batch, including an `observe` inside one, joins the outer transaction: only the outermost commits or rolls back, so an error you swallow inside keeps its writes.
+- **Update changed signatures.** `Corpus::open` and `save` take `impl AsRef<Path>`. `save` exports only JSON: pointed at another path with a SQLite extension, it returns `Error::Unsupported` and writes nothing (it used to write JSON under a `.db` name nothing could reopen). `normalize_identifier(&iri, hints)` and `trace_identifier(&iri, hints)` no longer take a classifier, and `Corpus::normalize_identifier` gains the same `hints` argument (`true` gives the old output). `cross_host_shapes(&corpus, n)` is now `corpus.cross_host_shapes(n)`. `activate_proposals_above` returns only the recognizers it newly activated. `ParamSummary`'s `value_distribution`, `subtype_distribution` and `kind_distribution` are ordered `Vec<(K, f64)>`, and `SegmentPositionStat.values` is a `Vec<(String, usize)>`, most common first (see `cluster -J` below).
+- **`.clone()` a `SegmentType`; it's no longer `Copy`.** Clone one you copied out of a borrow (`row.ty`, `hint.ty`), and match through a reference (`matches!(&t, SegmentType::Custom(c) if …)`). `SegmentType::as_str` and `CustomType::as_str` return a `&str` borrowed from the value; `.to_string()` it to keep it. Why: custom type names were interned and never freed, so every distinct name a process read stayed in memory until exit.
+- **Build custom types with `segment_type_from_name`.** `SegmentType::Custom` wraps a `CustomType` instead of a `&'static str`, so `SegmentType::Custom("integer")`, which silently differed from `SegmentType::Integer`, no longer compiles. `segment_type_from_name("ghp")` returns the built-in variant for a built-in name.
+- **Allow for `#[non_exhaustive]`.** Add a `_ =>` arm when matching `SegmentType`, `FileKind`, `Kind`, `PositionScope`, `Classification` or `HostStrategy`. The structs `Cluster`, `ParamSummary`, `SegmentPositionStat`, `PositionStats`, `Extractor`, `Identifier`, `Position`, `CorpusEntry`, `CrossHostShape`, `RecognizerProposal`, `ProposalOptions`, `SegmentHint`, `TraceResult` and `TraceRow` can't be built from literals: use `Cluster::new`, `PositionStats::new`, `Extractor::new()` or `ProposalOptions::default()`, then assign fields. `FileKind` gains `Unknown`, used only as `kind_distribution`'s bucket for unrecognized extensions.
+
+**CLI users.**
+
+- **Rebuild a `--host reg` corpus that saw hostnames ending in a dot.** The Rust CLI filed `api.foo.com.` and `api.bar.com.` together under `com.`. `iriq --corpus PATH --host reg --reinfer` splits them.
+- **Run `--reinfer` once if a `.db` may be out of step with its log.** Three ways that happened: the Rust CLI's `--reinfer` (or `--activate-above`, which reinfers) overlapped another process writing the corpus; a long-running writer (`tail -f … | iriq -n`) kept its old recognizers after one was activated elsewhere; or an activation was killed after storing its recognizer but before its reinfer finished. Rerunning `--activate-above` doesn't repair the last one, because it now sees the recognizer as already held. Once writes stop, `iriq --corpus PATH --reinfer`.
+- **Reinfer a Rust-CLI corpus with an activated recognizer** if it saw values that start with the prefix but aren't just letters and digits after it (`ghp_abc-def`, `ghp_a.b`, a bare `ghp_`). The Rust CLI filed them under the recognizer's type.
+- **Reinfer a `.json` corpus the Rust CLI wrote or opened** to get back its params' numeric ranges (`min`, `max`, `avg`).
+- **Expect status 141 from `iriq … | head`.** The Rust CLI now stops when its reader goes away, so a `set -o pipefail` pipeline ending in `| head` fails with 141, as it already did with the Ruby CLI.
+- **Expect corpus-mode `-n` output to change** at slots seen fewer than 5 times (see the evidence rule below). Script against it accordingly.
+- **Clear recognizers activated under a built-in type name.** `--propose-recognizers` could name a proposal after a built-in type (`literal_` proposed `literal`), and activating it turned every matching value into a fixed literal. `literal_` now proposes `literal_id`, but a recognizer already activated under a built-in name stays until `--reset` or a fresh corpus.
+
+#### Changed
+
+Both runtimes unless noted.
+
+- **A positional that names an existing file is read as that file,** unless it contains `://`. `iriq access.log` used to parse as the host `https://access.log/`, and `iriq --stats access.log` reported 0 observations. A path-like argument (`/x`, `./x`, `../x`) that doesn't exist, or any missing file after `cluster`, is now `iriq: no such file: PATH` (exit 1, JSON code `file_not_found`) instead of a parse error or a Ruby backtrace. A bare name that isn't a file (`nope.log`) still parses as a host.
+- **The corpus changes a shape only where it has evidence:** a position or param it has seen at least 5 times. Below that, output is exactly what `-C` gives. One sighting used to be enough to print a raw `v1` / `abc-123` / `usd` instead of `{version}` / `{post_id}` / `USD`, and corpus mode skipped param-name hints (`?phone=unknown` → `{phone}`).
+- **Dates and currencies always print canonically** (`2024-01-15`, `USD`), with or without a corpus; corpus mode no longer renders `{currency}`. Currency upcasing is ASCII-only (Ruby turned `uſd` into `USD`).
+- **`-N` / `--no-hints` works with a corpus.** It was ignored whenever a corpus was in play, which is by default. A slot that only the corpus knows is variable renders `{value}`.
+- **Piped `-n`, `-c`, `-p` and `-e` stream, rendering from the corpus as it stands.** Each extracted URL is observed, then rendered from the corpus at that moment, and output is flushed as it goes, so `tail -f app.log | iriq -n` works with the default corpus. The Rust CLI used to print nothing until stdin closed whenever a corpus was in use; Ruby rendered piped input mechanically, and piped `-e` printed only the `# URL` header. With a warm corpus, the output matches single-input mode; with a cold one, a slot stays literal until it has been seen 5 times. On SQLite each chunk of lines commits before it prints, so a killed `iriq` never printed a line its corpus lost. `--json` (one array) still prints at the end.
+- **A closed stdout stops the Rust CLI** quietly at its next write, with status 141 (the status Ruby's CLI gets from SIGPIPE); the corpus still saves everything read. `iriq -C -n big.log | head -1` used to read the whole file, and `tail -f access.log | iriq -n | head -1` never exited. Any other stdout failure exits 1 with `iriq: stdout: CAUSE` (JSON code `stdout_error`) instead of exiting 0 with the output lost.
+- **`--host` applies with `-C`.** It was ignored there, so `-C --host reg cluster` kept subdomains apart.
+- **Corpus files iriq can't safely use are refused and left untouched**, exiting 1 with `iriq: corpus PATH: REASON` (JSON code `corpus_error`): a `.json` file that isn't valid JSON or has none of a corpus's top-level keys (`{}` is still an empty corpus); a SQLite corpus written by a newer iriq (`schema version N is newer than this iriq supports (4); upgrade iriq`); a `.db` that isn't a database or can't be opened or written. Every corpus failure, read or write, now uses this form.
+- **Unreadable input is a clean error:** invalid UTF-8 is `iriq: stream did not contain valid UTF-8` (JSON code `invalid_utf8`); a file without read permission, or stdin that is a directory, is `read_error`. Both exit 1 and honor `--json`. Ruby printed backtraces.
+- **JSON corpora save through a writer-unique temp file** (`PATH.PID.N.tmp`), so concurrent saves no longer crash with `No such file or directory`. A `.json` corpus is still single-writer, and the last save wins; use `.db` for concurrent writers. `--reset` removes these temp files along with the corpus, its `-wal`/`-shm` sidecars and `PATH.tmp`, and nothing else.
+- **Processes take turns with a SQLite corpus.** A writer waits up to 10 seconds for the write lock, then exits 1 with `iriq: corpus PATH: another process held the corpus lock for over 10s`. Long jobs share it: an ingest (`cluster`, `--stats`, or piped input with no section flag) commits about once a second, and `--reinfer` and `--activate-above` rebuild without the lock and take it only to install the result (about 1.4 s for 1.2 million observations), so a `tail -f app.log | iriq -n` stream keeps going beside them. Two catches: an ingest that's killed or fails keeps what it committed, so running the same input again counts those IRIs twice; and `--reinfer` needs free space in `TMPDIR`, more than the corpus file's own size. Ruby's `--reinfer` used to fail with `corpus PATH: database is locked` while another process was writing. For the Rust CLI's `--reinfer` bug, see Fixed.
+- **`cluster -J` lists keys in one order,** the same from both runtimes and every backend: a segment's `values`, and a param's `value_distribution` and `kind_distribution`, most common first, ties by value; `subtype_distribution` `integer` before `float`. The Rust CLI's order changed from run to run, and Ruby's segment `values` depended on the backend.
+- **Library: `observe_all`** (`Corpus::observe_all` in Rust, `Corpus#observe_all` in Ruby) observes many IRIs, committing about a second at a time on SQLite so other processes get turns. Inside `batch` it joins the batch's transaction.
+- **`--help` says what the CLI does:** a file argument needs no `./`; streaming NDJSON needs a section flag (`tail -f app.log | iriq -nJ`); `--host` keys IRIs as they're observed and on `--reinfer`, and doesn't re-key an existing corpus's report; `--reset` also removes SQLite sidecars and JSON temp files; `cluster` shows every cluster in the corpus; `-e` stays mechanical even with a corpus.
+- **The "created corpus" notice prints only once a corpus has opened.** Ruby printed it before an open that then failed (a `.db` in a read-only directory, say), so the error followed a false announcement.
+- `--propose-recognizers --activate-above F` with nothing to activate says `no proposals at or above confidence F`. It said "coverage", but `F` is a confidence.
+- `--host bogus` reports `invalid argument: --host bogus (expected full|registrable|reg|none)`, without the doubled `--host`.
+- Ruby `--explain`: an already-canonical date or currency shows its value, agreeing with the normalized line, instead of `{date}` / `{currency}`; the JSON omits `host` for an IRI without one instead of emitting `null`. Rust already did both.
+- **Rust crate:** `Corpus` implements `Debug`; `Cluster`, `ParamSummary`, `PositionStats`, `SegmentPositionStat`, `Identifier` and `OrderedMap` implement `PartialEq`. The crate declares `rust-version = "1.85"`. docs.rs builds with all features and labels the SQLite-only error variant. The crate README is the crate docs, so its samples are compiled and run as doctests. The published crate no longer ships integration tests, which could only fail outside the repository. The CLI detects a terminal with `std::io::IsTerminal` instead of an `unsafe`, Unix-only `isatty` call.
+
+#### Fixed
+
+**Rust**
+
+- **Non-ASCII digits crashed the classifier** (exit 101) in a segment or query value that mixed ASCII digits with another script's (`https://x.com/1०००००००`, `?d=2024-٠١-١٥`). "Digit" and "space" now mean ASCII, as in Ruby, which also settles values the runtimes classified differently: `?v=1.٥` was `{float}`, `?v=v𝟎` was `{version}`, `?u=http://x.com/a<NBSP>b` stayed literal, and `registrable_domain("١.٢.٣.٤")` was treated as an IPv4 address.
+- **Hostnames keyed differently from Ruby.** `--host reg` keyed trailing-dot hosts (`api.foo.com.`) as `com.`; they now key as `foo.com`. A host ending a word in capital sigma (`ΑΣ-x.com`) lowercased to the final form `ας-x.com`; it is now `ασ-x.com`.
+- **IP addresses rendered as `{ipv4}` / `{ipv6}` with a corpus**, which is by default. They're `{ip}`, as with `-C` and in Ruby.
+- **SQLite failures were silent.** A read-only, full or otherwise failing `.db` accepted every observation, wrote nothing, and the CLI exited 0, so a cron job could lose data indefinitely. A failed read was just as quiet: `-n` printed a mechanical shape, `--stats` dropped the row, and some reads panicked. Both now exit 1 with `iriq: corpus PATH: CAUSE`.
+- **An observation that failed part-way stayed half-recorded** in a `.db` (a full disk, a lock timeout), leaving its views out of step with the source log until `--reinfer`. Each observation is now one transaction, as in Ruby.
+- **`--reinfer` could corrupt a `.db` other processes were writing.** It read the log, cleared the views and replayed in separate steps, so it either exited 0 with views that no longer matched the log, or died with `UNIQUE constraint failed: cluster_examples…` after the clear had committed. What to do is under What you must do.
+- **An activated recognizer matched too much.** After activating `ghp_`, the Rust CLI rendered `ghp_abc-def`, `ghp_a.b`, `ghp_x_y` and a bare `ghp_` as `{ghp}`. A recognizer now matches only its prefix followed by letters and digits, as the whole segment, as in Ruby. What to do is under What you must do.
+- **`.json` corpora lost their numeric ranges.** The Rust CLI saved a `.json` corpus without the data behind a param's `min`, `max` and `avg`, so Ruby reading one showed no ranges, and a Rust run over a Ruby-written file removed them for good. It now writes them, and an unchanged corpus saves to the same bytes every run.
+- **Exiting waited on other processes.** A run that had already printed its output waited, before exiting, for another process's read snapshot or long write on the same `.db` to clear. It now exits right away.
+- **A `Corpus` on SQLite could wedge for the rest of the process.** A panic inside `batch` left its transaction open, so later "successful" writes rolled back at close; a panic while the connection lock was held poisoned it; iterating the corpus while normalizing against it deadlocked; and a `COMMIT` that SQLite refused but kept open made every later observation fail with `cannot start a transaction within a transaction`. Each now recovers.
+- **`Corpus::save` could overwrite a live SQLite corpus** with a JSON export when given another spelling of its own path (`dir/./c.db`). It now resolves both paths and flushes in place.
+- **`Corpus::open` accepted a JSON corpus in a directory that doesn't exist** and failed only at save, with an error that didn't name the file. It now fails at open, naming the path.
+- **The human cluster view printed large or rounded numbers wrong.** A param past 2^63 (`?v=18446744073709551616`) printed `9223372036854775807..9223372036854775807  avg 9223372036854775807`; it now prints the exact value, as Ruby does. Averages and ranges round like Ruby's elsewhere too: a non-whole value that rounds to a whole number keeps its `.0` (`avg 3.0`), `-0.004` rounds to `-0.0`, `1.005` to `1.01`, and values past 15 digits use `e+15` notation. `--json` is unchanged.
+- **`avg` could differ between runs over the same corpus** in its last digits (`100000000000002.52` vs `…2.53`), because reopening a corpus summed its values in hash-map order. The sum now follows the order the corpus stores them.
+- **`kind_distribution` dropped unrecognized extensions.** A `file` param seen as `b.pdf` ×3 and `c.zzz` reported `{"document":0.75}`; it now reports `{"document":0.75,"unknown":0.25}`, as in Ruby.
+
+**Both runtimes**
+
+- **A numeric param with a 310+ digit value** overflowed to infinity: Ruby crashed `cluster` (losing the batch before a JSON corpus saved), and Rust printed `null` for `max` and `avg`. Such values still count as observations but stay out of `min`, `max` and `avg`.
+- **An interrupted activation left its recognizer stored but not applied.** Activation stored the recognizer, then reinferred in a separate step; a process killed after the first step left the recognizer stored and the views not, or only partly, rebuilt. Storing and reinferring now commit together or not at all. What to do is under What you must do.
+- **Activating a recognizer the corpus already held wasn't a no-op.** It put a second copy on the live classifier and reran `reinfer`, and Ruby also stored a duplicate in a Memory or JSON corpus. Now it changes nothing, whichever runtime or binary stored the first: an activation is identified by its prefix and type, and the Rust CLI stores a recognizer's specificity as 1.0, as Ruby does. Recognizers older Rust binaries stored at 0.3 classify the same and need nothing.
+- **A running writer ignored recognizers another process activated,** and wrote clusters under the old types beside the new ones. Each write transaction (a streamed chunk, an ingest turn, a lone observation) now picks up activations committed before it began, at the cost of one small read, and only when another process has written since.
+
+**Ruby**
+
+- **Corpus failures were backtraces:** a `.db` that isn't a database, a read-only `.db`, a stored count hand-edited to a non-integer, an unreadable or directory `.json`, a corpus directory that can't be created, a failed JSON save. They're now `iriq: corpus PATH: REASON`, with the Rust CLI's reasons (`file is not a database`, `Invalid column type Text at index: 1, name: count`, `Permission denied (os error 13)`).
+
+#### Performance
+
+No action needed, and output is unchanged. Rust timings are single runs over 20,000 URLs, v0.34.0 against now.
+
+- **Rust: corpus-informed `-n` no longer slows down as the corpus grows.** Shaping each URL read every value and example the corpus held for its route, so each line cost more than the last; it now reads only the counts classification uses. One route into a fresh corpus: `.db` 22 s → 0.4 s, `.json` 2.2 s → 0.1 s.
+- **Rust, SQLite: observing skips redundant counting.** A known route no longer scans every route to number itself, and a new value in a path slot or query param no longer re-counts the values already there (a batch remembers the count, and forgets it when another process writes). `--stats` over one route: 3.1 s → 0.3 s; over 1,000 hosts: 0.9 s → 0.4 s; with a unique `?q=` on every URL: 2.2 s → 0.2 s. `-n` with a unique `?q=` on every URL is still slow (27 s → 17 s).
+- **SQLite: a big ingest no longer shuts out other writers.** `cluster` and `--stats` held the write lock from their first observation to their last; they now hold it about a second at a time. A one-line `-n` run beside a 1.2-million-line `cluster` waited at most 1.3 s; with v0.34.0 it waited 258 s, for the whole ingest.
+- **Ruby, SQLite: `-n` with a corpus does about 50× less work.** Normalizing read every value tracked at each path position and the whole cluster, observing reloaded the cluster, and each new value re-counted its position. 10,000 URLs into a fresh `.db` went from about 150 s to 3 s of CPU.
+
+###  0.34.0  (2026-08-12)
+- **Rust: SQLite is now an optional (default-on) feature.** `cargo install iriq`, the Homebrew formula, and the CLI are unchanged — `default = ["sqlite"]`. Library consumers who only need parsing, extraction, or normalization can now take `iriq = { version = "0.34", default-features = false }` and skip the bundled C SQLite build entirely (`rusqlite` is the crate's only non-Rust dependency). Without the feature, `open_storage` rejects `.db`/`.sqlite`/`.sqlite3` paths with a clear `Unsupported` error and the CLI's auto-default corpus becomes `default.json` instead of `default.db`; Memory and JSON backends are unaffected. No behavior change with default features, so Ruby parity and the shared schema are untouched. CI now gates the no-default-features build (test + clippy) so the configuration can't rot.
+
 ###  0.33.0  (2026-07-07)
 - **New: end-to-end URL calibration corpus** — 160 messy real-world-shaped inputs (tokens, i18n, encoding damage, legacy endpoints, non-http schemes, garbage) with adjudicated expected templates at `spec/fixtures/calibration/urls.json`, generated by `script/build_url_calibration.rb`, asserted by both runtimes, and CI-gated for staleness. Building it caught the three fixes below.
 - **Bugfix: opaque non-urn schemes were rewritten to `urn:` on normalize/canonical** — `mailto:support@foo.com` came back as `urn:support@foo.com`; same for `tel:`, `sms:`, `data:`, `blob:`, `magnet:`. The scheme is now preserved (both runtimes; cluster keys too).

data/README.md

@@ -62,26 +62,48 @@ $ iriq -n https://shop.com/pricing/usd?currency=eur
 https://shop.com/pricing/USD?currency=EUR     # currency upcased
 ```
 
+Pipe in text, or name a file, and iriq extracts every URL in it:
+
 ```sh
-$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
-[190] docs.example.com  /users/{user_id}
-[186] app.example.com   /users/{user_id}
-...
-
-$ cat access.log | iriq --stats               # rolling aggregates
-$ iriq ./access.log -n                        # auto-detect file → normalize each
-$ iriq -J < access.log                        # newline-delimited JSON
-$ iriq --corpus team.db < access.log          # use a specific corpus file
+$ cat urls.log | iriq                         # ≥ 10 IRIs → cluster view
+[6] api.example.com  /api/{version}/users/{user_id}
+    https://api.example.com/api/v1/users/123
+    https://api.example.com/api/v1/users/456
+    https://api.example.com/api/v1/users/789
+    + 3 more
+
+[3] api.example.com  /orders/{order_uuid}
+    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
+    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
+    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
+    status  string  conf 0.17  (2 distinct, 100%)
+
+[3] api.example.com  /products/{product_id}
+    https://api.example.com/products/blue-widget
+    https://api.example.com/products/red-gadget
+    https://api.example.com/products/green-gizmo
+
+$ cat urls.log | iriq --stats                 # rolling aggregates
+$ iriq urls.log -n                            # a file argument → normalize each URL
+$ iriq -nJ < urls.log                         # one JSON line per URL
+$ iriq --corpus team.db < urls.log            # use a specific corpus file
 ```
 
-Per-IRI output (`-n`, `-p`, `-c`, `-J`) streams — each line is read, classified,
-and flushed as it arrives, so iriq works on an unbounded live feed:
+Reading a web server's access log? Its request lines have no host, so see
+[Access logs](#access-logs) first.
+
+Per-IRI sections (`-n`, `-c`, `-p`, `-e`) stream: each line is read, observed,
+rendered from the corpus as it stands, and flushed, so iriq works on an
+unbounded live feed:
 
 ```sh
-$ tail -f access.log | iriq -n                # one shape per line, as logs land
-$ tail -f access.log | iriq -J                # same, as newline-delimited JSON
+$ tail -f app.log | iriq -n                   # one shape per line, as logs land
+$ tail -f app.log | iriq -nJ                  # same, as newline-delimited JSON
 ```
 
+`-J` on its own doesn't stream. Like the default view, it waits for the end of
+input, then prints the URL list (fewer than 10 IRIs) or one object per cluster.
+
 **Every invocation observes into a persistent corpus by default**, so iriq gets
 smarter the more you run it. The corpus-only types (e.g. `enum` / `http_status`)
 emerge from the *distribution* of values observed.
@@ -101,11 +123,20 @@ $ iriq --corpus demo.db cluster
     https://api.foo.com/orders/1?status=closed
     https://api.foo.com/orders/2?status=open
     + 37 more
-    status  enum  (2 distinct, 100%)
+    status  enum  conf 0.73  (2 distinct, 100%)
+```
+
+`conf` is how much evidence backs the type, from 0 to 1. These learned types
+also flow into normalized output:
+
+```sh
+$ iriq --corpus demo.db -n 'https://api.foo.com/orders/99?status=open'
+https://api.foo.com/orders/{order_id}?status={enum}
 ```
 
-These learned types also flow into normalized output: once the corpus has pegged
-`?status` as an enum, `iriq -n …?status=open` renders `?status={enum}`.
+The corpus only acts on evidence: it changes a shape only at a position or
+param it has seen at least 5 times. Until then, `-n` prints exactly what `-C`
+would. Dates and currencies always print canonically (`2024-01-15`, `USD`).
 
 The default corpus lives at `$XDG_DATA_HOME/iriq/default.db` (Linux),
 `~/Library/Application Support/iriq/default.db` (macOS), or
@@ -117,7 +148,7 @@ $ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
 $ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
 $ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
 $ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
-$ iriq --reset                                     # delete the corpus DB and exit
+$ iriq --reset                                     # delete the corpus and exit
 ```
 
 ### Two ways to normalize
@@ -129,8 +160,9 @@ Pick by the question you're asking:
   lowercased, default port dropped; path and query left alone). Handy, but
   table stakes — plenty of libraries do it.
 - **`--normalize`** *(the default)* — find the URL's *shape*, erasing the
-  specifics into placeholders. `…/pull/42` → `…/pull/{id}`. This is the part
-  you came to iriq for.
+  specifics into placeholders. `…/pull/42` → `…/pull/{pull_id}`. A shape
+  ignores the `#fragment`, so `-n` drops it. This is the part you came to iriq
+  for.
 
 Same input, two questions: "what's the clean form of *this* URL?" vs "what
 *kind* of URL is this?" The second is iriq's reason to exist.
@@ -141,7 +173,7 @@ Same input, two questions: "what's the clean form of *this* URL?" vs "what
 # Homebrew (recommended)
 brew install dpep/tools/iriq
 
-# Cargo, from crates.io
+# Cargo, from crates.io (Rust 1.85 or newer)
 cargo install iriq
 ```
 
@@ -186,7 +218,8 @@ what produces `{user_id}` from `/users/123` and `{order_id}` from `/orders/456`.
 Semantic types (`version`, `locale`, `currency`, `date`, `boolean`) skip the
 hint and surface as `{type}` — `/api/v1/status` renders as `/api/{version}/status`,
 not the misleading `/api/{api_id}/status`. Pass `-N` / `--no-hints` for
-mechanical placeholders (`{integer}` instead of `{user_id}`).
+mechanical placeholders (`{integer}` instead of `{user_id}`); a slot only the
+corpus knows is variable renders `{value}`.
 
 ### Types only the corpus can see
 
@@ -209,9 +242,51 @@ so classification improves as more data comes in — handy for an unbounded stre
 of identifiers. The default corpus already persists; `--corpus PATH` points iriq
 at a specific file instead, to keep separate corpora or share one across runs.
 
-A `.db` / `.sqlite` / `.sqlite3` path is stored in SQLite (WAL journaling, incremental
-UPSERTs — multiple `iriq --corpus` processes can write concurrently); a
-`.json` path writes a plain JSON file instead.
+The extension picks the backend, and the two behave differently:
+
+- **`.db` / `.sqlite` / `.sqlite3` (SQLite)** — the default, and the one to
+  share. Many `iriq` processes can write at once by taking turns: a writer
+  waits up to 10 seconds for its turn. A big `cluster` commits about a second
+  at a time, and `--reinfer` rebuilds on the side and holds the corpus only to
+  swap the result in, so a `tail -f` stream keeps flowing beside either. Use
+  SQLite for streams and concurrent writers.
+- **Anything else (JSON)** — read when iriq starts and written once, when it
+  exits cleanly. It's single-writer: when two processes use one file, the last
+  to exit wins. A streaming run that's killed, Ctrl-C included, saves nothing.
+
+A few things to know:
+
+- The cluster view (`iriq cluster`, or 10+ piped IRIs) shows the whole corpus,
+  not just this input. Add `-C` to cluster one input on its own.
+- iriq keeps every IRI it observes, repeats included, so `--reinfer` can replay
+  them. That log grows without bound.
+- On SQLite, `cluster` and `--stats` commit their input about a second at a
+  time. If one is killed part-way, what it committed stays: feed it the same
+  input again and those IRIs count twice.
+- `--reinfer` (and `--activate-above`) rebuilds in temporary tables before
+  swapping the result in, so it needs free space in `TMPDIR`: plan on more than
+  the corpus file's own size.
+- `--reset` deletes the corpus file, its SQLite `-wal` / `-shm` sidecars, and any
+  temp files a JSON save left behind. Don't reset a corpus another process is
+  writing: that process carries on, exits 0, and its writes are lost with the
+  deleted file.
+- iriq refuses a corpus file it can't safely use rather than overwrite it — a
+  JSON file that isn't an iriq corpus, or a SQLite corpus written by a newer
+  iriq (upgrade to open it).
+
+### Host keying
+
+By default every hostname gets its own clusters. `--host reg` keys by
+registrable domain, so `api.foo.com` and `www.foo.com` both cluster under
+`foo.com`; `--host none` ignores the host. The mode applies when observations
+are recorded, when you `--reinfer`, and to a `-C` run's throwaway corpus. It
+doesn't re-key a report of an existing corpus: `iriq --host reg cluster` shows
+the clusters as they were recorded. To re-key a corpus, reinfer it:
+
+```sh
+$ iriq --corpus c.db --host reg --reinfer
+reinferred 3 observations: 3 → 1 cluster
+```
 
 ### Re-runnable inference
 
@@ -271,6 +346,42 @@ Known limitations (intentional):
 
 Disable scheme-less extraction with `--no-scheme-less`.
 
+### Access logs
+
+Extraction needs URLs with a host. A web server's request line
+(`"GET /api/v1/users/123 HTTP/1.1"`) has none, so on a raw access log iriq
+finds only the full URLs on each line, usually the Referer:
+
+```sh
+$ cat access.log | iriq
+[12] example.com  /referrer
+    https://example.com/referrer
+    + 11 more
+```
+
+Pull out the path and give it a host first. In the common and combined log
+formats, the path is the seventh field:
+
+```sh
+$ awk '{print "https://api.example.com" $7}' access.log | iriq
+[6] api.example.com  /api/{version}/users/{user_id}
+    https://api.example.com/api/v1/users/123
+    https://api.example.com/api/v1/users/456
+    https://api.example.com/api/v1/users/789
+    + 3 more
+
+[3] api.example.com  /orders/{order_uuid}
+    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
+    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
+    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
+    status  string  conf 0.17  (2 distinct, 100%)
+
+[3] api.example.com  /products/{product_id}
+    https://api.example.com/products/blue-widget
+    https://api.example.com/products/red-gadget
+    https://api.example.com/products/green-gizmo
+```
+
 ## How it works
 
 Under the shape sits one idea: **Position + Evidence**. A *Position* is a slot
@@ -285,24 +396,26 @@ underneath. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full model.
 **Single input** — combined parse + normalize summary; trim with section flags
 (`-p`, `-n`).
 
-**Piped stdin** — extraction runs by default. Output auto-switches: small inputs
-get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the cluster view via
-an ephemeral corpus.
+**Piped stdin, or a file argument** — extraction runs by default. With no
+section flag, iriq reads all of the input, then prints a deduplicated URL list
+(fewer than 10 IRIs) or the cluster view of the corpus (10 or more). With a
+section flag, it prints each IRI's result as the line arrives. `-n` is
+corpus-informed; `-e` is mechanical even with a corpus.
 
 | Flag                | Effect                                                  |
 | ------------------- | ------------------------------------------------------- |
 | `-p, --parse`       | Show parsed fields                                      |
 | `-n, --normalize`   | Show the shape-normalized form                          |
 | `-c, --canonical`   | Show the canonical form (no shape normalization)        |
-| `-e, --explain`     | Annotated trace — per-segment notes about why each placeholder / canonical value was chosen |
+| `-e, --explain`     | Annotated trace — per-segment notes about why each placeholder / canonical value was chosen. Mechanical, even with a corpus |
 | `-j, --json`        | Emit JSON                                               |
-| `-J, --ndjson`      | Newline-delimited JSON (one object per line); implies `--json` |
+| `-J, --ndjson`      | Newline-delimited JSON; implies `--json`. With a section flag (`-nJ`), one line per IRI as it arrives; alone, the URL list or clusters at end of input |
 | `-N, --no-hints`    | Use `{integer}` etc. instead of `{user_id}`             |
 | `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
 | `--corpus PATH`     | Use a specific corpus file (`.json` or `.db`/`.sqlite`/`.sqlite3`). Overrides the default |
 | `-C, --no-corpus`   | Disable corpus persistence for this invocation (same as `IRIQ_NO_CORPUS=1`) |
-| `--reset`           | Delete the corpus database and exit                     |
-| `--host MODE`       | Host-keying for clustering: `full` (default), `reg` strips subdomains, `none` ignores host |
+| `--reset`           | Delete the corpus file, its SQLite sidecars and JSON temp files, and exit |
+| `--host MODE`       | Host keying: `full` (default), `reg` strips subdomains, `none` ignores host. Applies when observing, to `--reinfer`, and with `-C` |
 | `--stats`           | Print rolling aggregates                                |
 | `--reinfer`         | Drop the materialized views and replay the source-IRI log through the current classifier + reducers |
 | `--propose-recognizers` | Scan observed values for shape patterns that recur enough to suggest a new recognizer. Combine with `--json` for structured output |
@@ -311,6 +424,7 @@ an ephemeral corpus.
 | `--min-coverage F`      | Proposal threshold; default 0.7                   |
 | `--min-hosts N`         | Threshold for both proposals and cross-host shapes; default 1 / 2 respectively |
 | `--activate-above F`    | With `--propose-recognizers`, auto-activate every proposal whose confidence is ≥ F |
+| `cluster [file]`        | Force the cluster view                            |
 | `completion bash\|zsh`  | Print shell completion script (Homebrew installs this automatically) |
 | `-V, --version`     | Print version                                           |
 
@@ -321,11 +435,21 @@ Environment variables:
 | `IRIQ_CORPUS=PATH`   | Set the corpus path (overrides the default)             |
 | `IRIQ_NO_CORPUS=1`   | Disable the default corpus (equivalent to `-C`)         |
 
-A positional argument that doesn't parse as an IRI but IS an existing file is
-read and extracted from automatically — `iriq ./access.log` and
-`iriq /var/log/foo.log` Just Work. (pipe with `cat` to disambiguate)
+A positional argument that names an existing file is read as a file, unless it
+contains `://` — `iriq access.log` and `iriq /var/log/foo.log` both work. A
+path-like argument (`/x`, `./x`, `../x`) that doesn't exist is an error; a bare
+name that isn't a file, like `foo.log`, parses as a host (`https://foo.log/`).
+
+Errors go to stderr as `iriq: MESSAGE`, or, with `--json` / `-J`, as
+`{"error":{"code":"…","message":"…"}}`. A corpus error names the file:
+`iriq: corpus team.db: attempt to write a readonly database`.
 
-Exit codes: `0` success, `1` usage error, `2` parse error.
+| Exit  | Meaning | JSON codes |
+| ----- | ------- | ---------- |
+| `0`   | Success | |
+| `1`   | Bad option or argument, missing or unreadable input, unusable corpus, or stdout failed | `option_error`, `unknown_shell`, `file_not_found`, `read_error`, `invalid_utf8`, `corpus_error`, `stdout_error` |
+| `2`   | The input isn't a parseable IRI | `parse_error` |
+| `141` | The reader went away (`iriq … \| head`); iriq stops quietly | |
 
 ## Rust library
 
@@ -334,23 +458,36 @@ cargo add iriq
 ```
 
 ```rust
-use iriq::{parse, normalize, Corpus};
-
-let iri = parse("https://foo.com/users/123")?;
-iri.host;             // "foo.com"
-iri.path_segments;    // ["users", "123"]
-iri.canonical();      // "https://foo.com/users/123"
-
-normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"
-
-// Streaming clustering against a persistent corpus.
-let mut corpus = Corpus::open("c.db")?;
-corpus.observe("https://foo.com/users/1")?;
-corpus.save("c.db")?;
+use iriq::{normalize, parse, Corpus};
+
+fn main() -> iriq::Result<()> {
+    let iri = parse("https://foo.com/users/123")?;
+    println!("{} {:?}", iri.host, iri.path_segments); // foo.com ["users", "123"]
+    println!("{}", normalize("https://foo.com/users/123")?); // https://foo.com/users/{user_id}
+
+    // A persistent corpus: SQLite for .db, JSON otherwise.
+    let mut corpus = Corpus::open("c.db")?;
+    for n in 1..=3 {
+        corpus.observe(&format!("https://foo.com/users/{n}"))?;
+    }
+    for cluster in corpus.clusters()? {
+        println!("[{}] {} {}", cluster.count, cluster.host, cluster.shape); // [3] foo.com /users/{user_id}
+    }
+    corpus.save("c.db")?; // flushes in place; a .json corpus is written only here
+    Ok(())
+}
 ```
 
-Full API on [docs.rs/iriq](https://docs.rs/iriq); see the
-[crate README](rust/iriq/README.md) for the library tour.
+Every `Corpus` operation returns `iriq::Result`, whose `iriq::Error` names the
+corpus that failed. SQLite comes from the default-on `sqlite` feature;
+`cargo add iriq --no-default-features` drops it and keeps in-memory and JSON
+corpora. Requires Rust 1.85 or newer. A long-lived `Corpus` sees recognizers
+another process activated at its next `batch` or `observe`, not in reads
+outside one.
+
+The [crate README](rust/iriq/README.md) is the library tour: reading clusters
+and params, batches, sharing a corpus, and errors. Full API on
+[docs.rs/iriq](https://docs.rs/iriq).
 
 ## Limitations (intentional)