iriq 0.30.2 → 0.33.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 598c04e3c1777787ae9e5d1be98e2bc68d441e2020ebe7743bdf8075b20fdaec
-  data.tar.gz: 396ad6b0b0acffb76b7bc2b4e31792b02ac65749c6ac769fd70a47ce5d806496
+  metadata.gz: 4290f75faee81b1292a3404fd01221299490731378ace043999e6ab7e7a0dc81
+  data.tar.gz: 0b1b074eb746ec2b50a790fb38da54c0715a420ec752929f3eb2af85bed50670
 SHA512:
-  metadata.gz: 16637ff46f4648a2cfc14404ed074d3cedc6fc08cabf0c46a6fa7a39553b8b78020907fde1d9024005cec3a4f2cdb4cb3e999802ec866a942c20c15be6c7af34
-  data.tar.gz: 458aa6deba73a571a07801bb3df445a4d08500a55c3d1d8c50b11df0811dbc785270fd1ef908ebbb58c6f53dfb1ecfc7013fe137e216e88bca81c9bc56d21fa4
+  metadata.gz: 81feb4d4b7c8fb0cd9f7e68dbec17fdc787642a6829f2f479849e458c14e5fa9812c251df08f57658b6cb338fcb112d8ec16393efb7aa4e4364e07b2dcf19ae1
+  data.tar.gz: 8979d705a2dd2d1db59c4b5fc794ba9307f061c2fbb7b3968c1585c8d633a7871d306c1ec372ad3dc15e11f1ec30c65a7f7833a6955dae8a3e11ca3f3a06d925

data/CHANGELOG.md

@@ -1,3 +1,35 @@
+###  0.33.0  (2026-07-07)
+- **New: end-to-end URL calibration corpus** — 160 messy real-world-shaped inputs (tokens, i18n, encoding damage, legacy endpoints, non-http schemes, garbage) with adjudicated expected templates at `spec/fixtures/calibration/urls.json`, generated by `script/build_url_calibration.rb`, asserted by both runtimes, and CI-gated for staleness. Building it caught the three fixes below.
+- **Bugfix: opaque non-urn schemes were rewritten to `urn:` on normalize/canonical** — `mailto:support@foo.com` came back as `urn:support@foo.com`; same for `tel:`, `sms:`, `data:`, `blob:`, `magnet:`. The scheme is now preserved (both runtimes; cluster keys too).
+- New `:font` file kind (`woff`/`woff2`/`ttf`/`otf`/`eot`), plus `m3u8` (video) and `map` (web) extensions — `Inter-Bold.woff2` now normalizes to `{file}` instead of `{font_id}`.
+- **Bugfix (Rust):** out-of-range ports (`:99999`) were rejected where Ruby's deliberately lenient parser accepts them — the port field is now `Option<u64>` matching Ruby's nil-vs-integer semantics.
+- **Bugfix (Ruby, SQLite backend):** rolling numeric stats (`min`/`max`/`avg` on numeric params) were never restored when loading a corpus from disk, so numeric ranges vanished from cluster output after a reopen — diverging from the Memory/JSON backends and from Rust. Now recomputed from tracked value counts on load; locked in by a new parity scenario and storage specs.
+- Perf: trimmed hot-path allocations in both runtimes — Rust's classifier no longer clones the recognizer list (and locks once, not three times) per cache miss; Ruby builds each observation's Shape once instead of twice, passes the recognizer array without a splat, and drops a redundant `to_s`. No behavior change.
+- **The Rust CLI's `completion` subcommand now matches Ruby exactly.** It previously carried its own divergent bash/zsh scripts plus a fish script Ruby never supported; it now embeds byte-identical copies of the shared bash/zsh scripts (parity-tested), defaults the shell from `$SHELL` like Ruby, and emits Ruby's `unknown_shell` error envelope. Fish is no longer supported. The `completion` subcommand is also listed in `--help` now.
+- **Rust now supports dynamic synthesized types**, closing the last known Ruby↔Rust divergence: `--activate-above` activates a proposal under its suggested type (`activated: ghp (ghp_)`) instead of falling back to `opaque_id`, and the activated type drives `{ghp}`-style placeholders, survives corpus reopen (JSON + SQLite), and round-trips storage — parity-tested. Implementation: `SegmentType::Custom(&'static str)` backed by a leak-once interned name, keeping the enum `Copy`. Also fixes a latent Rust bug where `string`-typed param counts were dropped when reloading a JSON corpus (a hand-copied type table was missing the `string` entry).
+- **The Go port is retired.** Ruby (reference) + Rust (shipped CLI via Homebrew/crates.io) continue; the `go/` module, its CI, and the Ruby↔Go / Rust↔Go parity harnesses are removed. `script/cli_parity.sh` now diffs Ruby ↔ Rust directly. Go consumers can pin `github.com/dpep/iriq/go@v0.32.1`, the last tag with Go support. `.db` corpora written by the Go binary still open cleanly (shared schema v4).
+
+###  0.32.1  (2026-06-26)
+- Go and Rust `cluster --json` now include the per-param `values` and `value_distribution` (plus `subtype_distribution`, `kind_distribution`, and numeric `min`/`max`/`avg`) that Ruby already emitted — the cluster JSON is now identical across all three runtimes. No change to the human-readable cluster view.
+- Test coverage: new `param_summary` golden fixture exercises the const → string → enum ladder + confidence across Ruby/Go/Rust (`go test` / `cargo test`), and the parity harnesses gained a key-order-agnostic JSON comparison (`jq -S` + number canonicalization) with a `cluster --json` scenario.
+
+###  0.32.0  (2026-06-26)
+- **Query params now climb a confidence ladder: constant → string → enum.** A param with a single observed value is a constant (rendered as its value); one that varies across free-form literal values is the new `string` type (renders `{string}`); a bounded, well-supported value set is `enum` (renders `{enum}`). Previously a varying literal param just echoed whatever value you passed.
+- **Enum detection is now coverage-based and straggler-robust.** An enum is promoted when its *established* values (each seen ≥ `ENUM_MIN_VALUE_COUNT`, now 3) number 2–10 and cover ≥ 90% of observations. A single brand-new value no longer knocks an established enum back down — fixing the observe-before-normalize order dependence where normalizing `?status=<new>` could flip the type. A lone repeated value is now correctly a constant, not a one-member enum.
+- **New `confidence` score on every param** (`total / (total + 15)`): a 0–1 figure of how much evidence backs the classification, shown in the cluster view (`status  enum  conf 0.93  ...`) and the cluster JSON. The type is the guess; confidence says how sure.
+- Ruby, Go, and Rust ship this together — same thresholds, same rendering, parity preserved (68/68 Ruby↔Go, 60/60 Rust↔Go).
+
+###  0.31.1  (2026-06-26)
+- Docs: clarified the README around corpus-on-by-default — sharper IRI definition (URLs are one member of the family, alongside URNs, `mailto:`, and internationalized addresses), an accurate worked example of the corpus learning a query param is an `enum`, and a streaming example (`tail -f access.log | iriq -J`). Added the streaming example to `--help` (Ruby + Go). No behavior change.
+
+###  0.31.0  (2026-06-26)
+- **Corpus is now on by default.** Every invocation observes into a persistent corpus, so classification gets sharper the more you use the tool — the streaming/learning behavior that was the selling point is no longer hidden behind a flag. Default location: `$XDG_DATA_HOME/iriq/default.db` on Linux, `~/Library/Application Support/iriq/default.db` on macOS, `%LOCALAPPDATA%/iriq/default.db` on Windows. First-run creation prints a one-line stderr notice.
+- New flag `-C` / `--no-corpus` (and `IRIQ_NO_CORPUS=1` env) — disables the default corpus for a single invocation. Explicit `--corpus PATH` always wins, even with `--no-corpus` set in the env.
+- New env var `IRIQ_CORPUS=PATH` — overrides the default location without needing `--corpus` on every call.
+- New flag `--reset` — deletes the resolved corpus file (and SQLite `-wal` / `-shm` sidecars) and exits. Honors `--corpus` / `IRIQ_CORPUS` so you can reset a non-default file.
+- **Go build simplification:** the slim / SQLite build split is gone. The Go binary now always links `modernc.org/sqlite` (pure Go, no cgo). `make build-sqlite`, `make release-sqlite`, and `-tags sqlite` are retired. The `dpep/tools/iriq-sqlite` Homebrew formula folds into `dpep/tools/iriq`.
+- Ruby, Go, and Rust ship this together — same defaults, same flags, parity preserved.
+
 ###  0.30.2  (2026-06-23)
 - Piped stdin and `--file` now **stream** the per-IRI sections (`-n`/`-p`/`-c`/`-e`) line by line, flushing each IRI as it's processed — `tail -f access.log | iriq -n` is live and memory stays bounded on huge inputs. Output is byte-identical to before; the aggregate views (deduped URL list, clusters, `--stats`) still read the whole input. Ruby, Go, and Rust.
 

data/README.md

@@ -1,5 +1,5 @@
-Iriq
-======
+# Iriq — IRI Query
+
 [![codecov](https://codecov.io/gh/dpep/iriq/branch/main/graph/badge.svg)](https://codecov.io/gh/dpep/iriq)
 
 **Iriq finds the *shape* of a URL** — the structural template you get when you
@@ -9,17 +9,18 @@ URLs — a log file, a column of links, free-text prose — and it collapses the
 into a small set of stable, deterministic route templates. Fifty thousand
 distinct URLs become twelve shapes.
 
-(An **IRI** is just a URL — the internationalized superset of URI/URL that also
-allows non-ASCII characters. If you know URLs, you know IRIs. The name is *IRI
-Query*: iriq queries an IRI for its structure.)
+What's an IRI? Internationalized Resource Identifiers cover everyday URLs `https://…`, plus URNs like `urn:isbn:0451450523`,
+other schemes like `mailto:`, and internationalized addresses with non-ASCII
+characters like `https://例え.jp/パス`. Formally it's the Unicode superset of
+URI/URL. The name is *IRI Query*: iriq queries an IRI for its structure.
 
 Everything iriq does — parsing, normalizing, classifying path and query
 components, clustering, learning new patterns — exists to derive, render, or
 group by that shape.
 
-And it gets sharper the more you feed it. Point a *corpus* at a stream and
-classifications improve as data flows in — high-churn slots get promoted to
-placeholders, and whole types emerge that you can't see in any single URL (a
+And it gets sharper the more you feed it. A *corpus* — on by default — records
+what it sees and improves classifications as data flows in: high-churn slots get
+promoted to placeholders, and whole types emerge that no single URL can reveal (a
 position that's always 100–599 is an HTTP status; one bounded to a dozen values
 is an enum).
 
@@ -70,20 +71,53 @@ $ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
 $ cat access.log | iriq --stats               # rolling aggregates
 $ iriq ./access.log -n                        # auto-detect file → normalize each
 $ iriq -J < access.log                        # newline-delimited JSON
-$ iriq --corpus c.db < access.log             # persist into a SQLite corpus
+$ iriq --corpus team.db < access.log          # use a specific corpus file
+```
+
+Per-IRI output (`-n`, `-p`, `-c`, `-J`) streams — each line is read, classified,
+and flushed as it arrives, so iriq works on an unbounded live feed:
+
+```sh
+$ tail -f access.log | iriq -n                # one shape per line, as logs land
+$ tail -f access.log | iriq -J                # same, as newline-delimited JSON
 ```
 
-Once a corpus has data, `-n` becomes corpus-informed — a position that only ever
-holds integers clusters to a single `{user_id}` shape, and new values normalize
-to it:
+**Every invocation observes into a persistent corpus by default**, so iriq gets
+smarter the more you run it. The corpus-only types (e.g. `enum` / `http_status`)
+emerge from the *distribution* of values observed.
 
 ```sh
-$ for n in 1 2 3 4 5 6 7 8 9 10; do
-    iriq --corpus c.db https://api.foo.com/users/$n >/dev/null
+# Feed a stream where ?status only ever holds a couple of words:
+$ for n in $(seq 1 20); do
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=open"   >/dev/null
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=closed" >/dev/null
   done
 
-$ iriq -n --corpus c.db https://api.foo.com/users/999
-https://api.foo.com/users/{user_id}
+# Ask what it learned. ?status is now an enum — a verdict no single URL
+# could support, since one URL shows only one value:
+$ iriq --corpus demo.db cluster
+[40] api.foo.com  /orders/{order_id}
+    https://api.foo.com/orders/1?status=open
+    https://api.foo.com/orders/1?status=closed
+    https://api.foo.com/orders/2?status=open
+    + 37 more
+    status  enum  (2 distinct, 100%)
+```
+
+These learned types also flow into normalized output: once the corpus has pegged
+`?status` as an enum, `iriq -n …?status=open` renders `?status={enum}`.
+
+The default corpus lives at `$XDG_DATA_HOME/iriq/default.db` (Linux),
+`~/Library/Application Support/iriq/default.db` (macOS), or
+`%LOCALAPPDATA%/iriq/default.db` (Windows). First-run creation prints a
+one-line stderr notice. Three knobs control it:
+
+```sh
+$ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
+$ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
+$ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
+$ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
+$ iriq --reset                                     # delete the corpus DB and exit
 ```
 
 ### Two ways to normalize
@@ -109,39 +143,11 @@ brew install dpep/tools/iriq
 
 # Cargo, from crates.io
 cargo install iriq
-
-# Cargo, from a source checkout
-cargo install --path rust/iriq
 ```
 
 One crate ships both the library and the `iriq` binary. Corpora persist to
 SQLite (bundled, WAL) out of the box — nothing to flag, install, or rebuild.
 
-## Use it as a Rust library
-
-```sh
-cargo add iriq
-```
-
-```rust
-use iriq::{parse, normalize, Corpus};
-
-let iri = parse("https://foo.com/users/123")?;
-iri.host;             // "foo.com"
-iri.path_segments;    // ["users", "123"]
-iri.canonical();      // "https://foo.com/users/123"
-
-normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"
-
-// Streaming clustering against a persistent corpus.
-let mut corpus = Corpus::open("c.db")?;
-corpus.observe("https://foo.com/users/1")?;
-corpus.save("c.db")?;
-```
-
-Full API on [docs.rs/iriq](https://docs.rs/iriq); see the
-[crate README](rust/iriq/README.md) for the library tour.
-
 ## Segment classification
 
 Iriq classifies each path/query segment into one of ~25 types — the first
@@ -184,8 +190,7 @@ mechanical placeholders (`{integer}` instead of `{user_id}`).
 
 ### Types only the corpus can see
 
-Four types never come from a single URL — they emerge from the *distribution*
-of values a position has held across many observations:
+Four types emerge from the *distribution* of values across many observations:
 
 | Type | Emerges when a position… |
 | --- | --- |
@@ -195,16 +200,16 @@ of values a position has held across many observations:
 | `enum` | holds a small, bounded set of distinct values |
 
 Mechanically, `200` is just an integer. Across ten thousand URLs where that
-slot is always 100–599, it's an HTTP status. That's the corpus earning its keep.
+slot is always 100–599, it's likely an HTTP status.
 
 ## Corpus (streaming + learning)
 
-For processing many identifiers — possibly an unbounded stream — point iriq at a
-corpus. It maintains rolling aggregates and per-(host, prefix) frequency stats,
-so classification improves as more data comes in.
+The corpus maintains rolling aggregates and per-(host, prefix) frequency stats,
+so classification improves as more data comes in — handy for an unbounded stream
+of identifiers. The default corpus already persists; `--corpus PATH` points iriq
+at a specific file instead, to keep separate corpora or share one across runs.
 
-`--corpus PATH` makes the corpus survive across invocations. A `.db` /
-`.sqlite` / `.sqlite3` path is stored in SQLite (WAL journaling, incremental
+A `.db` / `.sqlite` / `.sqlite3` path is stored in SQLite (WAL journaling, incremental
 UPSERTs — multiple `iriq --corpus` processes can write concurrently); a
 `.json` path writes a plain JSON file instead.
 
@@ -225,9 +230,7 @@ $ iriq --corpus c.db --reinfer
 Iriq doesn't just classify against a fixed list — it watches the stream and
 *proposes new recognizers* for patterns it keeps seeing. Notice `ghp_…` or
 `cus_…` recurring at a slug position and iriq will suggest a recognizer for it,
-with evidence: coverage, host count, confidence. Proposals are never
-auto-applied — you activate the ones you trust, and they persist with the
-corpus. Human-in-the-loop by design.
+with evidence: coverage, host count, confidence.
 
 ```sh
 # Print proposals (human-readable, or --json)
@@ -291,11 +294,14 @@ an ephemeral corpus.
 | `-p, --parse`       | Show parsed fields                                      |
 | `-n, --normalize`   | Show the shape-normalized form                          |
 | `-c, --canonical`   | Show the canonical form (no shape normalization)        |
+| `-e, --explain`     | Annotated trace — per-segment notes about why each placeholder / canonical value was chosen |
 | `-j, --json`        | Emit JSON                                               |
 | `-J, --ndjson`      | Newline-delimited JSON (one object per line); implies `--json` |
 | `-N, --no-hints`    | Use `{integer}` etc. instead of `{user_id}`             |
 | `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
-| `--corpus PATH`     | Load/create a corpus at PATH (`.json` or `.db`/`.sqlite`/`.sqlite3`) |
+| `--corpus PATH`     | Use a specific corpus file (`.json` or `.db`/`.sqlite`/`.sqlite3`). Overrides the default |
+| `-C, --no-corpus`   | Disable corpus persistence for this invocation (same as `IRIQ_NO_CORPUS=1`) |
+| `--reset`           | Delete the corpus database and exit                     |
 | `--host MODE`       | Host-keying for clustering: `full` (default), `reg` strips subdomains, `none` ignores host |
 | `--stats`           | Print rolling aggregates                                |
 | `--reinfer`         | Drop the materialized views and replay the source-IRI log through the current classifier + reducers |
@@ -308,13 +314,44 @@ an ephemeral corpus.
 | `completion bash\|zsh`  | Print shell completion script (Homebrew installs this automatically) |
 | `-V, --version`     | Print version                                           |
 
+Environment variables:
+
+| Variable             | Effect                                                  |
+| -------------------- | ------------------------------------------------------- |
+| `IRIQ_CORPUS=PATH`   | Set the corpus path (overrides the default)             |
+| `IRIQ_NO_CORPUS=1`   | Disable the default corpus (equivalent to `-C`)         |
+
 A positional argument that doesn't parse as an IRI but IS an existing file is
 read and extracted from automatically — `iriq ./access.log` and
-`iriq /var/log/foo.log` Just Work. (Bare filenames like `README.md` may still
-parse as a URL; pipe with `cat` to disambiguate.)
+`iriq /var/log/foo.log` Just Work. (pipe with `cat` to disambiguate)
 
 Exit codes: `0` success, `1` usage error, `2` parse error.
 
+## Rust library
+
+```sh
+cargo add iriq
+```
+
+```rust
+use iriq::{parse, normalize, Corpus};
+
+let iri = parse("https://foo.com/users/123")?;
+iri.host;             // "foo.com"
+iri.path_segments;    // ["users", "123"]
+iri.canonical();      // "https://foo.com/users/123"
+
+normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"
+
+// Streaming clustering against a persistent corpus.
+let mut corpus = Corpus::open("c.db")?;
+corpus.observe("https://foo.com/users/1")?;
+corpus.save("c.db")?;
+```
+
+Full API on [docs.rs/iriq](https://docs.rs/iriq); see the
+[crate README](rust/iriq/README.md) for the library tour.
+
 ## Limitations (intentional)
 
 Iriq does **not**:
@@ -330,15 +367,3 @@ Iriq does **not**:
 
 Iriq's focus is the analysis side: classification, normalization, and clustering
 — not a complete URL implementation.
-
-----
-## Contributing
-
-Yes please  :)
-
-1. Fork it
-1. Create your feature branch (`git checkout -b my-feature`)
-1. Ensure the tests pass (`cd rust && cargo test`)
-1. Commit your changes (`git commit -am 'awesome new feature'`)
-1. Push your branch (`git push origin my-feature`)
-1. Create a Pull Request

data/completions/_iriq

@@ -27,6 +27,8 @@ _iriq() {
         '--no-scheme-less[skip schemeless URL extraction]' \
         '--scheme-less[enable schemeless URL extraction]' \
         '--corpus[load/create a JSON or SQLite corpus]:corpus path:_files -g "*.(json|db|sqlite|sqlite3)"' \
+        '(-C --no-corpus)'{-C,--no-corpus}'[disable corpus persistence for this invocation]' \
+        '--reset[delete the corpus database and exit]' \
         '--host[host-keying strategy for clustering]:strategy:(full registrable reg none)' \
         '--stats[print rolling aggregates]' \
         '--reinfer[replay the source-IRI log]' \

data/completions/iriq.bash

@@ -44,7 +44,7 @@ _iriq() {
     if [[ "$cur" == -* ]]; then
         local flags="-h --help -V --version -p --parse -n --normalize -c --canonical -e --explain
                      -j --json -J --ndjson -N --no-hints --hints --no-scheme-less
-                     --scheme-less --corpus --host --stats --reinfer
+                     --scheme-less --corpus -C --no-corpus --reset --host --stats --reinfer
                      --propose-recognizers --activate-above --cross-host-shapes
                      --min-observations --min-coverage --min-hosts"
         COMPREPLY=( $(compgen -W "$flags" -- "$cur") )

data/iriq.gemspec

@@ -5,7 +5,7 @@ Gem::Specification.new do |s|
   s.version     = Iriq::VERSION
   s.authors     = ["Daniel Pepper"]
   s.description = "IRI extraction, normalization, and clustering."
-  s.files       = `git ls-files * ':!:spec' ':!:script' ':!:bin' ':!:rust' ':!:go'`.split("\n")
+  s.files       = `git ls-files lib exe completions README.md LICENSE.txt CHANGELOG.md iriq.gemspec`.split("\n")
   s.bindir      = "exe"
   s.executables = ["iriq"]
   s.homepage    = "https://github.com/dpep/iriq"

data/lib/iriq/cli.rb

@@ -1,3 +1,4 @@
+require "fileutils"
 require "json"
 require "optparse"
 require "stringio"
@@ -35,30 +36,37 @@ module Iriq
                               each placeholder / canonical value was chosen
 
       Corpus + stats:
-            --corpus PATH     Load/create a JSON corpus; observe and save atomically.
-                              -n becomes corpus-informed once it has data.
+            --corpus PATH     Use a specific corpus file (overrides the default).
+                              Extension picks the backend: .db/.sqlite/.sqlite3
+                              are SQLite; anything else is JSON.
+        -C, --no-corpus       Disable corpus persistence for this invocation.
+                              Same as IRIQ_NO_CORPUS=1 in the environment.
+            --reset           Delete the corpus database (default path or the
+                              one resolved via --corpus / IRIQ_CORPUS) and exit.
             --host MODE       Host-keying strategy for clustering:
                               full (default), registrable (or reg) strips
                               subdomains, none ignores host entirely.
             --stats           Print rolling aggregates
             --reinfer         Replay the source-IRI log through the current
                               classifier + reducers; rebuilds materialized
-                              views from scratch. Requires --corpus.
+                              views from scratch.
             --propose-recognizers
                               Scan observed values for shape patterns that
                               recur enough to suggest a new Recognizer.
                               Combine with --json for structured output.
-                              Requires --corpus.
             --cross-host-shapes
                               List route shapes that recur across
                               multiple hosts. Combine with --min-hosts.
-                              Requires --corpus.
             --activate-above F  With --propose-recognizers, promote every
                               proposal at or above CONFIDENCE F into a
                               live Recognizer on the corpus, then
                               reinfer. Confidence integrates coverage
                               and cross-host corroboration.
 
+      Environment:
+            IRIQ_CORPUS=PATH    Set the corpus path (overrides the default).
+            IRIQ_NO_CORPUS=1    Disable the default corpus (equivalent to -C).
+
       Thresholds (apply to --propose-recognizers / --cross-host-shapes):
             --min-observations N  proposal noise floor (default 20)
             --min-coverage F      proposal coverage floor (default 0.7)
@@ -76,12 +84,14 @@ module Iriq
 
       Subcommands:
         cluster [file]        Force cluster view (default for ≥10 IRIs anyway)
+        completion <shell>    Print shell completion script (bash | zsh)
 
       Examples:
         iriq foo.com/users/456
         iriq -n https://foo.com/users/123
         iriq ./access.log                     # auto-detect file → extract URLs
         cat README.md | iriq -n               # one normalized URL per line
+        tail -f access.log | iriq -J          # live stream → NDJSON per IRI
         cat README.md | iriq --corpus c.json
     TXT
 
@@ -123,9 +133,17 @@ module Iriq
       batch_mode = explicit_cluster || positional_is_file ||
                    (args.empty? && piped_stdin?)
 
+      # --reset short-circuits: delete the resolved corpus file (+ sidecars)
+      # and exit. Resolves through the same precedence chain as the normal
+      # path so `--reset --corpus other.db` and `IRIQ_CORPUS=… --reset` Just Work.
+      if opts[:reset]
+        return cmd_reset(opts)
+      end
+
       return print_usage(stdout, 0) if args.empty? && !batch_mode && !opts[:reinfer] && !opts[:propose] && !opts[:cross_host_shapes]
 
-      corpus = opts[:corpus] ? load_corpus(opts[:corpus], host_strategy: opts[:host_strategy]) : nil
+      corpus_path = resolve_corpus_path(opts)
+      corpus = corpus_path ? load_corpus(corpus_path, host_strategy: opts[:host_strategy], announce_create: true) : nil
 
       code = if opts[:reinfer]
         cmd_reinfer(corpus, opts)
@@ -141,7 +159,7 @@ module Iriq
         cmd_summary(args, opts, corpus)
       end
 
-      corpus.save(opts[:corpus]) if corpus && opts[:corpus]
+      corpus.save(corpus_path) if corpus && corpus_path
       code
     rescue Iriq::ParseError => e
       emit_error("parse_error", e.message, 2, human: "iriq: parse error: #{e.message}")
@@ -167,6 +185,8 @@ module Iriq
         hints:       true,
         sections:    [],
         corpus:        nil,
+        no_corpus:     false,
+        reset:         false,
         stats:         false,
         reinfer:       false,
         propose:       false,
@@ -191,6 +211,8 @@ module Iriq
         o.on("--[no-]hints")         { |v| opts[:hints] = v }
         o.on("-N")                   { opts[:hints] = false }
         o.on("--corpus PATH")        { |v| opts[:corpus] = v }
+        o.on("-C", "--no-corpus")    { opts[:no_corpus] = true }
+        o.on("--reset")              { opts[:reset] = true }
         o.on("--host MODE")          { |v| opts[:host_strategy] = host_strategy_arg(v) }
         o.on("--stats")              { opts[:stats]   = true }
         o.on("--reinfer")            { opts[:reinfer] = true }
@@ -220,10 +242,79 @@ module Iriq
       end
     end
 
-    def load_corpus(path, host_strategy: :full)
+    def load_corpus(path, host_strategy: :full, announce_create: false)
+      if announce_create && !File.exist?(path)
+        FileUtils.mkdir_p(File.dirname(path))
+        stderr.puts "iriq: created corpus at #{path} (disable with --no-corpus or IRIQ_NO_CORPUS=1)"
+      end
       Corpus.open(path, host_strategy: host_strategy)
     end
 
+    # Resolve the corpus file the CLI should use. Precedence:
+    #   1. --corpus PATH       — explicit always wins (you asked for it)
+    #   2. --no-corpus / IRIQ_NO_CORPUS=1 — opt out of the default
+    #   3. IRIQ_CORPUS=PATH    — env override of the default location
+    #   4. default_corpus_path — platform-aware location
+    def resolve_corpus_path(opts)
+      return opts[:corpus] if opts[:corpus]
+      return nil if opts[:no_corpus] || env_corpus_disabled?
+
+      env_path = ENV["IRIQ_CORPUS"].to_s
+      return env_path unless env_path.empty?
+
+      default_corpus_path
+    end
+
+    # Platform-aware default. XDG-honoring on Linux + BSD, Apple-style on
+    # macOS, %LOCALAPPDATA% on Windows. Same logic in Rust so both
+    # runtimes share the same default.db.
+    def default_corpus_path
+      base = if (xdg = ENV["XDG_DATA_HOME"].to_s) && !xdg.empty?
+        File.join(xdg, "iriq")
+      elsif RUBY_PLATFORM =~ /darwin/
+        File.expand_path("~/Library/Application Support/iriq")
+      elsif RUBY_PLATFORM =~ /mingw|mswin|cygwin/
+        File.join(ENV["LOCALAPPDATA"].to_s.empty? ? File.expand_path("~/AppData/Local") : ENV["LOCALAPPDATA"], "iriq")
+      else
+        File.expand_path("~/.local/share/iriq")
+      end
+      File.join(base, "default.db")
+    end
+
+    def env_corpus_disabled?
+      v = ENV["IRIQ_NO_CORPUS"].to_s.downcase
+      !v.empty? && v != "0" && v != "false" && v != "no"
+    end
+
+    def cmd_reset(opts)
+      path = resolve_reset_path(opts)
+      unless path
+        return emit_error("missing_argument", "no corpus path to reset (use --corpus PATH or unset --no-corpus)", 1)
+      end
+      removed = []
+      [path, "#{path}-wal", "#{path}-shm", "#{path}.tmp"].each do |p|
+        if File.exist?(p)
+          File.delete(p)
+          removed << p
+        end
+      end
+      if removed.empty?
+        stderr.puts "iriq: no corpus to reset at #{path}"
+      else
+        stderr.puts "iriq: reset corpus at #{path}"
+      end
+      0
+    end
+
+    # --reset honors --corpus / IRIQ_CORPUS even when --no-corpus is set —
+    # the user is explicitly addressing a stored file, not the runtime state.
+    def resolve_reset_path(opts)
+      return opts[:corpus] if opts[:corpus]
+      env_path = ENV["IRIQ_CORPUS"].to_s
+      return env_path unless env_path.empty?
+      default_corpus_path
+    end
+
     # Accept `--host=reg` as a short alias for the `registrable` mode.
     HOST_STRATEGY_ALIASES = {
       "full" => :full, "registrable" => :registrable, "reg" => :registrable, "none" => :none,
@@ -473,9 +564,10 @@ module Iriq
     end
 
     # `completion <shell>` — emit the bundled shell-completion script.
-    # Scripts live in completions/{iriq.bash,_iriq} alongside the gem;
-    # Homebrew installs them automatically, but the user can also do
-    # `source <(iriq completion bash)` in their shell rc.
+    # Scripts live in completions/{iriq.bash,_iriq} alongside the
+    # gem; Homebrew installs them automatically, but the user can also do
+    # `source <(iriq completion bash)` in their shell rc. The Rust CLI
+    # inlines the same scripts — keep them byte-identical (parity-tested).
     COMPLETIONS_DIR = File.expand_path("../../completions", __dir__).freeze
     COMPLETION_FILES = {
       "bash" => File.join(COMPLETIONS_DIR, "iriq.bash"),
@@ -653,7 +745,10 @@ module Iriq
     end
 
     def emit_clusters(clusters, opts)
-      sorted = clusters.sort_by { |c| -c.count }
+      # Stable sort: equal-count clusters keep first-seen order. Ruby's
+      # sort_by is unstable, so ties need the explicit index tie-break to
+      # match the Rust CLI's (stable) sort.
+      sorted = clusters.sort_by.with_index { |c, i| [-c.count, i] }
 
       if opts[:json]
         emit_json(sorted.map(&:to_h), opts)
@@ -672,8 +767,8 @@ module Iriq
       end
     end
 
-    # One line per param: type, range (numeric), cardinality, presence.
-    # `page  integer  1..100  avg 50.5  (10 distinct, 100%)`
+    # One line per param: type, confidence, range (numeric), cardinality,
+    # presence. `page  integer  conf 0.87  1..100  avg 50.5  (10 distinct, 100%)`
     def emit_param_summary(cluster)
       rows = cluster.param_summary
       return if rows.empty?
@@ -681,6 +776,7 @@ module Iriq
       width = rows.map { |r| r[:name].length }.max
       rows.each do |r|
         bits = ["#{r[:type]}"]
+        bits << "conf #{format_conf(r[:confidence])}" if r[:confidence]
         if r[:min] && r[:max]
           bits << format_range(r[:min], r[:max])
           bits << "avg #{format_num(r[:avg])}" if r[:avg]
@@ -690,6 +786,11 @@ module Iriq
       end
     end
 
+    # Two-decimal confidence, e.g. 0.87. Matches the Rust CLI formatting.
+    def format_conf(conf)
+      format("%.2f", conf)
+    end
+
     def format_range(lo, hi)
       "#{format_num(lo)}..#{format_num(hi)}"
     end
@@ -737,9 +838,8 @@ module Iriq
     end
 
     def top(hash)
-      # Lex tie-break on equal counts — Ruby Hash insertion order would
-      # otherwise diverge from Go's map iteration (which has no insertion
-      # order). Keeps Ruby ↔ Go --stats parity stable.
+      # Lex tie-break on equal counts — keeps --stats output deterministic
+      # and Ruby ↔ Rust parity stable regardless of insertion order.
       hash.sort_by { |k, n| [-n, k] }.first(TOP_N_STATS).to_h
     end
   end