iriq 0.2.0 → 0.33.0

data/README.md

@@ -1,46 +1,48 @@
-Iriq
-======
-![Gem](https://img.shields.io/gem/dt/iriq?style=plastic)
-[![codecov](https://codecov.io/gh/dpep/iriq/branch/main/graph/badge.svg)](https://codecov.io/gh/dpep/iriq)
+# Iriq — IRI Query
 
-IRI extraction, normalization, and clustering.
+[![codecov](https://codecov.io/gh/dpep/iriq/branch/main/graph/badge.svg)](https://codecov.io/gh/dpep/iriq)
 
-Iriq pulls IRIs out of free text, parses them, normalizes them into
-canonical shape-aware forms, classifies their path and query components,
-and clusters similar identifiers — surfacing what's stable vs. unique.
+**Iriq finds the *shape* of a URL** — the structural template you get when you
+erase the parts that vary and keep the parts that don't. `…/users/123` and
+`…/users/999` are the same shape: `/users/{user_id}`. Feed iriq a pile of messy
+URLs — a log file, a column of links, free-text prose — and it collapses them
+into a small set of stable, deterministic route templates. Fifty thousand
+distinct URLs become twelve shapes.
 
-Ships as both a **command-line tool** (`iriq`) and a **library** (Ruby and
-Go — same behavior, enforced by parity tests).
+What's an IRI? Internationalized Resource Identifiers cover everyday URLs `https://…`, plus URNs like `urn:isbn:0451450523`,
+other schemes like `mailto:`, and internationalized addresses with non-ASCII
+characters like `https://例え.jp/パス`. Formally it's the Unicode superset of
+URI/URL. The name is *IRI Query*: iriq queries an IRI for its structure.
 
-## Install
+Everything iriq does — parsing, normalizing, classifying path and query
+components, clustering, learning new patterns — exists to derive, render, or
+group by that shape.
 
-The CLI is available three ways. Pick whichever fits your workflow:
+And it gets sharper the more you feed it. A *corpus* — on by default — records
+what it sees and improves classifications as data flows in: high-churn slots get
+promoted to placeholders, and whole types emerge that no single URL can reveal (a
+position that's always 100–599 is an HTTP status; one bounded to a dozen values
+is an enum).
 
 ```sh
-# Homebrew (recommended)
-brew install dpep/tools/iriq
-
-# RubyGems — installs the CLI shim and the library
-gem install iriq
-
-# Go — installs the CLI binary into $GOBIN
-go install github.com/dpep/iriq/cmd/iriq@latest
+$ iriq -n https://foo.com/users/123
+https://foo.com/users/{user_id}
 ```
 
-For library use, depend on whichever runtime you're working in:
+It answers questions like:
 
-```ruby
-# Gemfile
-gem "iriq"
-```
+- "What routes does this service actually expose?" (cluster a log file)
+- "Which params are stable identifiers vs. churning IDs vs. enums?"
+  (`--stats`)
+- "Are these 50,000 distinct URLs really just 12 templates?" (clustering)
+- "What does `/api/v1/users/abc-123-def` become as a route shape?"
+  (`/api/{version}/users/{user_id}`)
 
-```go
-import "github.com/dpep/iriq"
-```
+Iriq ships as a **command-line tool** (`iriq`) and a **Rust library**.
 
-## CLI quick start
+## Quick start
 
-```
+```sh
 $ iriq https://foo.com/users/123
 # parse
 original:      https://foo.com/users/123
@@ -56,368 +58,303 @@ https://foo.com/users/{user_id}
 $ iriq -n https://foo.com/users/123
 https://foo.com/users/{user_id}
 
-$ cat access.log | iriq             # extract → URL list (or clusters at scale)
-$ cat access.log | iriq --stats     # rolling aggregates
-$ iriq ./access.log -n              # file auto-detected → normalize each found URL
+$ iriq -n https://shop.com/pricing/usd?currency=eur
+https://shop.com/pricing/USD?currency=EUR     # currency upcased
 ```
 
-Full CLI reference is below under [CLI](#cli).
-
-## Library quick start
-
-```ruby
-# Ruby
-iri = Iriq.parse("https://foo.com/users/123")
-iri.scheme         # => "https"
-iri.host           # => "foo.com"
-iri.path_segments  # => ["users", "123"]
-iri.canonical      # => "https://foo.com/users/123"
-
-Iriq.normalize("https://foo.com/users/123")
-# => "https://foo.com/users/{user_id}"
+```sh
+$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
+[190] docs.example.com  /users/{user_id}
+[186] app.example.com   /users/{user_id}
+...
 
-Iriq.explain("https://foo.com/users/123/orders/456")
-# => [
-#      { value: "users",  type: :literal,    variable: false, hint: nil        },
-#      { value: "123",    type: :integer_id, variable: true,  hint: "user_id"  },
-#      { value: "orders", type: :literal,    variable: false, hint: nil        },
-#      { value: "456",    type: :integer_id, variable: true,  hint: "order_id" },
-#    ]
+$ cat access.log | iriq --stats               # rolling aggregates
+$ iriq ./access.log -n                        # auto-detect file → normalize each
+$ iriq -J < access.log                        # newline-delimited JSON
+$ iriq --corpus team.db < access.log          # use a specific corpus file
 ```
 
-```go
-// Go (same surface)
-iri, _ := iriq.Parse("https://foo.com/users/123")
-iri.Scheme         // "https"
-iri.Host           // "foo.com"
-iri.PathSegments   // []string{"users", "123"}
-iri.Canonical()    // "https://foo.com/users/123"
+Per-IRI output (`-n`, `-p`, `-c`, `-J`) streams — each line is read, classified,
+and flushed as it arrives, so iriq works on an unbounded live feed:
 
-norm, _ := iriq.Normalize("https://foo.com/users/123")
-// "https://foo.com/users/{user_id}"
+```sh
+$ tail -f access.log | iriq -n                # one shape per line, as logs land
+$ tail -f access.log | iriq -J                # same, as newline-delimited JSON
 ```
 
-The Ruby gem is the reference implementation; Go mirrors its API and is
-kept in sync via JSON fixtures plus a CLI parity harness. See
-[CLAUDE.md](CLAUDE.md) for the dev process.
-
-Pass `hints: false` to `Iriq.normalize` (or `PathShape`) for mechanical
-placeholders (`{integer_id}` instead of `{user_id}`).
-
-## RESTful hints
+**Every invocation observes into a persistent corpus by default**, so iriq gets
+smarter the more you run it. The corpus-only types (e.g. `enum` / `http_status`)
+emerge from the *distribution* of values observed.
 
-When a variable segment follows a literal one, Iriq derives a hint by
-singularizing the literal and suffixing `_id` (or `_uuid` for UUIDs). This is
-what produces `{user_id}` from `/users/123` and `{order_id}` from
-`/orders/456`. Singularization uses `Iriq::Inflector`, which delegates to a
-swappable adapter:
-
-```ruby
-# Default: ActiveSupport::Inflector if `active_support/inflector` is loadable,
-# otherwise a built-in adapter with rules adapted from ActiveSupport.
-
-Iriq::Inflector.singularize("categories")  # => "category"
-Iriq::Inflector.singularize("people")      # => "person"
+```sh
+# Feed a stream where ?status only ever holds a couple of words:
+$ for n in $(seq 1 20); do
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=open"   >/dev/null
+    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=closed" >/dev/null
+  done
 
-# Override:
-Iriq::Inflector.adapter = MyAdapter        # must respond to .singularize(String)
-Iriq::Inflector.reset_adapter!
+# Ask what it learned. ?status is now an enum — a verdict no single URL
+# could support, since one URL shows only one value:
+$ iriq --corpus demo.db cluster
+[40] api.foo.com  /orders/{order_id}
+    https://api.foo.com/orders/1?status=open
+    https://api.foo.com/orders/1?status=closed
+    https://api.foo.com/orders/2?status=open
+    + 37 more
+    status  enum  (2 distinct, 100%)
 ```
 
-## Supported inputs
-
-| Input                                | Notes                                            |
-| ------------------------------------ | ------------------------------------------------ |
-| `https://foo.com/users/123`          | Standard URL                                     |
-| `foo.com/users/456`                  | Scheme-less; `https://` is assumed               |
-| `urn:isbn:0451450523`                | URN — `scheme` and `nss` are populated           |
-| `https://例え.テスト/こんにちは`         | Unicode IRI — display form preserved             |
-| `HTTPS://Foo.com:443/A`              | Scheme + host lowercased; default port dropped   |
-| `https://foo.com/a/./b/../c`         | Dot segments normalized                          |
+These learned types also flow into normalized output: once the corpus has pegged
+`?status` as an enum, `iriq -n …?status=open` renders `?status={enum}`.
 
-## Segment classification
+The default corpus lives at `$XDG_DATA_HOME/iriq/default.db` (Linux),
+`~/Library/Application Support/iriq/default.db` (macOS), or
+`%LOCALAPPDATA%/iriq/default.db` (Windows). First-run creation prints a
+one-line stderr notice. Three knobs control it:
 
-`Iriq::SegmentClassifier` returns one of:
-
-- `:literal` — plain word (`users`, `orders`, `Profile`, `こんにちは`)
-- `:integer_id` — pure digits below the timestamp range (`1`, `123`, `42`)
-- `:uuid` — `f47ac10b-58cc-4372-a567-0e02b2c3d479`
-- `:date` — `2024-05-23`
-- `:timestamp` — ISO 8601, or 10/13-digit UNIX epoch
-- `:hash` — 32+ hex chars (md5 / sha)
-- `:slug` — `my-cool-post`, `my_cool_post`
-- `:opaque_id` — short alphanumeric mix that doesn't fit elsewhere
-
-Heuristics are deterministic and ordered — the first matching rule wins.
-
-## Clustering
-
-```ruby
-clusterer = Iriq::Clusterer.new
-clusterer.add("https://foo.com/users/123")
-clusterer.add("https://foo.com/users/456")
-clusterer.add("https://foo.com/users/789/orders/1")
-
-clusterer.clusters.map(&:shape)
-# => ["/users/{user_id}", "/users/{user_id}/orders/{order_id}"]
-
-clusterer.clusters.first.segment_stats
-# => [
-#      { position: 0, stable: true,  values: { "users" => 2 } },
-#      { position: 1, stable: false, values: { "123" => 1, "456" => 1 } },
-#    ]
-
-clusterer.explain("https://foo.com/users/999")
-# => [
-#      { value: "users", type: :literal,    variable: false, hint: nil,       stable: true  },
-#      { value: "999",   type: :integer_id, variable: true,  hint: "user_id", stable: false },
-#    ]
+```sh
+$ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
+$ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
+$ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
+$ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
+$ iriq --reset                                     # delete the corpus DB and exit
 ```
 
-The clusterer combines classifier output with what it has actually observed:
-a position the classifier *would* call variable but that is empirically
-constant across all members of the cluster will be reported with
-`stable: true, variable: false`.
-
-## Corpus (streaming + learning)
-
-For processing many identifiers — possibly an unbounded stream — use
-`Iriq::Corpus`. It maintains rolling aggregates and per-(host, prefix)
-frequency stats so classification improves as more data comes in.
-
-```ruby
-corpus = Iriq::Corpus.new
-
-iris.each do |iri|
-  obs = corpus.observe(iri)
-  obs.fingerprint   # deterministic shape: "https://foo.com/users/{user_id}"
-  obs.cluster       # the Iriq::Cluster this fell into
-  obs.explanation   # per-segment annotations with corpus-informed classification
-end
-
-corpus.host_counts          # { "foo.com" => 1234, "bar.com" => 7 }
-corpus.path_length_counts   # { 2 => 800, 3 => 434 }
-corpus.fingerprint_counts   # shape → count
-corpus.raw_shape_counts     # hint-free shape → count
-corpus.clusters             # Iriq::Cluster instances
-```
+### Two ways to normalize
 
-### Deterministic vs. corpus-informed normalization
+Pick by the question you're asking:
 
-```ruby
-Iriq.normalize("https://foo.com/users/me")
-# => "https://foo.com/users/me"   # mechanical: "me" is a literal
+- **`--canonical`** — clean up *this* URL, keeping the specifics.
+  `HTTP://Foo.com:80/pull/42` → `http://foo.com/pull/42` (scheme/host
+  lowercased, default port dropped; path and query left alone). Handy, but
+  table stakes — plenty of libraries do it.
+- **`--normalize`** *(the default)* — find the URL's *shape*, erasing the
+  specifics into placeholders. `…/pull/42` → `…/pull/{id}`. This is the part
+  you came to iriq for.
 
-corpus.normalize("https://foo.com/users/me")
-# => depends on what the corpus has seen
-```
+Same input, two questions: "what's the clean form of *this* URL?" vs "what
+*kind* of URL is this?" The second is iriq's reason to exist.
 
-If many `/users/{integer_id}` paths flow in alongside a handful of
-`/users/me`, the cluster `/users/me` is preserved (mechanical clustering
-keeps literal routes distinct). If many *distinct literal handles*
-(`/users/alice`, `/users/bob`, `/users/carol`, ...) flow in, the corpus
-promotes that position to a `{user}` placeholder:
+## Install
 
-```ruby
-%w[alice bob carol dave erin frank gina hank ivan jane].each do |name|
-  corpus.observe("https://foo.com/users/#{name}/profile")
-end
+```sh
+# Homebrew (recommended)
+brew install dpep/tools/iriq
 
-corpus.normalize("https://foo.com/users/alice/profile")
-# => "https://foo.com/users/{user}/profile"
+# Cargo, from crates.io
+cargo install iriq
 ```
 
-### Explainability
+One crate ships both the library and the `iriq` binary. Corpora persist to
+SQLite (bundled, WAL) out of the box — nothing to flag, install, or rebuild.
 
-Each row of `corpus.explain(...)` (and `observation.explanation`) carries a
-`classification:` symbol on top of the deterministic fields:
+## Segment classification
 
-| Classification              | Meaning                                              |
-| --------------------------- | ---------------------------------------------------- |
-| `:stable_literal`           | Literal value dominates this position                |
-| `:variable_identifier`      | Classifier said variable (uuid, integer, etc.)       |
-| `:rare_literal`             | Literal seen here, but not dominant                  |
-| `:corpus_inferred_variable` | Classifier said literal, but position has high entropy |
-| `:ambiguous`                | Insufficient signal — never seen, or mixed           |
+Iriq classifies each path/query segment into one of ~25 types — the first
+matching rule wins, and heuristics are deterministic:
+
+- `literal` — plain word (`users`, `orders`, `Profile`, `こんにちは`)
+- `integer` — pure digits below the timestamp range
+- `float` — decimal with digits on both sides (`3.14`, `-2.5`, `1.0`)
+- `boolean` — `true` / `false` (any case)
+- `version` — semver-ish with `v` prefix (`v1`, `v2.0.1`, `v1.2.3-beta`)
+- `locale` — BCP 47-ish (`en-US`, `fr_CA`, `zh-Hant`, bare `en`/`fr`/`ja`)
+- `currency` — ISO 4217 codes (`USD`, `EUR`, `JPY`)
+- `uuid` — `f47ac10b-58cc-4372-a567-0e02b2c3d479`
+- `date` — `2024-05-23`, `2024/05/23`, `20240523`, `05/23/2024`. Canonicalized to ISO in `--normalize` output.
+- `timestamp` — ISO 8601, or 10/13-digit UNIX epoch
+- `hash` — 32+ hex chars (md5 / sha)
+- `slug` — `my-cool-post`, `my_cool_post`
+- `ipv4` / `ipv6` — collapsed to `{ip}` in normalized output
+- `url` — `https://...`, `ftp://...`, also scheme-less `foo.com/path`
+- `email` — `local@host.tld`
+- `phone` — E.164 (`+15551234567`) or NANP (`555-666-7777`, `(555) 666-7777`)
+- `jwt` — three base64url segments separated by dots
+- `mime` — `image/png`, `application/vnd.api+json`
+- `file` — `name.ext` for known extensions; per-kind grouping (image/document/data/...)
+- `color` — hex form (`#fff`, `#ffffff`, `#ffffff80`)
+- `coordinate` — `lat,lng` pair with plausible-range validation
+- `country` — ISO 3166-1 alpha-2 codes (`US`, `JP`, `GB`)
+- `base64` — standard base64 blobs with disambiguating `+`/`/`/`=`
+- `opaque_id` — short alphanumeric mix that doesn't fit elsewhere
+
+### RESTful hints
+
+When a variable segment follows a literal one, iriq derives a hint by
+singularizing the literal and suffixing `_id` (or `_uuid` for UUIDs). That's
+what produces `{user_id}` from `/users/123` and `{order_id}` from `/orders/456`.
+Semantic types (`version`, `locale`, `currency`, `date`, `boolean`) skip the
+hint and surface as `{type}` — `/api/v1/status` renders as `/api/{version}/status`,
+not the misleading `/api/{api_id}/status`. Pass `-N` / `--no-hints` for
+mechanical placeholders (`{integer}` instead of `{user_id}`).
+
+### Types only the corpus can see
+
+Four types emerge from the *distribution* of values across many observations:
+
+| Type | Emerges when a position… |
+| --- | --- |
+| `number` | holds both integers and floats |
+| `year` | holds integers that all land in 1900–2100 |
+| `http_status` | holds integers that all land in 100–599 |
+| `enum` | holds a small, bounded set of distinct values |
+
+Mechanically, `200` is just an integer. Across ten thousand URLs where that
+slot is always 100–599, it's likely an HTTP status.
 
-## Extracting IRIs from text
+## Corpus (streaming + learning)
 
-`Iriq::Extractor` is what powers pipe-mode in the CLI. Picks up explicit-
-scheme URLs (`http`, `https`, `ftp`, `ws`, `wss`, `urn`) and `foo.com/path`-
-style scheme-less URLs (small TLD allow-list, required path). Trims trailing
-sentence punctuation iteratively and preserves balanced parens
-(`https://en.wikipedia.org/wiki/Ruby_(programming_language)` stays intact;
-`(see https://foo.com)` drops the outer paren).
-
-```ruby
-Iriq.extract("Visit https://foo.com today, also hit foo.com/users.")
-# => [#<Iriq::Identifier https://foo.com>,
-#     #<Iriq::Identifier https://foo.com/users>]
-
-# Disable scheme-less:
-Iriq::Extractor.new(scheme_less: false).extract("hit foo.com/users today")
-# => []
-```
+The corpus maintains rolling aggregates and per-(host, prefix) frequency stats,
+so classification improves as more data comes in — handy for an unbounded stream
+of identifiers. The default corpus already persists; `--corpus PATH` points iriq
+at a specific file instead, to keep separate corpora or share one across runs.
 
-Known limitations (intentional):
+A `.db` / `.sqlite` / `.sqlite3` path is stored in SQLite (WAL journaling, incremental
+UPSERTs — multiple `iriq --corpus` processes can write concurrently); a
+`.json` path writes a plain JSON file instead.
 
-- Comma is a URL boundary, so query strings like `?q=37.7,-122.4` truncate.
-  Trade-off picked to keep CSV-shaped text working.
-- No HTML entity decoding (`&amp;` stays as-is).
-- Scheme-less mode skips bare hostnames without a path (too noisy in prose).
-
-### Memory bounds
+### Re-runnable inference
 
-- Per-position `value_counts` is capped (`max_values_per_position`, default
-  1000) — once full, `total` keeps growing but only existing keys count up.
-- Cluster examples are capped at `Iriq::Cluster::MAX_EXAMPLES`.
-- No raw IRI strings are retained outside the bounded cluster examples.
+A corpus persists the source-IRI log alongside the materialized views.
+`--reinfer` drops every view and replays the log through the current classifier
+and reducers. Tune a threshold, swap in a different classifier, or activate new
+recognizers (below) — then reinfer to see the new results without re-feeding
+URLs.
 
-```ruby
-Iriq::Corpus.new(max_values_per_position: 200)
+```sh
+$ iriq --corpus c.db --reinfer
 ```
 
-## Object model
+## Learning new types
 
-| Class                       | Responsibility                                       |
-| --------------------------- | ---------------------------------------------------- |
-| `Iriq::Parser`              | String → `Identifier`                                |
-| `Iriq::Identifier`          | Structured fields + `canonical` reconstruction       |
-| `Iriq::SegmentClassifier`   | Single segment → type symbol                         |
-| `Iriq::PathShape`           | Segments → `/users/{user_id}` route shape            |
-| `Iriq::SegmentHints`        | Derives `user_id`-style hints from neighbors         |
-| `Iriq::Inflector`           | Singularization with swappable adapter (AS or built-in) |
-| `Iriq::Normalizer`          | Identifier → canonical, shape-aware string           |
-| `Iriq::Explanation`         | Per-segment `{value, type, variable, hint}` rows     |
-| `Iriq::Cluster`             | One host + shape group, with examples & stats        |
-| `Iriq::Clusterer`           | Many identifiers → `Cluster` set + explain          |
-| `Iriq::PositionStats`       | Capped value/type frequencies for one position       |
-| `Iriq::Observation`         | What `Corpus#observe` returns                        |
-| `Iriq::Corpus`              | Streaming observer with rolling aggregates + learning |
-| `Iriq::Extractor`           | Pulls IRIs out of free text (scheme-anchored)        |
+Iriq doesn't just classify against a fixed list — it watches the stream and
+*proposes new recognizers* for patterns it keeps seeing. Notice `ghp_…` or
+`cus_…` recurring at a slug position and iriq will suggest a recognizer for it,
+with evidence: coverage, host count, confidence.
 
-## CLI
-
-Installing the gem installs an `iriq` executable. Two main modes:
-
-**Single input** — combined parse + normalize summary; trim with section
-flags (`-p`, `-n`).
+```sh
+# Print proposals (human-readable, or --json)
+$ iriq --corpus c.db --propose-recognizers
 
+# Auto-activate every proposal with confidence ≥ 0.9, then reinfer
+$ iriq --corpus c.db --propose-recognizers --activate-above 0.9
 ```
-$ iriq foo.com/users/456
-# parse
-original:      foo.com/users/456
-kind:          url
-scheme:        https
-host:          foo.com
-path_segments: ["users", "456"]
-canonical:     https://foo.com/users/456
-
-# normalize
-https://foo.com/users/{user_id}
 
-$ iriq -n https://foo.com/users/123
-https://foo.com/users/{user_id}
-```
+### Cross-host shape learning
 
-**Piped stdin** — extraction runs by default. Output auto-switches: small
-inputs get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the
-cluster view via an ephemeral corpus. Section flags work too — emit one
-normalized URL / parsed record per extracted IRI.
+A route shape that recurs across multiple hosts is independent evidence of a
+semantic pattern — two unrelated hosts inventing the same `/users/{integer}`
+structure by accident is unlikely.
 
+```sh
+$ iriq --corpus c.db --cross-host-shapes [--min-hosts N]
 ```
-$ cat short.txt | iriq
-[2] https://github.com/dpep/iriq
-[1] https://foo.com/users
 
-$ cat short.txt | iriq -n                     # normalized URL per line
-https://github.com/dpep/iriq
-https://foo.com/users
+The same signal feeds back into proposal `confidence`: each additional host
+beyond the first adds `0.05` to the score (capped at 1.0), so a prefix proposed
+on 5 hosts is meaningfully stronger than the same coverage seen on 1 host.
 
-$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
-[190] docs.example.com  /users/{user_id}
-[186] app.example.com   /users/{user_id}
-...
+## Extracting IRIs from text
 
-$ cat README.md | iriq --stats                # rolling aggregates
-$ cat README.md | iriq cluster                # force cluster view
-$ cat README.md | iriq --corpus c.json        # persist into a corpus
-```
+Pipe-mode extraction picks up explicit-scheme URLs (`http`, `https`, `ftp`,
+`ws`, `wss`, `urn`) and `foo.com/path`-style scheme-less URLs (small TLD
+allow-list, required path). It trims trailing sentence punctuation and preserves
+balanced parens (`https://en.wikipedia.org/wiki/Ruby_(programming_language)`
+stays intact; `(see https://foo.com)` drops the outer paren).
 
-`--corpus PATH` makes the corpus survive across invocations. The file
-extension picks the storage backend:
+Known limitations (intentional):
 
-- `.json` — a single atomically-written JSON file (default). Best for small
-  corpora and when you want the data human-readable.
-- `.db` / `.sqlite` / `.sqlite3` — a SQLite database with WAL journaling.
-  Each observation is an incremental UPSERT, so multiple `iriq --corpus`
-  processes can write concurrently without clobbering each other, and the
-  cost of opening doesn't scale with corpus size.
+- Comma is a URL boundary, so query strings like `?q=37.7,-122.4` truncate.
+  Trade-off picked to keep CSV-shaped text working.
+- No HTML entity decoding (`&amp;` stays as-is).
+- Scheme-less mode skips bare hostnames without a path (too noisy in prose).
 
-Once the corpus has data, `-n` becomes corpus-informed:
+Disable scheme-less extraction with `--no-scheme-less`.
 
-```
-$ for n in alice bob carol dave erin frank gina hank ivan jane; do
-    iriq --corpus c.db https://foo.com/users/$n/profile >/dev/null
-  done
+## How it works
 
-$ iriq -n --corpus c.db https://foo.com/users/zoe/profile
-https://foo.com/users/{user}/profile         # mechanical would keep "zoe"
-```
+Under the shape sits one idea: **Position + Evidence**. A *Position* is a slot
+in a host's structure — a typed path prefix, or a query-param name. *Evidence*
+is everything the corpus has observed about that slot: which values, how often,
+across how many hosts. Strings are observations; types are inferences drawn from
+the pile. Shape is the surface you see; Position + Evidence is the engine
+underneath. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full model.
+
+## CLI reference
 
-Library: `Iriq::Corpus.open("c.db")` (or `iriq.OpenCorpus("c.db")` in Go)
-dispatches on the same extension rules. `corpus.save("export.json")`
-exports any backend as JSON.
+**Single input** — combined parse + normalize summary; trim with section flags
+(`-p`, `-n`).
 
-Flags:
+**Piped stdin** — extraction runs by default. Output auto-switches: small inputs
+get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the cluster view via
+an ephemeral corpus.
 
 | Flag                | Effect                                                  |
 | ------------------- | ------------------------------------------------------- |
 | `-p, --parse`       | Show parsed fields                                      |
 | `-n, --normalize`   | Show the shape-normalized form                          |
+| `-c, --canonical`   | Show the canonical form (no shape normalization)        |
+| `-e, --explain`     | Annotated trace — per-segment notes about why each placeholder / canonical value was chosen |
 | `-j, --json`        | Emit JSON                                               |
-| `-N, --no-hints`    | Use `{integer_id}` etc. instead of `{user_id}`          |
+| `-J, --ndjson`      | Newline-delimited JSON (one object per line); implies `--json` |
+| `-N, --no-hints`    | Use `{integer}` etc. instead of `{user_id}`             |
 | `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
-| `--corpus PATH`     | Load/create a corpus at PATH (`.json` or `.db`/`.sqlite`/`.sqlite3`) |
+| `--corpus PATH`     | Use a specific corpus file (`.json` or `.db`/`.sqlite`/`.sqlite3`). Overrides the default |
+| `-C, --no-corpus`   | Disable corpus persistence for this invocation (same as `IRIQ_NO_CORPUS=1`) |
+| `--reset`           | Delete the corpus database and exit                     |
+| `--host MODE`       | Host-keying for clustering: `full` (default), `reg` strips subdomains, `none` ignores host |
 | `--stats`           | Print rolling aggregates                                |
+| `--reinfer`         | Drop the materialized views and replay the source-IRI log through the current classifier + reducers |
+| `--propose-recognizers` | Scan observed values for shape patterns that recur enough to suggest a new recognizer. Combine with `--json` for structured output |
+| `--cross-host-shapes`   | List route shapes that recur across multiple hosts |
+| `--min-observations N`  | Proposal threshold; default 20                    |
+| `--min-coverage F`      | Proposal threshold; default 0.7                   |
+| `--min-hosts N`         | Threshold for both proposals and cross-host shapes; default 1 / 2 respectively |
+| `--activate-above F`    | With `--propose-recognizers`, auto-activate every proposal whose confidence is ≥ F |
+| `completion bash\|zsh`  | Print shell completion script (Homebrew installs this automatically) |
 | `-V, --version`     | Print version                                           |
 
-A positional argument that doesn't parse as an IRI but IS an existing
-file is read and extracted from automatically — `iriq ./access.log` and
-`iriq /var/log/foo.log` Just Work. (Bare filenames like `README.md`
-may still parse as a URL; pipe with `cat` to disambiguate.)
+Environment variables:
+
+| Variable             | Effect                                                  |
+| -------------------- | ------------------------------------------------------- |
+| `IRIQ_CORPUS=PATH`   | Set the corpus path (overrides the default)             |
+| `IRIQ_NO_CORPUS=1`   | Disable the default corpus (equivalent to `-C`)         |
+
+A positional argument that doesn't parse as an IRI but IS an existing file is
+read and extracted from automatically — `iriq ./access.log` and
+`iriq /var/log/foo.log` Just Work. (pipe with `cat` to disambiguate)
 
 Exit codes: `0` success, `1` usage error, `2` parse error.
 
-## Performance
+## Rust library
 
-Measured on the deterministic `IriGenerator` fixture (Ruby 3.4.9, single
-thread):
+```sh
+cargo add iriq
+```
 
-| Operation                | Throughput   |
-| ------------------------ | ------------ |
-| `Iriq.parse`             | ~260k URLs/s |
-| `Iriq.normalize`         | ~148k URLs/s |
-| `Iriq.explain`           | ~205k URLs/s |
-| `Iriq.extract` (prose)   | ~9.6 MB/s    |
-| `Corpus#observe`         | ~80k URLs/s  |
-| Corpus save/load (10k)   | ~135 ms      |
+```rust
+use iriq::{parse, normalize, Corpus};
 
-Linear scaling holds through 100k observations; per-observation retained
-memory amortizes to ~100 bytes at that scale. Memoization caches are
-bounded by `CACHE_MAX = 10_000` (cleared when full) — overhead is a few
-hundred KB regardless of corpus size.
+let iri = parse("https://foo.com/users/123")?;
+iri.host;             // "foo.com"
+iri.path_segments;    // ["users", "123"]
+iri.canonical();      // "https://foo.com/users/123"
 
-Re-run anytime with:
+normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"
 
+// Streaming clustering against a persistent corpus.
+let mut corpus = Corpus::open("c.db")?;
+corpus.observe("https://foo.com/users/1")?;
+corpus.save("c.db")?;
 ```
-bundle exec script/benchmark.rb       # throughput
-bundle exec script/memory.rb          # retained memory + cache footprints
-```
+
+Full API on [docs.rs/iriq](https://docs.rs/iriq); see the
+[crate README](rust/iriq/README.md) for the library tour.
 
 ## Limitations (intentional)
 
-This is an MVP. Iriq does **not**:
+Iriq does **not**:
 
 - Implement RFC 3986, RFC 3987, or the WHATWG URL standard fully.
 - Convert between Unicode (IRI) and punycode (URI) — the display form is
@@ -425,42 +362,8 @@ This is an MVP. Iriq does **not**:
 - Percent-encode or decode path/query bytes. Bytes are kept as written.
 - Validate scheme-specific structure beyond URL vs. URN.
 - Resolve relative references against a base URL.
-- Round-trip `canonical` back to the exact original byte-for-byte (whitespace
-  is stripped, default ports are dropped, dot segments are collapsed).
-
-For richer IRI handling, see `addressable`. Iriq's focus is the analysis
-side: classification, normalization, and clustering — not a complete URL
-implementation.
-
-----
-## Go port
-
-A Go implementation lives under [`go/`](go/) — same public surface, same
-behavior, ~10× faster CLI on extraction-heavy workloads. The Ruby gem is
-the reference; the Go port stays in sync via golden JSON fixtures
-(`spec/fixtures/`) and a CLI parity harness (`script/cli_parity.sh`), both
-checked in CI.
-
-```go
-import "github.com/dpep/iriq/go/iriq"
-
-iri, _ := iriq.Parse("https://foo.com/users/123")
-norm, _ := iriq.Normalize("https://foo.com/users/123")
-// "https://foo.com/users/{user_id}"
-```
-
-See [`go/README.md`](go/README.md) for the full API table and porting workflow.
-
-----
-## Contributing
-
-Yes please  :)
+- Round-trip `canonical` back to the exact original byte-for-byte (whitespace is
+  stripped, default ports are dropped, dot segments are collapsed).
 
-1. Fork it
-1. Create your feature branch (`git checkout -b my-feature`)
-1. Ensure the tests pass (`bundle exec rspec`)
-1. If you changed library behavior, port the change to Go (or open an
-   issue) and regenerate fixtures: `bundle exec ruby script/generate_fixtures.rb`
-1. Commit your changes (`git commit -am 'awesome new feature'`)
-1. Push your branch (`git push origin my-feature`)
-1. Create a Pull Request
+Iriq's focus is the analysis side: classification, normalization, and clustering
+— not a complete URL implementation.