iriq 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 95d6bc09f7de65bcb4acc5db3ad68d1b83c326360d913c45aede66038a100461
-  data.tar.gz: ae58d39b77fce3041cc5561b575dea52706b38f4cfab11d4881cf2f644f6cc59
+  metadata.gz: e629988f23137ecb0c0f4737e246f65a949fa2843f4bd16d244566ca76dc37ed
+  data.tar.gz: c36cae38205a6a6f63a8a38e40849c922bbb6cf7046e9599aaaefc37fb443303
 SHA512:
-  metadata.gz: a8fa85f112d9766ff4e9e4ad60c1043ce9701fe42e6dbd01f70da39fa0dd554bff573f40c858d88ab1e1539de0c1f017609535cc57e8e4fcc6651f0d20697e60
-  data.tar.gz: 97c714e664874c08278a305b8beff470b68bd2b7e4f58977cc44cf2e3716e619d17011a33d77b8ea9356eec0715acb6c815252551edf64db16eec4599f816274
+  metadata.gz: 75d0329756d16dd7b9c8e5ca7b8ba447aa51c4d3c34a1ec31549bb6e6725338bfbdb2ee4badb18c52342c8ae0ece5f005e9288c617fa5e140d7cfff400274561
+  data.tar.gz: cb670e665a2e67feeb5f1bcad157a27e15da1aa61efc53e4ae66388a67ee498ef7caa1605be85d27e087fda44c399a9f93b6701242d1a22dec2653b667918c6d

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+###  0.2.0  (2026-05-25)
+- Corpus storage backends: JSON (default) and SQLite, dispatched by file extension
+- Go: `iriq.OpenCorpus(path)`; Ruby: `Iriq::Corpus.open(path)`
+- SQLite backend: incremental UPSERTs, WAL mode, concurrent-safe via busy_timeout + BEGIN IMMEDIATE; checkpoints on close so the WAL sidecar doesn't grow unbounded
+- Batch mode: `corpus.batch { ... }` (Ruby) / `corpus.Batch(fn)` (Go) wraps many observations in one transaction
+- Clusterer now wraps the in-memory Storage backend; only one cluster code path
+- script/bench_storage.sh — JSON vs SQLite timing across single-process, incremental, and concurrent workloads
+- **Breaking (Go)**: `Corpus.HostCounts` / `PathLengthCounts` / `RawShapeCounts` / `FingerprintCounts` are methods now, not fields
+
 ###  0.1.0  (2026-05-24)
 - CLI: auto-detect file argument, retire --extract flag
 - CLI: section flags work in pipe mode + clean up help text

data/CLAUDE.md

@@ -0,0 +1,121 @@
+# Iriq development conventions
+
+## Repo layout — Ruby and Go intermixed at the root
+
+We chose to mix Ruby and Go at the repo root rather than nest the Go module
+under `/go/`. The signal is "both implementations are peers, not one-is-primary."
+
+```
+iriq/
+  lib/ exe/ spec/      ← Ruby gem (library, CLI, specs)
+  iriq.gemspec
+  Gemfile
+
+  go.mod               ← module github.com/dpep/iriq
+  *.go                 ← Go package `iriq` at the root
+  cmd/iriq/            ← Go CLI binary
+  bin/                 ← built Go binary (gitignored)
+
+  script/              ← shared dev scripts (fixture gen, parity, benches)
+  spec/fixtures/       ← golden JSON shared by Ruby specs + Go tests
+  .github/workflows/   ← Ruby CI, Go CI, parity CI
+```
+
+Trade-offs of this layout:
+
+- Clean import path: `github.com/dpep/iriq` (no `/go/` artifact in consumers' code).
+- One version tag (`vX.Y.Z`) serves both runtimes — Ruby's gemspec and Go's
+  module use the same tag stream.
+- Root `ls` is busier (~15 `.go` files next to Ruby ones), accepted in exchange.
+- The gemspec explicitly excludes Go files so `gem build` doesn't ship them:
+  `git ls-files * ':!:spec' ':!:script' ':!:cmd' ':!:bin' ':!:*.go' ':!:go.mod' ':!:go.sum'`.
+
+## Building
+
+```sh
+# Ruby gem
+bundle install
+bundle exec exe/iriq --help     # runs the CLI from source
+
+# Go binary — convenience targets in the Makefile
+make build                      # → ./bin/iriq
+make install                    # go install into $GOBIN
+make uninstall                  # remove from $GOBIN
+make clean                      # remove ./bin/
+make test                       # go test ./...
+
+# Both via Homebrew
+brew install dpep/tools/iriq    # uses the Ruby gem under the hood
+```
+
+## Keeping Ruby and Go in sync
+
+The Ruby gem is the **reference implementation**. Go mirrors its public API
+and behavior. Two layers of parity testing keep them aligned:
+
+1. **Golden JSON fixtures** (`spec/fixtures/*.json`)
+   Generated by `script/generate_fixtures.rb` from the Ruby implementation
+   over a curated set of inputs. Go's `fixtures_test.go` loads each file
+   and asserts the same outputs from the Go side.
+
+2. **CLI parity harness** (`script/cli_parity.sh`)
+   Runs the same input through `bundle exec exe/iriq` and the Go binary and
+   diffs stdout. Lives in CI as the `Ruby ↔ Go parity` job.
+
+When changing behavior:
+
+1. Update the Ruby code + specs first.
+2. Regenerate fixtures: `bundle exec ruby script/generate_fixtures.rb`.
+3. Port the change to Go.
+4. `go test ./...` (uses the updated fixtures).
+5. `script/cli_parity.sh` should pass.
+6. Commit fixtures with the change — CI will fail if they're stale.
+
+## Tests
+
+```sh
+bundle exec rspec        # Ruby suite (305+ examples)
+go test ./...            # Go suite (native + fixture parity tests)
+script/cli_parity.sh     # CLI parity (13+ scenarios)
+```
+
+## Releases
+
+- One version tag covers both runtimes — bump `lib/iriq/version.rb` (and
+  optionally a matching constant on the Go side if we add one), tag `vX.Y.Z`,
+  push.
+- `gem push iriq-X.Y.Z.gem` to publish to RubyGems.
+- Update `Formula/iriq.rb` in the homebrew-tools tap to the new version.
+- Go consumers pick up the tag automatically via `go get @vX.Y.Z`.
+
+## Corpus storage backends
+
+The `Corpus` class delegates state to a `Storage` backend; three backends ship:
+
+- **Memory** — default, in-process only.
+- **JSON** — Memory wrapped with atomic load/save against a JSON file
+  (`.json` by default). Same shape both runtimes have always written.
+- **SQLite** — incremental UPSERTs against a `.db` / `.sqlite` / `.sqlite3`
+  file with WAL journaling. Supports concurrent observers and avoids
+  loading the whole corpus into memory.
+
+`Corpus.open(path)` (Ruby) / `iriq.OpenCorpus(path)` (Go) picks the backend
+by file extension. `corpus.save(other_path)` exports as JSON regardless of
+the live backend; `corpus.save(same_path)` is idempotent (no clobbering a
+SQLite file with JSON, etc.).
+
+The Ruby `sqlite3` gem is loaded lazily (only when a `.db` path is opened),
+keeping the iriq install footprint minimal for users that stick with JSON.
+On the Go side we use `modernc.org/sqlite` (pure Go — no cgo).
+
+When adding a new backend, replicate the contract in both languages and
+add a parity scenario in `script/cli_parity.sh`'s `corpus_pair` section.
+
+## What lives where in scripts
+
+- `script/benchmark.rb` — Ruby-only throughput benchmark.
+- `script/memory.rb` — Ruby-only memory profile.
+- `script/generate_fixtures.rb` — produces `spec/fixtures/*.json` for cross-runtime parity.
+- `script/cli_parity.sh` — Ruby ↔ Go CLI diff.
+- `script/bench_compare.sh` — Ruby vs Go CLI wall-time comparison.
+- `script/bench_storage.sh` — JSON vs SQLite backend timing (single-process, incremental, concurrent).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    iriq (0.1.0)
+    iriq (0.2.0)
 
 GEM
   remote: https://rubygems.org/
@@ -19,6 +19,7 @@ GEM
       prism (>= 1.3.0)
       rdoc (>= 4.0.0)
       reline (>= 0.4.2)
+    mini_portile2 (2.8.9)
     pp (0.6.3)
       prettyprint
     prettyprint (0.2.0)
@@ -53,6 +54,8 @@ GEM
       simplecov_json_formatter (~> 0.1)
     simplecov-html (0.13.2)
     simplecov_json_formatter (0.1.4)
+    sqlite3 (2.9.4)
+      mini_portile2 (~> 2.8.0)
     stringio (3.2.0)
     tsort (0.2.0)
 
@@ -65,6 +68,7 @@ DEPENDENCIES
   rspec (>= 3.10)
   rspec-debugging
   simplecov (>= 0.22)
+  sqlite3 (>= 1.6)
 
 CHECKSUMS
   date (3.5.1) sha256=750d06384d7b9c15d562c76291407d89e368dda4d4fff957eb94962d325a0dc0
@@ -74,7 +78,8 @@ CHECKSUMS
   erb (6.0.4) sha256=38e3803694be357fe2bfe312487c74beaf9fb4e5beb3e22498952fe1645b95d9
   io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc
   irb (1.17.0) sha256=168c4ddb93d8a361a045c41d92b2952c7a118fa73f23fe14e55609eb7a863aae
-  iriq (0.1.0)
+  iriq (0.2.0)
+  mini_portile2 (2.8.9) sha256=0cd7c7f824e010c072e33f68bc02d85a00aeb6fce05bb4819c03dfd3c140c289
   pp (0.6.3) sha256=2951d514450b93ccfeb1df7d021cae0da16e0a7f95ee1e2273719669d0ab9df6
   prettyprint (0.2.0) sha256=2bc9e15581a94742064a3cc8b0fb9d45aae3d03a1baa6ef80922627a0766f193
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
@@ -90,6 +95,7 @@ CHECKSUMS
   simplecov (0.22.0) sha256=fe2622c7834ff23b98066bb0a854284b2729a569ac659f82621fc22ef36213a5
   simplecov-html (0.13.2) sha256=bd0b8e54e7c2d7685927e8d6286466359b6f16b18cb0df47b508e8d73c777246
   simplecov_json_formatter (0.1.4) sha256=529418fbe8de1713ac2b2d612aa3daa56d316975d307244399fa4838c601b428
+  sqlite3 (2.9.4) sha256=6161c5b9c17886b289558e6c8082b28a22a814736d2433c9a67f4c6bfcde5c97
   stringio (3.2.0) sha256=c37cb2e58b4ffbd33fe5cd948c05934af997b36e0b6ca6fdf43afa234cf222e1
   tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f
 

data/Makefile

@@ -0,0 +1,56 @@
+# Iriq Go binary — build/install/clean/uninstall helpers.
+#
+#   make           - same as `make help`
+#   make build     - build into ./bin/iriq
+#   make install   - go install into $GOBIN (defaults to $GOPATH/bin)
+#   make test      - go test ./...
+#   make clean     - remove ./bin/
+#   make uninstall - remove the binary from $GOBIN
+#
+# Ruby gem build/install is handled by Bundler/RubyGems; see CLAUDE.md.
+
+GO        ?= go
+BIN_DIR   := bin
+BIN       := $(BIN_DIR)/iriq
+PKG       := ./cmd/iriq
+
+# Resolve $GOBIN, falling back to $GOPATH/bin (Go's default install location).
+GOBIN     := $(shell $(GO) env GOBIN)
+ifeq ($(GOBIN),)
+GOBIN     := $(shell $(GO) env GOPATH)/bin
+endif
+INSTALLED := $(GOBIN)/iriq
+
+.DEFAULT_GOAL := help
+.PHONY: help build install test clean uninstall
+
+help:
+	@echo "Iriq Go targets:"
+	@echo "  make build       build into $(BIN)"
+	@echo "  make install     go install into $(GOBIN)"
+	@echo "  make test        run go test ./..."
+	@echo "  make clean       remove $(BIN_DIR)/"
+	@echo "  make uninstall   remove $(INSTALLED)"
+
+build:
+	@mkdir -p $(BIN_DIR)
+	$(GO) build -o $(BIN) $(PKG)
+	@echo "built $(BIN)"
+
+install:
+	$(GO) install $(PKG)
+	@echo "installed $(INSTALLED)"
+
+test:
+	$(GO) test ./...
+
+clean:
+	rm -rf $(BIN_DIR)
+	@echo "removed $(BIN_DIR)/"
+
+uninstall:
+	@if [ -f "$(INSTALLED)" ]; then \
+		rm "$(INSTALLED)" && echo "removed $(INSTALLED)"; \
+	else \
+		echo "not installed at $(INSTALLED)"; \
+	fi

data/README.md

@@ -3,19 +3,70 @@ Iriq
 ![Gem](https://img.shields.io/gem/dt/iriq?style=plastic)
 [![codecov](https://codecov.io/gh/dpep/iriq/branch/main/graph/badge.svg)](https://codecov.io/gh/dpep/iriq)
 
-Semantic IRI / URI / URL / URN normalization and clustering for Ruby.
+IRI extraction, normalization, and clustering.
 
-Iriq parses resource identifiers, normalizes them into canonical IRI-like
-forms, classifies path and query components, clusters similar identifiers,
-and explains which parts are stable vs. unique.
+Iriq pulls IRIs out of free text, parses them, normalizes them into
+canonical shape-aware forms, classifies their path and query components,
+and clusters similar identifiers — surfacing what's stable vs. unique.
+
+Ships as both a **command-line tool** (`iriq`) and a **library** (Ruby and
+Go — same behavior, enforced by parity tests).
+
+## Install
+
+The CLI is available three ways. Pick whichever fits your workflow:
+
+```sh
+# Homebrew (recommended)
+brew install dpep/tools/iriq
+
+# RubyGems — installs the CLI shim and the library
+gem install iriq
+
+# Go — installs the CLI binary into $GOBIN
+go install github.com/dpep/iriq/cmd/iriq@latest
+```
+
+For library use, depend on whichever runtime you're working in:
 
 ```ruby
-require "iriq"
+# Gemfile
+gem "iriq"
+```
+
+```go
+import "github.com/dpep/iriq"
+```
+
+## CLI quick start
+
+```
+$ iriq https://foo.com/users/123
+# parse
+original:      https://foo.com/users/123
+kind:          url
+scheme:        https
+host:          foo.com
+path_segments: ["users", "123"]
+canonical:     https://foo.com/users/123
+
+# normalize
+https://foo.com/users/{user_id}
+
+$ iriq -n https://foo.com/users/123
+https://foo.com/users/{user_id}
+
+$ cat access.log | iriq             # extract → URL list (or clusters at scale)
+$ cat access.log | iriq --stats     # rolling aggregates
+$ iriq ./access.log -n              # file auto-detected → normalize each found URL
 ```
 
-## Quick start
+Full CLI reference is below under [CLI](#cli).
+
+## Library quick start
 
 ```ruby
+# Ruby
 iri = Iriq.parse("https://foo.com/users/123")
 iri.scheme         # => "https"
 iri.host           # => "foo.com"
@@ -34,6 +85,22 @@ Iriq.explain("https://foo.com/users/123/orders/456")
 #    ]
 ```
 
+```go
+// Go (same surface)
+iri, _ := iriq.Parse("https://foo.com/users/123")
+iri.Scheme         // "https"
+iri.Host           // "foo.com"
+iri.PathSegments   // []string{"users", "123"}
+iri.Canonical()    // "https://foo.com/users/123"
+
+norm, _ := iriq.Normalize("https://foo.com/users/123")
+// "https://foo.com/users/{user_id}"
+```
+
+The Ruby gem is the reference implementation; Go mirrors its API and is
+kept in sync via JSON fixtures plus a CLI parity harness. See
+[CLAUDE.md](CLAUDE.md) for the dev process.
+
 Pass `hints: false` to `Iriq.normalize` (or `PathShape`) for mechanical
 placeholders (`{integer_id}` instead of `{user_id}`).
 
@@ -277,18 +344,31 @@ $ cat README.md | iriq cluster                # force cluster view
 $ cat README.md | iriq --corpus c.json        # persist into a corpus
 ```
 
-`--corpus PATH` makes the corpus survive across invocations (atomic JSON
-file). Once it has data, `-n` becomes corpus-informed:
+`--corpus PATH` makes the corpus survive across invocations. The file
+extension picks the storage backend:
+
+- `.json` — a single atomically-written JSON file (default). Best for small
+  corpora and when you want the data human-readable.
+- `.db` / `.sqlite` / `.sqlite3` — a SQLite database with WAL journaling.
+  Each observation is an incremental UPSERT, so multiple `iriq --corpus`
+  processes can write concurrently without clobbering each other, and the
+  cost of opening doesn't scale with corpus size.
+
+Once the corpus has data, `-n` becomes corpus-informed:
 
 ```
 $ for n in alice bob carol dave erin frank gina hank ivan jane; do
-    iriq --corpus c.json https://foo.com/users/$n/profile >/dev/null
+    iriq --corpus c.db https://foo.com/users/$n/profile >/dev/null
   done
 
-$ iriq -n --corpus c.json https://foo.com/users/zoe/profile
+$ iriq -n --corpus c.db https://foo.com/users/zoe/profile
 https://foo.com/users/{user}/profile         # mechanical would keep "zoe"
 ```
 
+Library: `Iriq::Corpus.open("c.db")` (or `iriq.OpenCorpus("c.db")` in Go)
+dispatches on the same extension rules. `corpus.save("export.json")`
+exports any backend as JSON.
+
 Flags:
 
 | Flag                | Effect                                                  |
@@ -298,7 +378,7 @@ Flags:
 | `-j, --json`        | Emit JSON                                               |
 | `-N, --no-hints`    | Use `{integer_id}` etc. instead of `{user_id}`          |
 | `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
-| `--corpus PATH`     | Load/create a JSON corpus at PATH; observe and save     |
+| `--corpus PATH`     | Load/create a corpus at PATH (`.json` or `.db`/`.sqlite`/`.sqlite3`) |
 | `--stats`           | Print rolling aggregates                                |
 | `-V, --version`     | Print version                                           |
 
@@ -352,6 +432,25 @@ For richer IRI handling, see `addressable`. Iriq's focus is the analysis
 side: classification, normalization, and clustering — not a complete URL
 implementation.
 
+----
+## Go port
+
+A Go implementation lives under [`go/`](go/) — same public surface, same
+behavior, ~10× faster CLI on extraction-heavy workloads. The Ruby gem is
+the reference; the Go port stays in sync via golden JSON fixtures
+(`spec/fixtures/`) and a CLI parity harness (`script/cli_parity.sh`), both
+checked in CI.
+
+```go
+import "github.com/dpep/iriq/go/iriq"
+
+iri, _ := iriq.Parse("https://foo.com/users/123")
+norm, _ := iriq.Normalize("https://foo.com/users/123")
+// "https://foo.com/users/{user_id}"
+```
+
+See [`go/README.md`](go/README.md) for the full API table and porting workflow.
+
 ----
 ## Contributing
 
@@ -360,6 +459,8 @@ Yes please  :)
 1. Fork it
 1. Create your feature branch (`git checkout -b my-feature`)
 1. Ensure the tests pass (`bundle exec rspec`)
+1. If you changed library behavior, port the change to Go (or open an
+   issue) and regenerate fixtures: `bundle exec ruby script/generate_fixtures.rb`
 1. Commit your changes (`git commit -am 'awesome new feature'`)
 1. Push your branch (`git push origin my-feature`)
 1. Create a Pull Request

data/iriq.gemspec

@@ -4,13 +4,13 @@ Gem::Specification.new do |s|
   s.name        = "iriq"
   s.version     = Iriq::VERSION
   s.authors     = ["Daniel Pepper"]
-  s.description = "Semantic IRI/URI/URL/URN parsing, normalization, classification, and clustering."
-  s.files       = `git ls-files * ':!:spec'`.split("\n")
+  s.description = "IRI extraction, normalization, and clustering."
+  s.files       = `git ls-files * ':!:spec' ':!:script' ':!:cmd' ':!:bin' ':!:*.go' ':!:go.mod' ':!:go.sum'`.split("\n")
   s.bindir      = "exe"
   s.executables = ["iriq"]
   s.homepage    = "https://github.com/dpep/iriq"
   s.license     = "MIT"
-  s.summary     = "Semantic IRI normalization and clustering."
+  s.summary     = "IRI extraction, normalization, and clustering."
 
   s.required_ruby_version = ">= 3.2"
 
@@ -18,4 +18,5 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'rspec', '>= 3.10'
   s.add_development_dependency 'rspec-debugging'
   s.add_development_dependency 'simplecov', '>= 0.22'
+  s.add_development_dependency 'sqlite3', '>= 1.6'
 end

data/lib/iriq/cli.rb

@@ -150,9 +150,7 @@ module Iriq
     end
 
     def load_corpus(path)
-      return Corpus.load(path) if File.exist?(path)
-
-      Corpus.new
+      Corpus.open(path)
     end
 
     def print_usage(io, code)
@@ -193,7 +191,7 @@ module Iriq
     def cmd_batch(args, opts, corpus, explicit_cluster: false)
       corpus ||= Corpus.new
       iris = extract_text(read_text(args.first), opts)
-      iris.each { |iri| corpus.observe(iri) }
+      corpus.batch { iris.each { |iri| corpus.observe(iri) } }
 
       if opts[:sections].any?
         emit_per_iri_sections(iris, opts)
@@ -298,6 +296,9 @@ module Iriq
       end
     end
 
+    # Compact identifier hash for parse output (both JSON and human). Drops
+    # nil values and empty collections so URN dumps don't carry empty
+    # host/path/query slots, and URL dumps don't include null fragment/nss.
     def identifier_hash(iri)
       {
         original:      iri.original,
@@ -310,7 +311,7 @@ module Iriq
         fragment:      iri.fragment,
         nss:           iri.nss,
         canonical:     iri.canonical,
-      }
+      }.reject { |_, v| v.nil? || (v.respond_to?(:empty?) && v.empty?) }
     end
 
     def emit_sections(data, sections)

data/lib/iriq/cluster.rb

@@ -77,5 +77,29 @@ module Iriq
       cluster.instance_variable_set(:@segment_counts, h["segment_counts"].map { |sub| Hash.new(0).merge(sub) })
       cluster
     end
+
+    # Shared cluster-key derivation. Returns [key, host, scheme, shape] —
+    # callers that already have a hinted shape can pass it in to skip the
+    # recomputation; URN inputs ignore the override and always derive their
+    # own shape from the NSS value.
+    def self.key_for(iri, classifier:, shape: nil)
+      if iri.urn?
+        ns, value = (iri.nss || "").split(":", 2)
+        derived = value ? urn_value_shape(ns, value, classifier) : nil
+        key     = "urn:#{ns}:#{derived}"
+        [key, nil, "urn", key]
+      else
+        shape ||= PathShape.new(classifier: classifier).for(iri.path_segments)
+        key = "#{iri.scheme}://#{iri.host}#{shape}"
+        [key, iri.host, iri.scheme, shape]
+      end
+    end
+
+    def self.urn_value_shape(ns, value, classifier)
+      entry = SegmentHints.derive([ns, value], classifier).last
+      return entry[:value] unless entry[:variable]
+
+      "{#{entry[:hint] || entry[:type]}}"
+    end
   end
 end

data/lib/iriq/clusterer.rb

@@ -3,31 +3,28 @@ module Iriq
   # `clusters` to read out the groups. `explain` annotates a single identifier
   # against the cluster it would fall into, including which positions are
   # stable across all observed members.
+  #
+  # Implemented as a thin wrapper over Storage::Memory — the same code path
+  # Corpus uses for the cluster portion of its state, so there's only one
+  # place that knows how clusters get stored.
   class Clusterer
     def initialize(classifier: SegmentClassifier::DEFAULT)
       @classifier = classifier
-      @clusters   = {}
+      @storage    = Storage::Memory.new(classifier: classifier)
     end
 
     def add(input, shape: nil)
       iri = coerce(input)
-      key, host, scheme, shape = cluster_key(iri, shape: shape)
-      cluster = @clusters[key] ||= Cluster.new(
-        key:    key,
-        host:   host,
-        scheme: scheme,
-        shape:  shape,
-      )
-      cluster.add(iri)
-      cluster
+      key, host, scheme, derived = Cluster.key_for(iri, classifier: @classifier, shape: shape)
+      @storage.add_to_cluster(key, host, scheme, derived, iri)
     end
 
     def clusters
-      @clusters.values
+      @storage.clusters
     end
 
     def size
-      @clusters.size
+      @storage.cluster_size
     end
 
     # Returns a per-segment explanation for the input, merging classifier
@@ -36,8 +33,8 @@ module Iriq
     # would otherwise call them variable).
     def explain(input)
       iri = coerce(input)
-      key, * = cluster_key(iri)
-      cluster = @clusters[key]
+      key, * = Cluster.key_for(iri, classifier: @classifier)
+      cluster = clusters.find { |c| c.key == key }
       stats   = cluster ? cluster.segment_stats : []
       hinted  = SegmentHints.derive(iri.path_segments, @classifier)
 
@@ -50,43 +47,21 @@ module Iriq
       end
     end
 
-    private
-
-    def coerce(input)
-      input.is_a?(Identifier) ? input : Parser.parse(input)
-    end
-
-    def cluster_key(iri, shape: nil)
-      if iri.urn?
-        ns, value = (iri.nss || "").split(":", 2)
-        shape = value ? urn_value_shape(ns, value) : nil
-        key   = "urn:#{ns}:#{shape}"
-        [key, nil, "urn", key]
-      else
-        shape ||= PathShape.new(classifier: @classifier).for(iri.path_segments)
-        key   = "#{iri.scheme}://#{iri.host}#{shape}"
-        [key, iri.host, iri.scheme, shape]
-      end
-    end
-
-    def urn_value_shape(ns, value)
-      entry = SegmentHints.derive([ns, value], @classifier).last
-      return entry[:value] unless entry[:variable]
-
-      "{#{entry[:hint] || entry[:type]}}"
-    end
-
-    public
-
     def dump
-      { "clusters" => @clusters.transform_values(&:dump) }
+      { "clusters" => clusters.each_with_object({}) { |c, h| h[c.key] = c.dump } }
     end
 
     def self.from_dump(h, classifier: SegmentClassifier::DEFAULT)
       c = new(classifier: classifier)
       restored = h["clusters"].transform_values { |cdump| Cluster.from_dump(cdump) }
-      c.instance_variable_set(:@clusters, restored)
+      c.instance_variable_get(:@storage).instance_variable_set(:@clusters, restored)
       c
     end
+
+    private
+
+    def coerce(input)
+      input.is_a?(Identifier) ? input : Parser.parse(input)
+    end
   end
 end