iriq 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d1f6ebe248ed57192adc8f5c9600dfe14d9d0c85160dd5d6588d7fbfd7995e72
-  data.tar.gz: d80356a646effb078ebe78b62417534746ded550fd4145faa53998aa35866bde
+  metadata.gz: 95d6bc09f7de65bcb4acc5db3ad68d1b83c326360d913c45aede66038a100461
+  data.tar.gz: ae58d39b77fce3041cc5561b575dea52706b38f4cfab11d4881cf2f644f6cc59
 SHA512:
-  metadata.gz: d355eef90433cef5cd807d9c68d6259ad49d79dd87d842770fded377a535ed39e4d24228594b24ce96015f8c9e7246bdc96d7bcfa58f3cdf3350bb389bff211f
-  data.tar.gz: b0e1ddf1c6bcebadbe2578fe0760076a8401fd4fc2cd9a5f345b651371cf2fb47ebaf302565df61b05a389e0253f3febdbccc4e2389cf7d2e810abc8e4156a40
+  metadata.gz: a8fa85f112d9766ff4e9e4ad60c1043ce9701fe42e6dbd01f70da39fa0dd554bff573f40c858d88ab1e1539de0c1f017609535cc57e8e4fcc6651f0d20697e60
+  data.tar.gz: 97c714e664874c08278a305b8beff470b68bd2b7e4f58977cc44cf2e3716e619d17011a33d77b8ea9356eec0715acb6c815252551edf64db16eec4599f816274

data/CHANGELOG.md

@@ -1,2 +1,18 @@
+###  0.1.0  (2026-05-24)
+- CLI: auto-detect file argument, retire --extract flag
+- CLI: section flags work in pipe mode + clean up help text
+- script/memory.rb — track retained memory + cache footprints
+- Perf: classifier + inflector memoization, singleton classifier, combined extractor regex
+- Perf: derive SegmentHints once per Corpus.observe (~2x faster)
+- script/benchmark.rb — measure the main hot paths
+- README: replace fabricated example numbers with real fixture output
+- Pipe mode: extraction by default, auto-switch to cluster view at scale
+- Iriq::Extractor — pull IRIs out of free text
+- E2E spec: pipe IriGenerator stream through real iriq binary
+- IriGenerator fixture + popular-outlier heuristic
+- CLI --corpus persistence, pipe batch mode, --stats, E2E specs
+- Streaming Corpus with rolling stats and learning
+- RESTful hints, flag-based CLI, swappable inflector
+
 ###  0.0.1  (2026-05-24)
  - prototype

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    iriq (0.0.1)
+    iriq (0.1.0)
 
 GEM
   remote: https://rubygems.org/
@@ -74,7 +74,7 @@ CHECKSUMS
   erb (6.0.4) sha256=38e3803694be357fe2bfe312487c74beaf9fb4e5beb3e22498952fe1645b95d9
   io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc
   irb (1.17.0) sha256=168c4ddb93d8a361a045c41d92b2952c7a118fa73f23fe14e55609eb7a863aae
-  iriq (0.0.1)
+  iriq (0.1.0)
   pp (0.6.3) sha256=2951d514450b93ccfeb1df7d021cae0da16e0a7f95ee1e2273719669d0ab9df6
   prettyprint (0.2.0) sha256=2bc9e15581a94742064a3cc8b0fb9d45aae3d03a1baa6ef80922627a0766f193
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85

data/README.md

@@ -23,17 +23,40 @@ iri.path_segments  # => ["users", "123"]
 iri.canonical      # => "https://foo.com/users/123"
 
 Iriq.normalize("https://foo.com/users/123")
-# => "https://foo.com/users/{integer_id}"
+# => "https://foo.com/users/{user_id}"
 
 Iriq.explain("https://foo.com/users/123/orders/456")
 # => [
-#      { value: "users",  type: :literal,    variable: false },
-#      { value: "123",    type: :integer_id, variable: true  },
-#      { value: "orders", type: :literal,    variable: false },
-#      { value: "456",    type: :integer_id, variable: true  },
+#      { value: "users",  type: :literal,    variable: false, hint: nil        },
+#      { value: "123",    type: :integer_id, variable: true,  hint: "user_id"  },
+#      { value: "orders", type: :literal,    variable: false, hint: nil        },
+#      { value: "456",    type: :integer_id, variable: true,  hint: "order_id" },
 #    ]
 ```
 
+Pass `hints: false` to `Iriq.normalize` (or `PathShape`) for mechanical
+placeholders (`{integer_id}` instead of `{user_id}`).
+
+## RESTful hints
+
+When a variable segment follows a literal one, Iriq derives a hint by
+singularizing the literal and suffixing `_id` (or `_uuid` for UUIDs). This is
+what produces `{user_id}` from `/users/123` and `{order_id}` from
+`/orders/456`. Singularization uses `Iriq::Inflector`, which delegates to a
+swappable adapter:
+
+```ruby
+# Default: ActiveSupport::Inflector if `active_support/inflector` is loadable,
+# otherwise a built-in adapter with rules adapted from ActiveSupport.
+
+Iriq::Inflector.singularize("categories")  # => "category"
+Iriq::Inflector.singularize("people")      # => "person"
+
+# Override:
+Iriq::Inflector.adapter = MyAdapter        # must respond to .singularize(String)
+Iriq::Inflector.reset_adapter!
+```
+
 ## Supported inputs
 
 | Input                                | Notes                                            |
@@ -69,7 +92,7 @@ clusterer.add("https://foo.com/users/456")
 clusterer.add("https://foo.com/users/789/orders/1")
 
 clusterer.clusters.map(&:shape)
-# => ["/users/{integer_id}", "/users/{integer_id}/orders/{integer_id}"]
+# => ["/users/{user_id}", "/users/{user_id}/orders/{order_id}"]
 
 clusterer.clusters.first.segment_stats
 # => [
@@ -79,8 +102,8 @@ clusterer.clusters.first.segment_stats
 
 clusterer.explain("https://foo.com/users/999")
 # => [
-#      { value: "users", type: :literal,    variable: false, stable: true  },
-#      { value: "999",   type: :integer_id, variable: true,  stable: false },
+#      { value: "users", type: :literal,    variable: false, hint: nil,       stable: true  },
+#      { value: "999",   type: :integer_id, variable: true,  hint: "user_id", stable: false },
 #    ]
 ```
 
@@ -89,6 +112,104 @@ a position the classifier *would* call variable but that is empirically
 constant across all members of the cluster will be reported with
 `stable: true, variable: false`.
 
+## Corpus (streaming + learning)
+
+For processing many identifiers — possibly an unbounded stream — use
+`Iriq::Corpus`. It maintains rolling aggregates and per-(host, prefix)
+frequency stats so classification improves as more data comes in.
+
+```ruby
+corpus = Iriq::Corpus.new
+
+iris.each do |iri|
+  obs = corpus.observe(iri)
+  obs.fingerprint   # deterministic shape: "https://foo.com/users/{user_id}"
+  obs.cluster       # the Iriq::Cluster this fell into
+  obs.explanation   # per-segment annotations with corpus-informed classification
+end
+
+corpus.host_counts          # { "foo.com" => 1234, "bar.com" => 7 }
+corpus.path_length_counts   # { 2 => 800, 3 => 434 }
+corpus.fingerprint_counts   # shape → count
+corpus.raw_shape_counts     # hint-free shape → count
+corpus.clusters             # Iriq::Cluster instances
+```
+
+### Deterministic vs. corpus-informed normalization
+
+```ruby
+Iriq.normalize("https://foo.com/users/me")
+# => "https://foo.com/users/me"   # mechanical: "me" is a literal
+
+corpus.normalize("https://foo.com/users/me")
+# => depends on what the corpus has seen
+```
+
+If many `/users/{integer_id}` paths flow in alongside a handful of
+`/users/me`, the cluster `/users/me` is preserved (mechanical clustering
+keeps literal routes distinct). If many *distinct literal handles*
+(`/users/alice`, `/users/bob`, `/users/carol`, ...) flow in, the corpus
+promotes that position to a `{user}` placeholder:
+
+```ruby
+%w[alice bob carol dave erin frank gina hank ivan jane].each do |name|
+  corpus.observe("https://foo.com/users/#{name}/profile")
+end
+
+corpus.normalize("https://foo.com/users/alice/profile")
+# => "https://foo.com/users/{user}/profile"
+```
+
+### Explainability
+
+Each row of `corpus.explain(...)` (and `observation.explanation`) carries a
+`classification:` symbol on top of the deterministic fields:
+
+| Classification              | Meaning                                              |
+| --------------------------- | ---------------------------------------------------- |
+| `:stable_literal`           | Literal value dominates this position                |
+| `:variable_identifier`      | Classifier said variable (uuid, integer, etc.)       |
+| `:rare_literal`             | Literal seen here, but not dominant                  |
+| `:corpus_inferred_variable` | Classifier said literal, but position has high entropy |
+| `:ambiguous`                | Insufficient signal — never seen, or mixed           |
+
+## Extracting IRIs from text
+
+`Iriq::Extractor` is what powers pipe-mode in the CLI. Picks up explicit-
+scheme URLs (`http`, `https`, `ftp`, `ws`, `wss`, `urn`) and `foo.com/path`-
+style scheme-less URLs (small TLD allow-list, required path). Trims trailing
+sentence punctuation iteratively and preserves balanced parens
+(`https://en.wikipedia.org/wiki/Ruby_(programming_language)` stays intact;
+`(see https://foo.com)` drops the outer paren).
+
+```ruby
+Iriq.extract("Visit https://foo.com today, also hit foo.com/users.")
+# => [#<Iriq::Identifier https://foo.com>,
+#     #<Iriq::Identifier https://foo.com/users>]
+
+# Disable scheme-less:
+Iriq::Extractor.new(scheme_less: false).extract("hit foo.com/users today")
+# => []
+```
+
+Known limitations (intentional):
+
+- Comma is a URL boundary, so query strings like `?q=37.7,-122.4` truncate.
+  Trade-off picked to keep CSV-shaped text working.
+- No HTML entity decoding (`&amp;` stays as-is).
+- Scheme-less mode skips bare hostnames without a path (too noisy in prose).
+
+### Memory bounds
+
+- Per-position `value_counts` is capped (`max_values_per_position`, default
+  1000) — once full, `total` keeps growing but only existing keys count up.
+- Cluster examples are capped at `Iriq::Cluster::MAX_EXAMPLES`.
+- No raw IRI strings are retained outside the bounded cluster examples.
+
+```ruby
+Iriq::Corpus.new(max_values_per_position: 200)
+```
+
 ## Object model
 
 | Class                       | Responsibility                                       |
@@ -96,51 +217,124 @@ constant across all members of the cluster will be reported with
 | `Iriq::Parser`              | String → `Identifier`                                |
 | `Iriq::Identifier`          | Structured fields + `canonical` reconstruction       |
 | `Iriq::SegmentClassifier`   | Single segment → type symbol                         |
-| `Iriq::PathShape`           | Segments → `/users/{integer_id}` route shape         |
+| `Iriq::PathShape`           | Segments → `/users/{user_id}` route shape            |
+| `Iriq::SegmentHints`        | Derives `user_id`-style hints from neighbors         |
+| `Iriq::Inflector`           | Singularization with swappable adapter (AS or built-in) |
 | `Iriq::Normalizer`          | Identifier → canonical, shape-aware string           |
-| `Iriq::Explanation`         | Per-segment `{value, type, variable}` annotations    |
+| `Iriq::Explanation`         | Per-segment `{value, type, variable, hint}` rows     |
 | `Iriq::Cluster`             | One host + shape group, with examples & stats        |
 | `Iriq::Clusterer`           | Many identifiers → `Cluster` set + explain          |
+| `Iriq::PositionStats`       | Capped value/type frequencies for one position       |
+| `Iriq::Observation`         | What `Corpus#observe` returns                        |
+| `Iriq::Corpus`              | Streaming observer with rolling aggregates + learning |
+| `Iriq::Extractor`           | Pulls IRIs out of free text (scheme-anchored)        |
 
 ## CLI
 
-Installing the gem also installs an `iriq` executable.
+Installing the gem installs an `iriq` executable. Two main modes:
+
+**Single input** — combined parse + normalize summary; trim with section
+flags (`-p`, `-n`).
 
 ```
-$ iriq parse https://foo.com/users/123
-original:      https://foo.com/users/123
+$ iriq foo.com/users/456
+# parse
+original:      foo.com/users/456
 kind:          url
 scheme:        https
 host:          foo.com
-path_segments: ["users", "123"]
-canonical:     https://foo.com/users/123
+path_segments: ["users", "456"]
+canonical:     https://foo.com/users/456
 
-$ iriq normalize foo.com/posts/2024-05-23/hello-world
-https://foo.com/posts/{date}/{slug}
+# normalize
+https://foo.com/users/{user_id}
 
-$ iriq explain https://foo.com/users/123/orders/456
-  literal      users
-* integer_id   123
-  literal      orders
-* integer_id   456
+$ iriq -n https://foo.com/users/123
+https://foo.com/users/{user_id}
+```
 
-$ iriq classify f47ac10b-58cc-4372-a567-0e02b2c3d479
-uuid
+**Piped stdin** — extraction runs by default. Output auto-switches: small
+inputs get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the
+cluster view via an ephemeral corpus. Section flags work too — emit one
+normalized URL / parsed record per extracted IRI.
 
-$ cat urls.txt | iriq cluster
-[2] foo.com  /users/{integer_id}
-    https://foo.com/users/1
-    https://foo.com/users/2
-[1] foo.com  /posts/{slug}/edit
-    https://foo.com/posts/abc-123/edit
+```
+$ cat short.txt | iriq
+[2] https://github.com/dpep/iriq
+[1] https://foo.com/users
+
+$ cat short.txt | iriq -n                     # normalized URL per line
+https://github.com/dpep/iriq
+https://foo.com/users
+
+$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
+[190] docs.example.com  /users/{user_id}
+[186] app.example.com   /users/{user_id}
+...
+
+$ cat README.md | iriq --stats                # rolling aggregates
+$ cat README.md | iriq cluster                # force cluster view
+$ cat README.md | iriq --corpus c.json        # persist into a corpus
 ```
 
-Add `--json` to any command for machine-readable output. `iriq cluster` reads
-identifiers (one per line) from a file argument or stdin; lines that fail to
-parse are skipped with a warning on stderr.
+`--corpus PATH` makes the corpus survive across invocations (atomic JSON
+file). Once it has data, `-n` becomes corpus-informed:
+
+```
+$ for n in alice bob carol dave erin frank gina hank ivan jane; do
+    iriq --corpus c.json https://foo.com/users/$n/profile >/dev/null
+  done
+
+$ iriq -n --corpus c.json https://foo.com/users/zoe/profile
+https://foo.com/users/{user}/profile         # mechanical would keep "zoe"
+```
+
+Flags:
+
+| Flag                | Effect                                                  |
+| ------------------- | ------------------------------------------------------- |
+| `-p, --parse`       | Show parsed fields                                      |
+| `-n, --normalize`   | Show the shape-normalized form                          |
+| `-j, --json`        | Emit JSON                                               |
+| `-N, --no-hints`    | Use `{integer_id}` etc. instead of `{user_id}`          |
+| `--no-scheme-less`  | Skip `foo.com/path`-style extraction (explicit-scheme only) |
+| `--corpus PATH`     | Load/create a JSON corpus at PATH; observe and save     |
+| `--stats`           | Print rolling aggregates                                |
+| `-V, --version`     | Print version                                           |
+
+A positional argument that doesn't parse as an IRI but IS an existing
+file is read and extracted from automatically — `iriq ./access.log` and
+`iriq /var/log/foo.log` Just Work. (Bare filenames like `README.md`
+may still parse as a URL; pipe with `cat` to disambiguate.)
 
 Exit codes: `0` success, `1` usage error, `2` parse error.
 
+## Performance
+
+Measured on the deterministic `IriGenerator` fixture (Ruby 3.4.9, single
+thread):
+
+| Operation                | Throughput   |
+| ------------------------ | ------------ |
+| `Iriq.parse`             | ~260k URLs/s |
+| `Iriq.normalize`         | ~148k URLs/s |
+| `Iriq.explain`           | ~205k URLs/s |
+| `Iriq.extract` (prose)   | ~9.6 MB/s    |
+| `Corpus#observe`         | ~80k URLs/s  |
+| Corpus save/load (10k)   | ~135 ms      |
+
+Linear scaling holds through 100k observations; per-observation retained
+memory amortizes to ~100 bytes at that scale. Memoization caches are
+bounded by `CACHE_MAX = 10_000` (cleared when full) — overhead is a few
+hundred KB regardless of corpus size.
+
+Re-run anytime with:
+
+```
+bundle exec script/benchmark.rb       # throughput
+bundle exec script/memory.rb          # retained memory + cache footprints
+```
+
 ## Limitations (intentional)
 
 This is an MVP. Iriq does **not**: