iriq 0.0.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d1f6ebe248ed57192adc8f5c9600dfe14d9d0c85160dd5d6588d7fbfd7995e72
-  data.tar.gz: d80356a646effb078ebe78b62417534746ded550fd4145faa53998aa35866bde
+  metadata.gz: e629988f23137ecb0c0f4737e246f65a949fa2843f4bd16d244566ca76dc37ed
+  data.tar.gz: c36cae38205a6a6f63a8a38e40849c922bbb6cf7046e9599aaaefc37fb443303
 SHA512:
-  metadata.gz: d355eef90433cef5cd807d9c68d6259ad49d79dd87d842770fded377a535ed39e4d24228594b24ce96015f8c9e7246bdc96d7bcfa58f3cdf3350bb389bff211f
-  data.tar.gz: b0e1ddf1c6bcebadbe2578fe0760076a8401fd4fc2cd9a5f345b651371cf2fb47ebaf302565df61b05a389e0253f3febdbccc4e2389cf7d2e810abc8e4156a40
+  metadata.gz: 75d0329756d16dd7b9c8e5ca7b8ba447aa51c4d3c34a1ec31549bb6e6725338bfbdb2ee4badb18c52342c8ae0ece5f005e9288c617fa5e140d7cfff400274561
+  data.tar.gz: cb670e665a2e67feeb5f1bcad157a27e15da1aa61efc53e4ae66388a67ee498ef7caa1605be85d27e087fda44c399a9f93b6701242d1a22dec2653b667918c6d

data/CHANGELOG.md

@@ -1,2 +1,27 @@
+###  0.2.0  (2026-05-25)
+- Corpus storage backends: JSON (default) and SQLite, dispatched by file extension
+- Go: `iriq.OpenCorpus(path)`; Ruby: `Iriq::Corpus.open(path)`
+- SQLite backend: incremental UPSERTs, WAL mode, concurrent-safe via busy_timeout + BEGIN IMMEDIATE; checkpoints on close so the WAL sidecar doesn't grow unbounded
+- Batch mode: `corpus.batch { ... }` (Ruby) / `corpus.Batch(fn)` (Go) wraps many observations in one transaction
+- Clusterer now wraps the in-memory Storage backend; only one cluster code path
+- script/bench_storage.sh — JSON vs SQLite timing across single-process, incremental, and concurrent workloads
+- **Breaking (Go)**: `Corpus.HostCounts` / `PathLengthCounts` / `RawShapeCounts` / `FingerprintCounts` are methods now, not fields
+
+###  0.1.0  (2026-05-24)
+- CLI: auto-detect file argument, retire --extract flag
+- CLI: section flags work in pipe mode + clean up help text
+- script/memory.rb — track retained memory + cache footprints
+- Perf: classifier + inflector memoization, singleton classifier, combined extractor regex
+- Perf: derive SegmentHints once per Corpus.observe (~2x faster)
+- script/benchmark.rb — measure the main hot paths
+- README: replace fabricated example numbers with real fixture output
+- Pipe mode: extraction by default, auto-switch to cluster view at scale
+- Iriq::Extractor — pull IRIs out of free text
+- E2E spec: pipe IriGenerator stream through real iriq binary
+- IriGenerator fixture + popular-outlier heuristic
+- CLI --corpus persistence, pipe batch mode, --stats, E2E specs
+- Streaming Corpus with rolling stats and learning
+- RESTful hints, flag-based CLI, swappable inflector
+
 ###  0.0.1  (2026-05-24)
  - prototype

data/CLAUDE.md

@@ -0,0 +1,121 @@
+# Iriq development conventions
+
+## Repo layout — Ruby and Go intermixed at the root
+
+We chose to mix Ruby and Go at the repo root rather than nest the Go module
+under `/go/`. The signal is "both implementations are peers, not one-is-primary."
+
+```
+iriq/
+  lib/ exe/ spec/      ← Ruby gem (library, CLI, specs)
+  iriq.gemspec
+  Gemfile
+
+  go.mod               ← module github.com/dpep/iriq
+  *.go                 ← Go package `iriq` at the root
+  cmd/iriq/            ← Go CLI binary
+  bin/                 ← built Go binary (gitignored)
+
+  script/              ← shared dev scripts (fixture gen, parity, benches)
+  spec/fixtures/       ← golden JSON shared by Ruby specs + Go tests
+  .github/workflows/   ← Ruby CI, Go CI, parity CI
+```
+
+Trade-offs of this layout:
+
+- Clean import path: `github.com/dpep/iriq` (no `/go/` artifact in consumers' code).
+- One version tag (`vX.Y.Z`) serves both runtimes — Ruby's gemspec and Go's
+  module use the same tag stream.
+- Root `ls` is busier (~15 `.go` files next to Ruby ones), accepted in exchange.
+- The gemspec explicitly excludes Go files so `gem build` doesn't ship them:
+  `git ls-files * ':!:spec' ':!:script' ':!:cmd' ':!:bin' ':!:*.go' ':!:go.mod' ':!:go.sum'`.
+
+## Building
+
+```sh
+# Ruby gem
+bundle install
+bundle exec exe/iriq --help     # runs the CLI from source
+
+# Go binary — convenience targets in the Makefile
+make build                      # → ./bin/iriq
+make install                    # go install into $GOBIN
+make uninstall                  # remove from $GOBIN
+make clean                      # remove ./bin/
+make test                       # go test ./...
+
+# Both via Homebrew
+brew install dpep/tools/iriq    # uses the Ruby gem under the hood
+```
+
+## Keeping Ruby and Go in sync
+
+The Ruby gem is the **reference implementation**. Go mirrors its public API
+and behavior. Two layers of parity testing keep them aligned:
+
+1. **Golden JSON fixtures** (`spec/fixtures/*.json`)
+   Generated by `script/generate_fixtures.rb` from the Ruby implementation
+   over a curated set of inputs. Go's `fixtures_test.go` loads each file
+   and asserts the same outputs from the Go side.
+
+2. **CLI parity harness** (`script/cli_parity.sh`)
+   Runs the same input through `bundle exec exe/iriq` and the Go binary and
+   diffs stdout. Lives in CI as the `Ruby ↔ Go parity` job.
+
+When changing behavior:
+
+1. Update the Ruby code + specs first.
+2. Regenerate fixtures: `bundle exec ruby script/generate_fixtures.rb`.
+3. Port the change to Go.
+4. `go test ./...` (uses the updated fixtures).
+5. `script/cli_parity.sh` should pass.
+6. Commit fixtures with the change — CI will fail if they're stale.
+
+## Tests
+
+```sh
+bundle exec rspec        # Ruby suite (305+ examples)
+go test ./...            # Go suite (native + fixture parity tests)
+script/cli_parity.sh     # CLI parity (13+ scenarios)
+```
+
+## Releases
+
+- One version tag covers both runtimes — bump `lib/iriq/version.rb` (and
+  optionally a matching constant on the Go side if we add one), tag `vX.Y.Z`,
+  push.
+- `gem push iriq-X.Y.Z.gem` to publish to RubyGems.
+- Update `Formula/iriq.rb` in the homebrew-tools tap to the new version.
+- Go consumers pick up the tag automatically via `go get @vX.Y.Z`.
+
+## Corpus storage backends
+
+The `Corpus` class delegates state to a `Storage` backend; three backends ship:
+
+- **Memory** — default, in-process only.
+- **JSON** — Memory wrapped with atomic load/save against a JSON file
+  (`.json` by default). Same shape both runtimes have always written.
+- **SQLite** — incremental UPSERTs against a `.db` / `.sqlite` / `.sqlite3`
+  file with WAL journaling. Supports concurrent observers and avoids
+  loading the whole corpus into memory.
+
+`Corpus.open(path)` (Ruby) / `iriq.OpenCorpus(path)` (Go) picks the backend
+by file extension. `corpus.save(other_path)` exports as JSON regardless of
+the live backend; `corpus.save(same_path)` is idempotent (no clobbering a
+SQLite file with JSON, etc.).
+
+The Ruby `sqlite3` gem is loaded lazily (only when a `.db` path is opened),
+keeping the iriq install footprint minimal for users that stick with JSON.
+On the Go side we use `modernc.org/sqlite` (pure Go — no cgo).
+
+When adding a new backend, replicate the contract in both languages and
+add a parity scenario in `script/cli_parity.sh`'s `corpus_pair` section.
+
+## What lives where in scripts
+
+- `script/benchmark.rb` — Ruby-only throughput benchmark.
+- `script/memory.rb` — Ruby-only memory profile.
+- `script/generate_fixtures.rb` — produces `spec/fixtures/*.json` for cross-runtime parity.
+- `script/cli_parity.sh` — Ruby ↔ Go CLI diff.
+- `script/bench_compare.sh` — Ruby vs Go CLI wall-time comparison.
+- `script/bench_storage.sh` — JSON vs SQLite backend timing (single-process, incremental, concurrent).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    iriq (0.0.1)
+    iriq (0.2.0)
 
 GEM
   remote: https://rubygems.org/
@@ -19,6 +19,7 @@ GEM
       prism (>= 1.3.0)
       rdoc (>= 4.0.0)
       reline (>= 0.4.2)
+    mini_portile2 (2.8.9)
     pp (0.6.3)
       prettyprint
     prettyprint (0.2.0)
@@ -53,6 +54,8 @@ GEM
       simplecov_json_formatter (~> 0.1)
     simplecov-html (0.13.2)
     simplecov_json_formatter (0.1.4)
+    sqlite3 (2.9.4)
+      mini_portile2 (~> 2.8.0)
     stringio (3.2.0)
     tsort (0.2.0)
 
@@ -65,6 +68,7 @@ DEPENDENCIES
   rspec (>= 3.10)
   rspec-debugging
   simplecov (>= 0.22)
+  sqlite3 (>= 1.6)
 
 CHECKSUMS
   date (3.5.1) sha256=750d06384d7b9c15d562c76291407d89e368dda4d4fff957eb94962d325a0dc0
@@ -74,7 +78,8 @@ CHECKSUMS
   erb (6.0.4) sha256=38e3803694be357fe2bfe312487c74beaf9fb4e5beb3e22498952fe1645b95d9
   io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc
   irb (1.17.0) sha256=168c4ddb93d8a361a045c41d92b2952c7a118fa73f23fe14e55609eb7a863aae
-  iriq (0.0.1)
+  iriq (0.2.0)
+  mini_portile2 (2.8.9) sha256=0cd7c7f824e010c072e33f68bc02d85a00aeb6fce05bb4819c03dfd3c140c289
   pp (0.6.3) sha256=2951d514450b93ccfeb1df7d021cae0da16e0a7f95ee1e2273719669d0ab9df6
   prettyprint (0.2.0) sha256=2bc9e15581a94742064a3cc8b0fb9d45aae3d03a1baa6ef80922627a0766f193
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
@@ -90,6 +95,7 @@ CHECKSUMS
   simplecov (0.22.0) sha256=fe2622c7834ff23b98066bb0a854284b2729a569ac659f82621fc22ef36213a5
   simplecov-html (0.13.2) sha256=bd0b8e54e7c2d7685927e8d6286466359b6f16b18cb0df47b508e8d73c777246
   simplecov_json_formatter (0.1.4) sha256=529418fbe8de1713ac2b2d612aa3daa56d316975d307244399fa4838c601b428
+  sqlite3 (2.9.4) sha256=6161c5b9c17886b289558e6c8082b28a22a814736d2433c9a67f4c6bfcde5c97
   stringio (3.2.0) sha256=c37cb2e58b4ffbd33fe5cd948c05934af997b36e0b6ca6fdf43afa234cf222e1
   tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f
 

data/Makefile

@@ -0,0 +1,56 @@
+# Iriq Go binary — build/install/clean/uninstall helpers.
+#
+#   make           - same as `make help`
+#   make build     - build into ./bin/iriq
+#   make install   - go install into $GOBIN (defaults to $GOPATH/bin)
+#   make test      - go test ./...
+#   make clean     - remove ./bin/
+#   make uninstall - remove the binary from $GOBIN
+#
+# Ruby gem build/install is handled by Bundler/RubyGems; see CLAUDE.md.
+
+GO        ?= go
+BIN_DIR   := bin
+BIN       := $(BIN_DIR)/iriq
+PKG       := ./cmd/iriq
+
+# Resolve $GOBIN, falling back to $GOPATH/bin (Go's default install location).
+GOBIN     := $(shell $(GO) env GOBIN)
+ifeq ($(GOBIN),)
+GOBIN     := $(shell $(GO) env GOPATH)/bin
+endif
+INSTALLED := $(GOBIN)/iriq
+
+.DEFAULT_GOAL := help
+.PHONY: help build install test clean uninstall
+
+help:
+	@echo "Iriq Go targets:"
+	@echo "  make build       build into $(BIN)"
+	@echo "  make install     go install into $(GOBIN)"
+	@echo "  make test        run go test ./..."
+	@echo "  make clean       remove $(BIN_DIR)/"
+	@echo "  make uninstall   remove $(INSTALLED)"
+
+build:
+	@mkdir -p $(BIN_DIR)
+	$(GO) build -o $(BIN) $(PKG)
+	@echo "built $(BIN)"
+
+install:
+	$(GO) install $(PKG)
+	@echo "installed $(INSTALLED)"
+
+test:
+	$(GO) test ./...
+
+clean:
+	rm -rf $(BIN_DIR)
+	@echo "removed $(BIN_DIR)/"
+
+uninstall:
+	@if [ -f "$(INSTALLED)" ]; then \
+		rm "$(INSTALLED)" && echo "removed $(INSTALLED)"; \
+	else \
+		echo "not installed at $(INSTALLED)"; \
+	fi