purplelight 0.1.13 → 0.1.17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c500768ed34e247a92be3979f56c1036314b51003cb1ff7953562f1ca8278677
-  data.tar.gz: f12b305e35b201192e219aec23a0c66c13a7b9eb262c038dbac9d1e2a3c77bc5
+  metadata.gz: 006653025fe8b2a5afa958c08992076f171550c7e1a1da4d57442df76d0e659d
+  data.tar.gz: 2476b1a5c4f078b539cd83c552e7ec5938277943e46ae6412b4c04df3830a042
 SHA512:
-  metadata.gz: 0f6959b02fb695a4c28802ec555d7b6ce6e7020d348024d2df8381efbb022632916d2915c688131cf79aa042f05aa3e2ec8780ad866fdb774b45b51f53093d6b
-  data.tar.gz: aa2885e120f57bd492c1a98aee91f52b7c83bee15492ce34a0b6d3dae985e0355ea77489e69dfde2f27fb5b51974263b7f7358a51103697d8222a03cd98a4185
+  metadata.gz: d31c3447814135d0f723f6bf35e955618932971f28bdcdd15a3831daee2c8c2d6a22ad0dae0bb8af1572f0e570fdbb71cc606f31241a83ee741742a783f3a673
+  data.tar.gz: 30bd7c0a089f57b6f8957d93b538b5e0f698e6440b7180dea86e3922cee58e639cace618782d888284883bcd5930456bd6cda9e270b338d54caea538b0804943

data/PERFORMANCE_NOTES.md

@@ -0,0 +1,53 @@
+# Purplelight upgrade and performance notes
+
+Updated: 2026-07-21
+
+## Completed
+
+- Pinned managed development Ruby 4.0.2; no Homebrew Ruby is used.
+- Updated all Ruby 3.2-compatible dependencies: MongoDB Ruby driver 2.24.1, BSON 5.2.0, zstd-ruby 2.0.6, red-arrow/red-parquet 25.0.0, Rake 13.4.2, RSpec 3.13.2, RuboCop 1.88.2, SimpleCov 1.0.2, StackProf 0.2.28, and current transitives. `parallel` remains on 1.28 because 2.x requires Ruby 3.3; `diff-lcs` 2.0 remains blocked by RSpec's `< 2.0` constraint.
+- Updated CI to MongoDB 7 and 8, Ruby 3.2 and 4.0, Apache Arrow 25, and `actions/checkout@v7.0.1`.
+- Added 24 bounded microbenchmarks covering all eight runtime paths. The harness enforces 100% path registration, takes the median of five samples, measures allocations, and aborts each timed sample or allocation pass after two seconds. The slowest observed timed sample was 0.386 seconds.
+- Added CPU and object-allocation StackProf tasks plus generated flamegraph support.
+- Added SimpleCov line and branch enforcement. The full suite verifies 100% line and branch coverage, including CLI and MongoDB integration behavior.
+- Added focused behavioral coverage for queue backpressure, manifests, telemetry, partition planning, compression backends, rotations, Parquet row groups, resumability, concurrent JSONL writing, progress, and telemetry output.
+- Made persistent MongoDB fixtures rerunnable by dropping static test collections before seeding.
+- Persisted the exact Extended JSON partition plan in manifest version 2. Resume now reuses the original boundaries rather than replanning against a growing collection, preventing boundary replay and duplicate rows.
+
+## Retained performance changes
+
+- ByteQueue stores payloads and sizes in parallel arrays instead of allocating a pair per enqueue. Confirmed round-trip runs improved from 1,792,596.54 to 2,637,130.80–2,657,101.17 operations/second: **+47.1% to +48.2%**. Oversized single items are also admitted when the queue is empty, avoiding a producer deadlock.
+- JSONL readers pass row and byte metadata with pre-encoded batches, removing a full `String#count("\n")` scan in the writer. The bounded accounting benchmark rose from 37,313.43 to more than 320,000 operations/second: **over +750%**.
+- JSONL writers cache their thread-local telemetry sink after the first write. Two confirmation runs improved the pre-encoded path by **+3.2% to +6.1%** without a confirmed JSONL pipeline regression.
+- Snapshot progress reporting no longer creates a mandatory two-second shutdown delay. Runs without a callback create no progress thread; callback runs use a condition variable and wake immediately on shutdown.
+- Two JSONL writer threads now perform real concurrent serialization/compression and allocate globally unique part numbers. On the 20,000-document incompressible pipeline, throughput improved from 9.56 to 15.14 exports/second: **+58.4%**, with complete row-count verification.
+- JSONL batching honors `batch_size` as well as the 1 MB byte ceiling, bounding latency and memory for small documents.
+- CSV uses an allocation-conscious serializer instead of constructing one `CSV.generate_line` result per row. Confirmed batch throughput improved from 1,211.39 to 1,716.50–1,738.04 operations/second: **+41.7% to +43.5%**.
+- Manifest configuration avoids redundant atomic rewrites. Confirmed progress runs improved from 1,343.24 to 1,753.49–1,764.93 operations/second: **+30.5% to +31.4%**.
+- Parquet now constructs Arrow record batches directly, caches the first row group's schema, packs primitive buffers without per-value GObject calls, preserves requested row-group boundaries through `Parquet::WriterProperties`, and keeps one native writer open per output part. The 200-document benchmark rose from 306.10 to 871.99–1,004.42 operations/second across three runs (**+184.9% to +228.1%**) while allocations fell from 12,281.0 to 2,263.2 per export (**-81.6%**). At 10,000 documents, median throughput rose from 7.23 to 46.71 operations/second (**+545.7%**) and allocations fell from 565,487 to 54,413 (**-90.4%**); output sizes remained unchanged for the compared row-group layouts.
+- Parquet batches each BSON ObjectId column's raw 12-byte values before one hexadecimal conversion, instead of allocating a hexadecimal string and `String#unpack` result per document. Alternating matched runs improved the 200-document export from 918.40 to 982.90 operations/second (**+7.0%**) and cut allocations from 2,266.15 to 1,872.15 (**-17.4%**). The 10,000-document row-group export improved from 47.05 to 50.23 operations/second (**+6.8%**) and from 54,413 to 34,443 allocations (**-36.7%**); a single 10,000-row group improved **+8.3%** with **-39.3%** allocations. Schemas, values, row-group counts, and artifact sizes remained unchanged.
+- Compression defaults were selected from bounded comparisons: zstd level 3 for JSONL, zstd level 9 for CSV, and gzip level 1 for throughput-oriented fallback.
+- CSV manifests count only rows actually written; pre-encoded strings are skipped without inflating progress.
+- Cached integer schemas now reject later values outside their represented range instead of allowing Arrow's integer builder to wrap them silently.
+
+## Rejected or unnecessary changes
+
+- Compact manifest JSON was measured and reverted: it remained below the 3% retention threshold.
+- A ring-buffer rewrite for ByteQueue was rejected. MRI's front-removal behavior kept the bulk-drain case stable, while the lower-allocation parallel-array representation supplied the retained gain.
+- Alternative CSV gzip and zstd levels that did not produce a repeatable throughput/size improvement were rejected.
+- In-place escaping for nested JSON in CSV was measured and reverted. Dedicated row serialization moved by +2.4% and +3.1% across two runs, so it did not repeatedly clear the 3% threshold.
+- Parquet `RecordBatch` substitution alone, native writer batch-size tuning, dictionary disabling, and extra row-group coalescing did not address the dominant per-value GObject conversion cost and were rejected.
+- Preallocating string capacity, avoiding the integer `compact` copy, borrowing caller buffers, bulk-joining ObjectId bytes, direct native writer calls, caching writer properties, and skipping redundant directory creation each stayed below 3% or regressed and were rejected.
+- Direct three-argument `RecordBatch` construction improved the 200-document case by 7.0% but only -0.6% to +1.9% on the 10,000-document cases, so it was rejected as a first-row-group-only gain that did not persist at scale.
+- No replacement gems or external `alexandernicholson/gems` repository were needed. Current maintained dependencies met the compatibility and measured performance requirements.
+
+## Follow-up suggestions
+
+- Run the existing opt-in one-million-document load benchmark only on a dedicated, named production-like host; record CPU, storage, MongoDB topology, and compression ratio with every result. It intentionally remains outside the sub-two-second microbenchmark harness.
+- Define schema evolution for CSV and Parquet. Columns are inferred from the first available batch, so fields first appearing in later batches are not added; an explicit Parquet schema option would also let callers avoid narrow first-row-group integer inference.
+- Further Parquet gains likely require an upstream/native bulk-buffer API: after the retained rewrite, CPU samples were dominated by GObject-introspection constructor calls and the native Parquet writer rather than Ruby row transposition.
+- Nested Ruby hashes now dominate the remaining avoidable Parquet allocations because compatibility requires their existing `Hash#inspect` string representation. An explicit nested Struct/JSON encoding option could remove that cost without silently changing current files.
+- Consider batching or journaling manifest checkpoints for high-latency filesystems. Current per-batch atomic rewrites prioritize resumability over maximum throughput.
+- Define an optional ordering contract if consumers need global document order with multiple JSONL writer threads. Current parts are collision-free and complete, but independently scheduled.
+- Track the 3% regression threshold on a pinned benchmark runner. Compression timings vary with host load and should not become a hard gate on shared CI workers.
+- Recheck `diff-lcs` when RSpec permits 2.x, and continue testing both MongoDB 7 and 8 before future driver upgrades.

data/README.md

@@ -6,18 +6,29 @@ Snapshot MongoDB collections efficiently from Ruby with resumable, partitioned e
 
 Purplelight is published on RubyGems: [purplelight on RubyGems](https://rubygems.org/gems/purplelight).
 
+Requires Ruby 3.2 or newer and MongoDB 7 or 8.
+
 Add to your Gemfile:
 
 ```ruby
-gem 'purplelight', '~> 0.1.13'
+gem 'purplelight', '~> 0.1.17'
 ```
 
 Or install directly:
 
 ```bash
-gem install purplelight
+gem install purplelight -v '~> 0.1.17'
+```
+
+`bigdecimal`, `logger`, and the MongoDB Ruby driver are installed automatically.
+To use zstd compression, add its optional backend:
+
+```ruby
+gem 'zstd-ruby', '~> 2.0'
 ```
 
+Without `zstd-ruby` or `ruby-zstds`, a requested zstd output safely falls back to gzip.
+
 ### Quick start
 
 ```ruby
@@ -124,8 +135,8 @@ Add optional dependencies:
 ```ruby
 # Gemfile
 group :parquet do
-  gem 'red-arrow', '~> 15.0'
-  gem 'red-parquet', '~> 15.0'
+  gem 'red-arrow', '>= 25.0'
+  gem 'red-parquet', '>= 25.0'
 end
 ```
 
@@ -166,6 +177,7 @@ Notes for Parquet:
 - Parquet multi-part sizing is controlled by rows via `parquet_max_rows`.
 - `--rotate-mb` / `part_bytes` do not affect Parquet part size; they apply to JSONL/CSV.
 - Use `sharding: { mode: :single_file }` to force a single `.parquet` file.
+- The first row group defines the Arrow schema. Later values must remain type-compatible; integer values outside the inferred width raise `RangeError` instead of wrapping.
 
 ### Environment variables (optional)
 
@@ -214,7 +226,7 @@ bundle exec bin/purplelight \
 - `--parquet-row-group N`: Parquet row group size (rows).
 - `--parquet-max-rows N`: Parquet max rows per part (enables multi-part when not `--single-file`).
 - `--write-chunk-mb MB`: JSONL encode/write chunk size before enqueueing.
-- `--writer-threads N` (experimental): Number of writer threads (JSONL only).
+- `--writer-threads N`: Number of concurrent JSONL writer threads.
 - `--telemetry on|off`: Force enable/disable telemetry output.
 - `--resume-overwrite-incompatible`: Overwrite an existing incompatible manifest to safely resume anew.
 - `--dry-run`: Print effective read preference JSON and exit (no snapshot).
@@ -223,7 +235,7 @@ bundle exec bin/purplelight \
 Notes:
 - Compression backend selection order is: requested format → `zstd-ruby` → `zstds` → `gzip`.
 - `--single-file` and `--by-size` update only the sharding mode/params and preserve any provided `--prefix`.
-– Parquet multi-part sizing can be controlled via `parquet_max_rows` programmatically, or via CLI `--parquet-max-rows`.
+- Parquet multi-part sizing can be controlled via `parquet_max_rows` programmatically, or via CLI `--parquet-max-rows`.
 - To increase concurrent connections, set `maxPoolSize` on your Mongo URI (used by `--uri`), e.g., `mongodb://.../?maxPoolSize=32`. A good starting point is `maxPoolSize >= --partitions`.
 
 ### Architecture
@@ -261,6 +273,7 @@ Key points:
 - **Compression**: prefer `zstd`; adjust level to balance speed/ratio. CLI: `--compression zstd --compression-level N`. For max speed, try `--compression gzip --compression-level 1`.
 - **Rotation size**: larger (512MB–1GB) reduces finalize overhead for many parts. CLI: `--rotate-mb` (and/or `--by-size`).
 - **JSONL chunking**: tune builder write chunk size for throughput. CLI: `--write-chunk-mb`.
+- **JSONL writer threads**: use `--writer-threads 2` when serialization or compression is the bottleneck. Writers emit independently numbered parts; downstream consumers must not assume document order across parts.
 - **Parquet row groups**: choose a row group size that fits downstream readers. CLI: `--parquet-row-group`.
 - **Parquet parts (rows)**: split Parquet outputs by rows with `parquet_max_rows` (programmatic API). Set `sharding.mode` to anything other than `:single_file` to enable multi-part filenames.
 - **Read preference**: offload to secondaries or tagged analytics nodes when available. CLI: `--read-preference`, `--read-tags`.
@@ -268,7 +281,19 @@ Key points:
 - **Cursor timeout**: for very long scans, leave `noCursorTimeout` enabled. CLI: `--no-cursor-timeout true|false`.
 - **Telemetry**: enable to inspect timing breakdowns; disable for minimal output. CLI: `--telemetry on|off`.
 
-Benchmarking (optional):
+Bounded microbenchmarks:
+
+```bash
+bundle exec rake microbench
+
+# Compare against the checked-in baseline.
+bundle exec ruby benchmark/microbench.rb --baseline benchmark/baseline.json
+```
+
+The harness covers every runtime component, takes the median of five samples,
+and aborts any timed sample or allocation pass after two seconds.
+
+Optional production-scale load benchmark (not part of the microbenchmark suite):
 
 ```bash
 # 1M docs benchmark with tunables (JSONL)
@@ -330,7 +355,7 @@ bundle exec bin/purplelight \
   --read-preference secondary --read-tags nodeType=ANALYTICS --dry-run
 ```
 
-### Quick Benchmark
+### Example load benchmark
 ```
 % BENCH=1 BENCH_PARTITIONS=16 BENCH_BATCH_SIZE=8000 BENCH_QUEUE_MB=512 BENCH_ROTATE_MB=512 BENCH_COMPRESSION=gzip bundle exec rspec spec/benchmark_perf_spec.rb --format doc
 

data/Rakefile

@@ -12,3 +12,14 @@ rescue LoadError
     sh 'echo "RSpec not installed"'
   end
 end
+
+desc 'Run the bounded microbenchmark suite'
+task :microbench do
+  ruby 'benchmark/microbench.rb'
+end
+
+desc 'Profile all microbenchmarks and render CPU/allocation flamegraphs'
+task :profile do
+  ruby 'benchmark/profile.rb --mode cpu'
+  ruby 'benchmark/profile.rb --mode object'
+end

data/benchmark/baseline.json

@@ -0,0 +1,366 @@
+{
+  "generated_at": "2026-07-21T12:41:06Z",
+  "coverage": {
+    "covered": [
+      "manifest",
+      "partitioner",
+      "queue",
+      "snapshot",
+      "telemetry",
+      "writer_csv",
+      "writer_jsonl",
+      "writer_parquet"
+    ],
+    "required": [
+      "manifest",
+      "partitioner",
+      "queue",
+      "snapshot",
+      "telemetry",
+      "writer_csv",
+      "writer_jsonl",
+      "writer_parquet"
+    ],
+    "percent": 100.0
+  },
+  "max_case_seconds": 2.0,
+  "samples": 5,
+  "results": {
+    "byte_queue_round_trip": {
+      "iterations": 20000,
+      "seconds": 0.053941,
+      "sample_seconds": [
+        0.007595,
+        0.018067,
+        0.007425,
+        0.013365,
+        0.007489
+      ],
+      "operations_per_second": 2633311.36,
+      "allocated_objects": 100001,
+      "allocations_per_operation": 5.0
+    },
+    "byte_queue_bulk_drain": {
+      "iterations": 10,
+      "seconds": 0.058171,
+      "sample_seconds": [
+        0.011596,
+        0.011667,
+        0.01175,
+        0.011877,
+        0.011281
+      ],
+      "operations_per_second": 857.12,
+      "allocated_objects": 70,
+      "allocations_per_operation": 7.0
+    },
+    "telemetry_counter_and_timer": {
+      "iterations": 50000,
+      "seconds": 0.04747,
+      "sample_seconds": [
+        0.009476,
+        0.009558,
+        0.009509,
+        0.009478,
+        0.009449
+      ],
+      "operations_per_second": 5275374.56,
+      "allocated_objects": 0,
+      "allocations_per_operation": 0.0
+    },
+    "telemetry_construction": {
+      "iterations": 100000,
+      "seconds": 0.107426,
+      "sample_seconds": [
+        0.02273,
+        0.021634,
+        0.020193,
+        0.021322,
+        0.021547
+      ],
+      "operations_per_second": 4641017.33,
+      "allocated_objects": 300000,
+      "allocations_per_operation": 3.0
+    },
+    "manifest_progress": {
+      "iterations": 250,
+      "seconds": 0.714513,
+      "sample_seconds": [
+        0.144839,
+        0.145152,
+        0.143372,
+        0.140609,
+        0.140541
+      ],
+      "operations_per_second": 1743.72,
+      "allocated_objects": 21250,
+      "allocations_per_operation": 85.0
+    },
+    "partition_range_planning": {
+      "iterations": 100000,
+      "seconds": 0.045859,
+      "sample_seconds": [
+        0.009671,
+        0.009649,
+        0.009258,
+        0.008996,
+        0.008285
+      ],
+      "operations_per_second": 10801468.79,
+      "allocated_objects": 100000,
+      "allocations_per_operation": 1.0
+    },
+    "partition_cursor_sampling": {
+      "iterations": 10,
+      "seconds": 0.006557,
+      "sample_seconds": [
+        0.001337,
+        0.001296,
+        0.001314,
+        0.001294,
+        0.001316
+      ],
+      "operations_per_second": 7610.35,
+      "allocated_objects": 250,
+      "allocations_per_operation": 25.0
+    },
+    "snapshot_configuration": {
+      "iterations": 10000,
+      "seconds": 0.211803,
+      "sample_seconds": [
+        0.043064,
+        0.041384,
+        0.042928,
+        0.042796,
+        0.041631
+      ],
+      "operations_per_second": 233666.7,
+      "allocated_objects": 170000,
+      "allocations_per_operation": 17.0
+    },
+    "snapshot_empty_export": {
+      "iterations": 20,
+      "seconds": 0.046427,
+      "sample_seconds": [
+        0.008102,
+        0.007108,
+        0.008957,
+        0.014756,
+        0.007504
+      ],
+      "operations_per_second": 2468.53,
+      "allocated_objects": 3420,
+      "allocations_per_operation": 171.0
+    },
+    "snapshot_raw_batching": {
+      "iterations": 100,
+      "seconds": 0.059924,
+      "sample_seconds": [
+        0.011836,
+        0.012093,
+        0.012064,
+        0.011962,
+        0.011969
+      ],
+      "operations_per_second": 8354.92,
+      "allocated_objects": 1800,
+      "allocations_per_operation": 18.0
+    },
+    "snapshot_jsonl_serial_pipeline": {
+      "iterations": 3,
+      "seconds": 1.593689,
+      "sample_seconds": [
+        0.332539,
+        0.320695,
+        0.319876,
+        0.30983,
+        0.310749
+      ],
+      "operations_per_second": 9.38,
+      "allocated_objects": 543645,
+      "allocations_per_operation": 181215.0
+    },
+    "snapshot_jsonl_pipeline": {
+      "iterations": 3,
+      "seconds": 0.978552,
+      "sample_seconds": [
+        0.195724,
+        0.195973,
+        0.19579,
+        0.195125,
+        0.19594
+      ],
+      "operations_per_second": 15.32,
+      "allocated_objects": 543867,
+      "allocations_per_operation": 181289.0
+    },
+    "jsonl_batch_write": {
+      "iterations": 100,
+      "seconds": 0.11875,
+      "sample_seconds": [
+        0.022894,
+        0.025733,
+        0.021527,
+        0.024267,
+        0.024329
+      ],
+      "operations_per_second": 4120.82,
+      "allocated_objects": 182800,
+      "allocations_per_operation": 1828.0
+    },
+    "jsonl_zstd_batch_write": {
+      "iterations": 50,
+      "seconds": 0.071127,
+      "sample_seconds": [
+        0.016098,
+        0.012678,
+        0.013116,
+        0.013103,
+        0.016132
+      ],
+      "operations_per_second": 3812.14,
+      "allocated_objects": 92050,
+      "allocations_per_operation": 1841.0
+    },
+    "jsonl_preencoded_accounting": {
+      "iterations": 10000,
+      "seconds": 0.155715,
+      "sample_seconds": [
+        0.031233,
+        0.03071,
+        0.031271,
+        0.031395,
+        0.031106
+      ],
+      "operations_per_second": 320174.17,
+      "allocated_objects": 0,
+      "allocations_per_operation": 0.0
+    },
+    "csv_batch_write": {
+      "iterations": 100,
+      "seconds": 0.29281,
+      "sample_seconds": [
+        0.057804,
+        0.058016,
+        0.058681,
+        0.059051,
+        0.059258
+      ],
+      "operations_per_second": 1704.13,
+      "allocated_objects": 363700,
+      "allocations_per_operation": 3637.0
+    },
+    "csv_zstd_batch_write": {
+      "iterations": 50,
+      "seconds": 0.222587,
+      "sample_seconds": [
+        0.05143,
+        0.046745,
+        0.038643,
+        0.052235,
+        0.033534
+      ],
+      "operations_per_second": 1069.63,
+      "allocated_objects": 182500,
+      "allocations_per_operation": 3650.0
+    },
+    "csv_gzip_batch_write": {
+      "iterations": 50,
+      "seconds": 0.148172,
+      "sample_seconds": [
+        0.027482,
+        0.034167,
+        0.027431,
+        0.031337,
+        0.027755
+      ],
+      "operations_per_second": 1801.48,
+      "allocated_objects": 182550,
+      "allocations_per_operation": 3651.0
+    },
+    "csv_counting_io_write": {
+      "iterations": 100000,
+      "seconds": 0.041051,
+      "sample_seconds": [
+        0.008018,
+        0.008041,
+        0.008821,
+        0.008223,
+        0.007948
+      ],
+      "operations_per_second": 12436264.16,
+      "allocated_objects": 0,
+      "allocations_per_operation": 0.0
+    },
+    "csv_row_serialization": {
+      "iterations": 25000,
+      "seconds": 0.291323,
+      "sample_seconds": [
+        0.061351,
+        0.057502,
+        0.057275,
+        0.057677,
+        0.057518
+      ],
+      "operations_per_second": 434646.55,
+      "allocated_objects": 375000,
+      "allocations_per_operation": 15.0
+    },
+    "parquet_buffer_append": {
+      "iterations": 5000,
+      "seconds": 0.005627,
+      "sample_seconds": [
+        0.001129,
+        0.001121,
+        0.001122,
+        0.001125,
+        0.00113
+      ],
+      "operations_per_second": 4444444.64,
+      "allocated_objects": 0,
+      "allocations_per_operation": 0.0
+    },
+    "parquet_object_id_array": {
+      "iterations": 1000,
+      "seconds": 0.237818,
+      "sample_seconds": [
+        0.048408,
+        0.044886,
+        0.056985,
+        0.043491,
+        0.044048
+      ],
+      "operations_per_second": 22278.66,
+      "allocated_objects": 64000,
+      "allocations_per_operation": 64.0
+    },
+    "parquet_batch_write": {
+      "iterations": 10,
+      "seconds": 0.049999,
+      "sample_seconds": [
+        0.010937,
+        0.00987,
+        0.009626,
+        0.00971,
+        0.009856
+      ],
+      "operations_per_second": 1014.61,
+      "allocated_objects": 18692,
+      "allocations_per_operation": 1869.2
+    },
+    "parquet_row_group_write": {
+      "iterations": 2,
+      "seconds": 0.18688,
+      "sample_seconds": [
+        0.03926,
+        0.036353,
+        0.037264,
+        0.03747,
+        0.036533
+      ],
+      "operations_per_second": 53.67,
+      "allocated_objects": 68880,
+      "allocations_per_operation": 34440.0
+    }
+  }
+}

data/benchmark/harness.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+require 'benchmark'
+require 'json'
+require 'timeout'
+
+module Purplelight
+  module Microbench
+    Case = Data.define(:name, :paths, :iterations, :block)
+
+    # Runs registered microbenchmarks under strict duration and coverage limits.
+    class Harness
+      MAX_CASE_SECONDS = 2.0
+      SAMPLES = 5
+      REQUIRED_PATHS = %i[
+        manifest partitioner queue snapshot telemetry writer_csv writer_jsonl writer_parquet
+      ].freeze
+
+      def initialize
+        @cases = []
+      end
+
+      def register(name, paths:, iterations: 1, &block)
+        raise ArgumentError, 'benchmark block required' unless block
+
+        @cases << Case.new(name:, paths: Array(paths).map(&:to_sym), iterations:, block:)
+      end
+
+      def run(baseline_path: nil)
+        verify_coverage!
+        results = @cases.to_h { |benchmark_case| [benchmark_case.name, measure(benchmark_case)] }
+        report = {
+          generated_at: Time.now.utc.iso8601,
+          coverage: coverage,
+          max_case_seconds: MAX_CASE_SECONDS,
+          samples: SAMPLES,
+          results:
+        }
+        report[:comparison] = compare(results, JSON.parse(File.read(baseline_path))) if baseline_path
+        report
+      end
+
+      def coverage
+        covered = @cases.flat_map(&:paths).uniq
+        {
+          covered: covered.sort,
+          required: REQUIRED_PATHS,
+          percent: (covered.length.fdiv(REQUIRED_PATHS.length) * 100).round(2)
+        }
+      end
+
+      private
+
+      def verify_coverage!
+        missing = REQUIRED_PATHS - @cases.flat_map(&:paths)
+        raise "missing benchmark paths: #{missing.join(', ')}" unless missing.empty?
+      end
+
+      def measure(benchmark_case)
+        samples = Array.new(SAMPLES) do
+          Timeout.timeout(MAX_CASE_SECONDS) do
+            Benchmark.realtime do
+              benchmark_case.iterations.times { benchmark_case.block.call }
+            end
+          end
+        end
+        allocated_objects = Timeout.timeout(MAX_CASE_SECONDS) do
+          allocations_before = GC.stat(:total_allocated_objects)
+          benchmark_case.iterations.times { benchmark_case.block.call }
+          GC.stat(:total_allocated_objects) - allocations_before
+        end
+        median = samples.sort.fetch(SAMPLES / 2)
+        {
+          iterations: benchmark_case.iterations,
+          seconds: samples.sum.round(6),
+          sample_seconds: samples.map { |elapsed| elapsed.round(6) },
+          operations_per_second: (benchmark_case.iterations / median).round(2),
+          allocated_objects:,
+          allocations_per_operation: allocated_objects.fdiv(benchmark_case.iterations).round(2)
+        }
+      rescue Timeout::Error
+        raise "#{benchmark_case.name} exceeded #{MAX_CASE_SECONDS}s in a single measurement"
+      end
+
+      def compare(results, baseline)
+        old_results = baseline.fetch('results')
+        results.to_h do |name, result|
+          old_result = old_results[name.to_s]
+          next [name, { baseline: false }] unless old_result
+
+          old_rate = old_result.fetch('operations_per_second')
+          delta = ((result[:operations_per_second] / old_rate) - 1.0) * 100
+          comparison = { percent: delta.round(2), significant: delta.abs > 3.0 }
+          if old_result['allocations_per_operation']
+            old_allocations = old_result.fetch('allocations_per_operation')
+            new_allocations = result[:allocations_per_operation]
+            allocation_delta = if old_allocations.zero?
+                                 (new_allocations.zero? ? 0.0 : nil)
+                               else
+                                 (((new_allocations / old_allocations) - 1.0) * 100)
+                               end
+            comparison[:allocation_percent] = allocation_delta&.round(2)
+          end
+          [name, comparison]
+        end
+      end
+    end
+  end
+end