purplelight 0.1.18 → 0.1.19

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6d2a9cae59e9123e8cf63cfd369323fb0a8b42991ceaf8304f4a93f2d59733a3
-  data.tar.gz: b8c33bef8ca40f904a916a0ee661586bbdc1cf183d3fb81f352448cd487561c8
+  metadata.gz: 300107b11002d6ede77f9ce6adc7c8f8d30045a34a43952c2b9d0c0d60e784a1
+  data.tar.gz: 20093fbe1faca5ea331258cf410fb57e71bdf8055388b2304672f2aedb2c7943
 SHA512:
-  metadata.gz: b891e35b28ace87681772722d07a9678abc74edfbeb585ae5b4a09019978ddd7583e7c96628e3282108a2166e36cc2a0b81ce08ac273975f7bda281fcd67b20e
-  data.tar.gz: 28fc4c257671e6a48f4ccb4a61928e322d2b9403f51ae5e59e94d2906b15af80b1df5241ce05c6995c5def12148beafbee0e57f6ad7626b1b28d54c7e5e8dfda
+  metadata.gz: 0c9f55c415a779b42a7f95c1a6468d68538670235f5cc475dc6ef110145b50566b826a1dfec8f7f048539ed8949c7061fd120df85f59126aa1bfe11c1157e8f0
+  data.tar.gz: fc7486b767c89ff9a308a2bd88ddb580400a0e7e830f457065f7e1bc19db44ab688ce82b27d0bfe95b15d964cd211aab5399663c30b519151a55ceab9626fe58

data/PERFORMANCE_NOTES.md

@@ -1,18 +1,20 @@
 # Purplelight upgrade and performance notes
 
-Updated: 2026-07-21
+Updated: 2026-07-23
 
 ## Completed
 
 - Pinned managed development Ruby 4.0.2; no Homebrew Ruby is used.
 - Updated all Ruby 3.2-compatible dependencies: MongoDB Ruby driver 2.24.1, BSON 5.2.0, zstd-ruby 2.0.6, red-arrow/red-parquet 25.0.0, Rake 13.4.2, RSpec 3.13.2, RuboCop 1.88.2, SimpleCov 1.0.2, StackProf 0.2.28, and current transitives. `parallel` remains on 1.28 because 2.x requires Ruby 3.3; `diff-lcs` 2.0 remains blocked by RSpec's `< 2.0` constraint.
 - Updated CI to MongoDB 7 and 8, Ruby 3.2 and 4.0, Apache Arrow 25, and `actions/checkout@v7.0.1`.
-- Added 54 bounded microbenchmarks covering all eight runtime paths, including 27 Parquet column types plus full-table, row-group, and projection reads. The harness enforces 100% path registration, takes the median of five samples, measures allocations, and aborts each timed sample or allocation pass after two seconds. The slowest observed timed sample in the expanded suite was 0.360 seconds.
+- Added 55 bounded microbenchmarks covering all eight runtime paths, including 28 Parquet column types plus full-table, row-group, and projection reads. The harness enforces 100% path registration, takes the median of five samples, measures allocations, and aborts each timed sample or allocation pass after two seconds. The slowest observed timed sample in the expanded suite was 0.397 seconds.
 - Added CPU and object-allocation StackProf tasks plus generated flamegraph support.
 - Added SimpleCov line and branch enforcement. The full suite verifies 100% line and branch coverage, including CLI and MongoDB integration behavior.
 - Added focused behavioral coverage for queue backpressure, manifests, telemetry, partition planning, compression backends, rotations, Parquet row groups, resumability, concurrent JSONL writing, progress, and telemetry output.
 - Made persistent MongoDB fixtures rerunnable by dropping static test collections before seeding.
 - Persisted the exact Extended JSON partition plan in manifest version 2. Resume now reuses the original boundaries rather than replanning against a growing collection, preventing boundary replay and duplicate rows.
+- BSON Symbol values now use Purplelight's direct UTF-8 Arrow string builder for scalar columns and recursive list leaves. Cached row-group schemas therefore remain buildable instead of failing through red-arrow's unsupported `DictionaryDataType#build_array` path.
+- Snapshot reader and writer failures now atomically close and discard the shared byte queue, immediately waking blocked producers and preserving the first worker exception for the caller.
 
 ## Retained performance changes
 

data/README.md

@@ -11,13 +11,13 @@ Requires Ruby 3.2 or newer and MongoDB 7 or 8.
 Add to your Gemfile:
 
 ```ruby
-gem 'purplelight', '~> 0.1.18'
+gem 'purplelight', '~> 0.1.19'
 ```
 
 Or install directly:
 
 ```bash
-gem install purplelight -v '~> 0.1.18'
+gem install purplelight -v '~> 0.1.19'
 ```
 
 `bigdecimal`, `logger`, and the MongoDB Ruby driver are installed automatically.
@@ -178,6 +178,7 @@ Notes for Parquet:
 - `--rotate-mb` / `part_bytes` do not affect Parquet part size; they apply to JSONL/CSV.
 - Use `sharding: { mode: :single_file }` to force a single `.parquet` file.
 - The first row group defines the Arrow schema. Later values must remain type-compatible; integer values outside the inferred width raise `RangeError` instead of wrapping.
+- BSON Symbol values, which the MongoDB driver exposes as Ruby `Symbol` objects, are exported as UTF-8 strings. This applies to scalar values and list elements across every row group.
 - Dictionary encoding is chosen from the first row group for read performance: list leaves retain it, while scalar strings use it only when they have at most 16 distinct values and average at least four repetitions per value.
 - Dictionary selection changes physical Parquet page encodings and file checksums only; logical schemas, values, nullability, ordering, and row-group boundaries remain compatible. Consumers should validate logical data rather than requiring a particular page encoding.
 - In the matched 50,000-row, 27-column benchmark, this policy improved full-table reads by 15.9%, row-group streaming by 29.1%, projected reads by 30.9%, and reduced file size by 31.5%. Existing snapshots must be re-exported to receive these gains; see `PERFORMANCE_NOTES.md` for the complete matrix.