purplelight 0.1.17 → 0.1.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. checksums.yaml +4 -4
  2. data/PERFORMANCE_NOTES.md +5 -2
  3. data/README.md +5 -2
  4. data/benchmark/baseline.json +612 -192
  5. data/benchmark/microbench.rb +87 -0
  6. data/lib/purplelight/version.rb +1 -1
  7. data/lib/purplelight/writer_parquet.rb +50 -1
  8. metadata +1 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 006653025fe8b2a5afa958c08992076f171550c7e1a1da4d57442df76d0e659d
4
- data.tar.gz: 2476b1a5c4f078b539cd83c552e7ec5938277943e46ae6412b4c04df3830a042
3
+ metadata.gz: 6d2a9cae59e9123e8cf63cfd369323fb0a8b42991ceaf8304f4a93f2d59733a3
4
+ data.tar.gz: b8c33bef8ca40f904a916a0ee661586bbdc1cf183d3fb81f352448cd487561c8
5
5
  SHA512:
6
- metadata.gz: d31c3447814135d0f723f6bf35e955618932971f28bdcdd15a3831daee2c8c2d6a22ad0dae0bb8af1572f0e570fdbb71cc606f31241a83ee741742a783f3a673
7
- data.tar.gz: 30bd7c0a089f57b6f8957d93b538b5e0f698e6440b7180dea86e3922cee58e639cace618782d888284883bcd5930456bd6cda9e270b338d54caea538b0804943
6
+ metadata.gz: b891e35b28ace87681772722d07a9678abc74edfbeb585ae5b4a09019978ddd7583e7c96628e3282108a2166e36cc2a0b81ce08ac273975f7bda281fcd67b20e
7
+ data.tar.gz: 28fc4c257671e6a48f4ccb4a61928e322d2b9403f51ae5e59e94d2906b15af80b1df5241ce05c6995c5def12148beafbee0e57f6ad7626b1b28d54c7e5e8dfda
data/PERFORMANCE_NOTES.md CHANGED
@@ -7,7 +7,7 @@ Updated: 2026-07-21
7
7
  - Pinned managed development Ruby 4.0.2; no Homebrew Ruby is used.
8
8
  - Updated all Ruby 3.2-compatible dependencies: MongoDB Ruby driver 2.24.1, BSON 5.2.0, zstd-ruby 2.0.6, red-arrow/red-parquet 25.0.0, Rake 13.4.2, RSpec 3.13.2, RuboCop 1.88.2, SimpleCov 1.0.2, StackProf 0.2.28, and current transitives. `parallel` remains on 1.28 because 2.x requires Ruby 3.3; `diff-lcs` 2.0 remains blocked by RSpec's `< 2.0` constraint.
9
9
  - Updated CI to MongoDB 7 and 8, Ruby 3.2 and 4.0, Apache Arrow 25, and `actions/checkout@v7.0.1`.
10
- - Added 24 bounded microbenchmarks covering all eight runtime paths. The harness enforces 100% path registration, takes the median of five samples, measures allocations, and aborts each timed sample or allocation pass after two seconds. The slowest observed timed sample was 0.386 seconds.
10
+ - Added 54 bounded microbenchmarks covering all eight runtime paths, including 27 Parquet column types plus full-table, row-group, and projection reads. The harness enforces 100% path registration, takes the median of five samples, measures allocations, and aborts each timed sample or allocation pass after two seconds. The slowest observed timed sample in the expanded suite was 0.360 seconds.
11
11
  - Added CPU and object-allocation StackProf tasks plus generated flamegraph support.
12
12
  - Added SimpleCov line and branch enforcement. The full suite verifies 100% line and branch coverage, including CLI and MongoDB integration behavior.
13
13
  - Added focused behavioral coverage for queue backpressure, manifests, telemetry, partition planning, compression backends, rotations, Parquet row groups, resumability, concurrent JSONL writing, progress, and telemetry output.
@@ -26,6 +26,7 @@ Updated: 2026-07-21
26
26
  - Manifest configuration avoids redundant atomic rewrites. Confirmed progress runs improved from 1,343.24 to 1,753.49–1,764.93 operations/second: **+30.5% to +31.4%**.
27
27
  - Parquet now constructs Arrow record batches directly, caches the first row group's schema, packs primitive buffers without per-value GObject calls, preserves requested row-group boundaries through `Parquet::WriterProperties`, and keeps one native writer open per output part. The 200-document benchmark rose from 306.10 to 871.99–1,004.42 operations/second across three runs (**+184.9% to +228.1%**) while allocations fell from 12,281.0 to 2,263.2 per export (**-81.6%**). At 10,000 documents, median throughput rose from 7.23 to 46.71 operations/second (**+545.7%**) and allocations fell from 565,487 to 54,413 (**-90.4%**); output sizes remained unchanged for the compared row-group layouts.
28
28
  - Parquet batches each BSON ObjectId column's raw 12-byte values before one hexadecimal conversion, instead of allocating a hexadecimal string and `String#unpack` result per document. Alternating matched runs improved the 200-document export from 918.40 to 982.90 operations/second (**+7.0%**) and cut allocations from 2,266.15 to 1,872.15 (**-17.4%**). The 10,000-document row-group export improved from 47.05 to 50.23 operations/second (**+6.8%**) and from 54,413 to 34,443 allocations (**-36.7%**); a single 10,000-row group improved **+8.3%** with **-39.3%** allocations. Schemas, values, row-group counts, and artifact sizes remained unchanged.
29
+ - Parquet now disables dictionary encoding for scalar columns unless the first row group proves a string has at most 16 distinct values and at least four repetitions per value. It preserves dictionary encoding for every list leaf, avoiding reader-specific list regressions. Seven interleaved Arrow-reader samples over 50,000 rows and 27 column types improved geometric-mean raw-column throughput by **21.4%**, full-table reads by **15.9%**, row-group streaming by **29.2%**, and projected primitive reads by **30.9%**; no individual column regressed by 3%, and the file shrank from 3,959,054 to 2,713,148 bytes (**-31.5%**). Independent DuckDB checks improved integer aggregates by **18.4%**, string aggregates by **10.2%**, low-cardinality filters by **13.4%**, range filters by **13.6%**, projections by **5.0%**, and full materialization by **3.6%**, while list aggregation remained unchanged.
29
30
  - Compression defaults were selected from bounded comparisons: zstd level 3 for JSONL, zstd level 9 for CSV, and gzip level 1 for throughput-oriented fallback.
30
31
  - CSV manifests count only rows actually written; pre-encoded strings are skipped without inflating progress.
31
32
  - Cached integer schemas now reject later values outside their represented range instead of allowing Arrow's integer builder to wrap them silently.
@@ -36,7 +37,7 @@ Updated: 2026-07-21
36
37
  - A ring-buffer rewrite for ByteQueue was rejected. MRI's front-removal behavior kept the bulk-drain case stable, while the lower-allocation parallel-array representation supplied the retained gain.
37
38
  - Alternative CSV gzip and zstd levels that did not produce a repeatable throughput/size improvement were rejected.
38
39
  - In-place escaping for nested JSON in CSV was measured and reverted. Dedicated row serialization moved by +2.4% and +3.1% across two runs, so it did not repeatedly clear the 3% threshold.
39
- - Parquet `RecordBatch` substitution alone, native writer batch-size tuning, dictionary disabling, and extra row-group coalescing did not address the dominant per-value GObject conversion cost and were rejected.
40
+ - Blanket Parquet dictionary disabling was rejected despite strong aggregate gains: it regressed low-cardinality strings by 9.2%, integer lists by 6.7%, nested integer lists by 6.5%, and DuckDB list aggregation by about 10%. The retained adaptive policy preserves those dictionaries. Global dictionary-page limits, 20,000–50,000-row groups, and native batch/data-page changes were also rejected because they either stayed below 3% or regressed filtered, full-materialization, temporal, or list workloads.
40
41
  - Preallocating string capacity, avoiding the integer `compact` copy, borrowing caller buffers, bulk-joining ObjectId bytes, direct native writer calls, caching writer properties, and skipping redundant directory creation each stayed below 3% or regressed and were rejected.
41
42
  - Direct three-argument `RecordBatch` construction improved the 200-document case by 7.0% but only -0.6% to +1.9% on the 10,000-document cases, so it was rejected as a first-row-group-only gain that did not persist at scale.
42
43
  - No replacement gems or external `alexandernicholson/gems` repository were needed. Current maintained dependencies met the compatibility and measured performance requirements.
@@ -46,6 +47,8 @@ Updated: 2026-07-21
46
47
  - Run the existing opt-in one-million-document load benchmark only on a dedicated, named production-like host; record CPU, storage, MongoDB topology, and compression ratio with every result. It intentionally remains outside the sub-two-second microbenchmark harness.
47
48
  - Define schema evolution for CSV and Parquet. Columns are inferred from the first available batch, so fields first appearing in later batches are not added; an explicit Parquet schema option would also let callers avoid narrow first-row-group integer inference.
48
49
  - Further Parquet gains likely require an upstream/native bulk-buffer API: after the retained rewrite, CPU samples were dominated by GObject-introspection constructor calls and the native Parquet writer rather than Ruby row transposition.
50
+ - A custom native Parquet read extension is not justified: Arrow and DuckDB already perform decoding natively, while profiling isolated file encoding—not Ruby dispatch—as the controllable bottleneck. Reconsider only if a concrete reader profile still spends more than 3% in Purplelight-owned Ruby code.
51
+ - Parquet columns containing literal dots retain Arrow's default dictionary behavior because the Ruby writer API exposes dot-delimited physical paths without an escaping mechanism. Revisit if red-parquet adds a structured `ColumnPath` binding.
49
52
  - Nested Ruby hashes now dominate the remaining avoidable Parquet allocations because compatibility requires their existing `Hash#inspect` string representation. An explicit nested Struct/JSON encoding option could remove that cost without silently changing current files.
50
53
  - Consider batching or journaling manifest checkpoints for high-latency filesystems. Current per-batch atomic rewrites prioritize resumability over maximum throughput.
51
54
  - Define an optional ordering contract if consumers need global document order with multiple JSONL writer threads. Current parts are collision-free and complete, but independently scheduled.
data/README.md CHANGED
@@ -11,13 +11,13 @@ Requires Ruby 3.2 or newer and MongoDB 7 or 8.
11
11
  Add to your Gemfile:
12
12
 
13
13
  ```ruby
14
- gem 'purplelight', '~> 0.1.17'
14
+ gem 'purplelight', '~> 0.1.18'
15
15
  ```
16
16
 
17
17
  Or install directly:
18
18
 
19
19
  ```bash
20
- gem install purplelight -v '~> 0.1.17'
20
+ gem install purplelight -v '~> 0.1.18'
21
21
  ```
22
22
 
23
23
  `bigdecimal`, `logger`, and the MongoDB Ruby driver are installed automatically.
@@ -178,6 +178,9 @@ Notes for Parquet:
178
178
  - `--rotate-mb` / `part_bytes` do not affect Parquet part size; they apply to JSONL/CSV.
179
179
  - Use `sharding: { mode: :single_file }` to force a single `.parquet` file.
180
180
  - The first row group defines the Arrow schema. Later values must remain type-compatible; integer values outside the inferred width raise `RangeError` instead of wrapping.
181
+ - Dictionary encoding is chosen from the first row group for read performance: list leaves retain it, while scalar strings use it only when they have at most 16 distinct values and average at least four repetitions per value.
182
+ - Dictionary selection changes physical Parquet page encodings and file checksums only; logical schemas, values, nullability, ordering, and row-group boundaries remain compatible. Consumers should validate logical data rather than requiring a particular page encoding.
183
+ - In the matched 50,000-row, 27-column benchmark, this policy improved full-table reads by 15.9%, row-group streaming by 29.1%, projected reads by 30.9%, and reduced file size by 31.5%. Existing snapshots must be re-exported to receive these gains; see `PERFORMANCE_NOTES.md` for the complete matrix.
181
184
 
182
185
  ### Environment variables (optional)
183
186
 
data/benchmark/baseline.json CHANGED
@@ -1,5 +1,5 @@
1
1
  {
2
- "generated_at": "2026-07-21T12:41:06Z",
2
+ "generated_at": "2026-07-21T15:57:50Z",
3
3
  "coverage": {
4
4
  "covered": [
5
5
  "manifest",
@@ -28,339 +28,759 @@
28
28
  "results": {
29
29
  "byte_queue_round_trip": {
30
30
  "iterations": 20000,
31
- "seconds": 0.053941,
31
+ "seconds": 0.065391,
32
32
  "sample_seconds": [
33
- 0.007595,
34
- 0.018067,
35
- 0.007425,
36
- 0.013365,
37
- 0.007489
33
+ 0.025423,
34
+ 0.008077,
35
+ 0.007332,
36
+ 0.007017,
37
+ 0.017542
38
38
  ],
39
- "operations_per_second": 2633311.36,
39
+ "operations_per_second": 2476166.88,
40
40
  "allocated_objects": 100001,
41
41
  "allocations_per_operation": 5.0
42
42
  },
43
43
  "byte_queue_bulk_drain": {
44
44
  "iterations": 10,
45
- "seconds": 0.058171,
45
+ "seconds": 0.056242,
46
46
  "sample_seconds": [
47
- 0.011596,
48
- 0.011667,
49
- 0.01175,
50
- 0.011877,
51
- 0.011281
47
+ 0.011117,
48
+ 0.011141,
49
+ 0.011126,
50
+ 0.011349,
51
+ 0.011509
52
52
  ],
53
- "operations_per_second": 857.12,
53
+ "operations_per_second": 897.59,
54
54
  "allocated_objects": 70,
55
55
  "allocations_per_operation": 7.0
56
56
  },
57
57
  "telemetry_counter_and_timer": {
58
58
  "iterations": 50000,
59
- "seconds": 0.04747,
59
+ "seconds": 0.046463,
60
60
  "sample_seconds": [
61
- 0.009476,
62
- 0.009558,
63
- 0.009509,
64
- 0.009478,
65
- 0.009449
61
+ 0.009533,
62
+ 0.00932,
63
+ 0.00927,
64
+ 0.009117,
65
+ 0.009223
66
66
  ],
67
- "operations_per_second": 5275374.56,
67
+ "operations_per_second": 5393743.24,
68
68
  "allocated_objects": 0,
69
69
  "allocations_per_operation": 0.0
70
70
  },
71
71
  "telemetry_construction": {
72
72
  "iterations": 100000,
73
- "seconds": 0.107426,
73
+ "seconds": 0.110846,
74
74
  "sample_seconds": [
75
- 0.02273,
76
- 0.021634,
77
- 0.020193,
78
- 0.021322,
79
- 0.021547
75
+ 0.027056,
76
+ 0.020808,
77
+ 0.022361,
78
+ 0.02009,
79
+ 0.020531
80
80
  ],
81
- "operations_per_second": 4641017.33,
81
+ "operations_per_second": 4805843.93,
82
82
  "allocated_objects": 300000,
83
83
  "allocations_per_operation": 3.0
84
84
  },
85
85
  "manifest_progress": {
86
86
  "iterations": 250,
87
- "seconds": 0.714513,
87
+ "seconds": 0.708686,
88
88
  "sample_seconds": [
89
- 0.144839,
90
- 0.145152,
91
- 0.143372,
92
- 0.140609,
93
- 0.140541
89
+ 0.143975,
90
+ 0.140292,
91
+ 0.140702,
92
+ 0.141118,
93
+ 0.142599
94
94
  ],
95
- "operations_per_second": 1743.72,
96
- "allocated_objects": 21250,
97
- "allocations_per_operation": 85.0
95
+ "operations_per_second": 1771.57,
96
+ "allocated_objects": 21252,
97
+ "allocations_per_operation": 85.01
98
98
  },
99
99
  "partition_range_planning": {
100
100
  "iterations": 100000,
101
- "seconds": 0.045859,
101
+ "seconds": 0.061333,
102
102
  "sample_seconds": [
103
- 0.009671,
104
- 0.009649,
105
- 0.009258,
106
- 0.008996,
107
- 0.008285
103
+ 0.026656,
104
+ 0.009021,
105
+ 0.008832,
106
+ 0.008527,
107
+ 0.008297
108
108
  ],
109
- "operations_per_second": 10801468.79,
109
+ "operations_per_second": 11322463.82,
110
110
  "allocated_objects": 100000,
111
111
  "allocations_per_operation": 1.0
112
112
  },
113
113
  "partition_cursor_sampling": {
114
114
  "iterations": 10,
115
- "seconds": 0.006557,
115
+ "seconds": 0.006506,
116
116
  "sample_seconds": [
117
- 0.001337,
118
- 0.001296,
119
- 0.001314,
120
- 0.001294,
121
- 0.001316
117
+ 0.001315,
118
+ 0.001302,
119
+ 0.001286,
120
+ 0.001301,
121
+ 0.001302
122
122
  ],
123
- "operations_per_second": 7610.35,
123
+ "operations_per_second": 7680.49,
124
124
  "allocated_objects": 250,
125
125
  "allocations_per_operation": 25.0
126
126
  },
127
127
  "snapshot_configuration": {
128
128
  "iterations": 10000,
129
- "seconds": 0.211803,
129
+ "seconds": 0.204242,
130
130
  "sample_seconds": [
131
- 0.043064,
132
- 0.041384,
133
- 0.042928,
134
- 0.042796,
135
- 0.041631
131
+ 0.040722,
132
+ 0.041442,
133
+ 0.04068,
134
+ 0.040386,
135
+ 0.041012
136
136
  ],
137
- "operations_per_second": 233666.7,
138
- "allocated_objects": 170000,
137
+ "operations_per_second": 245567.51,
138
+ "allocated_objects": 170002,
139
139
  "allocations_per_operation": 17.0
140
140
  },
141
141
  "snapshot_empty_export": {
142
142
  "iterations": 20,
143
- "seconds": 0.046427,
143
+ "seconds": 0.055533,
144
144
  "sample_seconds": [
145
- 0.008102,
146
- 0.007108,
147
- 0.008957,
148
- 0.014756,
149
- 0.007504
145
+ 0.009149,
146
+ 0.00743,
147
+ 0.010262,
148
+ 0.020069,
149
+ 0.008623
150
150
  ],
151
- "operations_per_second": 2468.53,
152
- "allocated_objects": 3420,
153
- "allocations_per_operation": 171.0
151
+ "operations_per_second": 2186.03,
152
+ "allocated_objects": 3422,
153
+ "allocations_per_operation": 171.1
154
154
  },
155
155
  "snapshot_raw_batching": {
156
156
  "iterations": 100,
157
- "seconds": 0.059924,
157
+ "seconds": 0.057351,
158
158
  "sample_seconds": [
159
- 0.011836,
160
- 0.012093,
161
- 0.012064,
162
- 0.011962,
163
- 0.011969
159
+ 0.011604,
160
+ 0.011575,
161
+ 0.011411,
162
+ 0.011368,
163
+ 0.011393
164
164
  ],
165
- "operations_per_second": 8354.92,
165
+ "operations_per_second": 8763.47,
166
166
  "allocated_objects": 1800,
167
167
  "allocations_per_operation": 18.0
168
168
  },
169
169
  "snapshot_jsonl_serial_pipeline": {
170
170
  "iterations": 3,
171
- "seconds": 1.593689,
171
+ "seconds": 1.635496,
172
172
  "sample_seconds": [
173
- 0.332539,
174
- 0.320695,
175
- 0.319876,
176
- 0.30983,
177
- 0.310749
173
+ 0.34454,
174
+ 0.325758,
175
+ 0.340644,
176
+ 0.315536,
177
+ 0.309018
178
178
  ],
179
- "operations_per_second": 9.38,
180
- "allocated_objects": 543645,
181
- "allocations_per_operation": 181215.0
179
+ "operations_per_second": 9.21,
180
+ "allocated_objects": 543647,
181
+ "allocations_per_operation": 181215.67
182
182
  },
183
183
  "snapshot_jsonl_pipeline": {
184
184
  "iterations": 3,
185
- "seconds": 0.978552,
185
+ "seconds": 0.986188,
186
186
  "sample_seconds": [
187
- 0.195724,
188
- 0.195973,
189
- 0.19579,
190
- 0.195125,
191
- 0.19594
187
+ 0.197135,
188
+ 0.196786,
189
+ 0.197211,
190
+ 0.197905,
191
+ 0.197151
192
192
  ],
193
- "operations_per_second": 15.32,
194
- "allocated_objects": 543867,
195
- "allocations_per_operation": 181289.0
193
+ "operations_per_second": 15.22,
194
+ "allocated_objects": 543869,
195
+ "allocations_per_operation": 181289.67
196
196
  },
197
197
  "jsonl_batch_write": {
198
198
  "iterations": 100,
199
- "seconds": 0.11875,
199
+ "seconds": 0.111044,
200
200
  "sample_seconds": [
201
- 0.022894,
202
- 0.025733,
203
- 0.021527,
204
- 0.024267,
205
- 0.024329
201
+ 0.021916,
202
+ 0.021842,
203
+ 0.025587,
204
+ 0.020918,
205
+ 0.020781
206
206
  ],
207
- "operations_per_second": 4120.82,
208
- "allocated_objects": 182800,
209
- "allocations_per_operation": 1828.0
207
+ "operations_per_second": 4578.34,
208
+ "allocated_objects": 182802,
209
+ "allocations_per_operation": 1828.02
210
210
  },
211
211
  "jsonl_zstd_batch_write": {
212
212
  "iterations": 50,
213
- "seconds": 0.071127,
213
+ "seconds": 0.06922,
214
214
  "sample_seconds": [
215
- 0.016098,
216
- 0.012678,
217
- 0.013116,
218
- 0.013103,
219
- 0.016132
215
+ 0.01349,
216
+ 0.013059,
217
+ 0.01289,
218
+ 0.013011,
219
+ 0.01677
220
220
  ],
221
- "operations_per_second": 3812.14,
222
- "allocated_objects": 92050,
223
- "allocations_per_operation": 1841.0
221
+ "operations_per_second": 3828.78,
222
+ "allocated_objects": 92052,
223
+ "allocations_per_operation": 1841.04
224
224
  },
225
225
  "jsonl_preencoded_accounting": {
226
226
  "iterations": 10000,
227
- "seconds": 0.155715,
227
+ "seconds": 0.154764,
228
228
  "sample_seconds": [
229
- 0.031233,
230
- 0.03071,
231
- 0.031271,
232
- 0.031395,
233
- 0.031106
229
+ 0.030654,
230
+ 0.031015,
231
+ 0.030894,
232
+ 0.031364,
233
+ 0.030837
234
234
  ],
235
- "operations_per_second": 320174.17,
235
+ "operations_per_second": 323687.45,
236
236
  "allocated_objects": 0,
237
237
  "allocations_per_operation": 0.0
238
238
  },
239
239
  "csv_batch_write": {
240
240
  "iterations": 100,
241
- "seconds": 0.29281,
241
+ "seconds": 0.293376,
242
242
  "sample_seconds": [
243
- 0.057804,
244
- 0.058016,
245
- 0.058681,
246
- 0.059051,
247
- 0.059258
243
+ 0.053863,
244
+ 0.061627,
245
+ 0.056016,
246
+ 0.061445,
247
+ 0.060425
248
248
  ],
249
- "operations_per_second": 1704.13,
250
- "allocated_objects": 363700,
251
- "allocations_per_operation": 3637.0
249
+ "operations_per_second": 1654.94,
250
+ "allocated_objects": 363702,
251
+ "allocations_per_operation": 3637.02
252
252
  },
253
253
  "csv_zstd_batch_write": {
254
254
  "iterations": 50,
255
- "seconds": 0.222587,
255
+ "seconds": 0.233272,
256
256
  "sample_seconds": [
257
- 0.05143,
258
- 0.046745,
259
- 0.038643,
260
- 0.052235,
261
- 0.033534
257
+ 0.049286,
258
+ 0.041579,
259
+ 0.044768,
260
+ 0.04899,
261
+ 0.048649
262
262
  ],
263
- "operations_per_second": 1069.63,
264
- "allocated_objects": 182500,
265
- "allocations_per_operation": 3650.0
263
+ "operations_per_second": 1027.77,
264
+ "allocated_objects": 182502,
265
+ "allocations_per_operation": 3650.04
266
266
  },
267
267
  "csv_gzip_batch_write": {
268
268
  "iterations": 50,
269
- "seconds": 0.148172,
269
+ "seconds": 0.143877,
270
270
  "sample_seconds": [
271
- 0.027482,
272
- 0.034167,
273
- 0.027431,
274
- 0.031337,
275
- 0.027755
271
+ 0.027111,
272
+ 0.026331,
273
+ 0.036606,
274
+ 0.026717,
275
+ 0.027112
276
276
  ],
277
- "operations_per_second": 1801.48,
278
- "allocated_objects": 182550,
279
- "allocations_per_operation": 3651.0
277
+ "operations_per_second": 1844.27,
278
+ "allocated_objects": 182552,
279
+ "allocations_per_operation": 3651.04
280
280
  },
281
281
  "csv_counting_io_write": {
282
282
  "iterations": 100000,
283
- "seconds": 0.041051,
283
+ "seconds": 0.035209,
284
284
  "sample_seconds": [
285
- 0.008018,
286
- 0.008041,
287
- 0.008821,
288
- 0.008223,
289
- 0.007948
285
+ 0.007071,
286
+ 0.007051,
287
+ 0.007134,
288
+ 0.006939,
289
+ 0.007014
290
290
  ],
291
- "operations_per_second": 12436264.16,
291
+ "operations_per_second": 14182385.2,
292
292
  "allocated_objects": 0,
293
293
  "allocations_per_operation": 0.0
294
294
  },
295
295
  "csv_row_serialization": {
296
296
  "iterations": 25000,
297
- "seconds": 0.291323,
297
+ "seconds": 0.279785,
298
298
  "sample_seconds": [
299
- 0.061351,
300
- 0.057502,
301
- 0.057275,
302
- 0.057677,
303
- 0.057518
299
+ 0.054681,
300
+ 0.059591,
301
+ 0.05232,
302
+ 0.059951,
303
+ 0.053242
304
304
  ],
305
- "operations_per_second": 434646.55,
305
+ "operations_per_second": 457197.2,
306
306
  "allocated_objects": 375000,
307
307
  "allocations_per_operation": 15.0
308
308
  },
309
309
  "parquet_buffer_append": {
310
310
  "iterations": 5000,
311
- "seconds": 0.005627,
311
+ "seconds": 0.005628,
312
312
  "sample_seconds": [
313
- 0.001129,
314
- 0.001121,
315
- 0.001122,
316
- 0.001125,
317
- 0.00113
313
+ 0.001098,
314
+ 0.001186,
315
+ 0.001117,
316
+ 0.001106,
317
+ 0.001121
318
318
  ],
319
- "operations_per_second": 4444444.64,
319
+ "operations_per_second": 4476275.24,
320
320
  "allocated_objects": 0,
321
321
  "allocations_per_operation": 0.0
322
322
  },
323
323
  "parquet_object_id_array": {
324
324
  "iterations": 1000,
325
- "seconds": 0.237818,
325
+ "seconds": 0.229308,
326
326
  "sample_seconds": [
327
- 0.048408,
328
- 0.044886,
329
- 0.056985,
330
- 0.043491,
331
- 0.044048
327
+ 0.04412,
328
+ 0.044025,
329
+ 0.046154,
330
+ 0.050956,
331
+ 0.044053
332
332
  ],
333
- "operations_per_second": 22278.66,
333
+ "operations_per_second": 22665.46,
334
334
  "allocated_objects": 64000,
335
335
  "allocations_per_operation": 64.0
336
336
  },
337
337
  "parquet_batch_write": {
338
338
  "iterations": 10,
339
- "seconds": 0.049999,
339
+ "seconds": 0.09166,
340
340
  "sample_seconds": [
341
- 0.010937,
342
- 0.00987,
343
- 0.009626,
344
- 0.00971,
345
- 0.009856
341
+ 0.052028,
342
+ 0.010425,
343
+ 0.00997,
344
+ 0.010049,
345
+ 0.009188
346
346
  ],
347
- "operations_per_second": 1014.61,
348
- "allocated_objects": 18692,
349
- "allocations_per_operation": 1869.2
347
+ "operations_per_second": 995.12,
348
+ "allocated_objects": 19222,
349
+ "allocations_per_operation": 1922.2
350
350
  },
351
351
  "parquet_row_group_write": {
352
352
  "iterations": 2,
353
- "seconds": 0.18688,
353
+ "seconds": 0.186154,
354
354
  "sample_seconds": [
355
- 0.03926,
356
- 0.036353,
357
- 0.037264,
358
- 0.03747,
359
- 0.036533
355
+ 0.036422,
356
+ 0.039406,
357
+ 0.041695,
358
+ 0.034504,
359
+ 0.034127
360
360
  ],
361
- "operations_per_second": 53.67,
362
- "allocated_objects": 68880,
363
- "allocations_per_operation": 34440.0
361
+ "operations_per_second": 54.91,
362
+ "allocated_objects": 68986,
363
+ "allocations_per_operation": 34493.0
364
+ },
365
+ "parquet_read_full_table": {
366
+ "iterations": 10,
367
+ "seconds": 0.030373,
368
+ "sample_seconds": [
369
+ 0.007322,
370
+ 0.005853,
371
+ 0.005759,
372
+ 0.00575,
373
+ 0.005689
374
+ ],
375
+ "operations_per_second": 1736.41,
376
+ "allocated_objects": 380,
377
+ "allocations_per_operation": 38.0
378
+ },
379
+ "parquet_read_row_groups": {
380
+ "iterations": 10,
381
+ "seconds": 0.030306,
382
+ "sample_seconds": [
383
+ 0.005807,
384
+ 0.006112,
385
+ 0.006219,
386
+ 0.006126,
387
+ 0.006042
388
+ ],
389
+ "operations_per_second": 1636.13,
390
+ "allocated_objects": 480,
391
+ "allocations_per_operation": 48.0
392
+ },
393
+ "parquet_read_projection": {
394
+ "iterations": 25,
395
+ "seconds": 0.024149,
396
+ "sample_seconds": [
397
+ 0.00506,
398
+ 0.004772,
399
+ 0.004853,
400
+ 0.004763,
401
+ 0.004701
402
+ ],
403
+ "operations_per_second": 5238.89,
404
+ "allocated_objects": 1425,
405
+ "allocations_per_operation": 57.0
406
+ },
407
+ "parquet_read_category": {
408
+ "iterations": 50,
409
+ "seconds": 0.042582,
410
+ "sample_seconds": [
411
+ 0.008589,
412
+ 0.008581,
413
+ 0.00853,
414
+ 0.008443,
415
+ 0.008439
416
+ ],
417
+ "operations_per_second": 5861.66,
418
+ "allocated_objects": 2000,
419
+ "allocations_per_operation": 40.0
420
+ },
421
+ "parquet_read_date": {
422
+ "iterations": 50,
423
+ "seconds": 0.020498,
424
+ "sample_seconds": [
425
+ 0.004104,
426
+ 0.004145,
427
+ 0.004105,
428
+ 0.004065,
429
+ 0.004079
430
+ ],
431
+ "operations_per_second": 12183.24,
432
+ "allocated_objects": 2000,
433
+ "allocations_per_operation": 40.0
434
+ },
435
+ "parquet_read_flag": {
436
+ "iterations": 50,
437
+ "seconds": 0.020281,
438
+ "sample_seconds": [
439
+ 0.004084,
440
+ 0.004065,
441
+ 0.004033,
442
+ 0.004055,
443
+ 0.004044
444
+ ],
445
+ "operations_per_second": 12330.46,
446
+ "allocated_objects": 2000,
447
+ "allocations_per_operation": 40.0
448
+ },
449
+ "parquet_read_hash_list": {
450
+ "iterations": 50,
451
+ "seconds": 0.073747,
452
+ "sample_seconds": [
453
+ 0.015044,
454
+ 0.01472,
455
+ 0.014526,
456
+ 0.014728,
457
+ 0.014729
458
+ ],
459
+ "operations_per_second": 3394.89,
460
+ "allocated_objects": 2000,
461
+ "allocations_per_operation": 40.0
462
+ },
463
+ "parquet_read_i16": {
464
+ "iterations": 50,
465
+ "seconds": 0.023719,
466
+ "sample_seconds": [
467
+ 0.004704,
468
+ 0.004746,
469
+ 0.004758,
470
+ 0.004741,
471
+ 0.00477
472
+ ],
473
+ "operations_per_second": 10535.19,
474
+ "allocated_objects": 2000,
475
+ "allocations_per_operation": 40.0
476
+ },
477
+ "parquet_read_i32": {
478
+ "iterations": 50,
479
+ "seconds": 0.02468,
480
+ "sample_seconds": [
481
+ 0.004956,
482
+ 0.004946,
483
+ 0.004911,
484
+ 0.004953,
485
+ 0.004914
486
+ ],
487
+ "operations_per_second": 10109.18,
488
+ "allocated_objects": 2000,
489
+ "allocations_per_operation": 40.0
490
+ },
491
+ "parquet_read_i64": {
492
+ "iterations": 50,
493
+ "seconds": 0.032992,
494
+ "sample_seconds": [
495
+ 0.006369,
496
+ 0.00655,
497
+ 0.006616,
498
+ 0.006668,
499
+ 0.006789
500
+ ],
501
+ "operations_per_second": 7557.44,
502
+ "allocated_objects": 2000,
503
+ "allocations_per_operation": 40.0
504
+ },
505
+ "parquet_read_i8": {
506
+ "iterations": 50,
507
+ "seconds": 0.020171,
508
+ "sample_seconds": [
509
+ 0.004047,
510
+ 0.004107,
511
+ 0.004026,
512
+ 0.004013,
513
+ 0.003978
514
+ ],
515
+ "operations_per_second": 12419.27,
516
+ "allocated_objects": 2000,
517
+ "allocations_per_operation": 40.0
518
+ },
519
+ "parquet_read_int_list": {
520
+ "iterations": 50,
521
+ "seconds": 0.060468,
522
+ "sample_seconds": [
523
+ 0.012083,
524
+ 0.012062,
525
+ 0.012169,
526
+ 0.012051,
527
+ 0.012103
528
+ ],
529
+ "operations_per_second": 4138.05,
530
+ "allocated_objects": 2003,
531
+ "allocations_per_operation": 40.06
532
+ },
533
+ "parquet_read_long_text": {
534
+ "iterations": 50,
535
+ "seconds": 0.070799,
536
+ "sample_seconds": [
537
+ 0.014215,
538
+ 0.013996,
539
+ 0.014224,
540
+ 0.014229,
541
+ 0.014135
542
+ ],
543
+ "operations_per_second": 3517.41,
544
+ "allocated_objects": 2000,
545
+ "allocations_per_operation": 40.0
546
+ },
547
+ "parquet_read_mixed_id": {
548
+ "iterations": 50,
549
+ "seconds": 0.048812,
550
+ "sample_seconds": [
551
+ 0.009718,
552
+ 0.009602,
553
+ 0.009872,
554
+ 0.009823,
555
+ 0.009797
556
+ ],
557
+ "operations_per_second": 5103.6,
558
+ "allocated_objects": 2000,
559
+ "allocations_per_operation": 40.0
560
+ },
561
+ "parquet_read_nested": {
562
+ "iterations": 50,
563
+ "seconds": 0.053001,
564
+ "sample_seconds": [
565
+ 0.010346,
566
+ 0.010576,
567
+ 0.010622,
568
+ 0.010742,
569
+ 0.010715
570
+ ],
571
+ "operations_per_second": 4707.21,
572
+ "allocated_objects": 2000,
573
+ "allocations_per_operation": 40.0
574
+ },
575
+ "parquet_read_nested_int_list": {
576
+ "iterations": 50,
577
+ "seconds": 0.070874,
578
+ "sample_seconds": [
579
+ 0.014148,
580
+ 0.014801,
581
+ 0.014101,
582
+ 0.014109,
583
+ 0.013715
584
+ ],
585
+ "operations_per_second": 3543.84,
586
+ "allocated_objects": 2000,
587
+ "allocations_per_operation": 40.0
588
+ },
589
+ "parquet_read_nullable_flag": {
590
+ "iterations": 50,
591
+ "seconds": 0.021453,
592
+ "sample_seconds": [
593
+ 0.004375,
594
+ 0.004264,
595
+ 0.004161,
596
+ 0.004301,
597
+ 0.004352
598
+ ],
599
+ "operations_per_second": 11625.2,
600
+ "allocated_objects": 2000,
601
+ "allocations_per_operation": 40.0
602
+ },
603
+ "parquet_read_nullable_score": {
604
+ "iterations": 50,
605
+ "seconds": 0.036034,
606
+ "sample_seconds": [
607
+ 0.00719,
608
+ 0.007139,
609
+ 0.007193,
610
+ 0.007199,
611
+ 0.007313
612
+ ],
613
+ "operations_per_second": 6951.2,
614
+ "allocated_objects": 2000,
615
+ "allocations_per_operation": 40.0
616
+ },
617
+ "parquet_read_object_id": {
618
+ "iterations": 50,
619
+ "seconds": 0.048042,
620
+ "sample_seconds": [
621
+ 0.009593,
622
+ 0.009638,
623
+ 0.00957,
624
+ 0.009586,
625
+ 0.009655
626
+ ],
627
+ "operations_per_second": 5212.13,
628
+ "allocated_objects": 2000,
629
+ "allocations_per_operation": 40.0
630
+ },
631
+ "parquet_read_object_id_list": {
632
+ "iterations": 50,
633
+ "seconds": 0.077701,
634
+ "sample_seconds": [
635
+ 0.015525,
636
+ 0.015522,
637
+ 0.015646,
638
+ 0.015537,
639
+ 0.015471
640
+ ],
641
+ "operations_per_second": 3220.61,
642
+ "allocated_objects": 2000,
643
+ "allocations_per_operation": 40.0
644
+ },
645
+ "parquet_read_row_id": {
646
+ "iterations": 50,
647
+ "seconds": 0.033483,
648
+ "sample_seconds": [
649
+ 0.006813,
650
+ 0.006827,
651
+ 0.006824,
652
+ 0.00659,
653
+ 0.006429
654
+ ],
655
+ "operations_per_second": 7338.91,
656
+ "allocated_objects": 2000,
657
+ "allocations_per_operation": 40.0
658
+ },
659
+ "parquet_read_score": {
660
+ "iterations": 50,
661
+ "seconds": 0.035378,
662
+ "sample_seconds": [
663
+ 0.007036,
664
+ 0.007053,
665
+ 0.007043,
666
+ 0.00708,
667
+ 0.007166
668
+ ],
669
+ "operations_per_second": 7089.18,
670
+ "allocated_objects": 2000,
671
+ "allocations_per_operation": 40.0
672
+ },
673
+ "parquet_read_string_list": {
674
+ "iterations": 50,
675
+ "seconds": 0.083486,
676
+ "sample_seconds": [
677
+ 0.01681,
678
+ 0.016741,
679
+ 0.016704,
680
+ 0.016365,
681
+ 0.016866
682
+ ],
683
+ "operations_per_second": 2986.68,
684
+ "allocated_objects": 2000,
685
+ "allocations_per_operation": 40.0
686
+ },
687
+ "parquet_read_time": {
688
+ "iterations": 50,
689
+ "seconds": 0.031529,
690
+ "sample_seconds": [
691
+ 0.0063160000000000004,
692
+ 0.0063160000000000004,
693
+ 0.006309,
694
+ 0.00627,
695
+ 0.006318
696
+ ],
697
+ "operations_per_second": 7916.4,
698
+ "allocated_objects": 2000,
699
+ "allocations_per_operation": 40.0
700
+ },
701
+ "parquet_read_u16": {
702
+ "iterations": 50,
703
+ "seconds": 0.023675,
704
+ "sample_seconds": [
705
+ 0.004727,
706
+ 0.004744,
707
+ 0.004724,
708
+ 0.004749,
709
+ 0.004731
710
+ ],
711
+ "operations_per_second": 10568.59,
712
+ "allocated_objects": 2000,
713
+ "allocations_per_operation": 40.0
714
+ },
715
+ "parquet_read_u32": {
716
+ "iterations": 50,
717
+ "seconds": 0.02498,
718
+ "sample_seconds": [
719
+ 0.005005,
720
+ 0.004978,
721
+ 0.004969,
722
+ 0.005033,
723
+ 0.004995
724
+ ],
725
+ "operations_per_second": 10010.01,
726
+ "allocated_objects": 2000,
727
+ "allocations_per_operation": 40.0
728
+ },
729
+ "parquet_read_u64": {
730
+ "iterations": 50,
731
+ "seconds": 0.033401,
732
+ "sample_seconds": [
733
+ 0.006821,
734
+ 0.006775,
735
+ 0.006823,
736
+ 0.006583,
737
+ 0.006399
738
+ ],
739
+ "operations_per_second": 7380.07,
740
+ "allocated_objects": 2000,
741
+ "allocations_per_operation": 40.0
742
+ },
743
+ "parquet_read_u8": {
744
+ "iterations": 50,
745
+ "seconds": 0.019733,
746
+ "sample_seconds": [
747
+ 0.00399,
748
+ 0.003898,
749
+ 0.003976,
750
+ 0.00392,
751
+ 0.003949
752
+ ],
753
+ "operations_per_second": 12661.43,
754
+ "allocated_objects": 2000,
755
+ "allocations_per_operation": 40.0
756
+ },
757
+ "parquet_read_unique_text": {
758
+ "iterations": 50,
759
+ "seconds": 0.072319,
760
+ "sample_seconds": [
761
+ 0.014467,
762
+ 0.014536,
763
+ 0.014379,
764
+ 0.014257,
765
+ 0.01468
766
+ ],
767
+ "operations_per_second": 3456.14,
768
+ "allocated_objects": 2000,
769
+ "allocations_per_operation": 40.0
770
+ },
771
+ "parquet_read_utf8_text": {
772
+ "iterations": 50,
773
+ "seconds": 0.032683,
774
+ "sample_seconds": [
775
+ 0.006359,
776
+ 0.006559,
777
+ 0.00652,
778
+ 0.006601,
779
+ 0.006644
780
+ ],
781
+ "operations_per_second": 7623.11,
782
+ "allocated_objects": 2000,
783
+ "allocations_per_operation": 40.0
364
784
  }
365
785
  }
366
786
  }
data/benchmark/microbench.rb CHANGED
@@ -2,6 +2,7 @@
2
2
  # frozen_string_literal: true
3
3
 
4
4
  require 'bundler/setup'
5
+ require 'date'
5
6
  require 'fileutils'
6
7
  require 'json'
7
8
  require 'stringio'
@@ -28,6 +29,40 @@ PARQUET_DOCS = Array.new(10_000) do |index|
28
29
  'nested' => { 'value' => index }
29
30
  }
30
31
  end.freeze
32
+ PARQUET_READ_DOCS = Array.new(10_000) do |index|
33
+ object_id = BSON::ObjectId.from_string(format('%024x', index + 1))
34
+ nullable = (index % 11).zero?
35
+ {
36
+ 'row_id' => 4_294_967_296 + index,
37
+ 'u8' => index % 256,
38
+ 'u16' => 1_000 + index,
39
+ 'u32' => 1_000_000_000 + index,
40
+ 'u64' => 5_000_000_000 + index,
41
+ 'i8' => (index % 256) - 128,
42
+ 'i16' => index - 32_768,
43
+ 'i32' => index - 2_000_000_000,
44
+ 'i64' => index - 9_000_000_000,
45
+ 'flag' => index.even?,
46
+ 'nullable_flag' => (nullable ? nil : index.even?),
47
+ 'score' => index * 0.25,
48
+ 'nullable_score' => (nullable ? nil : index * 0.5),
49
+ 'category' => "category-#{index % 8}",
50
+ 'unique_text' => format('row-%<index>08d-value-%<encoded>08x',
51
+ index:, encoded: index * 2_654_435_761),
52
+ 'utf8_text' => "日本語-#{index % 32}",
53
+ 'long_text' => "#{format('%08d', index)}-#{'abcdefghijklmnopqrstuvwxyz' * 4}",
54
+ 'object_id' => object_id,
55
+ 'mixed_id' => (index.even? ? object_id : "literal-#{index}"),
56
+ 'nested' => { 'index' => index, 'flag' => index.even? },
57
+ 'date' => Date.new(2020, 1, 1) + (index % 365),
58
+ 'time' => Time.utc(2020, 1, 1) + index,
59
+ 'int_list' => (nullable ? nil : [index, index + 1, index + 2]),
60
+ 'nested_int_list' => (nullable ? nil : [[index, index + 1], [index + 2]]),
61
+ 'string_list' => (nullable ? nil : ["tag-#{index % 16}", "tag-#{(index + 1) % 16}"]),
62
+ 'object_id_list' => (nullable ? nil : [object_id]),
63
+ 'hash_list' => (nullable ? nil : [{ 'index' => index }])
64
+ }
65
+ end.freeze
31
66
  ENCODED_JSONL = DOCS.map { |document| "#{JSON.generate(document)}\n" }.join * 10
32
67
  ENCODED_BATCH = Purplelight::WriterJSONL::EncodedBatch.new(
33
68
  data: ENCODED_JSONL,
@@ -334,6 +369,58 @@ Dir.mktmpdir('purplelight-microbench') do |directory|
334
369
  writer.write_many(PARQUET_DOCS)
335
370
  writer.close
336
371
  end
372
+ parquet_read_writer = Purplelight::WriterParquet.new(
373
+ directory:,
374
+ prefix: 'parquet-read',
375
+ compression: :zstd,
376
+ row_group_size: 10_000,
377
+ single_file: true
378
+ )
379
+ parquet_read_writer.write_many(PARQUET_READ_DOCS)
380
+ parquet_read_writer.close
381
+ parquet_read_path = File.join(directory, 'parquet-read.parquet')
382
+ parquet_schema_reader = Parquet::ArrowFileReader.new(parquet_read_path)
383
+ parquet_column_indices = parquet_schema_reader.schema.fields.each_with_index.to_h do |field, index|
384
+ [field.name, index]
385
+ end.freeze
386
+ parquet_schema_reader.close
387
+
388
+ harness.register('parquet_read_full_table', paths: :writer_parquet, iterations: 10) do
389
+ reader = Parquet::ArrowFileReader.new(parquet_read_path)
390
+ reader.use_threads = true
391
+ table = reader.read_table
392
+ reader.close
393
+ raise 'Parquet full read row mismatch' unless table.n_rows == PARQUET_READ_DOCS.length
394
+ end
395
+
396
+ harness.register('parquet_read_row_groups', paths: :writer_parquet, iterations: 10) do
397
+ reader = Parquet::ArrowFileReader.new(parquet_read_path)
398
+ reader.use_threads = true
399
+ rows = reader.each_row_group.sum(&:n_rows)
400
+ reader.close
401
+ raise 'Parquet row-group read mismatch' unless rows == PARQUET_READ_DOCS.length
402
+ end
403
+
404
+ projected_indices = %w[row_id flag score].map { |name| parquet_column_indices.fetch(name) }.freeze
405
+ harness.register('parquet_read_projection', paths: :writer_parquet, iterations: 25) do
406
+ reader = Parquet::ArrowFileReader.new(parquet_read_path)
407
+ reader.use_threads = true
408
+ rows = reader.n_row_groups.times.sum do |index|
409
+ reader.read_row_group(index, projected_indices).n_rows
410
+ end
411
+ reader.close
412
+ raise 'Parquet projected read mismatch' unless rows == PARQUET_READ_DOCS.length
413
+ end
414
+
415
+ parquet_column_indices.each do |column, index|
416
+ harness.register("parquet_read_#{column}", paths: :writer_parquet, iterations: 50) do
417
+ reader = Parquet::ArrowFileReader.new(parquet_read_path)
418
+ reader.use_threads = true
419
+ values = reader.read_column_data(index)
420
+ reader.close
421
+ raise "Parquet #{column} read mismatch" unless values.length == PARQUET_READ_DOCS.length
422
+ end
423
+ end
337
424
 
338
425
  baseline_index = ARGV.index('--baseline')
339
426
  output_index = ARGV.index('--output')
data/lib/purplelight/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Purplelight
4
- VERSION = '0.1.17'
4
+ VERSION = '0.1.18'
5
5
  end
data/lib/purplelight/writer_parquet.rb CHANGED
@@ -13,6 +13,8 @@ module Purplelight
13
13
  # WriterParquet writes Parquet files via Apache Arrow when available.
14
14
  class WriterParquet
15
15
  DEFAULT_ROW_GROUP_SIZE = 10_000
16
+ DICTIONARY_CARDINALITY_LIMIT = 16
17
+ DICTIONARY_MIN_REPETITIONS = 4
16
18
 
17
19
  def initialize(directory:, prefix:, compression: :zstd, row_group_size: DEFAULT_ROW_GROUP_SIZE, logger: nil,
18
20
  manifest: nil, single_file: true, schema: nil, rotate_rows: nil)
@@ -63,6 +65,8 @@ module Purplelight
63
65
  @buffer_head = 0
64
66
  @arrow_schema = nil
65
67
  @arrow_data_types = nil
68
+ @dictionary_paths = nil
69
+ @dictionary_paths_resolved = false
66
70
  @writer_path = nil
67
71
  end
68
72
 
@@ -79,10 +83,21 @@ module Purplelight
79
83
 
80
84
  def build_record_batch(documents)
81
85
  @columns ||= infer_columns(documents)
86
+ resolve_dictionary_paths = !@dictionary_paths_resolved
87
+ dictionary_paths = [] if resolve_dictionary_paths && @columns.none? { |name| name.include?('.') }
88
+ @dictionary_paths_resolved = true if resolve_dictionary_paths
82
89
  arrays = @columns.each_with_index.map do |name, index|
83
90
  values = documents.map { |document| extract_value(document, name) }
84
- build_arrow_array(values, @arrow_data_types&.at(index))
91
+ array = build_arrow_array(values, @arrow_data_types&.at(index))
92
+ if dictionary_paths
93
+ data_type = array.value_data_type
94
+ path = list_dictionary_path(name, data_type)
95
+ path ||= name if data_type.is_a?(Arrow::StringDataType) && low_cardinality_strings?(values)
96
+ dictionary_paths << path if path
97
+ end
98
+ array
85
99
  end
100
+ @dictionary_paths = dictionary_paths if resolve_dictionary_paths
86
101
  batch = if @arrow_schema
87
102
  Arrow::RecordBatch.new(@arrow_schema, arrays)
88
103
  else
@@ -287,10 +302,44 @@ module Purplelight
287
302
  end
288
303
  end
289
304
 
305
+ def list_dictionary_path(name, data_type)
306
+ return unless data_type.is_a?(Arrow::ListDataType)
307
+
308
+ path = name.dup
309
+ while data_type.is_a?(Arrow::ListDataType)
310
+ path << '.list.element'
311
+ data_type = data_type.field.data_type
312
+ end
313
+ return if data_type.is_a?(Arrow::BooleanDataType) || data_type.is_a?(Arrow::NullDataType)
314
+
315
+ path
316
+ end
317
+
318
+ def low_cardinality_strings?(values)
319
+ first = values.find { |value| !value.nil? }
320
+ return false unless first.is_a?(String)
321
+
322
+ distinct = {}
323
+ count = 0
324
+ values.each do |value|
325
+ next if value.nil?
326
+ return false unless value.is_a?(String)
327
+
328
+ count += 1
329
+ distinct[value] = true
330
+ return false if distinct.length > DICTIONARY_CARDINALITY_LIMIT
331
+ end
332
+ !distinct.empty? && count >= distinct.length * DICTIONARY_MIN_REPETITIONS
333
+ end
334
+
290
335
  def write_record_batch(batch, path)
291
336
  unless @pq_writer
292
337
  properties = build_writer_properties_for_compression(@compression)
293
338
  properties ||= Parquet::WriterProperties.new
339
+ if @dictionary_paths
340
+ properties.disable_dictionary
341
+ @dictionary_paths.each { |column_path| properties.enable_dictionary(column_path) }
342
+ end
294
343
  properties.max_row_group_length = @row_group_size
295
344
  @pq_writer = Parquet::ArrowFileWriter.open(batch.schema, path, properties)
296
345
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: purplelight
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.17
4
+ version: 0.1.18
5
5
  platform: ruby
6
6
  authors:
7
7
  - Alexander Nicholson