purplelight 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b62be6d2a3810b6278d43fadcee4647efb1c758c9bd04ddc69051737e66d1716
-  data.tar.gz: 2950f98c90869bcc3d6619e00adf68113e12197196f43a4c129dab2c1270e47c
+  metadata.gz: '07534009e367f28d3374708991cb870f5fa168ee11a95142af8d357885af7abc'
+  data.tar.gz: e665d587dea94999326c0c42e88d2bcfd99bae01e305aee9e3051d3ddcd266e2
 SHA512:
-  metadata.gz: 635a9c3114bc1d6a017a8244dfd6b5cca15f82f19a9e79f59480af91c0e1ec61b72e40703393098f584a2a9d29f68cac9b81f8833c14ccf8d42f527c4cb40c2c
-  data.tar.gz: f24b2faaff4218481b8fa8345842dd1fadeb9174adb92d449008637e072e81861b13bb8f71cacbcb51254fbcca18ea799ca8dfa5ce5c99f418e82262dd5505c1
+  metadata.gz: e4cabf4d438a8afa0d00902aa07b01320013f4e8588630fb2d5c4f9b2432e1910ac94c19ccce78bdb396a415ac3ea949527b83d22eadfbc036520656c4273869
+  data.tar.gz: b038e1fa40f36e985571019d7b4d7fe9c5013ea17314640f8d484ca9cbbb68292af08c5f91b1a006f34c473a24f3b83b03aece33d333685afb749437ac920ca4

data/README.md

@@ -4,10 +4,18 @@ Snapshot MongoDB collections efficiently from Ruby with resumable, partitioned e
 
 ### Install
 
+Purplelight is published on RubyGems: [purplelight on RubyGems](https://rubygems.org/gems/purplelight).
+
 Add to your Gemfile:
 
 ```ruby
-gem 'purplelight'
+gem 'purplelight', '~> 0.1.2'
+```
+
+Or install directly:
+
+```bash
+gem install purplelight
 ```
 
 ### Quick start
@@ -33,6 +41,60 @@ Purplelight.snapshot(
 )
 ```
 
+### Filtering with `query`
+
+`query` is passed directly to MongoDB as the filter for the collection read. Use standard MongoDB query operators.
+
+Ruby examples:
+
+```ruby
+# Equality
+query: { status: 'active' }
+
+# Ranges
+query: { created_at: { '$gte' => Time.parse('2025-01-01'), '$lt' => Time.parse('2025-02-01') } }
+
+# $in / $nin
+query: { type: { '$in' => %w[user admin] } }
+
+# Nested fields (dot-notation also supported in Mongo)
+query: { 'profile.country' => 'US' }
+
+# By ObjectId boundary (works great with _id partitions)
+query: { _id: { '$gt' => BSON::ObjectId.from_time(Time.utc(2024, 1, 1)) } }
+```
+
+CLI examples (JSON):
+
+```bash
+# Equality
+--query '{"status":"active"}'
+
+# Date/time range (ISO8601 strings your app can parse downstream)
+--query '{"created_at":{"$gte":"2025-01-01T00:00:00Z","$lt":"2025-02-01T00:00:00Z"}}'
+
+# Nested field
+--query '{"profile.country":"US"}'
+
+# IN list
+--query '{"type":{"$in":["user","admin"]}}'
+```
+
+Notes:
+- Ensure values are serializable; when using Ruby, you can pass native `Time`, `BSON::ObjectId`, etc.
+- Consider adding an appropriate index to match your `query` and pass `hint:` to force indexed scans when needed:
+
+```ruby
+Purplelight.snapshot(
+  client: client,
+  collection: 'events',
+  output: '/data/exports',
+  format: :jsonl,
+  query: { created_at: { '$gte' => Time.parse('2025-01-01') } },
+  hint: { created_at: 1 }
+)
+```
+
 Outputs files like:
 
 ```
@@ -42,7 +104,158 @@ Outputs files like:
   users.manifest.json
 ```
 
-### Status
+### CSV usage (single-file)
+
+```ruby
+Purplelight.snapshot(
+  client: client,
+  collection: 'users',
+  output: '/data/exports',
+  format: :csv,
+  sharding: { mode: :single_file, prefix: 'users' },
+  resume: { enabled: true }
+)
+```
+
+### Parquet usage (requires Arrow and Parquet gems)
+
+Add optional dependencies:
+
+```ruby
+# Gemfile
+group :parquet do
+  gem 'red-arrow', '~> 15.0'
+  gem 'red-parquet', '~> 15.0'
+end
+```
+
+Then:
+
+```ruby
+Purplelight.snapshot(
+  client: client,
+  collection: 'users',
+  output: '/data/exports',
+  format: :parquet,
+  sharding: { mode: :single_file, prefix: 'users' },
+  resume: { enabled: true }
+)
+```
+
+### CLI
+
+```bash
+bundle exec bin/purplelight \
+  --uri "$MONGO_URL" \
+  --db mydb --collection users \
+  --output /data/exports \
+  --format jsonl --partitions 8 --by-size $((256*1024*1024)) --prefix users
+```
+
+### Architecture
+
+```mermaid
+flowchart LR
+  A[Partition planner] -->|filters/ranges| B[Reader pool - threads]
+  B -->|batches| C[Byte-bounded queue]
+  C --> D[Serializer]
+  D -->|JSONL/CSV/Parquet| E[Sink with size-based rotation]
+  E --> F[Parts + Manifest]
 
-Phase 1 (JSONL + zstd, partitioning, resume, size-based sharding) in progress.
+  subgraph Concurrency
+    B
+    C
+    D
+  end
 
+  subgraph Resume
+    F -->|checkpoints| A
+  end
+```
+
+Key points:
+- Partitions default to contiguous `_id` ranges with sorted reads and `no_cursor_timeout`.
+- Readers stream batches into a bounded, byte-aware queue to provide backpressure.
+- Writers serialize to JSONL/CSV/Parquet with default zstd compression and rotate by target size.
+- A manifest records parts and per-partition checkpoints for safe resume.
+
+### Tuning for performance
+
+- Partitions: start with `2 × cores` (default). Increase gradually if reads are underutilized; too high can add overhead.
+- Batch size: 2k–10k usually works well. Larger batches reduce cursor roundtrips, but can raise latency/memory.
+- Queue size: increase to 256–512MB to reduce backpressure on readers for fast disks.
+- Compression: use `:zstd` for good ratio; for max speed, try `:gzip` with low level.
+- Rotation size: larger (512MB–1GB) reduces finalize overhead for many parts.
+- Read preference: offload to secondaries or tagged analytics nodes when available.
+
+Benchmarking (optional):
+
+```bash
+# 1M docs benchmark with tunables
+BENCH=1 BENCH_PARTITIONS=16 BENCH_BATCH_SIZE=8000 BENCH_QUEUE_MB=512 BENCH_ROTATE_MB=512 BENCH_COMPRESSION=gzip \
+  bundle exec rspec spec/benchmark_perf_spec.rb --format doc
+```
+
+### Read preference and node pinning
+
+You can direct reads to non-primary members or specific tagged nodes in a replica set (e.g., MongoDB Atlas analytics nodes) via `read_preference`.
+
+Programmatic examples:
+
+```ruby
+# Secondary reads
+Purplelight.snapshot(
+  client: client,
+  collection: 'events',
+  output: '/data/exports',
+  format: :jsonl,
+  read_preference: :secondary
+)
+
+# Pin to tagged nodes (Atlas analytics nodes)
+Purplelight.snapshot(
+  client: client,
+  collection: 'events',
+  output: '/data/exports',
+  format: :jsonl,
+  read_preference: { mode: :secondary, tag_sets: [{ 'nodeType' => 'ANALYTICS' }] }
+)
+```
+
+Notes:
+- `read_preference` accepts a symbol (mode) or a full hash with `mode` and optional `tag_sets`.
+- Use tags that exist on your cluster. Atlas analytics nodes can be targeted with `{ 'nodeType' => 'ANALYTICS' }`.
+
+CLI examples:
+
+```bash
+# Secondary reads
+bundle exec bin/purplelight \
+  --uri "$MONGO_URL" --db mydb --collection events --output /data/exports \
+  --format jsonl --read-preference secondary
+
+# Pin to tagged nodes (Atlas analytics nodes)
+bundle exec bin/purplelight \
+  --uri "$MONGO_URL" --db mydb --collection events --output /data/exports \
+  --format jsonl --read-preference secondary \
+  --read-tags nodeType=ANALYTICS,region=EAST
+
+# Inspect effective read preference without running
+bundle exec bin/purplelight \
+  --uri "$MONGO_URL" --db mydb --collection events --output /tmp \
+  --read-preference secondary --read-tags nodeType=ANALYTICS --dry-run
+```
+
+### Quick Benchmark
+```
+% bash -lc 'BENCH=1 BENCH_PARTITIONS=16 BENCH_BATCH_SIZE=8000 BENCH_QUEUE_MB=512 BENCH_ROTATE_MB=512 BENCH_COMPRESSION=gzip bundle exec rspec spec/benchmark_perf_spec.rb --format doc | cat'
+
+Performance benchmark (1M docs, gated by BENCH=1)
+W, [2025-09-03T16:10:40.437304 #33546]  WARN -- : MONGODB | Error checking 127.0.0.1:27018: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:27018 (for 127.0.0.1:27018 (no TLS)) (on 127.0.0.1:27018)
+Benchmark results:
+  Inserted: 1000000 docs in 8.16s
+  Exported: 1000000 docs in 8.21s
+  Parts:    1, Bytes: 10646279
+  Throughput: 121729.17 docs/s, 1.24 MB/s
+  Settings: partitions=16, batch_size=8000, queue_mb=512, rotate_mb=512, compression=gzip
+```

data/bin/purplelight

@@ -0,0 +1,109 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'json'
+require 'mongo'
+require_relative '../lib/purplelight'
+
+options = {
+  format: :jsonl,
+  compression: :zstd,
+  partitions: nil,
+  batch_size: 2000,
+  output: nil,
+  query: {},
+  sharding: { mode: :by_size, part_bytes: 256 * 1024 * 1024, prefix: nil },
+  resume: { enabled: true },
+  read_preference: nil,
+  read_tags: nil,
+  dry_run: false
+}
+
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: purplelight snapshot [options]"
+
+  opts.on('-u', '--uri URI', 'MongoDB connection URI (required)') { |v| options[:uri] = v }
+  opts.on('-d', '--db NAME', 'Database name (required)') { |v| options[:db] = v }
+  opts.on('-c', '--collection NAME', 'Collection name (required)') { |v| options[:collection] = v }
+  opts.on('-o', '--output PATH', 'Output directory or file (required)') { |v| options[:output] = v }
+  opts.on('-f', '--format FORMAT', 'Format: jsonl|csv|parquet (default jsonl)') { |v| options[:format] = v.to_sym }
+  opts.on('--compression NAME', 'Compression: zstd|gzip|none') { |v| options[:compression] = v.to_sym }
+  opts.on('--partitions N', Integer, 'Number of partitions') { |v| options[:partitions] = v }
+  opts.on('--batch-size N', Integer, 'Mongo batch size (default 2000)') { |v| options[:batch_size] = v }
+  opts.on('--by-size BYTES', Integer, 'Shard by size (bytes); default 268435456') { |v| options[:sharding] = { mode: :by_size, part_bytes: v } }
+  opts.on('--single-file', 'Write a single output file') { options[:sharding] = { mode: :single_file } }
+  opts.on('--prefix NAME', 'Output file prefix') do |v|
+    options[:sharding] ||= {}
+    options[:sharding][:prefix] = v
+  end
+  opts.on('-q', '--query JSON', 'Filter query as JSON') { |v| options[:query] = JSON.parse(v) }
+  opts.on('--read-preference MODE', 'Read preference mode: primary|primary_preferred|secondary|secondary_preferred|nearest') { |v| options[:read_preference] = v.to_sym }
+  opts.on('--read-tags TAGS', 'Comma-separated key=value list to target tagged nodes (e.g., nodeType=ANALYTICS,region=EAST)') do |v|
+    tags = {}
+    v.split(',').each do |pair|
+      k, val = pair.split('=', 2)
+      next if k.nil? || val.nil?
+      tags[k] = val
+    end
+    options[:read_tags] = tags unless tags.empty?
+  end
+  opts.on('--dry-run', 'Parse options and print effective read preference JSON, then exit') { options[:dry_run] = true }
+  opts.on('--version', 'Show version') do
+    puts Purplelight::VERSION
+    exit 0
+  end
+  opts.on('-h', '--help', 'Show help') do
+    puts opts
+    exit 0
+  end
+end
+
+begin
+  parser.parse!(ARGV)
+rescue OptionParser::ParseError => e
+  warn e.message
+  warn parser
+  exit 1
+end
+
+%i[uri db collection output].each do |k|
+  if options[k].nil? || options[k].to_s.empty?
+    warn "Missing required option: --#{k}"
+    warn parser
+    exit 1
+  end
+end
+
+effective_read = nil
+if options[:read_tags]
+  effective_read = { mode: (options[:read_preference] || :secondary), tag_sets: [options[:read_tags]] }
+elsif options[:read_preference]
+  effective_read = { mode: options[:read_preference] }
+end
+
+if options[:dry_run]
+  puts JSON.generate({ read_preference: effective_read })
+  exit 0
+end
+
+client = Mongo::Client.new(options[:uri])
+options[:partitions] ||= (Etc.respond_to?(:nprocessors) ? [Etc.nprocessors * 2, 4].max : 4)
+
+ok = Purplelight.snapshot(
+  client: client.use(options[:db]),
+  collection: options[:collection],
+  output: options[:output],
+  format: options[:format],
+  compression: options[:compression],
+  partitions: options[:partitions],
+  batch_size: options[:batch_size],
+  query: options[:query],
+  sharding: options[:sharding],
+  read_preference: effective_read || options[:read_preference],
+  resume: { enabled: true },
+  on_progress: ->(s) { $stderr.puts("progress: #{s.to_json}") }
+)
+
+exit(ok ? 0 : 1)
+
+

data/lib/purplelight/manifest.rb

@@ -32,6 +32,7 @@ module Purplelight
         'partitions' => []
       }
       @mutex = Mutex.new
+      @last_save_at = Time.now
     end
 
     def self.load(path)
@@ -102,7 +103,7 @@ module Purplelight
         part = @data['parts'][index]
         part['rows'] += rows_delta
         part['bytes'] += bytes_delta
-        save!
+        save_maybe!
       end
     end
 
@@ -122,6 +123,16 @@ module Purplelight
     def partitions
       @data['partitions']
     end
+
+    private
+
+    def save_maybe!(interval_seconds: 2.0)
+      now = Time.now
+      if (now - @last_save_at) >= interval_seconds
+        save!
+        @last_save_at = now
+      end
+    end
   end
 end
 

data/lib/purplelight/snapshot.rb

@@ -6,6 +6,8 @@ require 'fileutils'
 require_relative 'partitioner'
 require_relative 'queue'
 require_relative 'writer_jsonl'
+require_relative 'writer_csv'
+require_relative 'writer_parquet'
 require_relative 'manifest'
 require_relative 'errors'
 
@@ -16,9 +18,9 @@ module Purplelight
       compression: :zstd,
       batch_size: 2_000,
       partitions: [Etc.respond_to?(:nprocessors) ? [Etc.nprocessors * 2, 4].max : 4, 32].min,
-      queue_size_bytes: 128 * 1024 * 1024,
+      queue_size_bytes: 256 * 1024 * 1024,
       rotate_bytes: 256 * 1024 * 1024,
-      read_concern: :majority,
+      read_concern: { level: :majority },
       read_preference: :primary,
       no_cursor_timeout: true
     }
@@ -79,7 +81,7 @@ module Purplelight
                  end
 
       manifest.configure!(collection: @collection.name, format: @format, compression: @compression, query_digest: query_digest, options: {
-        partitions: @partitions, batch_size: @batch_size, rotate_bytes: @rotate_bytes
+        partitions: @partitions, batch_size: @batch_size, rotate_bytes: @rotate_bytes, hint: @hint
       })
       manifest.ensure_partitions!(@partitions)
 
@@ -90,12 +92,18 @@ module Purplelight
       queue = ByteQueue.new(max_bytes: @queue_size_bytes)
 
       # Writer
-      case @format
-      when :jsonl
-        writer = WriterJSONL.new(directory: dir, prefix: prefix, compression: @compression, rotate_bytes: @rotate_bytes, logger: @logger, manifest: manifest)
-      else
-        raise ArgumentError, "format not implemented: #{@format}"
-      end
+      writer = case @format
+               when :jsonl
+                 WriterJSONL.new(directory: dir, prefix: prefix, compression: @compression, rotate_bytes: @rotate_bytes, logger: @logger, manifest: manifest)
+               when :csv
+                 single_file = (@sharding && @sharding[:mode].to_s == 'single_file')
+                 WriterCSV.new(directory: dir, prefix: prefix, compression: @compression, rotate_bytes: @rotate_bytes, logger: @logger, manifest: manifest, single_file: single_file)
+               when :parquet
+                 single_file = (@sharding && @sharding[:mode].to_s == 'single_file')
+                 WriterParquet.new(directory: dir, prefix: prefix, compression: @compression, logger: @logger, manifest: manifest, single_file: single_file)
+               else
+                 raise ArgumentError, "format not implemented: #{@format}"
+               end
 
       # Start reader threads
       readers = partition_filters.each_with_index.map do |pf, idx|
@@ -151,7 +159,7 @@ module Purplelight
     def read_partition(idx:, filter_spec:, queue:, batch_size:, manifest:)
       filter = filter_spec[:filter]
       sort = filter_spec[:sort] || { _id: 1 }
-      hint = filter_spec[:hint] || { _id: 1 }
+      hint = @hint || filter_spec[:hint] || { _id: 1 }
 
       # Resume from checkpoint if present
       checkpoint = manifest.partitions[idx] && manifest.partitions[idx]['last_id_exclusive']
@@ -164,11 +172,18 @@ module Purplelight
       opts[:projection] = @projection if @projection
       opts[:batch_size] = batch_size if batch_size
       opts[:no_cursor_timeout] = @no_cursor_timeout
-      opts[:read] = { mode: @read_preference }
-      opts[:read_concern] = @read_concern
+      # Read preference can be a symbol (mode) or a full hash with tag_sets
+      if @read_preference
+        opts[:read] = @read_preference.is_a?(Hash) ? @read_preference : { mode: @read_preference }
+      end
+      # Mongo driver expects read_concern as a hash like { level: :majority }
+      if @read_concern
+        opts[:read_concern] = @read_concern.is_a?(Hash) ? @read_concern : { level: @read_concern }
+      end
 
       cursor = @collection.find(filter, opts)
 
+      encode_lines = (@format == :jsonl)
       buffer = []
       buffer_bytes = 0
       last_id = checkpoint
@@ -176,9 +191,15 @@ module Purplelight
         cursor.each do |doc|
           last_id = doc['_id']
           doc = @mapper.call(doc) if @mapper
-          json = Oj.dump(doc, mode: :compat)
-          bytes = json.bytesize + 1 # newline later
-          buffer << doc
+          if encode_lines
+            line = Oj.dump(doc, mode: :compat) + "\n"
+            bytes = line.bytesize
+            buffer << line
+          else
+            # For CSV/Parquet keep raw docs to allow schema/row building
+            bytes = (Oj.dump(doc, mode: :compat).bytesize + 1)
+            buffer << doc
+          end
           buffer_bytes += bytes
           if buffer.length >= batch_size || buffer_bytes >= 1_000_000
             queue.push(buffer, bytes: buffer_bytes)

data/lib/purplelight/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Purplelight
-  VERSION = "0.1.0"
+  VERSION = "0.1.2"
 end
 
 

data/lib/purplelight/writer_csv.rb

@@ -0,0 +1,180 @@
+# frozen_string_literal: true
+
+require 'csv'
+require 'oj'
+require 'zlib'
+require 'fileutils'
+
+begin
+  require 'zstds'
+rescue LoadError
+end
+
+module Purplelight
+  class WriterCSV
+    DEFAULT_ROTATE_BYTES = 256 * 1024 * 1024
+
+    def initialize(directory:, prefix:, compression: :zstd, rotate_bytes: DEFAULT_ROTATE_BYTES, logger: nil, manifest: nil, single_file: false, columns: nil, headers: true)
+      @directory = directory
+      @prefix = prefix
+      @compression = compression
+      @rotate_bytes = rotate_bytes
+      @logger = logger
+      @manifest = manifest
+      @single_file = single_file
+
+      @columns = columns&.map(&:to_s)
+      @headers = headers
+
+      @part_index = nil
+      @io = nil
+      @csv = nil
+      @bytes_written = 0
+      @rows_written = 0
+      @file_seq = 0
+      @closed = false
+
+      @effective_compression = determine_effective_compression(@compression)
+      if @effective_compression.to_s != @compression.to_s
+        @logger&.warn("requested compression '#{@compression}' not available; using '#{@effective_compression}'")
+      end
+    end
+
+    def write_many(array_of_docs)
+      ensure_open!
+
+      # infer columns if needed from docs
+      if @columns.nil?
+        sample_docs = array_of_docs.is_a?(Array) ? array_of_docs : []
+        sample_docs = sample_docs.reject { |d| d.is_a?(String) }
+        @columns = infer_columns(sample_docs)
+        @csv << @columns if @headers
+      end
+
+      array_of_docs.each do |doc|
+        next if doc.is_a?(String)
+        row = @columns.map { |k| extract_value(doc, k) }
+        @csv << row
+        @rows_written += 1
+      end
+      @manifest&.add_progress_to_part!(index: @part_index, rows_delta: array_of_docs.size, bytes_delta: 0)
+
+      rotate_if_needed
+    end
+
+    def rotate_if_needed
+      return if @single_file
+      return if @rotate_bytes.nil?
+      raw_bytes = @io.respond_to?(:pos) ? @io.pos : @bytes_written
+      return if raw_bytes < @rotate_bytes
+      rotate!
+    end
+
+    def close
+      return if @closed
+      if @csv
+        @csv.flush
+      end
+      if @io
+        finalize_current_part!
+        @io.close
+      end
+      @closed = true
+    end
+
+    private
+
+    def ensure_open!
+      return if @io
+      FileUtils.mkdir_p(@directory)
+      path = next_part_path
+      @part_index = @manifest&.open_part!(path) if @manifest
+      raw = File.open(path, 'wb')
+      @io = build_compressed_io(raw)
+      @csv = CSV.new(@io)
+      @bytes_written = 0
+      @rows_written = 0
+    end
+
+    def build_compressed_io(raw)
+      case @effective_compression.to_s
+      when 'zstd'
+        if defined?(ZSTDS)
+          return ZSTDS::Writer.open(raw, level: 10)
+        else
+          @logger&.warn("zstd gem not loaded; using gzip")
+          return Zlib::GzipWriter.new(raw)
+        end
+      when 'gzip'
+        return Zlib::GzipWriter.new(raw)
+      when 'none'
+        return raw
+      else
+        raise ArgumentError, "unknown compression: #{@effective_compression}"
+      end
+    end
+
+    def rotate!
+      return unless @io
+      finalize_current_part!
+      @io.close
+      @io = nil
+      @csv = nil
+      ensure_open!
+    end
+
+    def finalize_current_part!
+      # Avoid flushing compressed writer explicitly to prevent Zlib::BufError; close will finish the stream.
+      @manifest&.complete_part!(index: @part_index, checksum: nil)
+      @file_seq += 1 unless @single_file
+    end
+
+    def next_part_path
+      ext = 'csv'
+      if @single_file
+        filename = format("%s.%s", @prefix, ext)
+      else
+        filename = format("%s-part-%06d.%s", @prefix, @file_seq, ext)
+      end
+      filename += ".zst" if @effective_compression.to_s == 'zstd'
+      filename += ".gz" if @effective_compression.to_s == 'gzip'
+      File.join(@directory, filename)
+    end
+
+    def determine_effective_compression(requested)
+      case requested.to_s
+      when 'zstd'
+        return (defined?(ZSTDS) ? :zstd : :gzip)
+      when 'gzip'
+        return :gzip
+      when 'none'
+        return :none
+      else
+        return :gzip
+      end
+    end
+
+    def infer_columns(docs)
+      keys = {}
+      docs.each do |d|
+        (d.keys - ['_id']).each { |k| keys[k.to_s] = true }
+      end
+      # Put _id first if present, then other keys sorted
+      cols = []
+      cols << '_id' if docs.first.key?('_id') || docs.first.key?(:_id)
+      cols + keys.keys.sort
+    end
+
+    def extract_value(doc, key)
+      val = doc[key] || doc[key.to_sym]
+      case val
+      when Hash, Array
+        Oj.dump(val, mode: :compat)
+      else
+        val
+      end
+    end
+  end
+end
+
+

data/lib/purplelight/writer_jsonl.rb

@@ -14,13 +14,14 @@ module Purplelight
   class WriterJSONL
     DEFAULT_ROTATE_BYTES = 256 * 1024 * 1024
 
-    def initialize(directory:, prefix:, compression: :zstd, rotate_bytes: DEFAULT_ROTATE_BYTES, logger: nil, manifest: nil)
+    def initialize(directory:, prefix:, compression: :zstd, rotate_bytes: DEFAULT_ROTATE_BYTES, logger: nil, manifest: nil, compression_level: nil)
       @directory = directory
       @prefix = prefix
       @compression = compression
       @rotate_bytes = rotate_bytes
       @logger = logger
       @manifest = manifest
+      @compression_level = compression_level
 
       @part_index = nil
       @io = nil
@@ -28,14 +29,26 @@ module Purplelight
       @rows_written = 0
       @file_seq = 0
       @closed = false
+
+      @effective_compression = determine_effective_compression(@compression)
+      if @effective_compression.to_s != @compression.to_s
+        @logger&.warn("requested compression '#{@compression}' not available; using '#{@effective_compression}'")
+      end
     end
 
     def write_many(array_of_docs)
       ensure_open!
-      buffer = array_of_docs.map { |doc| Oj.dump(doc, mode: :compat) + "\n" }.join
+      # If upstream already produced newline-terminated strings, join fast.
+      if array_of_docs.first.is_a?(String)
+        buffer = array_of_docs.join
+        rows = array_of_docs.size
+      else
+        buffer = array_of_docs.map { |doc| Oj.dump(doc, mode: :compat) + "\n" }.join
+        rows = array_of_docs.size
+      end
       write_buffer(buffer)
-      @rows_written += array_of_docs.size
-      @manifest&.add_progress_to_part!(index: @part_index, rows_delta: array_of_docs.size, bytes_delta: buffer.bytesize)
+      @rows_written += rows
+      @manifest&.add_progress_to_part!(index: @part_index, rows_delta: rows, bytes_delta: buffer.bytesize)
     end
 
     def rotate_if_needed
@@ -67,17 +80,20 @@ module Purplelight
     end
 
     def build_compressed_io(raw)
-      case @compression.to_s
+      case @effective_compression.to_s
       when 'zstd'
         if defined?(ZSTDS)
           # ZSTDS::Writer supports IO-like interface
-          return ZSTDS::Writer.open(raw, level: 10)
+          level = @compression_level || 3
+          return ZSTDS::Writer.open(raw, level: level)
         else
-          @logger&.warn("zstd not available, falling back to gzip")
-          return Zlib::GzipWriter.new(raw)
+          @logger&.warn("zstd gem not loaded; this should have been handled earlier")
+          level = @compression_level || Zlib::DEFAULT_COMPRESSION
+          return Zlib::GzipWriter.new(raw, level)
         end
       when 'gzip'
-        return Zlib::GzipWriter.new(raw)
+        level = @compression_level || 1
+        return Zlib::GzipWriter.new(raw, level)
       when 'none'
         return raw
       else
@@ -109,10 +125,23 @@ module Purplelight
     def next_part_path
       ext = 'jsonl'
       filename = format("%s-part-%06d.%s", @prefix, @file_seq, ext)
-      filename += ".zst" if @compression.to_s == 'zstd'
-      filename += ".gz" if @compression.to_s == 'gzip'
+      filename += ".zst" if @effective_compression.to_s == 'zstd'
+      filename += ".gz" if @effective_compression.to_s == 'gzip'
       File.join(@directory, filename)
     end
+
+    def determine_effective_compression(requested)
+      case requested.to_s
+      when 'zstd'
+        return (defined?(ZSTDS) ? :zstd : :gzip)
+      when 'gzip'
+        return :gzip
+      when 'none'
+        return :none
+      else
+        return :gzip
+      end
+    end
   end
 end
 

data/lib/purplelight/writer_parquet.rb

@@ -0,0 +1,137 @@
+# frozen_string_literal: true
+
+begin
+  require 'arrow'
+  require 'parquet'
+rescue LoadError
+  # Arrow/Parquet not available; writer will refuse to run
+end
+
+require 'fileutils'
+
+module Purplelight
+  class WriterParquet
+    DEFAULT_ROW_GROUP_SIZE = 10_000
+
+    def initialize(directory:, prefix:, compression: :zstd, row_group_size: DEFAULT_ROW_GROUP_SIZE, logger: nil, manifest: nil, single_file: true, schema: nil)
+      @directory = directory
+      @prefix = prefix
+      @compression = compression
+      @row_group_size = row_group_size
+      @logger = logger
+      @manifest = manifest
+      @single_file = single_file
+      @schema = schema
+
+      @closed = false
+      @file_seq = 0
+      @part_index = nil
+
+      ensure_dependencies!
+      reset_buffers
+    end
+
+    def write_many(array_of_docs)
+      ensure_open!
+      array_of_docs.each { |doc| @buffer_docs << doc }
+      @manifest&.add_progress_to_part!(index: @part_index, rows_delta: array_of_docs.length, bytes_delta: 0)
+    end
+
+    def close
+      return if @closed
+      ensure_open!
+      if !@buffer_docs.empty?
+        table = build_table(@buffer_docs)
+        write_table(table, @writer_path, append: false)
+      end
+      finalize_current_part!
+      @closed = true
+    end
+
+    private
+
+    def ensure_dependencies!
+      unless defined?(Arrow) && defined?(Parquet)
+        raise ArgumentError, "Parquet support requires gems: red-arrow and red-parquet. Add them to your Gemfile."
+      end
+    end
+
+    def reset_buffers
+      @buffer_docs = []
+      @columns = nil
+      @writer_path = nil
+    end
+
+    def ensure_open!
+      return if @writer_path
+      FileUtils.mkdir_p(@directory)
+      @writer_path = next_part_path
+      @part_index = @manifest&.open_part!(@writer_path) if @manifest
+    end
+
+    # No-op; we now write once on close for simplicity
+
+    def build_table(docs)
+      # Infer columns
+      @columns ||= infer_columns(docs)
+      columns = {}
+      @columns.each do |name|
+        values = docs.map { |d| extract_value(d, name) }
+        columns[name] = Arrow::ArrayBuilder.build(values)
+      end
+      Arrow::Table.new(columns)
+    end
+
+    def write_table(table, path, append: false)
+      # Prefer Arrow's save with explicit parquet format; compression defaults per build.
+      if table.respond_to?(:save)
+        table.save(path, format: :parquet)
+        return
+      end
+      # Fallback to red-parquet writer
+      if defined?(Parquet::ArrowFileWriter)
+        writer = Parquet::ArrowFileWriter.open(table.schema, path)
+        writer.write_table(table)
+        writer.close
+        return
+      end
+      raise "Parquet writer not available in this environment"
+    end
+
+    def finalize_current_part!
+      @manifest&.complete_part!(index: @part_index, checksum: nil)
+      @file_seq += 1 unless @single_file
+      @writer_path = nil
+    end
+
+    def next_part_path
+      ext = 'parquet'
+      filename = if @single_file
+                   format("%s.%s", @prefix, ext)
+                 else
+                   format("%s-part-%06d.%s", @prefix, @file_seq, ext)
+                 end
+      File.join(@directory, filename)
+    end
+
+    def infer_columns(docs)
+      keys = {}
+      docs.each do |d|
+        d.keys.each { |k| keys[k.to_s] = true }
+      end
+      keys.keys.sort
+    end
+
+    def extract_value(doc, key)
+      val = doc[key] || doc[key.to_sym]
+      case val
+      when Time
+        val
+      else
+        val
+      end
+    end
+  end
+end
+
+

metadata

@@ -1,10 +1,10 @@
 --- !ruby/object:Gem::Specification
 name: purplelight
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.2
 platform: ruby
 authors:
-- Purplelight Authors
+- Alexander Nicholson
 bindir: bin
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
@@ -37,6 +37,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '3.16'
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -68,13 +96,15 @@ dependencies:
 description: High-throughput, resumable snapshots of MongoDB collections with partitioning,
   multi-threaded readers, and size-based sharded outputs.
 email:
-- devnull@example.com
-executables: []
+- rubygems-maint@ctrl.tokyo
+executables:
+- purplelight
 extensions: []
 extra_rdoc_files: []
 files:
 - README.md
 - Rakefile
+- bin/purplelight
 - lib/purplelight.rb
 - lib/purplelight/errors.rb
 - lib/purplelight/manifest.rb
@@ -82,13 +112,15 @@ files:
 - lib/purplelight/queue.rb
 - lib/purplelight/snapshot.rb
 - lib/purplelight/version.rb
+- lib/purplelight/writer_csv.rb
 - lib/purplelight/writer_jsonl.rb
+- lib/purplelight/writer_parquet.rb
 licenses:
 - MIT
 metadata:
-  homepage_uri: https://github.com/example/purplelight
-  source_code_uri: https://github.com/example/purplelight
-  changelog_uri: https://github.com/example/purplelight/releases
+  homepage_uri: https://github.com/alexandernicholson/purplelight
+  source_code_uri: https://github.com/alexandernicholson/purplelight
+  changelog_uri: https://github.com/alexandernicholson/purplelight/releases
 rdoc_options: []
 require_paths:
 - lib