zlight_csv 0.4.0-x86_64-linux → 0.6.0-x86_64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6b96428d44c1acb33466a227063e7318cdfdd4233464f533a832accac4e0abf4
-  data.tar.gz: 633c2c530b5ddfd75dc1e006ba345e253c1f4f41e503f0094575421e6d92375f
+  metadata.gz: 4b9429002bef342c43edd29b084d759f6b2609e77295f7af997f47e783a39eea
+  data.tar.gz: 5216c05e5db7d8011cb4afa85505599a6c460b93537d521d6a47a352ee158f5e
 SHA512:
-  metadata.gz: 89b7b97b16cbae21d906f990cf5440ebb3438e403e0a20f02dda8a13a5c302e868de5e5d649e97c362623dbde050c225eea28ad20c424a511ed883d3fcc22d7b
-  data.tar.gz: f164fd92db98090061b241cb81da26fb9fda646d244d6d63afac992f5bfcc66f059aefb1e499883ce446d995f3e5597af3c927b8882043462f695b3e116495c1
+  metadata.gz: d67d65564a2910eb4e2b06d9666c8db4d8e95a6da610458941a78e58f77db7681fd8a1ed3c37a7c797a2661ae19d4ffe5d47917f8ac08d7fa98f2aea7e6017e1
+  data.tar.gz: 24dd4e179f55d42683892c572313388840143c2eeeed32d10cf7a15589596b13cd71ee9c9440a67e55deac6029ee272d5df06748530d239ff26d1ae55a22be43

data/ARCHITECTURE.md

@@ -0,0 +1,132 @@
+# Architecture
+
+This is a guide for changing the code, written on the assumption that you know
+Ruby or Python and may not know Rust. You do not need to read Rust fluently to
+find your way around — the layout is designed so that a given change lives in
+one predictable place.
+
+## The short version
+
+ZLight is a **Ruby gem with its hot loop written in Rust**. Ruby handles the
+public API and file conveniences; Rust does the parsing and writing.
+
+```
+your code
+    │
+    ▼
+lib/zlight_csv.rb        Ruby: the public API
+    │                    read / write / foreach / open are plain Ruby
+    │                    parse / generate / stream / stream_file are not —
+    ▼                    they are implemented in Rust
+ext/zlight_csv/src/      Rust: the parser, the writer, the streaming reader
+    │
+    ▼
+the `csv` crate          the actual CSV state machine
+```
+
+The `magnus` crate is the bridge: it is what lets Rust define a module Ruby can
+call and build Ruby objects (`Hash`, `Array`, `String`) from Rust data.
+
+## Which file does what
+
+### Ruby side — `lib/`
+
+| File | Owns |
+|---|---|
+| `zlight_csv.rb` | the public API and its documentation |
+| `zlight_csv/native.rb` | loading the compiled library |
+| `zlight_csv/errors.rb` | the exception hierarchy |
+| `zlight_csv/version.rb` | the version, read from the `VERSION` file |
+
+`native.rb` is worth understanding once. A compiled extension is built against
+one specific Ruby version's internals, so a precompiled gem ships **one library
+per Ruby minor version**:
+
+```
+lib/zlight_csv/3.1/zlight_csv.so
+lib/zlight_csv/3.4/zlight_csv.so
+```
+
+`native.rb` picks the right one at load time. A gem built from source has a
+single library instead, which is the fallback path.
+
+### Rust side — `ext/zlight_csv/src/`
+
+| File | Owns | Analogy |
+|---|---|---|
+| `lib.rs` | the list of what Ruby can call | a routes file |
+| `options.rs` | keyword arguments → typed settings | parsing params |
+| `convert.rs` | one field → one Ruby value | a type coercer |
+| `row.rs` | one record → one Ruby row | a serializer |
+| `reader.rs` | building the CSV reader | a factory |
+| `read/eager.rs` | `ZLight.parse` | |
+| `read/stream.rs` | `ZLight::StreamReader` | |
+| `write/mod.rs` | `ZLight.generate` | |
+| `error.rs` | Rust errors → Ruby exceptions | an error handler |
+
+The important structural point: **both read paths sit on the same three
+helpers.** `read/eager.rs` and `read/stream.rs` differ only in *when* they read
+— all at once, or a row at a time — and both build their reader with
+`reader.rs`, convert fields with `convert.rs`, and shape rows with `row.rs`.
+That is what stops the two from drifting apart.
+
+## Where to put a change
+
+| You want to… | Change | Why there |
+|---|---|---|
+| add a field type (dates, booleans, a Ruby proc) | `convert.rs` + one arm in `options.rs` | `Converter` is the only thing that turns bytes into values |
+| change what a row looks like (a Row object, a struct) | `row.rs` | `RowShape` is the only thing that builds rows |
+| add a parse or write option | `options.rs` | both readers and the writer read their settings from here |
+| add a whole new Ruby method | a file under `read/` or `write/`, plus one line in `lib.rs` | `lib.rs` is the complete list of the public surface |
+| change an error's Ruby class | `error.rs` | one table maps every Rust error to a Ruby class |
+
+If a change seems to require editing two files that do the same kind of work,
+that is a sign the seam is in the wrong place — say so rather than copying the
+code.
+
+## Two ideas that are easy to miss
+
+**Conversion is a value, not a flag.** `Converter` is an enum — think of it as
+a small sealed class hierarchy. It used to be a boolean (`convert_numeric`)
+threaded through every function, which meant a second converter could not be
+added without changing every signature it passed through. Now adding one is a
+new variant plus an arm in `options.rs`.
+
+**Row shape is a value too.** `RowShape` is either `Hashes(headers)` or
+`Arrays`. Deciding once, up front, means the per-row code does not re-ask "do
+we have headers?" for every row, and there is exactly one place that knows how
+a row is built.
+
+## Working on it
+
+You do not need Ruby or Rust installed on your machine — everything runs in
+containers:
+
+```bash
+./docker/test-matrix.sh        # every supported Ruby, 3.1 through 4.0
+./docker/test-matrix.sh 3.4    # just one
+```
+
+With a local Ruby (3.1–4.0) and a Rust toolchain:
+
+```bash
+bundle install
+bundle exec rake compile   # build the extension
+bundle exec rake spec      # run the suite
+bundle exec rake clobber   # remove every build artifact, cargo cache included
+```
+
+`rake compile` must be re-run after any change under `ext/`. Ruby-only changes
+take effect immediately.
+
+## The rule the tests enforce
+
+The RSpec suite is the contract. A refactor that changes behaviour will fail
+it; a refactor that does not, will not. If you find yourself editing a spec to
+make a refactor pass, you are changing behaviour — stop and decide whether you
+meant to.
+
+Several examples deliberately pin places where ZLight **differs** from Ruby's
+stdlib CSV (duplicate headers, empty fields, numeric edge cases). They assert
+both sides on purpose, so the documented differences and the code cannot drift
+apart. See the "Differences from stdlib CSV" section of the README.

data/CHANGELOG.md

@@ -7,22 +7,159 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.1.0] - 2024-XX-XX
+## [0.6.0] - 2026-09-12
+
+### Added
+
+- Custom converters. `converters:` now accepts any object answering `call`, a
+  built-in name, or an Array mixing the two, applied left to right:
+
+  ```ruby
+  ZLight.parse(csv, converters: ->(field) { field.strip })
+  ZLight.parse(csv, converters: [->(f) { f.strip }, :numeric])
+  ```
+
+  Semantics match Ruby's CSV, including that the chain stops as soon as a
+  converter returns something other than a String, and that converters are
+  never applied to the header row. An unusable converter raises ArgumentError
+  naming the problem, where Ruby's CSV raises NoMethodError from inside itself.
+
+  A converter must not read from or close the reader it is converting for, and
+  a reader running a converter must not be used from another thread. Either
+  raises `ZLight::Error` rather than corrupting a row. `ZLight.parse` is safe
+  from any thread, with or without a converter; Ractors are refused by Ruby.
+
+### Changed
+
+- The extension is organised around three seams — field conversion, row shape
+  and reader construction — each owned by one file and shared by both read
+  paths, so a feature is added in one place rather than two. See
+  ARCHITECTURE.md
+- Removed comments that restated the code they sat above, keeping those that
+  explain a decision, an invariant, or where new code belongs
+
+### Fixed
+
+- The documentation for `parse`, `generate` and `StreamReader` was attached to
+  no method, so YARD rendered none of it
+
+## [0.5.1] - 2026-09-12
+
+### Fixed
+
+- The extension now builds against a Ruby that rb-sys has no prebuilt bindings
+  for, via rb-sys `stable-api-compiled-fallback`, rather than failing outright.
+  This is the mechanism that breaks the build on every new Ruby release until
+  rb-sys catches up
+- Windows CI could not build the extension at all: the runner paired a MinGW
+  Ruby with an MSVC Rust toolchain. Ruby and Rust are now set up together so
+  their ABIs match, and Windows is verified across Ruby 3.1-4.0
+- Specs that asserted Unix-only error messages and paths now pass on Windows
+- Dropped an unreachable credentials check from the release workflow: the
+  credentials action fails first, so the check never ran
+
+### Note
+
+- 0.5.0 was tagged but never reached RubyGems: its release run built every gem
+  and then failed to publish, because trusted publishing still named the
+  project's former GitHub account. 0.5.1 is the first release carrying the
+  0.5.0 changes below.
+
+## [0.5.0] - 2026-09-12
+
+### Added
+
+- Ruby 4.0 support
+- `docker/test-matrix.sh`, which builds and tests the extension against every
+  supported Ruby in containers
+- Windows, Ruby 4.0 and a `ruby-head` early-warning job in CI, plus weekly
+  scheduled runs and Dependabot
+
+### Changed
+
+- Upgraded magnus from 0.6 to 0.8. magnus 0.6 read a field of Ruby's
+  RTypedData struct that Ruby 4.0 removed, so the extension could not compile
+  against it at all
+- Minimum Ruby is now 3.1. Ruby 3.0 is end-of-life and no longer supported by
+  magnus
+- Repository URLs now point at `codebyisaad/zlight`, which is where the code
+  is hosted
+
+### Fixed
+
+- The source gem shipped no extension and could not be compiled, so any Ruby
+  or platform without a prebuilt binary installed a gem that failed to load
+- `ZLight::ParseError`, `ZLight::EncodingError` and `ZLight::StreamClosedError`
+  were documented but never raised
+- Integers outside the i64 range were silently converted to lossy Floats
+- `col_sep` and `quote_char` longer than one byte were silently truncated when
+  reading, while writing rejected them
+- Releases no longer yank every previously published version
+
+
+## [0.4.0] - 2026-06-06
+
+### Added
+
+- `ZLight.generate` — build a CSV string from an array of hashes or an array
+  of arrays, with `headers`, `col_sep`, `quote_char` and `force_quotes` options
+- `ZLight.write` — generate a CSV and write it to a file in one call
+
+### Changed
+
+- Parsed fields are tagged with the encoding of the input string, matching
+  Ruby's stdlib CSV, so the parser and writer round-trip without
+  `force_encoding`
+
+## [0.3.0] - 2026-05-21
+
+### Changed
+
+- Release pipeline updates
+
+## [0.2.5] - 2026-05-21
+
+### Added
+
+- `ZLight.stream` and `ZLight.stream_file` — read rows one at a time without
+  loading the whole input into memory
+- `ZLight.open` — streaming with a block, closing the reader automatically
+- `ZLight::StreamReader`, including `Enumerable`, so `lazy`, `select`, `find`
+  and friends work over a stream
+
+### Fixed
+
+- Packaging and gemspec metadata corrections across 0.2.1–0.2.5
+
+## [0.2.0] - 2026-05-21
+
+### Added
+
+- Benchmark suite comparing ZLight against Ruby's stdlib CSV
+- Edge case coverage for quoting, encodings and flexible records
+
+## [0.1.2] - 2026-05-19
 
 ### Added
 
 - Initial release
-- `Zlight::Csv.parse` - Parse CSV strings with optional headers and numeric conversion
-- `Zlight::Csv.read` - Read and parse CSV files
-- `Zlight::Csv.foreach` - Iterate over CSV rows
-- `Zlight.parse` and `Zlight.read` convenience methods
-- Support for custom delimiters (`col_sep`)
-- Support for custom quote characters (`quote_char`)
-- Flexible record length support
-- Enum-based error handling with descriptive messages
-- Cross-platform native gem builds
+- `ZLight.parse` — parse CSV strings, with `headers`, `converters`, `col_sep`,
+  `quote_char` and `flexible` options
+- `ZLight.read` — read and parse a CSV file
+- `ZLight.foreach` — iterate over parsed rows
+- Cross-platform precompiled native gems
 
 ### Performance
 
-- Up to 10x faster than Ruby's standard CSV library
-- Efficient memory usage through Rust's zero-copy parsing
+- Substantially faster than Ruby's stdlib CSV; see the README for measured
+  figures
+
+[Unreleased]: https://github.com/codebyisaad/zlight/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/codebyisaad/zlight/compare/v0.5.1...v0.6.0
+[0.5.1]: https://github.com/codebyisaad/zlight/compare/v0.5.0...v0.5.1
+[0.5.0]: https://github.com/codebyisaad/zlight/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/codebyisaad/zlight/compare/v0.3.0...v0.4.0
+[0.3.0]: https://github.com/codebyisaad/zlight/compare/v0.2.5...v0.3.0
+[0.2.5]: https://github.com/codebyisaad/zlight/compare/v0.2.0...v0.2.5
+[0.2.0]: https://github.com/codebyisaad/zlight/compare/v0.1.2...v0.2.0
+[0.1.2]: https://github.com/codebyisaad/zlight/releases/tag/v0.1.2

data/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Zaidan Chaudhary
+Copyright (c) 2024 Zaidan Chaudhary (also known as Saad Chaudhary)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -7,7 +7,8 @@ A fast CSV parser for Ruby, powered by Rust.
 
 ## Why ZLight?
 
-Ruby's built-in CSV library is slow. ZLight parses CSV files **up to 30x faster** by using Rust under the hood.
+Ruby's built-in CSV library is slow. ZLight parses CSV files **up to 40x faster** by using Rust under the hood.
+The gap is widest on small inputs and narrows as they grow; the measurements below show the range.
 
 ### Benchmark Results
 
@@ -102,12 +103,64 @@ ZLight.parse (full):       73ms
 ZLight.stream (lazy):     0.3ms  ← 5,600x faster!
 ```
 
+### Custom Converters
+
+`converters:` takes a built-in name, any object answering `call`, or an Array
+of them applied left to right:
+
+```ruby
+# A single callable, given each field as a String
+ZLight.parse(csv, converters: ->(field) { field.strip })
+
+# Chained: trim, then recognise numbers
+ZLight.parse(csv, converters: [->(f) { f.strip }, :numeric])
+
+# Anything answering #call works
+ZLight.parse(csv, converters: MyConverter.new)
+```
+
+Two rules, both matching Ruby's CSV:
+
+- **The chain stops** as soon as a converter returns something other than a
+  String, so `[:numeric, other]` never calls `other` for a field that was
+  converted to a number.
+- **Converters never apply to the header row.** Headers are always symbols.
+
+An exception raised inside a converter propagates to the caller. A converter
+that is neither a known name nor callable raises `ArgumentError` naming the
+problem.
+
+A converter must not use the reader it is converting for — calling `next_row`,
+`headers` or `close` on it raises `ZLight::Error`. Any other reader, and
+`ZLight.parse` itself, are fine:
+
+```ruby
+# Raises ZLight::Error
+reader = ZLight.stream(csv, converters: ->(f) { reader.next_row })
+
+# Fine
+ZLight.parse(csv, converters: ->(f) { ZLight.parse(lookup_csv).first[:name] })
+```
+
+### Threads and Ractors
+
+| | |
+|---|---|
+| `ZLight.parse`, `.generate`, `.read`, `.write` | safe from any thread |
+| A `StreamReader` shared between threads | safe while no converter is running |
+| A `StreamReader` shared between threads, with a converter | raises `ZLight::Error` |
+| Ractors | not supported; Ruby refuses to cross the boundary |
+
+A converter runs Ruby code, which lets another thread take the GVL mid-row. The
+reader refuses that rather than returning a corrupted row. **Give each thread
+its own reader** and the question does not arise.
+
 ### Options (Reading)
 
 | Option | Default | Description |
 |--------|---------|-------------|
 | `headers` | `true` | Use first row as headers (returns hashes). Set `false` for arrays. |
-| `converters` | `nil` | Set to `:numeric` to convert numbers automatically |
+| `converters` | `nil` | `:numeric`, any object answering `call`, or an Array of them |
 | `col_sep` | `","` | Column separator (`"\t"` for TSV, `";"` for European CSV) |
 | `quote_char` | `"` | Quote character |
 | `flexible` | `true` | Allow rows with varying column counts |
@@ -156,21 +209,70 @@ CSV.parse(data, headers: true, header_converters: :symbol, converters: :numeric)
 ZLight.parse(data, converters: :numeric)
 ```
 
+### Differences from stdlib CSV
+
+ZLight is not a complete reimplementation of `CSV`. The differences below are
+deliberate, and are the ones most likely to matter when migrating.
+
+**API shape**
+
+- `ZLight.foreach` takes a **CSV string**, while `CSV.foreach` takes a **file
+  path**. Use `ZLight.open` to iterate a file.
+- `ZLight.foreach` parses the whole input before yielding. For genuinely lazy
+  iteration use `ZLight.stream` or `ZLight.open`.
+- `ZLight::StreamReader` is single-pass. It includes `Enumerable`, but each
+  row is consumed as it is read, so a second pass yields nothing and there is
+  no rewind. Call `ZLight.stream` again to re-read.
+
+**Parsing**
+
+- Duplicate headers collapse. Rows are `Hash`es, so `"a,a"` keeps only the
+  last `:a` column; `CSV` keeps both.
+- Fields beyond the header count are dropped rather than collected.
+- Headers become symbols always, equivalent to `header_converters: :symbol`.
+
+**`converters: :numeric`**
+
+| Input        | ZLight            | Ruby CSV   |
+|--------------|-------------------|------------|
+| `""`         | `""`              | `nil`      |
+| `"0x10"`     | `"0x10"`          | `16`       |
+| `"1_000"`    | `"1_000"`         | `1000`     |
+| `"Infinity"` | `Float::INFINITY` | `"Infinity"` |
+| `"NaN"`      | `Float::NAN`      | `"NaN"`    |
+
+Integers of any size are exact, as in `CSV`.
+
+**Writing**
+
+- `ZLight.generate` takes its column order from the keys of the **first**
+  hash. Keys that appear only in later rows are not written; keys missing
+  from a later row are written as empty fields.
+
 ## Requirements
 
-- Ruby 3.0+
+- Ruby 3.1 or newer, including Ruby 4.0
 - Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), or Windows (x64)
 
+Every supported Ruby is built and tested on each change; see
+`docker/test-matrix.sh` to run that matrix yourself. Rubies without a
+precompiled binary install from the source gem and need a Rust toolchain.
+
 ## Roadmap
 
 - [x] Streaming/lazy parsing for large files
 - [x] CSV writing support
-- [ ] Custom converter procs
+- [x] Custom converter procs
+- [ ] Streaming writer for large output
 
 ## Contributing
 
-Bug reports and pull requests are welcome on [GitHub](https://github.com/zaidanch/zlight).
+Bug reports and pull requests are welcome on [GitHub](https://github.com/codebyisaad/zlight).
+
+## Author
+
+Zaidan Chaudhary, who also publishes as Saad Chaudhary.
 
 ## License
 
-MIT
+MIT. See [LICENSE](LICENSE).

data/VERSION

@@ -1 +1 @@
-0.4.0
+0.6.0

data/ext/zlight_csv/Cargo.toml

@@ -0,0 +1,30 @@
+[package]
+name = "zlight_csv"
+# Synced from ../../VERSION by extconf.rb — do not edit manually.
+version = "0.6.0"
+edition = "2021"
+authors = ["Zaidan Chaudhary", "Saad Chaudhary"]
+description = "High-performance CSV parser for Ruby, powered by Rust"
+license = "MIT"
+repository = "https://github.com/codebyisaad/zlight"
+
+[lib]
+crate-type = ["cdylib"]
+path = "src/lib.rs"
+
+[dependencies]
+csv = "1"
+magnus = "0.8"
+# Depended on directly, only to turn on stable-api-compiled-fallback.
+# rb-sys ships prebuilt bindings for Ruby versions it knows about and fails
+# outright on ones it does not, which is what breaks the build on every new
+# Ruby before rb-sys catches up. The fallback compiles the shim from the
+# headers of whatever Ruby is present instead, so an unreleased Ruby builds.
+rb-sys = { version = "0.9", features = ["stable-api-compiled-fallback"] }
+thiserror = "1"
+
+[profile.release]
+lto = true
+codegen-units = 1
+opt-level = 3
+strip = true

data/ext/zlight_csv/extconf.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require "mkmf"
+require "rb_sys/mkmf"
+
+def sync_cargo_version!
+  version = File.read(File.expand_path("../../VERSION", __dir__), encoding: "UTF-8").strip
+  cargo_toml = File.join(__dir__, "Cargo.toml")
+  contents = File.read(cargo_toml, encoding: "UTF-8")
+  updated = contents.sub(/^version = ".*"$/, %(version = "#{version}"))
+  File.write(cargo_toml, updated) if contents != updated
+end
+
+sync_cargo_version!
+
+create_rust_makefile("zlight_csv/zlight_csv")