zlight_csv 0.5.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: be8e072a58943ca8b89b8f68c0943ad9daf80be6edb6274fad3f8fe7a20af34a
-  data.tar.gz: 5bf977f61545af25293a5e2b3560ee5e06df896b373a4771252d9a9dc482451a
+  metadata.gz: 7f7e500659c606d547762f3eb7a39b074686e6e68e8a9318ec97aa6af50e5905
+  data.tar.gz: 1b0283e7036596faa5d148edec0edbe700fd90d20f9c75e65e287fe3686fd538
 SHA512:
-  metadata.gz: fb006274ac24756d4336d27647343b64b6c753f6b6a5d4c35357b354558d379f8548dbef8825092c59ec9d8c4eb2431561177decb506816d22553f8d5a3eb10a
-  data.tar.gz: 88fff6de78e745e818cf22a3e92f3772550824e2453d93bc7e864a59d03a7131dafe1a4ccb7116086b30161559b65ce0f1696f236c0d339078dfa47acbd222c6
+  metadata.gz: e0a54efcc8e8f478fd8d6c95de46e16e903c506afe93aa2384b91f53a85fb3ebdb00d359963aa2ab26af1c9cc771122e9506453ab80b440213d97404bd6ad58f
+  data.tar.gz: a6017c7cb1e64c66279a7280c41dc57719559b455d9319e3497808edae213c1cbf5c3c99022984060ac0616561d64f2cb11b993c3b4eaca7e7beee45918d2223

data/ARCHITECTURE.md

@@ -0,0 +1,132 @@
+# Architecture
+
+This is a guide for changing the code, written on the assumption that you know
+Ruby or Python and may not know Rust. You do not need to read Rust fluently to
+find your way around — the layout is designed so that a given change lives in
+one predictable place.
+
+## The short version
+
+ZLight is a **Ruby gem with its hot loop written in Rust**. Ruby handles the
+public API and file conveniences; Rust does the parsing and writing.
+
+```
+your code
+    │
+    ▼
+lib/zlight_csv.rb        Ruby: the public API
+    │                    read / write / foreach / open are plain Ruby
+    │                    parse / generate / stream / stream_file are not —
+    ▼                    they are implemented in Rust
+ext/zlight_csv/src/      Rust: the parser, the writer, the streaming reader
+    │
+    ▼
+the `csv` crate          the actual CSV state machine
+```
+
+The `magnus` crate is the bridge: it is what lets Rust define a module Ruby can
+call and build Ruby objects (`Hash`, `Array`, `String`) from Rust data.
+
+## Which file does what
+
+### Ruby side — `lib/`
+
+| File | Owns |
+|---|---|
+| `zlight_csv.rb` | the public API and its documentation |
+| `zlight_csv/native.rb` | loading the compiled library |
+| `zlight_csv/errors.rb` | the exception hierarchy |
+| `zlight_csv/version.rb` | the version, read from the `VERSION` file |
+
+`native.rb` is worth understanding once. A compiled extension is built against
+one specific Ruby version's internals, so a precompiled gem ships **one library
+per Ruby minor version**:
+
+```
+lib/zlight_csv/3.1/zlight_csv.so
+lib/zlight_csv/3.4/zlight_csv.so
+```
+
+`native.rb` picks the right one at load time. A gem built from source has a
+single library instead, which is the fallback path.
+
+### Rust side — `ext/zlight_csv/src/`
+
+| File | Owns | Analogy |
+|---|---|---|
+| `lib.rs` | the list of what Ruby can call | a routes file |
+| `options.rs` | keyword arguments → typed settings | parsing params |
+| `convert.rs` | one field → one Ruby value | a type coercer |
+| `row.rs` | one record → one Ruby row | a serializer |
+| `reader.rs` | building the CSV reader | a factory |
+| `read/eager.rs` | `ZLight.parse` | |
+| `read/stream.rs` | `ZLight::StreamReader` | |
+| `write/mod.rs` | `ZLight.generate` | |
+| `error.rs` | Rust errors → Ruby exceptions | an error handler |
+
+The important structural point: **both read paths sit on the same three
+helpers.** `read/eager.rs` and `read/stream.rs` differ only in *when* they read
+— all at once, or a row at a time — and both build their reader with
+`reader.rs`, convert fields with `convert.rs`, and shape rows with `row.rs`.
+That is what stops the two from drifting apart.
+
+## Where to put a change
+
+| You want to… | Change | Why there |
+|---|---|---|
+| add a field type (dates, booleans, a Ruby proc) | `convert.rs` + one arm in `options.rs` | `Converter` is the only thing that turns bytes into values |
+| change what a row looks like (a Row object, a struct) | `row.rs` | `RowShape` is the only thing that builds rows |
+| add a parse or write option | `options.rs` | both readers and the writer read their settings from here |
+| add a whole new Ruby method | a file under `read/` or `write/`, plus one line in `lib.rs` | `lib.rs` is the complete list of the public surface |
+| change an error's Ruby class | `error.rs` | one table maps every Rust error to a Ruby class |
+
+If a change seems to require editing two files that do the same kind of work,
+that is a sign the seam is in the wrong place — say so rather than copying the
+code.
+
+## Two ideas that are easy to miss
+
+**Conversion is a value, not a flag.** `Converter` is an enum — think of it as
+a small sealed class hierarchy. It used to be a boolean (`convert_numeric`)
+threaded through every function, which meant a second converter could not be
+added without changing every signature it passed through. Now adding one is a
+new variant plus an arm in `options.rs`.
+
+**Row shape is a value too.** `RowShape` is either `Hashes(headers)` or
+`Arrays`. Deciding once, up front, means the per-row code does not re-ask "do
+we have headers?" for every row, and there is exactly one place that knows how
+a row is built.
+
+## Working on it
+
+You do not need Ruby or Rust installed on your machine — everything runs in
+containers:
+
+```bash
+./docker/test-matrix.sh        # every supported Ruby, 3.1 through 4.0
+./docker/test-matrix.sh 3.4    # just one
+```
+
+With a local Ruby (3.1–4.0) and a Rust toolchain:
+
+```bash
+bundle install
+bundle exec rake compile   # build the extension
+bundle exec rake spec      # run the suite
+bundle exec rake clobber   # remove every build artifact, cargo cache included
+```
+
+`rake compile` must be re-run after any change under `ext/`. Ruby-only changes
+take effect immediately.
+
+## The rule the tests enforce
+
+The RSpec suite is the contract. A refactor that changes behaviour will fail
+it; a refactor that does not, will not. If you find yourself editing a spec to
+make a refactor pass, you are changing behaviour — stop and decide whether you
+meant to.
+
+Several examples deliberately pin places where ZLight **differs** from Ruby's
+stdlib CSV (duplicate headers, empty fields, numeric edge cases). They assert
+both sides on purpose, so the documented differences and the code cannot drift
+apart. See the "Differences from stdlib CSV" section of the README.

data/CHANGELOG.md

@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-09-12
+
+### Added
+
+- Custom converters. `converters:` now accepts any object answering `call`, a
+  built-in name, or an Array mixing the two, applied left to right:
+
+  ```ruby
+  ZLight.parse(csv, converters: ->(field) { field.strip })
+  ZLight.parse(csv, converters: [->(f) { f.strip }, :numeric])
+  ```
+
+  Semantics match Ruby's CSV, including that the chain stops as soon as a
+  converter returns something other than a String, and that converters are
+  never applied to the header row. An unusable converter raises ArgumentError
+  naming the problem, where Ruby's CSV raises NoMethodError from inside itself.
+
+  A converter must not read from or close the reader it is converting for, and
+  a reader running a converter must not be used from another thread. Either
+  raises `ZLight::Error` rather than corrupting a row. `ZLight.parse` is safe
+  from any thread, with or without a converter; Ractors are refused by Ruby.
+
+### Changed
+
+- The extension is organised around three seams — field conversion, row shape
+  and reader construction — each owned by one file and shared by both read
+  paths, so a feature is added in one place rather than two. See
+  ARCHITECTURE.md
+- Removed comments that restated the code they sat above, keeping those that
+  explain a decision, an invariant, or where new code belongs
+
+### Fixed
+
+- The documentation for `parse`, `generate` and `StreamReader` was attached to
+  no method, so YARD rendered none of it
+
 ## [0.5.1] - 2026-09-12
 
 ### Fixed
@@ -118,7 +154,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Substantially faster than Ruby's stdlib CSV; see the README for measured
   figures
 
-[Unreleased]: https://github.com/codebyisaad/zlight/compare/v0.5.1...HEAD
+[Unreleased]: https://github.com/codebyisaad/zlight/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/codebyisaad/zlight/compare/v0.5.1...v0.6.0
 [0.5.1]: https://github.com/codebyisaad/zlight/compare/v0.5.0...v0.5.1
 [0.5.0]: https://github.com/codebyisaad/zlight/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/codebyisaad/zlight/compare/v0.3.0...v0.4.0

data/README.md

@@ -103,12 +103,64 @@ ZLight.parse (full):       73ms
 ZLight.stream (lazy):     0.3ms  ← 5,600x faster!
 ```
 
+### Custom Converters
+
+`converters:` takes a built-in name, any object answering `call`, or an Array
+of them applied left to right:
+
+```ruby
+# A single callable, given each field as a String
+ZLight.parse(csv, converters: ->(field) { field.strip })
+
+# Chained: trim, then recognise numbers
+ZLight.parse(csv, converters: [->(f) { f.strip }, :numeric])
+
+# Anything answering #call works
+ZLight.parse(csv, converters: MyConverter.new)
+```
+
+Two rules, both matching Ruby's CSV:
+
+- **The chain stops** as soon as a converter returns something other than a
+  String, so `[:numeric, other]` never calls `other` for a field that was
+  converted to a number.
+- **Converters never apply to the header row.** Headers are always symbols.
+
+An exception raised inside a converter propagates to the caller. A converter
+that is neither a known name nor callable raises `ArgumentError` naming the
+problem.
+
+A converter must not use the reader it is converting for — calling `next_row`,
+`headers` or `close` on it raises `ZLight::Error`. Any other reader, and
+`ZLight.parse` itself, are fine:
+
+```ruby
+# Raises ZLight::Error
+reader = ZLight.stream(csv, converters: ->(f) { reader.next_row })
+
+# Fine
+ZLight.parse(csv, converters: ->(f) { ZLight.parse(lookup_csv).first[:name] })
+```
+
+### Threads and Ractors
+
+| | |
+|---|---|
+| `ZLight.parse`, `.generate`, `.read`, `.write` | safe from any thread |
+| A `StreamReader` shared between threads | safe while no converter is running |
+| A `StreamReader` shared between threads, with a converter | raises `ZLight::Error` |
+| Ractors | not supported; Ruby refuses to cross the boundary |
+
+A converter runs Ruby code, which lets another thread take the GVL mid-row. The
+reader refuses that rather than returning a corrupted row. **Give each thread
+its own reader** and the question does not arise.
+
 ### Options (Reading)
 
 | Option | Default | Description |
 |--------|---------|-------------|
 | `headers` | `true` | Use first row as headers (returns hashes). Set `false` for arrays. |
-| `converters` | `nil` | Set to `:numeric` to convert numbers automatically |
+| `converters` | `nil` | `:numeric`, any object answering `call`, or an Array of them |
 | `col_sep` | `","` | Column separator (`"\t"` for TSV, `";"` for European CSV) |
 | `quote_char` | `"` | Quote character |
 | `flexible` | `true` | Allow rows with varying column counts |
@@ -210,7 +262,8 @@ precompiled binary install from the source gem and need a Rust toolchain.
 
 - [x] Streaming/lazy parsing for large files
 - [x] CSV writing support
-- [ ] Custom converter procs
+- [x] Custom converter procs
+- [ ] Streaming writer for large output
 
 ## Contributing
 

data/VERSION

@@ -1 +1 @@
-0.5.1
+0.6.0

data/ext/zlight_csv/src/convert.rs

@@ -0,0 +1,203 @@
+//! Turning a raw CSV field into a Ruby value.
+//!
+//! # Where new field types go
+//!
+//! `Converter` is the only thing that decides what a `&[u8]` field becomes. To
+//! support a new kind of value, add a `Step` variant here and map the option to
+//! it in `options.rs`; the read paths call `Converter::apply` and nothing else.
+
+use magnus::{
+    encoding::Index, gc, prelude::*, Error as MagnusError, RString, Ruby, Value,
+};
+
+/// One stage of a conversion chain.
+#[derive(Debug, Clone)]
+pub enum Step {
+    /// Recognise integers and floats, leaving anything else a String.
+    Numeric,
+    /// Any Ruby object answering `call`, given the field and returning its
+    /// replacement.
+    Callable(Value),
+}
+
+/// How each field is turned into a Ruby value.
+///
+/// An empty chain means every field stays a String, which is the default.
+#[derive(Debug, Clone, Default)]
+pub struct Converter {
+    steps: Vec<Step>,
+}
+
+impl Converter {
+    pub fn new(steps: Vec<Step>) -> Self {
+        Self { steps }
+    }
+
+    /// Converts one field, running each step in turn.
+    ///
+    /// Matching Ruby's CSV, the chain stops as soon as a step returns anything
+    /// other than a String: a converter that has already produced a typed value
+    /// is not handed to the next converter.
+    pub fn apply(
+        &self,
+        ruby: &Ruby,
+        field: &[u8],
+        encoding: Index,
+    ) -> Result<Value, MagnusError> {
+        let mut steps = self.steps.iter();
+
+        // The first step reads the bytes directly, so `converters: :numeric`
+        // never builds a Ruby String for a field it is about to replace.
+        let Some(first) = steps.next() else {
+            return Ok(string(ruby, field, encoding));
+        };
+
+        let mut value = match first {
+            Step::Numeric => numeric(ruby, field).unwrap_or_else(|| string(ruby, field, encoding)),
+            Step::Callable(callable) => {
+                callable.funcall("call", (string(ruby, field, encoding),))?
+            }
+        };
+
+        for step in steps {
+            let Some(text) = RString::from_value(value) else {
+                break;
+            };
+
+            value = match step {
+                Step::Numeric => {
+                    // Copied because the conversion below allocates, and an
+                    // embedded Ruby string may move when it does.
+                    let bytes = unsafe { text.as_slice() }.to_vec();
+                    numeric(ruby, &bytes).unwrap_or(value)
+                }
+                Step::Callable(callable) => callable.funcall("call", (value,))?,
+            };
+        }
+
+        Ok(value)
+    }
+
+    /// Keeps any Ruby callables in the chain alive.
+    ///
+    /// A `StreamReader` outlives the call that created it, so a proc passed as
+    /// a converter is reachable only through this struct. Without marking it,
+    /// Ruby could collect it and the reader would call freed memory.
+    pub fn mark(&self, marker: &gc::Marker) {
+        for step in &self.steps {
+            if let Step::Callable(value) = step {
+                marker.mark(*value);
+            }
+        }
+    }
+}
+
+fn string(ruby: &Ruby, field: &[u8], encoding: Index) -> Value {
+    ruby.enc_str_new(field, encoding).as_value()
+}
+
+/// Scans for `.`, `e` or `E` in one pass to decide whether the integer fast
+/// path is worth trying.
+#[inline(always)]
+fn looks_like_float(bytes: &[u8]) -> bool {
+    bytes.iter().any(|&b| b == b'.' || b == b'e' || b == b'E')
+}
+
+#[inline(always)]
+fn trim_ascii(bytes: &[u8]) -> &[u8] {
+    let start = bytes
+        .iter()
+        .position(|&b| !b.is_ascii_whitespace())
+        .unwrap_or(bytes.len());
+    let end = bytes
+        .iter()
+        .rposition(|&b| !b.is_ascii_whitespace())
+        .map_or(start, |i| i + 1);
+    &bytes[start..end]
+}
+
+enum IntScan {
+    Fits(i64),
+    /// A valid decimal integer, but too large in magnitude for an i64.
+    TooLarge,
+    NotAnInteger,
+}
+
+/// Reads an optionally signed run of ASCII digits without validating UTF-8.
+///
+/// Overflow is reported rather than rejected so the caller can fall back to an
+/// exact Ruby Integer instead of an inexact float.
+#[inline(always)]
+fn scan_int(bytes: &[u8]) -> IntScan {
+    if bytes.is_empty() {
+        return IntScan::NotAnInteger;
+    }
+
+    let digits = match bytes[0] {
+        b'-' | b'+' => &bytes[1..],
+        _ => bytes,
+    };
+    let negative = bytes[0] == b'-';
+
+    if digits.is_empty() {
+        return IntScan::NotAnInteger;
+    }
+
+    let mut magnitude: i64 = 0;
+    let mut overflowed = false;
+
+    for &b in digits {
+        if !b.is_ascii_digit() {
+            return IntScan::NotAnInteger;
+        }
+        if overflowed {
+            continue;
+        }
+        match magnitude
+            .checked_mul(10)
+            .and_then(|m| m.checked_add((b - b'0') as i64))
+        {
+            Some(next) => magnitude = next,
+            None => overflowed = true,
+        }
+    }
+
+    if overflowed {
+        IntScan::TooLarge
+    } else if negative {
+        IntScan::Fits(-magnitude)
+    } else {
+        IntScan::Fits(magnitude)
+    }
+}
+
+/// Builds an exact Ruby Integer from digits too large for an i64.
+///
+/// `scan_int` has already established the slice is ASCII digits with an
+/// optional sign, which is what makes `String#to_i` safe here.
+fn big_integer(ruby: &Ruby, digits: &[u8]) -> Option<Value> {
+    let text = std::str::from_utf8(digits).ok()?;
+    ruby.str_new(text).funcall("to_i", ()).ok()
+}
+
+/// Returns the numeric value of a field, or `None` if it is not a number.
+fn numeric(ruby: &Ruby, field: &[u8]) -> Option<Value> {
+    let trimmed = trim_ascii(field);
+    if trimmed.is_empty() {
+        return None;
+    }
+
+    if !looks_like_float(trimmed) {
+        match scan_int(trimmed) {
+            IntScan::Fits(i) => return Some(ruby.integer_from_i64(i).as_value()),
+            IntScan::TooLarge => return big_integer(ruby, trimmed),
+            IntScan::NotAnInteger => {}
+        }
+    }
+
+    std::str::from_utf8(trimmed)
+        .ok()?
+        .parse::<f64>()
+        .ok()
+        .map(|f| ruby.float_from_f64(f).as_value())
+}

data/ext/zlight_csv/src/error.rs

@@ -1,7 +1,6 @@
 use magnus::{prelude::*, Error as MagnusError, ExceptionClass, RModule, Ruby};
 use thiserror::Error;
 
-/// Enumerated error types for ZlightCsv operations.
 #[derive(Debug, Error)]
 pub enum ZlightError {
     #[error("CSV parsing error: {0}")]
@@ -25,11 +24,16 @@ pub enum ZlightError {
 
     #[error("Stream reader has been closed")]
     StreamClosed,
+
+    #[error(
+        "Stream reader is already in use. Either a converter used the reader it \
+         was converting for, or another thread used it while a converter was \
+         running. Give each thread its own reader."
+    )]
+    ReaderBusy,
 }
 
 impl ZlightError {
-    /// Converts to a Ruby exception with the appropriate exception class.
-    ///
     /// Requires the Ruby handle, which is always available here: an error is
     /// only converted on its way out of an extension method, and those run on
     /// a Ruby thread holding the GVL. Building an exception needs a class, and
@@ -56,6 +60,7 @@ impl ZlightError {
             ZlightError::CsvParse(_) => Some("ParseError"),
             ZlightError::InvalidHeaderEncoding(_) => Some("EncodingError"),
             ZlightError::StreamClosed => Some("StreamClosedError"),
+            ZlightError::ReaderBusy => Some("Error"),
             ZlightError::MissingArgument(_)
             | ZlightError::InvalidOption { .. }
             | ZlightError::Io(_) => None,
@@ -72,6 +77,7 @@ impl ZlightError {
                 ruby.exception_arg_error()
             }
             ZlightError::Io(_) | ZlightError::StreamClosed => ruby.exception_io_error(),
+            ZlightError::ReaderBusy => ruby.exception_runtime_error(),
         }
     }
 }

data/ext/zlight_csv/src/lib.rs

@@ -1,55 +1,57 @@
-mod converter;
+//! ZLight — a CSV parser and writer for Ruby, implemented in Rust.
+//!
+//! # Reading this codebase
+//!
+//! Every file has one job, and the layering is deliberate so that adding a
+//! feature touches one place rather than several:
+//!
+//! | Module        | Owns                                             |
+//! |---------------|--------------------------------------------------|
+//! | `lib.rs`      | what Ruby can call — the registration map, below |
+//! | `options`     | Ruby keyword arguments → typed settings          |
+//! | `convert`     | one field → one Ruby value                       |
+//! | `row`         | one record → one Ruby row (Hash or Array)        |
+//! | `reader`      | building the underlying `csv::Reader`            |
+//! | `read::eager` | `ZLight.parse`, whole input at once              |
+//! | `read::stream`| `ZLight::StreamReader`, one row at a time        |
+//! | `write`       | `ZLight.generate`                                |
+//! | `error`       | Rust errors → Ruby exception classes             |
+//!
+//! The two read paths sit on top of the same `reader`, `row` and `convert`,
+//! which is what stops them drifting apart. If you are adding something:
+//!
+//! * a new field type (dates, booleans, a Ruby proc) → `convert.rs`
+//! * a new result shape (a Row object, a struct) → `row.rs`
+//! * a new option → `options.rs`
+//! * a new Ruby method → a module under `read`/`write`, plus one line in
+//!   [`init`] so Ruby can see it
+//!
+//! See ARCHITECTURE.md for the same map written for someone who does not read
+//! Rust.
+
+mod convert;
 mod error;
 mod options;
-mod parser;
-mod stream;
-mod writer;
+mod read;
+mod reader;
+mod row;
+mod write;
 
-use magnus::{function, prelude::*, Error, RArray, Ruby, Value};
+use magnus::{function, prelude::*, Error, Ruby};
 
-use options::ParseOptions;
-use parser::{build_reader, parse_as_arrays, parse_as_hashes};
-
-fn parse(ruby: &Ruby, args: &[Value]) -> Result<RArray, Error> {
-    let (input, options) = ParseOptions::scan(ruby, args, "csv_string")?;
-    // Capture the input's encoding so parsed fields are tagged consistently
-    // (matching Ruby's stdlib CSV) rather than defaulting to BINARY.
-    let encoding = input.enc_get();
-
-    // SAFETY: `as_slice` borrows Ruby's own buffer, so Ruby must neither free
-    // nor move the string while `bytes` is live. It cannot be freed: `input`
-    // came from the argument list and stays reachable for the whole call.
-    // Parsing it without copying is the point of this extension; a copy here
-    // would double peak memory on large inputs.
-    //
-    // `input` is deliberately kept alive past the parse below, rather than
-    // ending its borrow at this line, so that Ruby's conservative stack scan
-    // still sees the string while the parser holds a pointer into it.
-    let bytes = unsafe { input.as_slice() };
-
-    let mut reader = build_reader(bytes, &options);
-
-    let result = if options.has_headers {
-        parse_as_hashes(ruby, &mut reader, options.convert_numeric, encoding)
-    } else {
-        parse_as_arrays(ruby, &mut reader, options.convert_numeric, encoding)
-    };
-
-    // Keeps `input` live until parsing has finished; see the note above.
-    std::hint::black_box(input);
-
-    result
-}
-
-/// Initializes the Ruby extension.
+/// Registers everything Ruby can call.
+///
+/// This function is the complete list of the extension's public surface; if a
+/// method is not named here, Ruby cannot reach it.
 #[magnus::init]
 fn init(ruby: &Ruby) -> Result<(), Error> {
-    let module = ruby.define_module("ZLight")?;
-    module.define_singleton_method("parse", function!(parse, -1))?;
-    module.define_singleton_method("generate", function!(writer::generate, -1))?;
+    let zlight = ruby.define_module("ZLight")?;
+
+    zlight.define_singleton_method("parse", function!(read::eager::parse, -1))?;
+    zlight.define_singleton_method("generate", function!(write::generate, -1))?;
 
-    // Initialize streaming support
-    stream::init(ruby)?;
+    // Defines ZLight::StreamReader along with .stream and .stream_file.
+    read::stream::init(ruby)?;
 
     Ok(())
 }