herringbone 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e9b95c84373ccf0215789628229613c8899227be711dc204c31be6778352f2d6
-  data.tar.gz: 9314fc6d6d55660e210db0fe62369d9732e33dc3910e306f500b3ebc5c7090e9
+  metadata.gz: c59941b3e4ed776a03d3596bf8b9288dd852b89f49934dfb1110f6b29d527ab8
+  data.tar.gz: 13ef32e91a12a11d249be06834f0c33dfb70c9ebb7746d29d28844f9af180fa5
 SHA512:
-  metadata.gz: 63292b6fde84386d41068903db5267fe7fa94b15c8c191d12c229f888cd94bfd1e6c78e3d01ed5f562643565e8126e9ee2f372e80ab96e08609293afeff1c3bf
-  data.tar.gz: e9946d5f2ac156222b3bca852f5fc26194ec304fa2b150fd9520311b7e3a323af3ba28e1662bc3b47e593508b22b216e148cf6af67d55318830272f1d87578e5
+  metadata.gz: '083624f293ce5d1825c9f6998e46b6e58767e9bbec15d6adbbd1cb556ef3771d2a0426a41ba8f097a43b2b937bef55ce018a01925a7df2d9f9d8d36a1b4f8da9'
+  data.tar.gz: 38b917be5c728a3837d3eafae3e9f060708574f5c6249792219d17a410a4fe14683df46dfd4b922d3bf4503a327a2384105c950e06ca1523a7482c042629f367

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 ## Unreleased
 
+## 0.9.0
+
+- ZSTD files are read without the `zstd-ruby` gem, by a pure-Ruby decompressor; `zstd-ruby` is
+  still used when loaded, and still needed to write ZSTD.
+
+## 0.8.0
+
+- `Herringbone::MissingCodecError` explains over several lines which gem to install and that it has
+  to be required before the read or write, and names the column it could not decompress up front.
+- `Herringbone::Schema#only(:id, :name)` and `#except(:ssn)` make a schema with some of the
+  top-level fields.
+- `time` columns convert a `Time` to UTC when writing, as `timestamp` columns already did, instead
+  of storing its time of day in its own zone.
+- **Breaking:** `local_timestamp` and `local_time` (`s.local_timestamp :opens_at`) write the wall
+  clock time without converting it to UTC, replacing the `utc: false` option, which now raises.
+
 ## 0.7.0
 
 - `Herringbone.redact` and `Herringbone::Redaction#apply` take a `Herringbone::Reader` as well as an

data/MANUAL.md

@@ -19,6 +19,7 @@ The full reference for Herringbone. For a quick start, see the [README](README.m
   - [SimpleWriter: coming from CSV](#simplewriter-coming-from-csv)
   - [Statistics, page indexes and bloom filters](#statistics-page-indexes-and-bloom-filters)
   - [Writing to S3](#writing-to-s3)
+  - [Writing to Google Cloud Storage](#writing-to-google-cloud-storage)
 - [Encryption](#encryption)
   - [Writing encrypted files](#writing-encrypted-files)
   - [Which settings other tools read](#which-settings-other-tools-read)
@@ -54,7 +55,7 @@ Add these libraries to speed up and enable certain functionality:
 
 ```ruby
 gem "snappy"          # native Snappy, 2-3x faster reads and writes of typical files
-gem "zstd-ruby"       # ZSTD
+gem "zstd-ruby"       # writing ZSTD; reading falls back to pure Ruby, 30-100x slower
 gem "brotli"          # Brotli
 gem "xxhash"          # faster bloom filters
 gem "numo-narray-alt" # read(as: :numo)
@@ -212,6 +213,17 @@ The block receives the schema builder, and the blocks of `struct`, `list` and `m
 builder of their own, so the block can still call your methods and read your instance variables.
 A block that takes no parameter raises `ArgumentError`.
 
+`only` and `except` make a schema with some of the top-level fields, keeping their order, for
+instance to write part of a file you are reading:
+
+```ruby
+schema.only(:id, :name)  # id and name
+schema.except(:ssn)      # every field but ssn
+```
+
+A name the schema has no top-level field for raises `ArgumentError`, and so does leaving out
+every field.
+
 ### Writing with an inferred schema
 
 `Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
@@ -232,11 +244,17 @@ file and rename it.
 ### Column types
 
 `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16 string binary
-json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list` and `map`;
+json bson enum uuid date int96 time local_time timestamp local_timestamp decimal fixed`, plus
+`struct`, `list` and `map`;
 `s.column :name, :int32` declares one by name. Fields are nullable unless `null: false` is given;
 list elements unless `element_null: false`, map values unless `value_null: false`. Nested lists:
-`s.list :matrix do |matrix| matrix.list :element, :double end`. `time` and `timestamp` take `unit:`
-(`:millis`, `:micros`, `:nanos`) and `utc:`.
+`s.list :matrix do |matrix| matrix.list :element, :double end`. The time and timestamp types take `unit:`
+(`:millis`, `:micros`, `:nanos`).
+
+`timestamp` stores instants: a `Time` in any zone is converted to UTC, so the file means the same
+thing wherever it was written. `local_timestamp` stores the wall clock date and time as given, with
+no zone, for values like "the shop opens at 09:00" that are not instants. `time` and `local_time`
+do the same for a time of day.
 
 `enum` is a string column. `values:` restricts what may be written, and also takes a Rails-style
 Hash (`values: Order.statuses`), in which case both labels and stored integers are accepted and
@@ -250,8 +268,8 @@ Columns accept the values Ruby and Rails code usually has at hand:
 | column | accepts |
 |---|---|
 | `date` | `Date`, `Time`/`DateTime` (their date), `"2024-05-01"` |
-| `timestamp` | `Time`, `DateTime`, `ActiveSupport::TimeWithZone`, `Date` (midnight UTC), ISO-8601 strings, Integers in the column's unit |
-| `time` | `Time` (its time of day, as Rails returns for `time` columns), `"13:45:30.25"`, Integers |
+| `timestamp`, `local_timestamp` | `Time`, `DateTime`, `ActiveSupport::TimeWithZone`, `Date` (midnight UTC), ISO-8601 strings, Integers in the column's unit |
+| `time`, `local_time` | `Time` (its time of day, in UTC for `time`, as Rails returns for `time` columns), `"13:45:30.25"`, Integers |
 | `json` | Strings as-is, anything else through `JSON.generate` |
 | `string`, `enum` | Strings, Symbols, anything with `to_s` |
 | `boolean` | `true`/`false`, `1`/`0`, `"t"`/`"f"`, `"true"`/`"false"`, `"yes"`/`"no"` |
@@ -342,6 +360,18 @@ If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::Mu
 so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
 the file at about 48GB; raise `part_size:` for bigger files.
 
+### Writing to Google Cloud Storage
+
+`google-cloud-storage` needs the size of an upload up front, so it can't take a stream. Use
+[gcs_put](https://github.com/julik/gcs_put), which streams into a resumable upload of unknown size:
+
+```ruby
+gcs_file = Google::Cloud::Storage.new.bucket("exports").file("orders.parquet", skip_lookup: true)
+GCSPut.open_gcs_file(gcs_file) { |io| Herringbone.write(io, Order.all) }
+```
+
+If the block raises, the upload is never finalized and no object appears.
+
 ## Encryption
 
 Herringbone reads and writes files encrypted with

data/README.md

@@ -4,7 +4,7 @@ A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/)
 
 - No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings
   and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems).
-  LZO, from Hadoop-era files, can be read but not written
+  ZSTD and LZO (from Hadoop-era files) are also read in pure Ruby; writing ZSTD needs `zstd-ruby`
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
 - Optimized reads with batches and pages
@@ -21,9 +21,9 @@ rules, combining files, type mapping and the inspector.
 gem "herringbone"
 ```
 
-Optionally add `snappy` (faster), `zstd-ruby` and `brotli` (more codecs), `xxhash` (faster bloom
-filters) or `numo-narray-alt` (`read(as: :numo)`). Herringbone uses whichever of them are loaded,
-see [Optional gems](MANUAL.md#optional-gems).
+Optionally add `snappy` (faster), `zstd-ruby` (writing ZSTD, much faster reading of it), `brotli`,
+`xxhash` (faster bloom filters) or `numo-narray-alt` (`read(as: :numo)`). Herringbone uses
+whichever of them are loaded, see [Optional gems](MANUAL.md#optional-gems).
 
 ## Exporting from Rails
 
@@ -46,7 +46,9 @@ end
 ```
 
 Any Enumerable of Hashes, Structs or Arrays works too; the schema is inferred from the first 1000
-rows. See [ActiveRecord](MANUAL.md#activerecord) and [Writing to S3](MANUAL.md#writing-to-s3).
+rows. See [ActiveRecord](MANUAL.md#activerecord) and [Writing to S3](MANUAL.md#writing-to-s3). For
+Google Cloud Storage, use [gcs_put](https://github.com/julik/gcs_put), see
+[Writing to Google Cloud Storage](MANUAL.md#writing-to-google-cloud-storage).
 
 ## Reading
 

data/lib/herringbone/active_record.rb

@@ -128,7 +128,7 @@ module Herringbone
         when :boolean then [:boolean, {}]
         when :binary then [:binary, {}]
         when :date then [:date, {}]
-        when :datetime, :timestamp, :timestamptz then [:timestamp, {unit: :micros, utc: true}]
+        when :datetime, :timestamp, :timestamptz then [:timestamp, {unit: :micros}]
         when :time then [:time, {unit: :micros}]
         when :json, :jsonb then [:json, {}]
         when :uuid then [:uuid, {}]