herringbone 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 826898400d423a36ccd89740d408da3d367726cf84c6cf33c4067a3c54b8861d
-  data.tar.gz: 0667170a3ede8376299aff04b4081259533c8877f9823ad8fcc9aee916a86650
+  metadata.gz: 4aca873b946a39da4bee947137c6914f0b321f95266f252f35487087e40ce3f4
+  data.tar.gz: 6d7f439f61d6d871cfbf8682e2af71bf2c502b5360355570aabeed373240d0d7
 SHA512:
-  metadata.gz: 21984e5695bf75e086fa091a4c69456edd4fc7ef21ea2d73c89e0ad3de1316238d2f66cf9d8c86183c5e962a1ce965aaeac8036dc1a5c1a43aa6a3b85c983405
-  data.tar.gz: 9377178bd66028014087cc55ed255c4abc2ed1c0217cdfbd86d5f9fe647a4ac0d741a973415f393c31fc7af83d653ce14df33d7a7629dee69aaed554e1fa5719
+  metadata.gz: 9cc7d70651be9426aec7608b1fa456545ea0c5318c4aaf193a67b14c7d2371a7173c62cca23a73f603d412dd2ddafa5d8f8e0833b08e6efc9c0632fe367e56d2
+  data.tar.gz: bb1b6ed8e32eb6bcbdb0c29ca1dbd773a74dd2f5d2587253a4a297d76a27ffddc7f0a508baabb9a3d45d0bcd40423f3f5ea531cdc4adf3b4564c3344f61f3fda

data/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 ## Unreleased
 
+## 0.5.0
+
+- `Herringbone.redact` and `Herringbone::Redaction` rewrite a file with rows deleted, values
+  replaced or columns dropped, for GDPR erasure and pseudonymization, copying the row groups they
+  don't touch byte for byte.
+- `Herringbone::Redaction#affects?` says whether a redaction would change a file, reading only
+  statistics, bloom filters and the `where` columns.
+- `Herringbone::Writer#inspect` (and `SimpleWriter`, `ByteValues`) prints a one-line summary with
+  the writer's state and row count instead of dumping the buffered rows.
+- **Breaking:** schema DSL blocks receive the builder as a parameter instead of running with
+  `instance_eval`: `Herringbone::Schema.define { |s| s.int64 :id }`, likewise for `Schema.infer`,
+  `Herringbone.write`, `SimpleWriter.new` and nested `struct`/`list`/`map` blocks. A block without
+  a parameter raises `ArgumentError`.
+
 ## 0.4.0
 
 - herringbone is now Ractor-enabled! Reading and writing work from any Ractor on Ruby 3.1+, and schemas can be passed to Ractors with

data/README.md

@@ -148,19 +148,19 @@ with `each_batch` it can differ between batches.
 ## Writing
 
 ```ruby
-schema = Herringbone::Schema.define do
-  int64 :id, null: false
-  string :name
-  enum :status, values: %w[pending paid shipped]
-  list :tags, :string
-  map :scores, :string, :double
-  struct :address do
-    string :city
-    string :zip
+schema = Herringbone::Schema.define do |s|
+  s.int64 :id, null: false
+  s.string :name
+  s.enum :status, values: %w[pending paid shipped]
+  s.list :tags, :string
+  s.map :scores, :string, :double
+  s.struct :address do |address|
+    address.string :city
+    address.string :zip
   end
-  decimal :price, precision: 12, scale: 2
-  json :payload
-  timestamp :created_at
+  s.decimal :price, precision: 12, scale: 2
+  s.json :payload
+  s.timestamp :created_at
 end
 
 File.open("out.parquet", "wb") do |file|
@@ -177,7 +177,7 @@ end
 
 `Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
 first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
-`Herringbone.write(io, rows) { json :payload }`. The rows are iterated once, holding back only
+`Herringbone.write(io, rows) { |s| s.json :payload }`. The rows are iterated once, holding back only
 those first 1000, so lazy Enumerators and cursors that can't be rewound work. A later row that
 doesn't fit the inferred types raises `Herringbone::SchemaMismatch`, which explains what was
 inferred and how to declare the column, and leaves the file unfinished.
@@ -188,11 +188,15 @@ the `Writer.open` block raises (or `#abort` is called), no footer is written and
 so far is left for you to discard. To replace a file only once it is complete, write to a temporary
 file and rename it.
 
+The block receives the schema builder, and the blocks of `struct`, `list` and `map` receive a
+builder of their own, so the block can still call your methods and read your instance variables.
+A block that takes no parameter raises `ArgumentError`.
+
 Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
 string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
-and `map`; `column :name, :int32` declares one by name. Fields are nullable unless `null: false`
+and `map`; `s.column :name, :int32` declares one by name. Fields are nullable unless `null: false`
 is given; list elements unless `element_null: false`, map values unless `value_null: false`.
-Nested lists: `list :matrix do list :element, :double end`. `time` and `timestamp` take `unit:`
+Nested lists: `s.list :matrix do |matrix| matrix.list :element, :double end`. `time` and `timestamp` take `unit:`
 (`:millis`, `:micros`, `:nanos`) and `utc:`.
 
 `enum` is a string column. `values:` restricts what may be written, and also takes a Rails-style
@@ -248,7 +252,7 @@ end
 ```
 
 Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
-`Herringbone::SimpleWriter.new(io) { string :code }` (then call `close` when done).
+`Herringbone::SimpleWriter.new(io) { |s| s.string :code }` (then call `close` when done).
 
 ### Statistics, page indexes and bloom filters
 
@@ -285,6 +289,136 @@ If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::Mu
 so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
 the file at about 48GB; raise `part_size:` for bigger files.
 
+## Redaction
+
+`Herringbone.redact` rewrites a file with rows removed or values replaced, for GDPR "forget me"
+requests and pseudonymization. One file goes in and one comes out, and the parts of the file
+nothing touches are copied byte for byte.
+
+```ruby
+# Forget me: remove the rows
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).delete
+end
+
+# Forget me, but keep the row for accounting: blank the personal columns
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).replace(email: nil, name: nil, address: nil)
+end
+
+# Forget me, but keep the row linkable: pseudonymize just that user's email
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email) }
+end
+
+# Pseudonymize a column across the whole file, mask another, drop a third
+Herringbone.redact(input, output) do |r|
+  r.replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
+  r.replace(:phone) { |phone| phone && "***#{phone[-3..]}" }
+  r.drop :ssn, :ip_address
+end
+```
+
+A `Herringbone::Redaction` is built once and applies itself to any number of files, which suits a
+forget-me job going over a bucket:
+
+```ruby
+forget = Herringbone::Redaction.new do |r|
+  r.where(user_id: 42).delete
+  r.where(email: "anna@example.com").delete     # separate statements OR together
+  r.where(created_at: ..2.years.ago).delete      # retention works the same way
+end
+
+if forget.affects?(input)      # statistics and bloom filters first, then the where columns only
+  report = forget.apply(input, output)
+  report.rows_deleted          # => 3
+  report.row_groups            # => { copied: 61, rewritten: 1 }
+end
+```
+
+The block receives the redaction being built, like `Writer.open` passes the writer, so it can use
+the methods and instance variables around it; a block without the `|r|` raises `ArgumentError`.
+`where` takes what `read(where:)` takes (values, Arrays, Ranges, `nil`, callables and dotted
+struct paths, but no columns inside lists or maps), and skips row groups and pages the same way.
+`where(...).delete` removes the matching rows. `replace` sets constants,
+`replace(email: nil, name: "[deleted]")`, or computes each value with a block,
+`replace(:email, :phone) { |value| ... }`. A block that takes two parameters also gets the whole
+row, as a Hash with String keys like `read` returns. Without `where`, `replace` applies to every
+row. A column is a top-level field or a struct member by dotted path (`"address.city"`; a member
+of a null struct is left alone). To change what is inside a list or map, replace the whole field
+with a block that receives the Array or Hash. `drop :a, :b` removes columns from the schema.
+
+Statements apply in declared order, row by row: a deleted row is gone for later statements, and
+later statements see what earlier ones replaced, in their conditions as well as in their blocks.
+Mistakes raise `ArgumentError` before anything is written: a `where` without a verb, a column that
+doesn't exist, `nil` (or a value of the wrong type) for a `null: false` column. A block that
+returns something its column can't store raises `Herringbone::EncodeError` when it gets there and
+leaves the output unfinished.
+
+`apply` and `Herringbone.redact` take IOs like `Reader` and `Writer` do: the input must be
+seekable, the output only needs `#write`, and neither is closed. To redact in place, write to a
+temporary file and rename it over the original; on S3, read the object and write the new one
+through `upload_stream`. The `Redaction::Report` that `apply` returns has `rows_read`,
+`rows_deleted`, `rows_changed` and `row_groups` (`{ copied:, rewritten: }`), which is the audit
+trail an erasure needs.
+
+Each row group is handled in the cheapest way that is still exact. If no statement can match it,
+judging by statistics, bloom filters and the page index, or by reading only the `where` columns,
+its column chunks are copied as they are and only their offsets are rebased. If rows match but
+none is deleted and only leaf columns change, just those column chunks are encoded again. If rows
+are deleted, or a nested field is replaced whole, the row group is rewritten, as one row group
+(or none, when every row goes). Rewritten chunks keep the codec of the original chunk (LZO, which
+Herringbone can't write, becomes Snappy) and get a new bloom filter if they had one. Writer
+options (`compression:`, `bloom_filters:`, `page_rows:`...) apply to the rewritten chunks;
+`row_group_bytes:` and `row_group_rows:` are refused, since row groups keep their boundaries. The
+footer's key/value metadata is copied (without `ARROW:schema` and `pandas` when columns are
+dropped, since those describe the columns) unless `metadata:` is given.
+
+No deleted or replaced value survives in the output: not in data pages, dictionary pages, column
+chunk min/max statistics, the page index or bloom filters. What Herringbone can't reach is up to
+you: the original file, its S3 versions and backups still hold the data until you delete them,
+only the columns you name are touched (an email that also sits in a free-text `notes` column stays
+there), and other files with the same person in them need their own pass.
+
+Some recipes. Keyed hashing keeps a column joinable without keeping the value; keep the key out
+of the data, and rotate or destroy it to cut the link:
+
+```ruby
+require "openssl"
+KEY = ENV.fetch("PSEUDONYM_KEY")
+Herringbone.redact(input, output) do |r|
+  r.replace(:email) { |email| email && OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
+end
+```
+
+Masking keeps enough to be recognizable to the person but not to anyone else:
+
+```ruby
+Herringbone.redact(input, output) do |r|
+  r.replace(:card_number) { |number| number && number[-4..].rjust(number.size, "*") }
+  r.replace(:email) { |email| email&.sub(/\A(.).*@/, '\1***@') }
+  r.replace(:birth_date) { |date| date && Date.new(date.year, 1, 1) }
+end
+```
+
+Fake values (with the `faker` gem) make a copy for staging that looks real. Seed Faker from the
+original value to get the same fake for the same person in every file:
+
+```ruby
+require "faker"
+require "zlib"
+Herringbone.redact(input, output) do |r|
+  r.replace(:name) do |name|
+    next nil unless name
+    Faker::Config.random = Random.new(Zlib.crc32(name))
+    Faker::Name.name
+  end
+  r.replace(:address) do |address|
+    address && { "city" => Faker::Address.city, "zip" => Faker::Address.zip_code }
+  end
+end
+```
+
 ## ActiveRecord
 
 `Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema

data/lib/herringbone/active_record.rb

@@ -92,7 +92,7 @@ module Herringbone
 
         if type == :hstore
           return builder.map(column.name, :string, :string, null: nullable) unless array
-          return builder.list(column.name, null: nullable) { map :element, :string, :string }
+          return builder.list(column.name, null: nullable) { |list| list.map :element, :string, :string }
         end
 
         dsl_type, opts = scalar_type(column, type, sql_type, primary)

data/lib/herringbone/byte_values.rb

@@ -45,6 +45,15 @@ module Herringbone
     # @return [Boolean] true while still dictionary-encoding (not yet switched to raw bytes)
     def dictionary? = !@indices.nil?
 
+    # Short summary for the console, without the values
+    #
+    # @return [String] value count, the dictionary size (or "raw" after switching to raw bytes),
+    #   the width for FIXED_LEN_BYTE_ARRAY and the estimated memory
+    def inspect
+      mode = dictionary? ? "dictionary=#{@dictionary.size}" : "raw"
+      "#<#{self.class.name} size=#{size} #{mode}#{" width=#{@width}" if @width} memory_bytes=#{memory_bytes}>"
+    end
+
     # Appends a value, switching to raw bytes when the dictionary limits are exceeded.
     # @param value [String] bytes of the value; for FIXED_LEN_BYTE_ARRAY it must be +width+ bytes
     #   long (not checked here)

data/lib/herringbone/inferring_writer.rb

@@ -73,6 +73,13 @@ module Herringbone
       @writer&.abort
     end
 
+    # Short summary for the console, without the held-back rows
+    #
+    # @return [String] the number of rows held back, or the Writer's summary once it is open
+    def inspect
+      "#<#{self.class.name} #{@writer ? "writer=#{@writer.inspect}" : "held_back=#{@sample.size}"}>"
+    end
+
     private
 
     # Builds the schema from the held-back rows, opens the Writer and writes them