herringbone 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/MANUAL.md +773 -0
- data/README.md +81 -316
- data/bin/herringbone +86 -7
- data/lib/herringbone/active_record.rb +1 -1
- data/lib/herringbone/bloom_filter.rb +0 -55
- data/lib/herringbone/byte_values.rb +9 -0
- data/lib/herringbone/compression.rb +27 -4
- data/lib/herringbone/encryption.rb +716 -0
- data/lib/herringbone/encryption_configuration.rb +304 -0
- data/lib/herringbone/format.rb +55 -3
- data/lib/herringbone/inferring_writer.rb +24 -2
- data/lib/herringbone/inspector.rb +158 -28
- data/lib/herringbone/key.rb +107 -0
- data/lib/herringbone/reader/bloom_filters.rb +91 -0
- data/lib/herringbone/reader/column_chunk_reader.rb +55 -1
- data/lib/herringbone/reader.rb +100 -15
- data/lib/herringbone/redaction/rewriter.rb +585 -0
- data/lib/herringbone/redaction.rb +302 -0
- data/lib/herringbone/schema.rb +123 -57
- data/lib/herringbone/simple_writer.rb +42 -4
- data/lib/herringbone/version.rb +1 -1
- data/lib/herringbone/writer.rb +231 -37
- data/lib/herringbone.rb +115 -29
- metadata +9 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 36e3b2444245fc90b36b577284aab2a81e4991ff04e30dd781f9ba94df43ae90
|
|
4
|
+
data.tar.gz: 9e2e4dbe2ab635a03aa743206cd5e6d74d6039cc5e790cefae8b04a7afd8304b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 162e38811c7e6371ed61a777fddcfa8e1e811de5225842e13dad1cf67da20490c7b7eb2347eaeb6515322a6ace971d453fa4ce1556cc75f4e70acfdcda7a9179
|
|
7
|
+
data.tar.gz: 0e5798748be214eb32a7939799ecb7a885283099908c05003065a98569c0572dc17a25fd4e6d017c5635256cefdd3b0db5072e28c1536fe4e26867c1d019eca0
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,48 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.6.0
|
|
6
|
+
|
|
7
|
+
- `require "herringbone"` loads the rest of the gem on first use: under 1 ms instead of about
|
|
8
|
+
40 ms, and reading or writing plain files no longer loads OpenSSL, the inspector or the
|
|
9
|
+
redaction code. `Herringbone.eager_load!` loads everything, which Ractors need before Ruby 3.4.
|
|
10
|
+
- `compression_level:` sets the level for `:zstd`, `:gzip` and `:brotli` in `Herringbone::Writer`,
|
|
11
|
+
`Herringbone.write`, `SimpleWriter` and `Herringbone.redact`.
|
|
12
|
+
- `Herringbone::Writer` (and `Herringbone.write`, `SimpleWriter`) take `encryption:` to write files
|
|
13
|
+
with Parquet modular encryption: footer and per-column keys, encrypted or signed plaintext
|
|
14
|
+
footer, AES-GCM or AES-GCM-CTR, AAD prefixes.
|
|
15
|
+
- `Herringbone::Key` (`Key.generate`, `Key.from_hex`) holds a key and its id: `encryption: key`
|
|
16
|
+
encrypts a whole file with it the way most readers (pyarrow 25+, Arrow, arrow-rs, Spark, Trino)
|
|
17
|
+
can decrypt, and `decryption: [key, older_key]` picks the key by the id stored in the file.
|
|
18
|
+
- `decryption:` takes a callable that returns the key for a key id.
|
|
19
|
+
- `Herringbone::SimpleWriter#encrypt!(key:)` encrypts the file with one key, given as its hex or as
|
|
20
|
+
a `Key`; Strings that are not the hex of a key are refused.
|
|
21
|
+
- `Herringbone::EncryptionConfiguration` and `Herringbone::DecryptionConfiguration` check the
|
|
22
|
+
`encryption:` and `decryption:` settings when they are built; Hashes are turned into them.
|
|
23
|
+
- `herringbone inspect` and `herringbone cat` take `--footer-key`, `--column-key`, `--key` and
|
|
24
|
+
`--aad-prefix` for encrypted files, and ask for missing keys on stdin.
|
|
25
|
+
- `Herringbone::Reader` and `Herringbone::Inspector` take `decryption:` with keys, or a `keys:`
|
|
26
|
+
lookup by the key metadata stored in the file, and read encrypted files from parquet-mr, Arrow
|
|
27
|
+
and Herringbone; `Reader#encryption` describes how a file is encrypted.
|
|
28
|
+
- `Herringbone.redact` takes `decryption:` and writes an encrypted input encrypted the same way,
|
|
29
|
+
unless `encryption:` says otherwise.
|
|
30
|
+
- **Breaking:** encrypted files raise `Herringbone::DecryptionError` (when their keys are missing
|
|
31
|
+
or wrong) instead of `Herringbone::UnsupportedError`.
|
|
32
|
+
|
|
33
|
+
## 0.5.0
|
|
34
|
+
|
|
35
|
+
- `Herringbone.redact` and `Herringbone::Redaction` rewrite a file with rows deleted, values
|
|
36
|
+
replaced or columns dropped, for GDPR erasure and pseudonymization, copying the row groups they
|
|
37
|
+
don't touch byte for byte.
|
|
38
|
+
- `Herringbone::Redaction#affects?` says whether a redaction would change a file, reading only
|
|
39
|
+
statistics, bloom filters and the `where` columns.
|
|
40
|
+
- `Herringbone::Writer#inspect` (and `SimpleWriter`, `ByteValues`) prints a one-line summary with
|
|
41
|
+
the writer's state and row count instead of dumping the buffered rows.
|
|
42
|
+
- **Breaking:** schema DSL blocks receive the builder as a parameter instead of running with
|
|
43
|
+
`instance_eval`: `Herringbone::Schema.define { |s| s.int64 :id }`, likewise for `Schema.infer`,
|
|
44
|
+
`Herringbone.write`, `SimpleWriter.new` and nested `struct`/`list`/`map` blocks. A block without
|
|
45
|
+
a parameter raises `ArgumentError`.
|
|
46
|
+
|
|
5
47
|
## 0.4.0
|
|
6
48
|
|
|
7
49
|
- herringbone is now Ractor-enabled! Reading and writing work from any Ractor on Ruby 3.1+, and schemas can be passed to Ractors with
|