parqueteur 1.1.1 → 1.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1805a7356005e238c3d8f4318a54ea498ff58a0fcd93819c51cc445c4f8b6ee0
-  data.tar.gz: 1360c3d481eac282e684f588e49cbcd142c7cc0f6c345d8a98e48e402a9edf22
+  metadata.gz: 6bae31969c118f27e2a1e3796a72dff3c7c63db4ce7b12db6f967d8f5b222ee5
+  data.tar.gz: 13bcbc2a1eaee55ba0b27d6aa2848147829c90d85a7089d35fb22e01ae586053
 SHA512:
-  metadata.gz: 28c157c39f25744a10223e8d11d165b7e4e25eb70e45cf2e22513795b4b1b4b1b9f44bc04d5ef8e10b74ebd82d1c87bc318a02b32fc25dbd6c32c0763e79801c
-  data.tar.gz: 6f0433e8090c49c48278bcd4d5f36f96c394bd8d8386fbf2bc22d224d42ef58036beaa0307c32420cbbb3b33eec463964b28e616f862385de6afa1f8c98c9310
+  metadata.gz: 04d7dc514fb2c0869a1b8ef071117b22da51ce162a31169b76a413c9bfb356b12ff7493e3fdbce5b8691033ad013254c53547d592f12f17bc554600e1ecc4d0d
+  data.tar.gz: fafaca859454f78f81b63961360c4baad6b0ec2614c0a8b675cb14a9df5c6aaef3bf4b72050effce84418415271567bf4a3032bef27abcb4bcbcd8e22ab160b2

data/Gemfile.lock

@@ -1,32 +1,32 @@
 PATH
   remote: .
   specs:
-    parqueteur (1.1.1)
-      red-parquet (~> 5.0)
+    parqueteur (1.3.2)
+      red-parquet (~> 6.0)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    bigdecimal (3.0.0)
-    extpp (0.0.9)
-    gio2 (3.4.6)
-      gobject-introspection (= 3.4.6)
-    glib2 (3.4.6)
+    bigdecimal (3.0.2)
+    extpp (0.1.0)
+    gio2 (3.4.9)
+      gobject-introspection (= 3.4.9)
+    glib2 (3.4.9)
       native-package-installer (>= 1.0.3)
       pkg-config (>= 1.3.5)
-    gobject-introspection (3.4.6)
-      glib2 (= 3.4.6)
+    gobject-introspection (3.4.9)
+      glib2 (= 3.4.9)
     native-package-installer (1.1.1)
     pkg-config (1.4.6)
     rake (13.0.6)
-    red-arrow (5.0.0)
+    red-arrow (6.0.0)
       bigdecimal (>= 2.0.3)
       extpp (>= 0.0.7)
-      gio2 (>= 3.4.5)
+      gio2 (>= 3.4.9)
       native-package-installer
       pkg-config
-    red-parquet (5.0.0)
-      red-arrow (= 5.0.0)
+    red-parquet (6.0.0)
+      red-arrow (= 6.0.0)
 
 PLATFORMS
   ruby

data/README.md

@@ -1,15 +1,31 @@
 # Parqueteur
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/parqueteur`. To experiment with that code, run `bin/console` for an interactive prompt.
+[![Gem Version](https://badge.fury.io/rb/parqueteur.svg)](https://badge.fury.io/rb/parqueteur)
 
-TODO: Delete this and the text above, and describe your gem
+Parqueteur enables you to generate Apache Parquet files from raw data.
 
+## Dependencies
+
+Since I only tested Parqueteur on Ubuntu, I don't have any install scripts for others operating systems.
+### Debian/Ubuntu packages
+- `libgirepository1.0-dev`
+- `libarrow-dev`
+- `libarrow-glib-dev`
+- `libparquet-dev`
+- `libparquet-glib-dev`
+
+You can check `scripts/apache-arrow-ubuntu-install.sh` script for a quick way to install all of them.
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'parqueteur'
+gem 'parqueteur', '~> 1.0'
+```
+
+> (optional) If you don't want to require Parqueteur globally you can add `require: false` to the Gemfile instruction:
+```ruby
+gem 'parqueteur', '~> 1.0', require: false
 ```
 
 And then execute:
@@ -22,14 +38,127 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+Parqueteur provides an elegant way to generate Apache Parquet files from a defined schema.
+
+Converters accepts any object that implements `Enumerable` as data source.
+
+### Working example
+
+```ruby
+require 'parqueteur'
+
+class FooParquetConverter < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :datetime, :timestamp
+end
+
+data = [
+  { 'id' => 1, 'reference' => 'hello world 1', 'datetime' => Time.now },
+  { 'id' => 2, 'reference' => 'hello world 2', 'datetime' => Time.now },
+  { 'id' => 3, 'reference' => 'hello world 3', 'datetime' => Time.now }
+]
+
+# initialize Converter with Parquet GZIP compression mode
+converter = FooParquetConverter.new(data, compression: :gzip)
 
-## Development
+# write result to file
+converter.write('hello_world.parquet')
 
-After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+# in-memory result (StringIO)
+converter.to_io
+
+# write to temporary file (Tempfile)
+# don't forget to `close` / `unlink` it after usage
+converter.to_tmpfile
+
+# convert to Arrow::Table
+pp converter.to_arrow_table
+```
+
+### Using transformers
+
+You can use transformers to apply data items transformations.
+
+From `examples/cars.rb`:
+
+```ruby
+require 'parqueteur'
+
+class Car
+  attr_reader :name, :production_year
+
+  def initialize(name, production_year)
+    @name = name
+    @production_year = production_year
+  end
+end
+
+class CarParquetConverter < Parqueteur::Converter
+  column :name, :string
+  column :production_year, :integer
+
+  transform do |car|
+    {
+      'name' => car.name,
+      'production_year' => car.production_year
+    }
+  end
+end
+
+cars = [
+  Car.new('Alfa Romeo 75', 1985),
+  Car.new('Alfa Romeo 33', 1983),
+  Car.new('Audi A3', 1996),
+  Car.new('Audi A4', 1994),
+  Car.new('BMW 503', 1956),
+  Car.new('BMW X5', 1999)
+]
+
+# initialize Converter with Parquet GZIP compression mode
+converter = CarParquetConverter.new(data, compression: :gzip)
+
+# write result to file
+pp converter.to_arrow_table
+```
+
+Output:
+```
+#<Arrow::Table:0x7fc1fb24b958 ptr=0x7fc1faedd910>
+#     name           production_year
+0     Alfa Romeo 75  1985
+1     Alfa Romeo 33  1983
+2     Audi A3        1996
+3     Audi A4        1994
+4     BMW 503        1956
+5     BMW X5         1999
+```
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+### Available Types
+
+| Name (Symbol) | Apache Parquet Type |
+| ------------- | --------- |
+| `:array` | `Array` |
+| `:bigdecimal` | `Decimal256` |
+| `:bigint` | `Int64` or `UInt64` with `unsigned: true` option |
+| `:boolean` | `Boolean` |
+| `:date` | `Date32` |
+| `:date32` | `Date32` |
+| `:date64` | `Date64` |
+| `:decimal` | `Decimal128` |
+| `:decimal128` | `Decimal128` |
+| `:decimal256` | `Decimal256` |
+| `:int32` | `Int32` or `UInt32` with `unsigned: true` option |
+| `:int64` | `Int64` or `UInt64` with `unsigned: true` option |
+| `:integer` | `Int32` or `UInt32` with `unsigned: true` option |
+| `:map` | `Map` |
+| `:string` | `String` |
+| `:struct` | `Struct` |
+| `:time` | `Time32` |
+| `:time32` | `Time32` |
+| `:time64` | `Time64` |
+| `:timestamp` | `Timestamp` |
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/parqueteur.
+Bug reports and pull requests are welcome on GitHub at https://github.com/pocketsizesun/parqueteur-ruby.

data/examples/cars.rb

@@ -0,0 +1,40 @@
+require 'bundler/setup'
+require 'parqueteur'
+
+class Car
+  attr_reader :name, :production_year
+
+  def initialize(name, production_year)
+    @name = name
+    @production_year = production_year
+  end
+end
+
+class CarParquetConverter < Parqueteur::Converter
+  column :name, :string
+  column :production_year, :integer
+
+  transform do |car|
+    {
+      'name' => car.name,
+      'production_year' => car.production_year
+    }
+  end
+end
+
+cars = [
+  Car.new('Alfa Romeo 75', 1985),
+  Car.new('Alfa Romeo 33', 1983),
+  Car.new('Audi A3', 1996),
+  Car.new('Audi A4', 1994),
+  Car.new('BMW 503', 1956),
+  Car.new('BMW X5', 1999)
+]
+
+# initialize Converter with Parquet GZIP compression mode
+converter = CarParquetConverter.new(
+  cars, compression: :gzip
+)
+
+# write result to file
+pp converter.to_arrow_table

data/examples/convert-and-compression.rb

@@ -0,0 +1,56 @@
+require 'bundler/setup'
+require 'parqueteur'
+require 'securerandom'
+require 'benchmark'
+
+class FooParquetConverter < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :hash, :map, key: :string, value: :string
+  # column :hash2, :map, key: :string, value: :string
+  # column :hash3, :map, key: :string, value: :string
+  column :valid, :boolean
+  column :total, :integer
+  column :numbers, :array, elements: :integer
+  column :my_struct, :struct do
+    field :test, :string
+    field :mon_nombre, :integer
+  end
+end
+
+def random_hash
+  {
+    'a' => SecureRandom.hex(128),
+    'b' => SecureRandom.hex(128),
+    'c' => SecureRandom.hex(128),
+    'd' => SecureRandom.hex(128),
+    'e' => SecureRandom.hex(128),
+    'f' => SecureRandom.hex(128),
+    'g' => SecureRandom.hex(128),
+    'h' => SecureRandom.hex(128),
+    'i' => SecureRandom.hex(128),
+    'j' => SecureRandom.hex(128),
+    'k' => SecureRandom.hex(128),
+  }
+end
+
+data = 10000.times.collect do |i|
+  {
+    'id' => i + 1,
+    'reference' => "coucou:#{i}",
+    'hash' => random_hash,
+    # 'hash2' => random_hash,
+    # 'hash3' => random_hash,
+    'valid' => rand < 0.5,
+    'total' => rand(100..500),
+    'numbers' => [1, 2, 3]
+  }
+end
+puts "data generation OK"
+
+# initialize Converter with Parquet GZIP compression mode
+converter = FooParquetConverter.new(data, compression: :gzip)
+
+# write result to file
+converter.write('tmp/example.gzip-compressed.parquet')
+converter.write('tmp/example.no-gzip.parquet', compression: false)

data/examples/convert-methods.rb

@@ -0,0 +1,54 @@
+require 'bundler/setup'
+require 'parqueteur'
+require 'securerandom'
+require 'benchmark'
+
+class Foo < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :hash, :map, key: :string, value: :string
+  # column :hash2, :map, key: :string, value: :string
+  # column :hash3, :map, key: :string, value: :string
+  column :valid, :boolean
+  column :total, :integer
+  column :numbers, :array, elements: :integer
+  column :my_struct, :struct do
+    field :test, :string
+    field :mon_nombre, :integer
+  end
+end
+
+def random_hash
+  {
+    'a' => SecureRandom.hex(128),
+    'b' => SecureRandom.hex(128),
+    'c' => SecureRandom.hex(128),
+    'd' => SecureRandom.hex(128),
+    'e' => SecureRandom.hex(128),
+    'f' => SecureRandom.hex(128),
+    'g' => SecureRandom.hex(128),
+    'h' => SecureRandom.hex(128),
+    'i' => SecureRandom.hex(128),
+    'j' => SecureRandom.hex(128),
+    'k' => SecureRandom.hex(128),
+  }
+end
+
+data = 10000.times.collect do |i|
+  {
+    'id' => i + 1,
+    'reference' => "coucou:#{i}",
+    'hash' => random_hash,
+    # 'hash2' => random_hash,
+    # 'hash3' => random_hash,
+    'valid' => rand < 0.5,
+    'total' => rand(100..500),
+    'numbers' => [1, 2, 3]
+  }
+end
+puts "data generation OK"
+
+converter = Foo.new(data, compression: :gzip)
+pp converter.to_io
+pp converter.to_arrow_table
+converter.write('tmp/test.parquet')

data/examples/convert-to-io.rb

@@ -0,0 +1,52 @@
+require 'bundler/setup'
+require 'parqueteur'
+require 'securerandom'
+require 'benchmark'
+
+class Foo < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :hash, :map, key: :string, value: :string
+  # column :hash2, :map, key: :string, value: :string
+  # column :hash3, :map, key: :string, value: :string
+  column :valid, :boolean
+  column :total, :integer
+  column :numbers, :array, elements: :integer
+  column :my_struct, :struct do
+    field :test, :string
+    field :mon_nombre, :integer
+  end
+end
+
+def random_hash
+  {
+    'a' => SecureRandom.hex(128),
+    'b' => SecureRandom.hex(128),
+    'c' => SecureRandom.hex(128),
+    'd' => SecureRandom.hex(128),
+    'e' => SecureRandom.hex(128),
+    'f' => SecureRandom.hex(128),
+    'g' => SecureRandom.hex(128),
+    'h' => SecureRandom.hex(128),
+    'i' => SecureRandom.hex(128),
+    'j' => SecureRandom.hex(128),
+    'k' => SecureRandom.hex(128),
+  }
+end
+
+data = 10000.times.collect do |i|
+  {
+    'id' => i + 1,
+    'reference' => "coucou:#{i}",
+    'hash' => random_hash,
+    # 'hash2' => random_hash,
+    # 'hash3' => random_hash,
+    'valid' => rand < 0.5,
+    'total' => rand(100..500),
+    'numbers' => [1, 2, 3]
+  }
+end
+puts "data generation OK"
+
+io = Foo.convert(data)
+pp io.read

data/examples/convert-with-chunks.rb

@@ -0,0 +1,54 @@
+require 'bundler/setup'
+require 'parqueteur'
+require 'securerandom'
+require 'benchmark'
+
+class Foo < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :hash, :map, key: :string, value: :string
+  # column :hash2, :map, key: :string, value: :string
+  # column :hash3, :map, key: :string, value: :string
+  column :valid, :boolean
+  column :total, :integer
+  column :numbers, :array, elements: :integer
+  column :my_struct, :struct do
+    field :test, :string
+    field :mon_nombre, :integer
+  end
+end
+
+def random_hash
+  {
+    'a' => SecureRandom.hex(128),
+    'b' => SecureRandom.hex(128),
+    'c' => SecureRandom.hex(128),
+    'd' => SecureRandom.hex(128),
+    'e' => SecureRandom.hex(128),
+    'f' => SecureRandom.hex(128),
+    'g' => SecureRandom.hex(128),
+    'h' => SecureRandom.hex(128),
+    'i' => SecureRandom.hex(128),
+    'j' => SecureRandom.hex(128),
+    'k' => SecureRandom.hex(128),
+  }
+end
+
+data = 10000.times.collect do |i|
+  {
+    'id' => i + 1,
+    'reference' => "coucou:#{i}",
+    'hash' => random_hash,
+    # 'hash2' => random_hash,
+    # 'hash3' => random_hash,
+    'valid' => rand < 0.5,
+    'total' => rand(100..500),
+    'numbers' => [1, 2, 3]
+  }
+end
+puts "data generation OK"
+
+converter = Foo.new(data, compression: :gzip)
+converter.split(200).each_with_index do |chunk, idx|
+  puts "#{idx}: #{chunk.path}"
+end

data/examples/convert-without-compression.rb

@@ -0,0 +1,52 @@
+require 'bundler/setup'
+require 'parqueteur'
+require 'securerandom'
+require 'benchmark'
+
+class Foo < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :hash, :map, key: :string, value: :string
+  # column :hash2, :map, key: :string, value: :string
+  # column :hash3, :map, key: :string, value: :string
+  column :valid, :boolean
+  column :total, :integer
+  column :numbers, :array, elements: :integer
+  column :my_struct, :struct do
+    field :test, :string
+    field :mon_nombre, :integer
+  end
+end
+
+def random_hash
+  {
+    'a' => SecureRandom.hex(128),
+    'b' => SecureRandom.hex(128),
+    'c' => SecureRandom.hex(128),
+    'd' => SecureRandom.hex(128),
+    'e' => SecureRandom.hex(128),
+    'f' => SecureRandom.hex(128),
+    'g' => SecureRandom.hex(128),
+    'h' => SecureRandom.hex(128),
+    'i' => SecureRandom.hex(128),
+    'j' => SecureRandom.hex(128),
+    'k' => SecureRandom.hex(128),
+  }
+end
+
+data = 10000.times.collect do |i|
+  {
+    'id' => i + 1,
+    'reference' => "coucou:#{i}",
+    'hash' => random_hash,
+    # 'hash2' => random_hash,
+    # 'hash3' => random_hash,
+    'valid' => rand < 0.5,
+    'total' => rand(100..500),
+    'numbers' => [1, 2, 3]
+  }
+end
+puts "data generation OK"
+
+path = 'tmp/test.parquet'
+Foo.convert_to(data, path)

data/examples/hello-world.rb

@@ -0,0 +1,57 @@
+require 'bundler/setup'
+require 'parqueteur'
+
+class FooParquetConverter < Parqueteur::Converter
+  column :id, :bigint
+  column :my_string_array, :array, elements: :string
+  column :my_date, :date
+  column :my_decimal, :decimal, precision: 12, scale: 4
+  column :my_int, :integer
+  column :my_map, :map, key: :string, value: :string
+  column :my_string, :string
+  column :my_struct, :struct do
+    field :my_struct_str, :string
+    field :my_struct_int, :integer
+  end
+  column :my_time, :time
+  column :my_timestamp, :timestamp
+end
+
+data = 100.times.collect do |i|
+  {
+    'id' => i,
+    'my_string_array' => %w[a b c],
+    'my_date' => Date.today,
+    'my_decimal' => BigDecimal('0.03'),
+    'my_int' => rand(1..10),
+    'my_map' => { 'a' => 'b' },
+    'my_string' => 'Hello World',
+    'my_struct' => {
+      'my_struct_str' => 'Hello World',
+      'my_struct_int' => 1
+    },
+    'my_time' => 3600,
+    'my_timestamp' => Time.now
+  }
+end
+
+# initialize Converter with Parquet GZIP compression mode
+converter = FooParquetConverter.new(data, compression: :gzip)
+
+# write result to file
+converter.write('tmp/hello_world.compressed.parquet')
+converter.write('tmp/hello_world.parquet', compression: false)
+
+# in-memory result (StringIO)
+converter.to_io
+
+# write to temporary file (Tempfile)
+# don't forget to `close` / `unlink` it after usage
+converter.to_tmpfile
+
+# Arrow Table
+table = converter.to_arrow_table
+table.each_record do |record|
+  # pp record['my_decimal'].to_f
+  pp record.to_h
+end

data/examples/readme-example.rb

@@ -0,0 +1,44 @@
+require 'bundler/setup'
+require 'parqueteur'
+
+class FooParquetConverter < Parqueteur::Converter
+  column :id, :bigint
+  column :reference, :string
+  column :datetime, :timestamp
+  column :beers_count, :integer
+
+  transform do |item|
+    item.merge(
+      'datetime' => Time.now
+    )
+  end
+
+  transform :add_beers
+
+  private
+
+  def add_beers(item)
+    item['beers_count'] += rand(1..3)
+    item
+  end
+end
+
+data = 10.times.lazy.map do |i|
+  { 'id' => i + 1, 'reference' => 'hello world 1', 'beers_count' => 0 }
+end
+
+# initialize Converter with Parquet GZIP compression mode
+converter = FooParquetConverter.new(data, compression: :gzip)
+
+# write result to file
+converter.write('tmp/hello_world.parquet')
+
+# in-memory result (StringIO)
+converter.to_io
+
+# write to temporary file (Tempfile)
+# don't forget to `close` / `unlink` it after usage
+converter.to_tmpfile
+
+# convert to Arrow::Table
+pp converter.to_arrow_table