ductr-postgres 0.1.0

data/COPYING.LESSER

@@ -0,0 +1,165 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/Gemfile

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in ductr-postgres.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,138 @@
+PATH
+  remote: .
+  specs:
+    ductr-postgres (0.1.0)
+      ductr (~> 0.1)
+      pg (~> 1.4)
+      sequel (~> 5.66)
+      sequel_pg (~> 1.8)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activejob (7.0.4.3)
+      activesupport (= 7.0.4.3)
+      globalid (>= 0.3.6)
+    activesupport (7.0.4.3)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 1.6, < 2)
+      minitest (>= 5.1)
+      tzinfo (~> 2.0)
+    annotable (0.1.2)
+    ast (2.4.2)
+    colorize (0.8.1)
+    commander (4.6.0)
+      highline (~> 2.0.0)
+    concurrent-ruby (1.2.2)
+    diff-lcs (1.5.0)
+    docile (1.4.0)
+    ductr (0.1.2)
+      activejob (~> 7.0)
+      annotable (~> 0.1)
+      colorize (~> 0.8)
+      kiba (~> 4.0)
+      rufus-scheduler (~> 3.8)
+      thor (~> 1.2)
+      zeitwerk (~> 2.6)
+    et-orbi (1.2.7)
+      tzinfo
+    fugit (1.8.1)
+      et-orbi (~> 1, >= 1.2.7)
+      raabro (~> 1.4)
+    globalid (1.1.0)
+      activesupport (>= 5.0)
+    highline (2.0.3)
+    i18n (1.12.0)
+      concurrent-ruby (~> 1.0)
+    json (2.6.3)
+    kiba (4.0.0)
+    minitest (5.18.0)
+    parallel (1.22.1)
+    parlour (5.0.0)
+      commander (~> 4.5)
+      parser
+      rainbow (~> 3.0)
+      sorbet-runtime (>= 0.5)
+    parser (3.2.1.1)
+      ast (~> 2.4.1)
+    pg (1.4.6)
+    raabro (1.4.0)
+    rainbow (3.1.1)
+    rake (13.0.6)
+    redcarpet (3.6.0)
+    regexp_parser (2.7.0)
+    rexml (3.2.5)
+    rspec (3.12.0)
+      rspec-core (~> 3.12.0)
+      rspec-expectations (~> 3.12.0)
+      rspec-mocks (~> 3.12.0)
+    rspec-core (3.12.1)
+      rspec-support (~> 3.12.0)
+    rspec-expectations (3.12.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.12.0)
+    rspec-mocks (3.12.4)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.12.0)
+    rspec-support (3.12.0)
+    rubocop (1.48.1)
+      json (~> 2.3)
+      parallel (~> 1.10)
+      parser (>= 3.2.0.0)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 1.8, < 3.0)
+      rexml (>= 3.2.5, < 4.0)
+      rubocop-ast (>= 1.26.0, < 2.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 2.4.0, < 3.0)
+    rubocop-ast (1.28.0)
+      parser (>= 3.2.1.0)
+    rubocop-capybara (2.17.1)
+      rubocop (~> 1.41)
+    rubocop-rspec (2.19.0)
+      rubocop (~> 1.33)
+      rubocop-capybara (~> 2.17)
+    ruby-progressbar (1.13.0)
+    rufus-scheduler (3.8.2)
+      fugit (~> 1.1, >= 1.1.6)
+    sequel (5.66.0)
+    sequel_pg (1.17.1)
+      pg (>= 0.18.0, != 1.2.0)
+      sequel (>= 4.38.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
+    sorbet-runtime (0.5.10736)
+    sord (4.0.0)
+      commander (~> 4.5)
+      parlour (~> 5.0)
+      sorbet-runtime
+      yard
+    thor (1.2.1)
+    tzinfo (2.0.6)
+      concurrent-ruby (~> 1.0)
+    unicode-display_width (2.4.2)
+    webrick (1.7.0)
+    yard (0.9.28)
+      webrick (~> 1.7.0)
+    zeitwerk (2.6.7)
+
+PLATFORMS
+  x86_64-linux
+
+DEPENDENCIES
+  ductr-postgres!
+  rake (~> 13.0)
+  redcarpet (~> 3.5)
+  rspec (~> 3.11)
+  rubocop (~> 1.29)
+  rubocop-rspec (~> 2.11)
+  simplecov (~> 0.21)
+  sord (~> 4.0)
+  yard (~> 0.9)
+
+BUNDLED WITH
+   2.4.8

data/README.md

@@ -0,0 +1,67 @@
+# PostgreSQL adapter for Ductr ETL
+This gem provides useful controls to operate Ductr with PostgreSQL databases.
+
+## Installation
+
+Add this line to your application's `Gemfile`:
+
+```ruby
+gem 'ductr-postgres'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+Require `ductr/postgres` in the `config/app.rb` file:
+
+```ruby
+require "ductr/postgres"
+```
+
+## Usage
+
+You can now use the adapter in your YAML configuration:
+
+```yml
+adapters:
+  some_postgres_database:
+    adapter: postgres
+    host: localhost
+    user: postgres
+    password: s3cr3t
+    database: example
+```
+
+You can pass any option recognized by sequel.
+See [the Sequel Sequel general options list](https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-General+connection+options) for further details.
+See [the Sequel PostgreSQL specific options list](https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-postgres) for further details.
+
+The configured adapter can now be used in Ductr jobs e.g.:
+
+```ruby
+source :some_postgres_database, :streamed
+def select_some_stuff(db, offset, limit)
+  db[:items].offset(offset).limit(limit)
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/ductr-io/ductr. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ductr-io/ductr-postgres/blob/main/CODE_OF_CONDUCT.md).
+
+## Code of Conduct
+
+Everyone interacting in the Ductr::PostgreSQL project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ductr-io/ductr-postgres/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [LGPLv3 or later](https://opensource.org/license/lgpl-3-0/).

data/Rakefile

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]
+
+require "sord"
+
+desc "Generates RBS file from the yard documentation"
+task :sord do
+  options = {
+    rbs: true,
+    sord_comments: true,
+    regenerate: true,
+    break_params: 4,
+    replace_errors_with_untyped: false,
+    replace_unresolved_with_untyped: false,
+    exclude_messages: nil,
+    include_messages: nil,
+    keep_original_comments: false,
+    skip_constants: false,
+    use_original_initialize_return: false,
+    exclude_untyped: false
+  }
+
+  plugin = Sord::ParlourPlugin.new(options)
+  plugin.parlour = Parlour::RbsGenerator.new(break_params: options[:break_params])
+  plugin.generate(plugin.parlour.root)
+
+  File.write("sig/ductr/postgres.rbs", plugin.parlour.send(:rbs))
+end

data/ductr-postgres.gemspec

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require_relative "lib/ductr/postgres/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "ductr-postgres"
+  spec.version = Ductr::Postgres::VERSION
+  spec.authors = ["Mathieu MOREL"]
+  spec.email = ["mathieu@lamanufacture.dev"]
+  spec.licenses = ["LGPL-3.0-or-later"]
+
+  spec.summary = "PostgreSQL adapter for Ductr using the `sequel` gem"
+  spec.description = "Allows ductr to interact with PostgreSQL DMBS."
+  spec.homepage = "https://github.com/ductr-io/ductr"
+  spec.required_ruby_version = ">= 3.1.0"
+
+  spec.metadata["allowed_push_host"] = "https://rubygems.org"
+  spec.metadata["rubygems_mfa_required"] = "true"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "#{spec.homepage}/releases"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path(__dir__)) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (f == __FILE__) || f.match(%r{\A(?:(?:bin|test|spec|features)/|\.(?:git|travis|circleci)|appveyor)})
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "ductr", "~> 0.1"
+  spec.add_dependency "pg", "~> 1.4"
+  spec.add_dependency "sequel", "~> 5.66"
+  spec.add_dependency "sequel_pg", "~> 1.8"
+
+  spec.add_development_dependency "rake", "~> 13.0"
+  spec.add_development_dependency "redcarpet", "~> 3.5"
+  spec.add_development_dependency "rspec", "~> 3.11"
+  spec.add_development_dependency "rubocop", "~> 1.29"
+  spec.add_development_dependency "rubocop-rspec", "~> 2.11"
+  spec.add_development_dependency "simplecov", "~> 0.21"
+  spec.add_development_dependency "sord", "~> 4.0"
+  spec.add_development_dependency "yard", "~> 0.9"
+end

data/lib/ductr/postgres/adapter.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module Ductr
+  module Postgres
+    #
+    # The PostgreSQL adapter implement the required #open! and #close! methods to handle the database connection.
+    # The adapter is registered as `:postgres` to use it, add `adapter: postgres` to the YAML configuration e.g.:
+    #
+    # ```yml
+    # # config/development.yml
+    # adapters:
+    #   some_postgres_database:
+    #     adapter: postgres
+    #     host: localhost
+    #     user: postgres
+    #     password: s3cr3t
+    #     database: example
+    # ```
+    #
+    # @see https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-General+connection+options
+    #   General sequel options
+    # @see https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-postgres
+    #   PostgreSQL specific options
+    #
+    class Adapter < Ductr::Adapter
+      Ductr.adapter_registry.add(self, as: :postgres)
+
+      # @return [Sequel::Database, nil] The database connection instance
+      attr_reader :db
+
+      #
+      # Opens the database connection with the adapter's configuration.
+      #
+      # @return [Sequel::Database] The database connection instance
+      #
+      def open!
+        @db = Sequel.postgres(**config)
+
+        @db.extension(:pg_streaming)
+        @db.stream_all_queries = true
+
+        @db
+      end
+
+      #
+      # Closes the database connection.
+      #
+      # @return [void]
+      #
+      def close!
+        @db.disconnect
+      end
+    end
+  end
+end

data/lib/ductr/postgres/basic_lookup.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Ductr
+  module Postgres
+    #
+    # A lookup control that execute one query per row, registered as `:basic`.
+    # The job's method must return a row which will merged with the current row:
+    #
+    #   lookup :some_postgres_database, :basic
+    #   def my_lookup(db, row)
+    #     db[:items_bis].where(item: row[:id]).limit(1)
+    #   end
+    #
+    # As the control merge the looked up row with the current row,
+    # ensure that column names are different or they will be overwritten.
+    #
+    # If the lookup returns a falsy value, nothing won't be merged with the current row.
+    #
+    class BasicLookup < Ductr::ETL::Transform
+      Adapter.lookup_registry.add(self, as: :basic)
+
+      #
+      # Calls the job's method to merge its result with the current row.
+      #
+      # @param [Hash<Symbol, Object>] row The current row, preferably a Hash
+      #
+      # @return [Hash<Symbol, Object>] The row merged with looked up row or the untouched row if nothing was found
+      #
+      def process(row)
+        matching_row = call_method(adapter.db, row).first
+        return row unless matching_row
+
+        row.merge matching_row
+      end
+    end
+  end
+end

data/lib/ductr/postgres/buffered_destination.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module Ductr
+  module Postgres
+    #
+    # A destination control that accumulates rows in a buffer to write them by batch, registered as `:buffered`.
+    # Accept the `:buffer_size` option, default value is 10 000:
+    #
+    #   destination :some_postgres_database, :buffered, buffer_size: 42
+    #   def my_destination(db, buffer)
+    #     db[:items].multi_insert(buffer)
+    #   end
+    #
+    # @see more Ductr::ETL::BufferedDestination
+    #
+    class BufferedDestination < Ductr::ETL::BufferedDestination
+      Adapter.destination_registry.add(self, as: :buffered)
+
+      #
+      # Open the database if needed and call the job's method to run the query.
+      #
+      # @return [void]
+      #
+      def on_flush
+        call_method(adapter.db, buffer)
+      end
+    end
+  end
+end

data/lib/ductr/postgres/buffered_lookup.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Ductr
+  module Postgres
+    #
+    # A lookup control that execute the query for a bunch of rows, registered as `:buffered`.
+    # Accept the `:buffer_size` option, default value is 10 000.
+    # You have to implement your own row matching logic:
+    #
+    #   lookup :some_postgres_database, :buffered, buffer_size: 42
+    #   def my_lookup(db, buffer, &)
+    #     ids = buffer.map {|row| row[:id]}
+    #     db[:items].where(item: ids).each do |row|
+    #       match = buffer.find { |r| r[:id] == row[:item] }
+    #
+    #       next yield(row) unless match
+    #
+    #       yield(row.merge match)
+    #     end
+    #   end
+    #
+    class BufferedLookup < Ductr::ETL::BufferedTransform
+      Adapter.lookup_registry.add(self, as: :buffered)
+
+      #
+      # Opens the database if needed, calls the job's method and pass the each block to it.
+      #
+      # @yield The each block
+      #
+      # @return [void]
+      #
+      def on_flush(&)
+        call_method(adapter.db, buffer, &)
+      end
+    end
+  end
+end

data/lib/ductr/postgres/buffered_upsert_destination.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Ductr
+  module Postgres
+    #
+    # A destination control that accumulates rows in a buffer to upsert them by batch, registered as `:buffered_upsert`.
+    # Accept the `:buffer_size` option, default value is 10 000:
+    #
+    #   destination :some_postgres_database, :buffered_upsert, buffer_size: 42
+    #   def my_destination(buffer, excluded, db)
+    #     db[:items].insert_conflict(target: :id, update: excluded).multi_insert(buffer)
+    #   end
+    #
+    # @see more Ductr::ETL::BufferedDestination
+    #
+    class BufferedUpsertDestination < Ductr::ETL::BufferedDestination
+      Adapter.destination_registry.add(self, as: :buffered_upsert)
+
+      #
+      # Open the database if needed and call the job's method to run the query.
+      #
+      # @return [void]
+      #
+      def on_flush
+        call_method(adapter.db, excluded, buffer)
+      end
+
+      private
+
+      #
+      # Generate the excluded keys hash e.g.
+      #
+      # ```ruby
+      # {a: Sequel[:excluded][:a]}
+      # ```
+      #
+      # @return [Hash<Symbol, Sequel::SQL::QualifiedIdentifier>] The excluded keys hash
+      #
+      def excluded
+        keys = buffer.first.keys
+
+        excluded_keys = keys.map do |key|
+          Sequel[:excluded][key]
+        end
+
+        keys.zip(excluded_keys).to_h
+      end
+    end
+  end
+end