gunwale 0.5.0

data/README.md

@@ -0,0 +1,102 @@
+# RowBoat
+
+Created by &nbsp;&nbsp;&nbsp; [<img src="https://raw.githubusercontent.com/devmynd/row_boat/master/devmynd-logo.png" alt="DevMynd Logo" />](https://www.devmynd.com/)
+
+[![Gem Version](https://badge.fury.io/rb/row_boat.svg)](http://badge.fury.io/rb/row_boat) &nbsp;&nbsp;&nbsp;[![Build Status](https://travis-ci.org/devmynd/row_boat.svg?branch=master)](https://travis-ci.org/devmynd/row_boat)
+
+A simple gem to help you import CSVs into your ActiveRecord models.
+
+[Check out the documentation!](/API.md#rowboat-api)
+
+It uses [SmarterCSV](https://github.com/tilo/smarter_csv) and [`activerecord-import`](https://github.com/zdennis/activerecord-import) to import database records from your CSVs.
+
+## Contents
+
+- [Installation](#installation)
+- [Basic Usage](#basic-usage)
+- [Development](#development)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "row_boat", "~> 0.4"
+```
+
+And then execute:
+
+    $ bundle
+
+## Basic Usage
+
+#### [Full documentation can be found here.](/API.md#rowboat-api)
+
+Below we're defining the required methods ([`import_into`](/API.md#import_into) and [`column_mapping`](/API.md#column_mapping)) and a few additional options as well (via [`value_converters`](/API.md#value_converters) and [`options`](/API.md#options)). Checkout [API.md](/API.md#rowboat-api) for the full documentation for more details :)
+
+```ruby
+class ImportProduct < RowBoat::Base
+  # required
+  def import_into
+    Product # The ActiveRecord class we want to import records into.
+  end
+
+  # required
+  def column_mapping
+    {
+      # `:prdct_name` is the downcased and symbolized version
+      # of our column header, while `:name` is the attribute
+      # of our model we want to receive `:prdct_name`'s value
+      prdct_name: :name,
+      dllr_amnt: :price_in_cents,
+      desc: :description
+    }
+  end
+
+  # optional
+  def value_converters
+    {
+      # Allows us to change values we want to import
+      # before we import them
+      price_in_cents: -> (value) { value * 1000 }
+    }
+  end
+
+  # optional
+  def preprocess_row(row)
+    if row[:name] && row[:description] && row[:price]
+      row
+    else
+      nil # return nil to skip a row
+    end
+    # we could also remove some attributes or do any
+    # other kind of work we want with the given row.
+  end
+
+  #optional
+  def options
+    {
+      # These are additional configurations that
+      # are generally passed through to SmarterCSV
+      # and activerecord-import
+      validate: false, # this defaults to `true`
+      wrap_in_transaction: false # this defaults to `true`
+    }
+  end
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake` to run the tests (run `appraisal install` and `appraisal rake` to run the tests against different combinations of dependencies). You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/devmynd/row_boat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+require "standalone_migrations"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+StandaloneMigrations::Tasks.load_tasks
+
+task default: %i[rubocop spec]

data/bin/console

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "row_boat"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here
+bundle exec rake db:create
+bundle exec rake db:migrate

data/db/config.yml

@@ -0,0 +1,12 @@
+default: &default
+  adapter: postgresql
+  database: row_boat
+  username: postgres
+  password: postgres
+  host: localhost
+
+development:
+  <<: *default
+
+test:
+  <<: *default

data/db/migrate/20170511160154_create_products.rb

@@ -0,0 +1,13 @@
+class CreateProducts < ActiveRecord::Migration[5.1]
+  def change
+    create_table :products do |t|
+      t.string :name, null: false
+      t.integer :rank, null: false
+      t.text :description
+
+      t.timestamps
+    end
+
+    add_index :products, :rank, unique: true
+  end
+end

data/db/schema.rb

@@ -0,0 +1,26 @@
+# This file is auto-generated from the current state of the database. Instead
+# of editing this file, please use the migrations feature of Active Record to
+# incrementally modify your database, and then regenerate this schema definition.
+#
+# This file is the source Rails uses to define your schema when running `bin/rails
+# db:schema:load`. When creating a new database, `bin/rails db:schema:load` tends to
+# be faster and is potentially less error prone than running all of your
+# migrations from scratch. Old migrations may fail to apply correctly if those
+# migrations use external dependencies or application code.
+#
+# It's strongly recommended that you check this file into your version control system.
+
+ActiveRecord::Schema[7.1].define(version: 2017_05_11_160154) do
+  # These are extensions that must be enabled in order to support this database
+  enable_extension "plpgsql"
+
+  create_table "products", force: :cascade do |t|
+    t.string "name", null: false
+    t.integer "rank", null: false
+    t.text "description"
+    t.datetime "created_at", precision: nil, null: false
+    t.datetime "updated_at", precision: nil, null: false
+    t.index ["rank"], name: "index_products_on_rank", unique: true
+  end
+
+end

data/gemfiles/csv_import_1.1_and_0.18.gemfile

@@ -0,0 +1,8 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "activerecord-import", "~> 0.18.2"
+gem "smarter_csv", "~> 1.1.0"
+
+gemspec path: "../"

data/gemfiles/csv_import_1.1_and_0.18.gemfile.lock

@@ -0,0 +1,144 @@
+PATH
+  remote: ..
+  specs:
+    row_boat (0.5.0)
+      activerecord (>= 5.0.0)
+      activerecord-import (~> 0.18.2)
+      smarter_csv (~> 1.1)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    actionpack (5.1.0)
+      actionview (= 5.1.0)
+      activesupport (= 5.1.0)
+      rack (~> 2.0)
+      rack-test (~> 0.6.3)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.0, >= 1.0.2)
+    actionview (5.1.0)
+      activesupport (= 5.1.0)
+      builder (~> 3.1)
+      erubi (~> 1.4)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.0, >= 1.0.3)
+    activemodel (5.1.0)
+      activesupport (= 5.1.0)
+    activerecord (5.1.0)
+      activemodel (= 5.1.0)
+      activesupport (= 5.1.0)
+      arel (~> 8.0)
+    activerecord-import (0.18.3)
+      activerecord (>= 3.2)
+    activesupport (5.1.0)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (~> 0.7)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+    appraisal (2.2.0)
+      bundler
+      rake
+      thor (>= 0.14.0)
+    arel (8.0.0)
+    ast (2.3.0)
+    awesome_print (1.7.0)
+    builder (3.2.3)
+    coderay (1.1.1)
+    concurrent-ruby (1.0.5)
+    database_cleaner (1.6.1)
+    diff-lcs (1.3)
+    erubi (1.6.0)
+    i18n (0.8.1)
+    loofah (2.0.3)
+      nokogiri (>= 1.5.9)
+    method_source (0.8.2)
+    mini_portile2 (2.1.0)
+    minitest (5.10.2)
+    nokogiri (1.7.2)
+      mini_portile2 (~> 2.1.0)
+    parser (2.4.0.0)
+      ast (~> 2.2)
+    pg (0.20.0)
+    powerpack (0.1.1)
+    pry (0.10.4)
+      coderay (~> 1.1.0)
+      method_source (~> 0.8.1)
+      slop (~> 3.4)
+    pry-doc (0.10.0)
+      pry (~> 0.9)
+      yard (~> 0.9)
+    pry-nav (0.2.4)
+      pry (>= 0.9.10, < 0.11.0)
+    rack (2.0.2)
+    rack-test (0.6.3)
+      rack (>= 1.0)
+    rails-dom-testing (2.0.3)
+      activesupport (>= 4.2.0)
+      nokogiri (>= 1.6)
+    rails-html-sanitizer (1.0.3)
+      loofah (~> 2.0)
+    railties (5.1.0)
+      actionpack (= 5.1.0)
+      activesupport (= 5.1.0)
+      method_source
+      rake (>= 0.8.7)
+      thor (>= 0.18.1, < 2.0)
+    rainbow (2.2.2)
+      rake
+    rake (10.5.0)
+    rspec (3.6.0)
+      rspec-core (~> 3.6.0)
+      rspec-expectations (~> 3.6.0)
+      rspec-mocks (~> 3.6.0)
+    rspec-core (3.6.0)
+      rspec-support (~> 3.6.0)
+    rspec-expectations (3.6.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.6.0)
+    rspec-mocks (3.6.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.6.0)
+    rspec-support (3.6.0)
+    rubocop (0.48.1)
+      parser (>= 2.3.3.1, < 3.0)
+      powerpack (~> 0.1)
+      rainbow (>= 1.99.1, < 3.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.0, >= 1.0.1)
+    ruby-progressbar (1.8.1)
+    slop (3.6.0)
+    smarter_csv (1.1.4)
+    standalone_migrations (5.2.1)
+      activerecord (>= 4.2.7, < 5.2.0)
+      railties (>= 4.2.7, < 5.2.0)
+      rake (~> 10.0)
+    thor (0.19.4)
+    thread_safe (0.3.6)
+    tzinfo (1.2.3)
+      thread_safe (~> 0.1)
+    unicode-display_width (1.2.1)
+    yard (0.9.9)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activerecord-import (~> 0.18.2)
+  appraisal (~> 2.2.0)
+  awesome_print
+  bundler (~> 1.14)
+  database_cleaner (~> 1.6.0)
+  pg
+  pry
+  pry-doc
+  pry-nav
+  rake (~> 10.0)
+  row_boat!
+  rspec (~> 3.0)
+  rubocop (~> 0.48.1)
+  smarter_csv (~> 1.1.0)
+  standalone_migrations (~> 5.2.0)
+  yard (~> 0.9.9)
+
+BUNDLED WITH
+   1.15.1

data/lib/row_boat/base.rb

@@ -0,0 +1,318 @@
+# frozen_string_literal: true
+
+require "active_record"
+require "activerecord-import"
+require "smarter_csv"
+
+module RowBoat
+  class Base
+    InvalidColumnMapping = Class.new(StandardError)
+
+    attr_reader :csv_source
+
+    class << self
+      # Imports database records from the given CSV-like object.
+      #
+      # @overload import(csv_source)
+      #   @param csv_source [String, #read] a CSV-like object that SmarterCSV can read.
+      #
+      # @return [Hash] a hash with +:invalid_records+, +:total_inserts+, +:inserted_ids+, and +:skipped_rows+.
+      #
+      # @see https://github.com/tilo/smarter_csv#documentation SmarterCSV Docs
+      def import(*args, &block)
+        new(*args, &block).import
+      end
+    end
+
+    # Makes a new instance with the given +csv_source+.
+    #
+    # @abstract Override this method if you need additional arguments to process your CSV (like defaults).
+    #
+    # @example
+    #   def initialize(csv_source, default_name)
+    #     super(csv_source)
+    #     @default_name = default_name
+    #   end
+    def initialize(csv_source)
+      @csv_source = csv_source
+    end
+
+    # Parses the csv and inserts/updates the database. You probably won't call this method directly,
+    # instead you would call {RowBoat::Base.import}.
+    #
+    # @return [Hash] a hash with +:invalid_records+, +:total_inserts+, +:inserted_ids+, and +:skipped_rows+.
+    def import
+      import_results = []
+
+      transaction_if_needed do
+        parse_rows do |rows|
+          import_results << import_rows(rows)
+        end
+      end
+
+      process_import_results(import_results).tap do |total_results|
+        handle_failed_rows(total_results[:invalid_records])
+      end
+    end
+
+    # Override with the ActiveRecord class that the CSV should be imported into.
+    #
+    # @abstract
+    #
+    # @note You must implement this method.
+    #
+    # @example
+    #   def import_into
+    #     Product
+    #   end
+    def import_into
+      raise NotImplementedError, not_implemented_error_message(__method__)
+    end
+
+    # Override with a hash that maps CSV column names to their preferred names.
+    #   Oftentimes these are the names of the attributes on the model class from {#import_into}.
+    #
+    # @abstract
+    #
+    # @note You must implement this method.
+    #
+    # @example
+    #   def column_mapping
+    #     {
+    #       prdct_name: :name,
+    #       price: :price,
+    #       sl_exp: :sale_expires_at
+    #     }
+    #   end
+    #
+    # @see #import_into
+    def column_mapping
+      raise NotImplementedError, not_implemented_error_message(__method__)
+    end
+
+    # Override this method if you need to do some work on the row before the record is
+    #   inserted/updated or want to skip the row in the import. Simply return +nil+ to skip the row.
+    #
+    # @abstract
+    #
+    # @note If you only need to manipulate one attribute (ie parse a date from a string, etc), then
+    #   you should probably use {#value_converters}
+    #
+    # @return [Hash,NilClass] a hash of attributes, +nil+, or even and instance of the class returned
+    #   in {#import_into}.
+    #
+    # @see #import_into
+    def preprocess_row(row)
+      row
+    end
+
+    # @api private
+    def import_rows(rows)
+      import_options = ::RowBoat::Helpers.extract_import_options(merged_options)
+      preprocessed_rows = preprocess_rows(rows)
+      import_into.import(preprocessed_rows, import_options)
+    end
+
+    # Override this method if you need to do something with a chunk of rows.
+    #
+    # @abstract
+    #
+    # @note If you want to filter out a row, you can just return +nil+ from {#preprocess_row}.
+    #
+    # @see #preprocess_row
+    def preprocess_rows(rows)
+      rows.each_with_object([]) do |row, preprocessed_rows|
+        increment_row_number
+        preprocessed_row = preprocess_row(row)
+        preprocessed_row ? preprocessed_rows << preprocessed_row : add_skipped_row(row)
+      end
+    end
+
+    # Override this method to specify CSV parsing and importing options.
+    #   All SmarterCSV and activerecord-import options can be listed here along with
+    #   +:wrap_in_transaction+. The defaults provided by RowBoat can be found in {#default_options}
+    #
+    # @abstract
+    #
+    # @note If you want to use the +:value_converters+ option provided by SmarterCSV
+    #   just override {#value_converters}.
+    #
+    # @return [Hash] a hash of configuration options.
+    #
+    # @see https://github.com/tilo/smarter_csv#documentation SmarterCSV docs
+    # @see https://github.com/zdennis/activerecord-import/wiki activerecord-import docs
+    # @see #value_converters
+    def options
+      {}
+    end
+
+    # Default options provided by RowBoat for CSV parsing and importing.
+    #
+    # @note Do not override.
+    #
+    # @return [Hash] a hash of configuration options.
+    #
+    # @api private
+    def default_options
+      {
+        chunk_size: 500,
+        recursive: false,
+        validate: true,
+        value_converters: csv_value_converters,
+        wrap_in_transaction: true
+      }.merge(column_mapping_options)
+    end
+
+    # @api private
+    def merged_options
+      default_options.merge(options)
+    end
+
+    # Override this method to do some work with a row that has failed to import.
+    #
+    # @abstract
+    #
+    # @note +row+ here is actually an instance of the class returned in {#import_into}
+    #
+    # @see #import_into
+    def handle_failed_row(row)
+      row
+    end
+
+    # Override this method to do some work will all of the rows that failed to import.
+    #
+    # @abstract
+    #
+    # @note If you override this method and {#handle_failed_row}, be sure to call +super+.
+    def handle_failed_rows(rows)
+      rows.each { |row| handle_failed_row(row) }
+    end
+
+    # Override this method to specify how to translate values from the CSV
+    #   into ruby objects.
+    #
+    #   You can provide an object that implements +convert+, a proc or lambda, or the
+    #   the name of a method as a Symbol
+    #
+    # @abstract
+    #
+    # @example
+    #   def value_converters
+    #     {
+    #       name: -> (value) { value.titleize }
+    #       price: :convert_price,
+    #       expires_at: CustomDateConverter
+    #     }
+    #   end
+    #
+    #   def convert_price(value)
+    #     value || 0
+    #   end
+    def value_converters
+      {}
+    end
+
+    # @api private
+    def csv_value_converters
+      value_converters.each_with_object({}) do |(key, potential_converter), converters_hash|
+        case potential_converter
+        when Proc
+          converters_hash[key] = ::RowBoat::ValueConverter.new(&potential_converter)
+        when Symbol
+          converters_hash[key] = ::RowBoat::ValueConverter.new { |value| public_send(potential_converter, value) }
+        when nil
+          next
+        else
+          converters_hash[key] = potential_converter
+        end
+      end
+    end
+
+    # Implement this method if you'd like to rollback the transaction
+    #   after it otherwise has completed.
+    #
+    # @abstract
+    #
+    # @note Only works if the `wrap_in_transaction` option is `true`
+    #   (which is the default)
+    #
+    # @return [Boolean]
+    def rollback_transaction?
+      false
+    end
+
+    private
+
+    # @private
+    attr_reader :row_number
+
+    # @api private
+    # @private
+    attr_reader :skipped_rows
+
+    # @api private
+    # @private
+    def increment_row_number
+      @row_number = row_number.to_i + 1
+    end
+
+    def add_skipped_row(row)
+      @skipped_rows ||= []
+      skipped_rows << row
+    end
+
+    # @api private
+    # @private
+    def column_mapping_options
+      case column_mapping
+      when Hash
+        { key_mapping: column_mapping, remove_unmapped_keys: true }
+      when Array
+        { user_provided_headers: column_mapping }
+      else
+        raise InvalidColumnMapping, "#column_mapping must be a Hash or an Array: got `#{column_mapping}`"
+      end
+    end
+
+    # @api private
+    # @private
+    def not_implemented_error_message(method_name)
+      "Subclasses of #{self.class.name} must implement `#{method_name}`"
+    end
+
+    # @api private
+    # @private
+    def parse_rows(&block)
+      csv_options = ::RowBoat::Helpers.extract_csv_options(merged_options)
+      ::SmarterCSV.process(csv_source, csv_options, &block)
+    end
+
+    # @api private
+    # @private
+    def transaction_if_needed
+      if merged_options[:wrap_in_transaction]
+        import_into.transaction do
+          yield
+          raise ActiveRecord::Rollback if rollback_transaction?
+        end
+      else
+        yield
+      end
+    end
+
+    # @api private
+    # @private
+    def process_import_results(import_results)
+      import_results.each_with_object(
+        invalid_records: [],
+        total_inserts: 0,
+        inserted_ids: [],
+        skipped_rows: skipped_rows
+      ) do |import_result, total_results|
+        total_results[:invalid_records] += import_result.failed_instances
+        total_results[:total_inserts] += import_result.num_inserts
+        total_results[:inserted_ids] += import_result.ids
+      end
+    end
+  end
+end