shopify_transporter 1.0.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e731ee32bf65a13db82226613fcdfe4de06e5dcd
-  data.tar.gz: 6f52f02f93b67cc3bb96a709ccc895126bd2dfe6
+  metadata.gz: 51ea8531c355faaa542a5cac769df64be88ab1c7
+  data.tar.gz: 8b46bab5dade9319600c9da397967f55aa7724ad
 SHA512:
-  metadata.gz: 624777e136f3cb0d659935f6fbe5152ccba71e17fa38e0b884bad43074f5e730315916aa52250758255dd28aec3c47fabc2042567f22b5a217a40d6a5ed47c2c
-  data.tar.gz: 505991f408682f301e7af3be5e4805e74ed2130bbb3c8ed3d42ae85ec1f5159bf6ea49350979e374924163d85ddb0db9fc43432415b5ad6e096129405a3a55dd
+  metadata.gz: 60200c955aa3dd02d952835992f0b59e0cf1a8667daf45f7907ec346ed0b6df0b10cd3ff1b466d3a989c4ebf37bff1668a348f932e3308cbc6e31454107f0b87
+  data.tar.gz: ff9d1c553e7617b9d84ceb0c933bbcfa3477a91e13e92986ef0e93f953ad69c030f5f92e5e5e90825f74e7b6fd672593a5b074fccb5873009f4b6e70a7a552b2

data/Gemfile

@@ -6,7 +6,7 @@ gemspec
 
 group :development do
   gem 'package_cloud'
-  gem 'rubocop'
+  gem 'rubocop', '~> 0.56.0'
   gem 'rubocop-git', '~> 0.1'
 end
 

data/README.md

@@ -9,29 +9,25 @@ data, will be made available in the near future.
 
 Note: the Transporter app is available to Shopify Plus plans only.
 
-## Installation
+## Submitting Issues
 
-### Building and installing locally
-To build locally run the following command:
+Please open an issue here if you encounter a specific bug with this library or if something is documented
+incorrectly.
 
-```
-$ bundle exec rake build
+When filing an issue, please ensure that:
 
-shopify_transporter 1.0.0 built to pkg/shopify_transporter-1.0.0.gem.
-```
+- The issue is not already covered in another open issue
+- The issue does not contain any confidential or personally identifiable information
+- The issue is specifically regarding the `shopify_transporter` gem and not related to the Transporter App.
 
-Then, you can install the gem system-wide by running:
 
-```
-$ gem install pkg/shopify_transporter-1.0.0.gem
+## Installation
 
-Successfully installed shopify_transporter-1.0.0
-Parsing documentation for shopify_transporter-1.0.0
-Installing ri documentation for shopify_transporter-1.0.0
-Done installing documentation for shopify_transporter after 0 seconds
-1 gem installed
-```
-Your locally built gem is now installed system-wide.
+### Requirements:
+1. A Ruby version of 2.4.0 or higher
+2. You are able to install Ruby gems. Please visit [the Bundler website](https://bundler.io/) to troubleshoot issues with installing gems.
+
+We test and support the gem for Mac OS environments. While not officially supported, the gem may work on other operating systems provided they meet the above requirements.
 
 ### Installing the Transporter tool gem from rubygems:
 
@@ -60,7 +56,7 @@ Commands:
   shopify_transporter new PROJECTNAME --platform=PLATFORM                             # Generates a new project structure for a platform
 ```
 
-## Create a conversion project
+### Create a conversion project
 
 It's convenient to create a conversion project for each store that you want to migrate to Shopify.
 To create it, use the `new` sub-command:
@@ -96,7 +92,7 @@ $ bundle install
 ```
 
 
-## Convert records from the third-party platform to Shopify
+### Convert records from the third-party platform to Shopify
 
 Run `shopify_transporter` and use the `convert` command to convert your objects from the
 third-party platform to the Shopify format. For example, the following command converts a
@@ -109,7 +105,7 @@ shopify_transporter convert --config=config.yml --object=customer magento_custom
 In this example, the converted customer objects are saved to *shopify_customers.csv*. If errors occur during
 the conversion, then they appear in your terminal.
 
-## Convert multiple files
+### Convert multiple files
 
 To convert multiple files to Shopify, separate the file names with a space:
 
@@ -117,7 +113,7 @@ To convert multiple files to Shopify, separate the file names with a space:
 shopify_transporter convert --config=config.yml --object=customer magento_customers_1.json magento_customers_2.json ...
 ```
 
-## Configuration file (_config.yml_)
+### Configuration file (_config.yml_)
 
 The configuraton file is generated when you create your conversion project. This file is specific to the
 third-party platform that you are converting to Shopify.
@@ -209,7 +205,6 @@ Existing pipeline stages and the attributes are populated below.
 
 ### Magento v1.x customer
 
-
 ### AddressesAttribute
 
 Addresses are built from the `shipping_` and `billing_` prefixed fields.  The Shopify object's `addresses`
@@ -266,25 +261,38 @@ A file named `your_custom_stage.rb` is added to the `lib/magento/custom_pipeline
 The `convert` command currently only converts customer and order JSON objects that have been exported by SOAP API
 from Magento 1.x.
 
-## Running unit tests
+## Contributing
+
+### Running unit tests
 
 We use rspec to run our test suite:
 `bundle exec rspec`
 
-## Running the linter
+### Running the linter
 
 It's important that all of the code introduced passes linting. To run it manually:
 `bundle exec rake rubocop`
 To automatically resolve any basic linting issues:
 `bundle exec rake rubocop:autocorrect`
 
-# Submitting Issues
+### Building and installing locally
+To build locally run the following command:
 
-Please open an issue here if you encounter a specific bug with this library or if something is documented
-incorrectly.
+```
+$ bundle exec rake build
 
-When filing an issue, please ensure that:
+shopify_transporter 1.0.0 built to pkg/shopify_transporter-1.0.0.gem.
+```
 
-- The issue is not already covered in another open issue
-- The issue does not contain any confidential or personally identifying information
-- The issue is specifically regarding the `shopify_transporter` gem and not related to the Transporter App.
+Then, you can install the gem system-wide by running:
+
+```
+$ gem install pkg/shopify_transporter-1.0.0.gem
+
+Successfully installed shopify_transporter-1.0.0
+Parsing documentation for shopify_transporter-1.0.0
+Installing ri documentation for shopify_transporter-1.0.0
+Done installing documentation for shopify_transporter after 0 seconds
+1 gem installed
+```
+Your locally built gem is now installed system-wide.

data/exe/shopify_transporter

@@ -7,6 +7,8 @@ require 'thor'
 require 'thor/group'
 require 'shopify_transporter'
 require 'yaml'
+require 'io/console'
+require 'active_support/core_ext/string'
 
 module ShopifyTransporter
   class CLI < Thor
@@ -15,9 +17,9 @@ module ShopifyTransporter
     end
 
     method_option :config, type: :string, default: 'config.yml',
-      desc: "The config file that lists the conversions to run."
+      desc: 'The config file that lists the conversions to run.'
     method_option :object, type: :string, required: true,
-      desc: "The object type (customer, product, or order) to convert.",
+      desc: 'The object type (customer, product, or order) to convert.',
       enum: %w(customer product order)
 
     desc 'convert FILE_NAMES', 'Converts objects into a Shopify-format. (accepts a list of space-separated files).'
@@ -29,15 +31,44 @@ module ShopifyTransporter
       conversion_error(error.message)
     end
 
+    method_option :config, type: :string, default: 'config.yml',
+      desc: 'The config file that describes the third party platform to extract from.'
+    method_option :object, type: :string, required: true,
+      desc: 'The object type (customer, product, or order) to extract.',
+      enum: %w(customer product order)
+    method_option :first_id, type: :numeric, required: true,
+      desc: 'The first product_id, customer_id, or order_id for the range of objects you are trying to export.'
+    method_option :last_id, type: :numeric, required: true,
+      desc: 'The last product_id, customer_id, or order_id (inclusive) for the range of objects you are trying' \
+        ' to export.'
+    method_option :batch_size, type: :numeric, default: 1000,
+      desc: 'The number of records that are retrieved from the source platform at one time. ' \
+        'Use a smaller batch size if you are running into performance issues extracting your data.'
+
+    desc 'extract', 'Extracts objects from a third-party platform into a format compatible for conversion.'
+    def extract
+      batch_config = options.slice('first_id', 'last_id', 'batch_size')
+      exporter = Exporters::Exporter.new(options[:config], options[:object], batch_config)
+      exporter.run
+    rescue Exporters::ExportError => error
+      extract_error(error.message)
+    end
+
     no_commands do
       def conversion_error(message)
         say("Conversion error: #{message}", :red)
         exit 1
       end
+
+      def extract_error(message)
+        say("Extract error: #{message}", :red)
+        exit 1
+      end
     end
 
     register ShopifyTransporter::New,
       'new', 'new PROJECTNAME --platform=PLATFORM', 'Generate a project for the platform'
+
     register ShopifyTransporter::Generate,
       'generate', 'generate STAGE_NAME --object=OBJECT',
       'Generate a custom pipeline stage for the object'

data/lib/shopify_transporter/exporters/exporter.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module ShopifyTransporter
+  module Exporters
+    class ExportError < StandardError; end
+
+    class InvalidConfigError < ExportError
+      def initialize(error_message)
+        super("Invalid configuration: #{error_message}")
+      end
+    end
+
+    class Exporter
+      def initialize(config_filename, object_type, batch_config)
+        @object_type = object_type
+        @batch_config = batch_config
+
+        load_config(config_filename)
+      end
+
+      def run
+        print_exported_objects
+      end
+
+      private
+
+      attr_reader :config, :object_type
+
+      def print_exported_objects
+        $stderr.puts 'Starting extraction...'
+        puts '['
+        first = true
+        object_exporter.export do |object|
+          puts ',' unless first
+          first = false
+          $stderr.puts "Extracting #{object_type}: #{object[object_exporter.key]}..."
+          print '  ' + JSON.pretty_generate(object, object_nl: "\n  ")
+        end
+      ensure
+        print "\n]"
+      end
+
+      def object_exporter
+        @exporter ||= Magento::MagentoExporter
+          .for(type: object_type)
+          .new(
+            soap_client: soap_client,
+            database_adapter: database_adapter
+          )
+      end
+
+      def soap_client
+        Magento::Soap.new(
+          hostname: config['extract_configuration']['soap']['hostname'],
+          username: config['extract_configuration']['soap']['username'],
+          api_key: config['extract_configuration']['soap']['api_key'],
+          batch_config: @batch_config
+        )
+      end
+
+      def database_adapter
+        Magento::SQL.new(
+          database: config['extract_configuration']['database']['database'],
+          host: config['extract_configuration']['database']['host'],
+          user: config['extract_configuration']['database']['user'],
+          port: config['extract_configuration']['database']['port'],
+          password: config['extract_configuration']['database']['password'],
+        )
+      end
+
+      def load_config(config_filename)
+        @config ||= begin
+          raise InvalidConfigError, "cannot find file name '#{config_filename}'" unless File.exist?(config_filename)
+          YAML.load_file(config_filename)
+        end
+        ensure_config_has_required_keys
+      end
+
+      def ensure_config_has_required_keys
+        base_required_keys = [
+          %w(extract_configuration),
+          %w(extract_configuration soap hostname),
+          %w(extract_configuration soap username),
+          %w(extract_configuration soap api_key),
+        ]
+
+        product_required_keys = [
+          %w(extract_configuration database host),
+          %w(extract_configuration database port),
+          %w(extract_configuration database database),
+          %w(extract_configuration database user),
+          %w(extract_configuration database password),
+        ]
+
+        required_keys = base_required_keys + (@object_type == 'product' ? product_required_keys : [])
+
+        required_keys.each do |keys|
+          raise InvalidConfigError, "missing required key '#{keys.join(' > ')}'" unless config.dig(*keys)
+        end
+      end
+    end
+  end
+end

data/lib/shopify_transporter/exporters/magento/customer_exporter.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module ShopifyTransporter
+  module Exporters
+    module Magento
+      class CustomerExporter
+        def initialize(soap_client: nil, database_adapter: nil)
+          @client = soap_client
+          @database_adapter = database_adapter
+        end
+
+        def key
+          :customer_id
+        end
+
+        def export
+          base_customers.each do |customer|
+            yield with_attributes(customer)
+          end
+        end
+
+        private
+
+        def with_attributes(base_customer)
+          base_customer.merge(address_list: customer_address_list(base_customer[:customer_id]))
+        end
+
+        def base_customers
+          Enumerator.new do |enumerator|
+            @client.call_in_batches(method: :customer_customer_list, batch_index_column: 'customer_id').each do |batch|
+              result = batch.body[:customer_customer_list_response][:store_view][:item] || []
+              result = [result] unless result.is_a? Array
+              result.each { |customer| enumerator << customer }
+            end
+          end
+        end
+
+        def customer_address_list(customer_id)
+          @client.call(:customer_address_list, customer_id: customer_id).body
+        end
+      end
+    end
+  end
+end

data/lib/shopify_transporter/exporters/magento/database_cache.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+require 'csv'
+require 'sequel'
+
+module ShopifyTransporter
+  module Exporters
+    module Magento
+      class DatabaseCache
+        DB_CACHE_FOLDER = './cache/magento_db'
+
+        def initialize
+          @cache = {}
+        end
+
+        def table(table_name)
+          @cache[table_name] ||= CSV.read("#{DB_CACHE_FOLDER}/#{table_name}.csv", headers: true)
+        end
+      end
+    end
+  end
+end

data/lib/shopify_transporter/exporters/magento/database_table_exporter.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+require 'sequel'
+
+module ShopifyTransporter
+  module Exporters
+    module Magento
+      class DatabaseTableExporter
+        BATCH_SIZE = 1000
+        DB_CACHE_FOLDER = './cache/magento_db'
+
+        def initialize(database_adapter)
+          @database_adapter = database_adapter
+          FileUtils.mkdir_p(DB_CACHE_FOLDER)
+        end
+
+        def export_table(table_name, index_column)
+          export_file_path = "#{DB_CACHE_FOLDER}/#{table_name}.csv"
+
+          return if File.file? export_file_path
+
+          index_key = index_column.to_sym
+
+          @database_adapter.connect do |db|
+            ordered_table = db
+              .from(table_name)
+              .order(index_key)
+
+            headers = ordered_table.columns
+
+            write_headers(export_file_path, headers)
+
+            in_batches(ordered_table, index_key) do |batch|
+              write_batch(export_file_path, headers, batch)
+            end
+          end
+        end
+
+        private
+
+        def write_headers(export_file_path, headers)
+          File.open(export_file_path, 'w') do |file|
+            file << headers.to_csv
+          end
+        end
+
+        def write_batch(export_file_path, headers, batch)
+          File.open(export_file_path, 'a') do |file|
+            batch.each do |row|
+              data = headers.map { |header| row[header] }
+              file << data.to_csv
+            end
+          end
+        end
+
+        def in_batches(table, index_key)
+          current_id = table.first[index_key]
+          max_id = table.last[index_key]
+
+          while current_id <= max_id
+            batch = table.where(index_key => current_id...(current_id + BATCH_SIZE))
+            yield batch
+            current_id += BATCH_SIZE
+          end
+        end
+      end
+    end
+  end
+end