seedster 0.1.6 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 4836e8e3dd8e804c6e96d74f65c6c1d818f6962a
-  data.tar.gz: f260085eb875b10d583a573baead0710890b9e62
+SHA256:
+  metadata.gz: cb77b9ee39a1723d22f4a59fa6cc400d9ec5f17e11a7aa335527d662a75a46c9
+  data.tar.gz: a548994be971c55220f0156193254759b8394db7cf638d3af690aabbe11a5ec8
 SHA512:
-  metadata.gz: 9ff5886df60737e366312a3248c8dba305c077a516dc3de230d19793c6bc079716dd7cdbfc0a9584e764f60204a9e4289b7114406b8a8f5f75d18cb654a72a97
-  data.tar.gz: 28fa8e49f4486531c4f7e15c7cd48e8284e41c4c31b45fd93d628640c53f0a5af239608c93c005c64400a732c373af42822e91d5d2b7b04af65e4b8d204cb0f7
+  metadata.gz: b3d43b971a2a312fbf8e877f200554168a882f27c23e8ef912a79962481a1c381c861be980f592fcca92b7774cbc8038505a28cba57befab114cda0058f129f2
+  data.tar.gz: ab2b18d2ca6485445b660dd056e7ed616da620e33f7f27166a7805530e4335d839c27f5c86629435e7d537cd34206c07e4ad64dd4a0f74cac2942e640ba62aa0

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    seedster (0.1.1)
+    seedster (0.1.6)
 
 GEM
   remote: https://rubygems.org/
@@ -19,4 +19,4 @@ DEPENDENCIES
   seedster!
 
 BUNDLED WITH
-   1.16.1
+   2.0.2

data/README.md

@@ -4,19 +4,19 @@ Seedster is a way to work with real content in a development database, for a Rai
 
 > Why not use the built-in Rails seed mechanism?
 
-We preferred a custom solution so that the development data was based on real production data, with referential integrity intact. What we settled on was exporting particular rows of data all driven from a specific user, to provide a meaningful set of related data that could periodically be refreshed with minimal effort over time.
+We preferred a custom solution so that the development data was based on real production data, with referential integrity intact. What we settled on was exporting particular rows of data for a specific user, to provide a meaningful set of their data that could periodically be refreshed and added to with minimal effort.
 
-> Briefly, how is it different from Rails' db seeds?
+> How is it different from Rails' db seeds?
 
-With Seedster, you write SQL queries for the tables you want data from. The queries can have a parameter like a user ID. A Rake task is provided to dump data from a production database, and a separate task is provided to load data locally on developers machines.
+With Seedster, you write SQL queries for the tables you want data from. The queries can have a parameter like a `user_id`. A Rake task is provided to dump data from a production database, and a separate task is provided to load data locally on developers machines.
 
 > This works with real user data?
 
-Seedster uses content from a real user, so the recommendation is to use an employee user, test user, or something along those lines, so that the content being used for development is appropriate to share on a development team. Dump files can be viewed by all team members, to ensure they are based on an agreed upon user's data.
+Seedster uses content from a real user, so the recommendation is to use an employee user, test user, or something along those lines, so that the content being used for development is appropriate to share on a development team. Dump files can be viewed by all team members, to ensure they are based on an agreed upon user's data. At this time the dump file is not encrypted although that could be added.
 
 > What are the dependencies?
 
-Seedster has developed for use with a Postgres database and depends on `psql`, `ssh`, `scp`, and `tar`.
+Seedster was developed for use with a Rails application, using PostgreSQL database and depends on `psql`, `ssh`, `scp`, and `tar` commands.
 
 
 ## Process Overview
@@ -25,20 +25,20 @@ Requirements and expectations:
 
  * Local and remote database schema versions are in same. The development database is empty prior to load.
  * Use the provided Rake task to dump data from production. `rake seedster:dump` Individual dump files are consolidated into a single tar file.
- * Use the provided Rake task to download data file (`rake seedster:load`) to download, extract, and load the data
- * Use ENV variables to pass in dynamic data to SQL queries responsible for selecting data, for example supply `USER_ID` with a value on the command line, and add a parameter to your SQL query with syntax like this `%{}`, for example: `SELECT * FROM users WHERE id = '%{user_id}'`.
+ * Use the provided Rake task to download the data file (`rake seedster:load`) to download, extract, and load the data. NEW: Use the `skip_download` option if you wish to load from a local data file.
+ * Use `ENV` variables to pass in dynamic data to SQL queries responsible for selecting data, for example supply `USER_ID` with a value on the command line, and add a parameter to your SQL query with syntax like this `%{}`, for example: `SELECT * FROM users WHERE id = '%{user_id}'`.
 
 
 In order to dump the data for user ID 1, using a parameterized SQL query that expects a value for `user_id`:
 
-```
-prod_shell> USER_ID=1 rake seedster:dump
+```sh
+$ rake seedster:dump USER_ID=1
 ```
 
 To download, extract, and load the data file:
 
-```
-dev_shell> rake seedster:load
+```sh
+$ rake seedster:load
 ```
 
 ## Table of Contents
@@ -77,59 +77,24 @@ The gem provides an initializer. Run:
 
 This will create a file `config/initializers/seedster.rb` where you can begin putting in values for your application. The configuration is both for DB credentials, SSH credentials, but also the configuration of the tables you wish to dump.
 
-The initializer looks like this:
-
-```ruby
-Seedster.configure do |c|
-  c.db_host = 'host.com' # DB host. Fill this in, or reference Rails config/development.yml database values or ar-octopus config
-  c.db_name = 'db_nameXXX'
-  c.db_username = 'db_usernameXXX'
-  c.db_password = 'passwordXXX'
-  c.remote_host_path = "/var/company/www/app/current" # where the root of the app is deployed on the host
-  c.query_params = { } # pass in values to interpolate into queries,
-                       # e.g. USER_ID=XXX would be {user_id: ENV['USER_ID']}
-
-  c.ssh_user = 'ssh_userXXX' # which user will connect to the host
-  c.dump_host = 'app.host.com' # host where app is deployed
-
-  #
-  # Help:
-  # Comma-separated list of hashes, with keys `query` and `name`, one hash per DB table
-  #
-  # Keys:
-  # query: the SQL query for the table to be dumped. Add a parameter like `user_id` to be passed in.
-  # name: the name of the database table
-  #
-  c.tables = [
-    # BEGIN example ---
-    # {
-    #   query: %{SELECT u.* FROM users
-    #       where u.id = '%{user_id}'},
-    #   name: 'users'
-    # }
-    # END example -----
-  ]
-end
-```
-
 
 ### Configuration Options
 
-##### `db_host`
+##### `dump_host` and `load_host`
 
-Production database host. We use a read-only replica
+These are both configured to read your Rails database configuration for defaults to get started. However the intention is the dump host is specified to be a Production host, and the load host is the Development host (or Staging etc.).
 
-##### `db_name`
+##### `dump_database` and `load_database`
 
-Production database name
+Database name
 
-##### `db_username`
+##### `dump_username` and `load_username`
 
-Production database username 
+Database connection username 
 
-##### `db_password`
+##### `dump_password` and `load_password`
 
-Production database password
+Database connection password
 
 ##### `remote_host_path`
 
@@ -143,15 +108,19 @@ To use a parameter like `user_id` in SQL queries, passing the value in via an en
 
 An SSH user that can connect (for purposes of downloading the file with `scp`) to the host where the production dump is.
 
-##### `dump_host`
+##### `ssh_host`
 
-The host (a hostname) where the dump file is stored. We made this configurable to be able to dump data from production and staging.
+The SSH host (a hostname) where the dump file is stored. We made this configurable to be able to dump data from production and staging.
 
 ##### `tables`
 
-e.g. `tables = [ /* {name: '', query: '' } */ ]`
+e.g. `tables = [ {name: '', query: '' } ]`
     
-The tables option is where each query is stored. This is currently an array of hashes, with two items, `name` and `query`. `name` is the name of the database table, and `query` is the SQL query. We are dumping 10-15 tables worth of data for the same user, so that means we have 10-15 items specified here. This is where we would modify those queries over time to include or exclude more tables or fields of data.
+The tables option is where each table being dumped is configured. This is currently an array of hashes, with two items, `name` and `query`. `name` is the name of the database table, and `query` is the SQL query.
+
+We are dumping 10-15 tables worth of data for the same user, so that means we have 10-15 hashes in this array.
+
+This is where we would modify those queries over time to include or exclude more tables or fields of data.
 
 
 

data/lib/seedster.rb

@@ -53,7 +53,7 @@ module Seedster
       @remote_host_path = nil
       @query_params = {}
       @ssh_user = nil
-      @ssh_host= nil
+      @ssh_host = nil
       @tables = []
       @skip_download = false
     end

data/lib/seedster/data_dumper.rb

@@ -15,15 +15,18 @@
 #
 module Seedster
   class DataDumper
-    attr_reader :dump_password, :dump_host, :dump_username, :dump_database
+    attr_reader :dump_password, :dump_host, :dump_username, :dump_database,
+      :file_manager
 
     def initialize(dump_password:, dump_host:, dump_username:, dump_database:)
+      print_greeting
       @dump_password  = dump_password
       @dump_host      = dump_host
       @dump_username  = dump_username
       @dump_database  = dump_database
-      FileManager.create_dump_dir
-      FileManager.create_seed_file_dir
+      @file_manager = FileManager.new(app_root: Rails.root)
+      @file_manager.create_dump_dir
+      @file_manager.create_seed_file_dir
     end
 
     def dump!
@@ -49,18 +52,28 @@ module Seedster
     end
 
     def create_consolidated_dump_file
-      dump_file = Rails.root.join(FileManager.dump_dir, FileManager.dump_file_name)
-      puts "Creating dump file '#{dump_file}' from '#{FileManager.seed_file_dir}'"
-      tar_command = "tar -zcvf #{dump_file} -C #{FileManager.seed_file_dir} ." # use relative paths
+      dump_file = file_manager.consolidated_dump_file_name
+      puts "Creating dump file: '#{dump_file}' from '#{file_manager.seed_file_dir}'"
+      tar_command = "tar -zcvf #{dump_file} -C #{file_manager.seed_file_dir} ." # use relative paths
       puts "Running tar command: '#{tar_command}'"
       system(tar_command)
     end
 
     def sql_results_to_file(sql:, table_name:)
-      filename = FileManager.get_filename(table_name: table_name)
-      puts "Dumping table '#{table_name}' to file '#{filename}' with query '#{sql}'"
+      filename = file_manager.get_filename(table_name: table_name)
+      puts "Dumping table '#{table_name}' to file '#{filename}' with query --#{sql}--"
       psql_cmd = %{PGPASSWORD=#{dump_password} psql -h #{dump_host} -d #{dump_database} -U #{dump_username} -c "\\copy (#{sql}) TO '#{filename}' WITH CSV"}
       system(psql_cmd)
     end
+
+    private
+
+    def print_greeting
+      puts
+      puts "🌱🌱🌱🌱🌱"
+      puts "SEEDSTER DUMP"
+      puts "🌱🌱🌱🌱🌱"
+      puts
+    end
   end
 end

data/lib/seedster/data_loader.rb

@@ -16,7 +16,8 @@
 module Seedster
   class DataLoader
     attr_reader :ssh_user, :ssh_host, :remote_host_path,
-      :load_host, :load_database, :load_username, :load_password
+      :load_host, :load_database, :load_username, :load_password,
+      :file_manager
 
     def initialize(ssh_user:, ssh_host:, remote_host_path:,
                    load_host:, load_database:, load_username:, load_password:)
@@ -27,6 +28,7 @@ module Seedster
       @load_database = load_database
       @load_username = load_username
       @load_password = load_password
+      @file_manager = FileManager.new(app_root: Rails.root)
       print_greeting
     end
 
@@ -42,18 +44,18 @@ module Seedster
 
     def download_and_extract_file
       remote_host_path = Seedster.configuration.remote_host_path
-      remote_file = "#{remote_host_path}/#{File.join(FileManager.dump_dir, FileManager.dump_file_name)}"
+      remote_file = "#{remote_host_path}/#{file_manager.consolidated_dump_file_name}"
       scp_command = "scp -r #{ssh_user}@#{ssh_host}:#{remote_file} ."
       puts "Downloading file: '#{scp_command}'"
       system(scp_command)
 
-      untar_command = "tar -zxvf #{FileManager.dump_file_name} -C #{FileManager.seed_file_dir}"
+      untar_command = "tar -zxvf #{FileManager.dump_file_name} -C #{file_manager.seed_file_dir}"
       puts "Extracting file: '#{untar_command}'"
       system(untar_command)
     end
 
     def load_data(table_name:)
-      filename = FileManager.get_filename(table_name: table_name)
+      filename = file_manager.get_filename(table_name: table_name)
       psql_load(filename: filename, table_name: table_name)
     end
 
@@ -68,7 +70,8 @@ module Seedster
     def print_greeting
       puts
       puts "🌱🌱🌱🌱🌱"
-      puts "Hello from Seedster. A note:"
+      puts "SEEDSTER LOAD"
+      puts "🌱🌱🌱🌱🌱"
       puts
       puts "`rake seedster:load` expects a compatible schema version and empty tables."
       puts "Run `rake db:reset` prior to running `rake seedster:load`"
@@ -76,7 +79,6 @@ module Seedster
       puts "Before loading data locally, truncate the relevant tables (or you may see constraint violation errors)."
       puts "Bugs and issues: https://github.com/groupon/seedster"
       puts "Thanks!"
-      puts "🌱🌱🌱🌱🌱"
       puts
     end
   end

data/lib/seedster/file_manager.rb

@@ -20,33 +20,45 @@ module Seedster
     DATA_DUMPS_DIR = 'data_dumps'
     SEED_FILE_DIR = 'seedster_data_dumps'
 
-    class << self
-      def dump_dir
-        # Capistrano note:
-        # set up as a linked_dir in config/deploy.rb
-        Rails.root.join(TMP_DIR, DATA_DUMPS_DIR)
-      end
+    # Expects Pathname
+    # https://ruby-doc.org/stdlib-2.5.1/libdoc/pathname/rdoc/Pathname.html
+    attr_reader :app_root
 
-      def seed_file_dir
-        Rails.root.join(TMP_DIR, SEED_FILE_DIR)
-      end
+    def initialize(app_root:)
+      @app_root = app_root
+    end
 
-      def get_filename(table_name:)
-        File.join(seed_file_dir, "#{SEEDSTER}-#{table_name}.csv")
-      end
+    def dump_dir
+      # Capistrano note:
+      # set up as a linked_dir in config/deploy.rb
+      app_root.join(TMP_DIR, DATA_DUMPS_DIR)
+    end
 
-      def dump_file_name
-        "#{SEEDSTER}-dump-latest.tar.gz"
-      end
+    def seed_file_dir
+      app_root.join(TMP_DIR, SEED_FILE_DIR)
+    end
 
-      def create_dump_dir
-        puts "Creating directory: '#{FileManager.dump_dir}'"
-        FileUtils.mkdir_p(dump_dir)
-      end
+    def get_filename(table_name:)
+      File.join(seed_file_dir, "#{SEEDSTER}-#{table_name}.csv")
+    end
+
+    def create_dump_dir
+      puts "Creating directory: '#{dump_dir}'"
+      FileUtils.mkdir_p(dump_dir)
+    end
+
+    def create_seed_file_dir
+      puts "Creating directory: '#{seed_file_dir}'"
+      FileUtils.mkdir_p(seed_file_dir)
+    end
 
-      def create_seed_file_dir
-        puts "Creating directory: '#{FileManager.seed_file_dir}'"
-        FileUtils.mkdir_p(seed_file_dir)
+    def consolidated_dump_file_name
+      app_root.join(dump_dir, self.class.dump_file_name)
+    end
+
+    class << self
+      def dump_file_name
+        "#{SEEDSTER}-dump-latest.tar.gz"
       end
     end
   end

data/lib/seedster/tasks/seedster.rb

@@ -18,8 +18,6 @@ require 'seedster'
 namespace :seedster do
   desc "Dump application seed data to a data dump file"
   task dump: :environment do
-    puts "Seedster loading..."
-
     Seedster::DataDumper.new(
       dump_host: Seedster.configuration.dump_host,
       dump_database: Seedster.configuration.dump_database,

data/lib/seedster/version.rb

@@ -14,5 +14,5 @@
 # limitations under the License.
 #
 module Seedster
-  VERSION = '0.1.6'
+  VERSION = '0.1.7'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: seedster
 version: !ruby/object:Gem::Version
-  version: 0.1.6
+  version: 0.1.7
 platform: ruby
 authors:
 - Groupon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-08-02 00:00:00.000000000 Z
+date: 2019-08-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -100,7 +100,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.11
+rubygems_version: 2.7.6.2
 signing_key: 
 specification_version: 4
 summary: Work with real content in development