seedster 0.1.6 → 0.1.7
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/Gemfile.lock +2 -2
- data/README.md +27 -58
- data/lib/seedster.rb +1 -1
- data/lib/seedster/data_dumper.rb +21 -8
- data/lib/seedster/data_loader.rb +8 -6
- data/lib/seedster/file_manager.rb +34 -22
- data/lib/seedster/tasks/seedster.rb +0 -2
- data/lib/seedster/version.rb +1 -1
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: cb77b9ee39a1723d22f4a59fa6cc400d9ec5f17e11a7aa335527d662a75a46c9
|
4
|
+
data.tar.gz: a548994be971c55220f0156193254759b8394db7cf638d3af690aabbe11a5ec8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b3d43b971a2a312fbf8e877f200554168a882f27c23e8ef912a79962481a1c381c861be980f592fcca92b7774cbc8038505a28cba57befab114cda0058f129f2
|
7
|
+
data.tar.gz: ab2b18d2ca6485445b660dd056e7ed616da620e33f7f27166a7805530e4335d839c27f5c86629435e7d537cd34206c07e4ad64dd4a0f74cac2942e640ba62aa0
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -4,19 +4,19 @@ Seedster is a way to work with real content in a development database, for a Rai
|
|
4
4
|
|
5
5
|
> Why not use the built-in Rails seed mechanism?
|
6
6
|
|
7
|
-
We preferred a custom solution so that the development data was based on real production data, with referential integrity intact. What we settled on was exporting particular rows of data
|
7
|
+
We preferred a custom solution so that the development data was based on real production data, with referential integrity intact. What we settled on was exporting particular rows of data for a specific user, to provide a meaningful set of their data that could periodically be refreshed and added to with minimal effort.
|
8
8
|
|
9
|
-
>
|
9
|
+
> How is it different from Rails' db seeds?
|
10
10
|
|
11
|
-
With Seedster, you write SQL queries for the tables you want data from. The queries can have a parameter like a
|
11
|
+
With Seedster, you write SQL queries for the tables you want data from. The queries can have a parameter like a `user_id`. A Rake task is provided to dump data from a production database, and a separate task is provided to load data locally on developers machines.
|
12
12
|
|
13
13
|
> This works with real user data?
|
14
14
|
|
15
|
-
Seedster uses content from a real user, so the recommendation is to use an employee user, test user, or something along those lines, so that the content being used for development is appropriate to share on a development team. Dump files can be viewed by all team members, to ensure they are based on an agreed upon user's data.
|
15
|
+
Seedster uses content from a real user, so the recommendation is to use an employee user, test user, or something along those lines, so that the content being used for development is appropriate to share on a development team. Dump files can be viewed by all team members, to ensure they are based on an agreed upon user's data. At this time the dump file is not encrypted although that could be added.
|
16
16
|
|
17
17
|
> What are the dependencies?
|
18
18
|
|
19
|
-
Seedster
|
19
|
+
Seedster was developed for use with a Rails application, using PostgreSQL database and depends on `psql`, `ssh`, `scp`, and `tar` commands.
|
20
20
|
|
21
21
|
|
22
22
|
## Process Overview
|
@@ -25,20 +25,20 @@ Requirements and expectations:
|
|
25
25
|
|
26
26
|
* Local and remote database schema versions are in same. The development database is empty prior to load.
|
27
27
|
* Use the provided Rake task to dump data from production. `rake seedster:dump` Individual dump files are consolidated into a single tar file.
|
28
|
-
* Use the provided Rake task to download data file (`rake seedster:load`) to download, extract, and load the data
|
29
|
-
* Use ENV variables to pass in dynamic data to SQL queries responsible for selecting data, for example supply `USER_ID` with a value on the command line, and add a parameter to your SQL query with syntax like this `%{}`, for example: `SELECT * FROM users WHERE id = '%{user_id}'`.
|
28
|
+
* Use the provided Rake task to download the data file (`rake seedster:load`) to download, extract, and load the data. NEW: Use the `skip_download` option if you wish to load from a local data file.
|
29
|
+
* Use `ENV` variables to pass in dynamic data to SQL queries responsible for selecting data, for example supply `USER_ID` with a value on the command line, and add a parameter to your SQL query with syntax like this `%{}`, for example: `SELECT * FROM users WHERE id = '%{user_id}'`.
|
30
30
|
|
31
31
|
|
32
32
|
In order to dump the data for user ID 1, using a parameterized SQL query that expects a value for `user_id`:
|
33
33
|
|
34
|
-
```
|
35
|
-
|
34
|
+
```sh
|
35
|
+
$ rake seedster:dump USER_ID=1
|
36
36
|
```
|
37
37
|
|
38
38
|
To download, extract, and load the data file:
|
39
39
|
|
40
|
-
```
|
41
|
-
|
40
|
+
```sh
|
41
|
+
$ rake seedster:load
|
42
42
|
```
|
43
43
|
|
44
44
|
## Table of Contents
|
@@ -77,59 +77,24 @@ The gem provides an initializer. Run:
|
|
77
77
|
|
78
78
|
This will create a file `config/initializers/seedster.rb` where you can begin putting in values for your application. The configuration is both for DB credentials, SSH credentials, but also the configuration of the tables you wish to dump.
|
79
79
|
|
80
|
-
The initializer looks like this:
|
81
|
-
|
82
|
-
```ruby
|
83
|
-
Seedster.configure do |c|
|
84
|
-
c.db_host = 'host.com' # DB host. Fill this in, or reference Rails config/development.yml database values or ar-octopus config
|
85
|
-
c.db_name = 'db_nameXXX'
|
86
|
-
c.db_username = 'db_usernameXXX'
|
87
|
-
c.db_password = 'passwordXXX'
|
88
|
-
c.remote_host_path = "/var/company/www/app/current" # where the root of the app is deployed on the host
|
89
|
-
c.query_params = { } # pass in values to interpolate into queries,
|
90
|
-
# e.g. USER_ID=XXX would be {user_id: ENV['USER_ID']}
|
91
|
-
|
92
|
-
c.ssh_user = 'ssh_userXXX' # which user will connect to the host
|
93
|
-
c.dump_host = 'app.host.com' # host where app is deployed
|
94
|
-
|
95
|
-
#
|
96
|
-
# Help:
|
97
|
-
# Comma-separated list of hashes, with keys `query` and `name`, one hash per DB table
|
98
|
-
#
|
99
|
-
# Keys:
|
100
|
-
# query: the SQL query for the table to be dumped. Add a parameter like `user_id` to be passed in.
|
101
|
-
# name: the name of the database table
|
102
|
-
#
|
103
|
-
c.tables = [
|
104
|
-
# BEGIN example ---
|
105
|
-
# {
|
106
|
-
# query: %{SELECT u.* FROM users
|
107
|
-
# where u.id = '%{user_id}'},
|
108
|
-
# name: 'users'
|
109
|
-
# }
|
110
|
-
# END example -----
|
111
|
-
]
|
112
|
-
end
|
113
|
-
```
|
114
|
-
|
115
80
|
|
116
81
|
### Configuration Options
|
117
82
|
|
118
|
-
##### `
|
83
|
+
##### `dump_host` and `load_host`
|
119
84
|
|
120
|
-
|
85
|
+
These are both configured to read your Rails database configuration for defaults to get started. However the intention is the dump host is specified to be a Production host, and the load host is the Development host (or Staging etc.).
|
121
86
|
|
122
|
-
##### `
|
87
|
+
##### `dump_database` and `load_database`
|
123
88
|
|
124
|
-
|
89
|
+
Database name
|
125
90
|
|
126
|
-
##### `
|
91
|
+
##### `dump_username` and `load_username`
|
127
92
|
|
128
|
-
|
93
|
+
Database connection username
|
129
94
|
|
130
|
-
##### `
|
95
|
+
##### `dump_password` and `load_password`
|
131
96
|
|
132
|
-
|
97
|
+
Database connection password
|
133
98
|
|
134
99
|
##### `remote_host_path`
|
135
100
|
|
@@ -143,15 +108,19 @@ To use a parameter like `user_id` in SQL queries, passing the value in via an en
|
|
143
108
|
|
144
109
|
An SSH user that can connect (for purposes of downloading the file with `scp`) to the host where the production dump is.
|
145
110
|
|
146
|
-
##### `
|
111
|
+
##### `ssh_host`
|
147
112
|
|
148
|
-
The host (a hostname) where the dump file is stored. We made this configurable to be able to dump data from production and staging.
|
113
|
+
The SSH host (a hostname) where the dump file is stored. We made this configurable to be able to dump data from production and staging.
|
149
114
|
|
150
115
|
##### `tables`
|
151
116
|
|
152
|
-
e.g. `tables = [
|
117
|
+
e.g. `tables = [ {name: '', query: '' } ]`
|
153
118
|
|
154
|
-
The tables option is where each
|
119
|
+
The tables option is where each table being dumped is configured. This is currently an array of hashes, with two items, `name` and `query`. `name` is the name of the database table, and `query` is the SQL query.
|
120
|
+
|
121
|
+
We are dumping 10-15 tables worth of data for the same user, so that means we have 10-15 hashes in this array.
|
122
|
+
|
123
|
+
This is where we would modify those queries over time to include or exclude more tables or fields of data.
|
155
124
|
|
156
125
|
|
157
126
|
|
data/lib/seedster.rb
CHANGED
data/lib/seedster/data_dumper.rb
CHANGED
@@ -15,15 +15,18 @@
|
|
15
15
|
#
|
16
16
|
module Seedster
|
17
17
|
class DataDumper
|
18
|
-
attr_reader :dump_password, :dump_host, :dump_username, :dump_database
|
18
|
+
attr_reader :dump_password, :dump_host, :dump_username, :dump_database,
|
19
|
+
:file_manager
|
19
20
|
|
20
21
|
def initialize(dump_password:, dump_host:, dump_username:, dump_database:)
|
22
|
+
print_greeting
|
21
23
|
@dump_password = dump_password
|
22
24
|
@dump_host = dump_host
|
23
25
|
@dump_username = dump_username
|
24
26
|
@dump_database = dump_database
|
25
|
-
FileManager.
|
26
|
-
|
27
|
+
@file_manager = FileManager.new(app_root: Rails.root)
|
28
|
+
@file_manager.create_dump_dir
|
29
|
+
@file_manager.create_seed_file_dir
|
27
30
|
end
|
28
31
|
|
29
32
|
def dump!
|
@@ -49,18 +52,28 @@ module Seedster
|
|
49
52
|
end
|
50
53
|
|
51
54
|
def create_consolidated_dump_file
|
52
|
-
dump_file =
|
53
|
-
puts "Creating dump file '#{dump_file}' from '#{
|
54
|
-
tar_command = "tar -zcvf #{dump_file} -C #{
|
55
|
+
dump_file = file_manager.consolidated_dump_file_name
|
56
|
+
puts "Creating dump file: '#{dump_file}' from '#{file_manager.seed_file_dir}'"
|
57
|
+
tar_command = "tar -zcvf #{dump_file} -C #{file_manager.seed_file_dir} ." # use relative paths
|
55
58
|
puts "Running tar command: '#{tar_command}'"
|
56
59
|
system(tar_command)
|
57
60
|
end
|
58
61
|
|
59
62
|
def sql_results_to_file(sql:, table_name:)
|
60
|
-
filename =
|
61
|
-
puts "Dumping table '#{table_name}' to file '#{filename}' with query
|
63
|
+
filename = file_manager.get_filename(table_name: table_name)
|
64
|
+
puts "Dumping table '#{table_name}' to file '#{filename}' with query --#{sql}--"
|
62
65
|
psql_cmd = %{PGPASSWORD=#{dump_password} psql -h #{dump_host} -d #{dump_database} -U #{dump_username} -c "\\copy (#{sql}) TO '#{filename}' WITH CSV"}
|
63
66
|
system(psql_cmd)
|
64
67
|
end
|
68
|
+
|
69
|
+
private
|
70
|
+
|
71
|
+
def print_greeting
|
72
|
+
puts
|
73
|
+
puts "🌱🌱🌱🌱🌱"
|
74
|
+
puts "SEEDSTER DUMP"
|
75
|
+
puts "🌱🌱🌱🌱🌱"
|
76
|
+
puts
|
77
|
+
end
|
65
78
|
end
|
66
79
|
end
|
data/lib/seedster/data_loader.rb
CHANGED
@@ -16,7 +16,8 @@
|
|
16
16
|
module Seedster
|
17
17
|
class DataLoader
|
18
18
|
attr_reader :ssh_user, :ssh_host, :remote_host_path,
|
19
|
-
:load_host, :load_database, :load_username, :load_password
|
19
|
+
:load_host, :load_database, :load_username, :load_password,
|
20
|
+
:file_manager
|
20
21
|
|
21
22
|
def initialize(ssh_user:, ssh_host:, remote_host_path:,
|
22
23
|
load_host:, load_database:, load_username:, load_password:)
|
@@ -27,6 +28,7 @@ module Seedster
|
|
27
28
|
@load_database = load_database
|
28
29
|
@load_username = load_username
|
29
30
|
@load_password = load_password
|
31
|
+
@file_manager = FileManager.new(app_root: Rails.root)
|
30
32
|
print_greeting
|
31
33
|
end
|
32
34
|
|
@@ -42,18 +44,18 @@ module Seedster
|
|
42
44
|
|
43
45
|
def download_and_extract_file
|
44
46
|
remote_host_path = Seedster.configuration.remote_host_path
|
45
|
-
remote_file = "#{remote_host_path}/#{
|
47
|
+
remote_file = "#{remote_host_path}/#{file_manager.consolidated_dump_file_name}"
|
46
48
|
scp_command = "scp -r #{ssh_user}@#{ssh_host}:#{remote_file} ."
|
47
49
|
puts "Downloading file: '#{scp_command}'"
|
48
50
|
system(scp_command)
|
49
51
|
|
50
|
-
untar_command = "tar -zxvf #{FileManager.dump_file_name} -C #{
|
52
|
+
untar_command = "tar -zxvf #{FileManager.dump_file_name} -C #{file_manager.seed_file_dir}"
|
51
53
|
puts "Extracting file: '#{untar_command}'"
|
52
54
|
system(untar_command)
|
53
55
|
end
|
54
56
|
|
55
57
|
def load_data(table_name:)
|
56
|
-
filename =
|
58
|
+
filename = file_manager.get_filename(table_name: table_name)
|
57
59
|
psql_load(filename: filename, table_name: table_name)
|
58
60
|
end
|
59
61
|
|
@@ -68,7 +70,8 @@ module Seedster
|
|
68
70
|
def print_greeting
|
69
71
|
puts
|
70
72
|
puts "🌱🌱🌱🌱🌱"
|
71
|
-
puts "
|
73
|
+
puts "SEEDSTER LOAD"
|
74
|
+
puts "🌱🌱🌱🌱🌱"
|
72
75
|
puts
|
73
76
|
puts "`rake seedster:load` expects a compatible schema version and empty tables."
|
74
77
|
puts "Run `rake db:reset` prior to running `rake seedster:load`"
|
@@ -76,7 +79,6 @@ module Seedster
|
|
76
79
|
puts "Before loading data locally, truncate the relevant tables (or you may see constraint violation errors)."
|
77
80
|
puts "Bugs and issues: https://github.com/groupon/seedster"
|
78
81
|
puts "Thanks!"
|
79
|
-
puts "🌱🌱🌱🌱🌱"
|
80
82
|
puts
|
81
83
|
end
|
82
84
|
end
|
@@ -20,33 +20,45 @@ module Seedster
|
|
20
20
|
DATA_DUMPS_DIR = 'data_dumps'
|
21
21
|
SEED_FILE_DIR = 'seedster_data_dumps'
|
22
22
|
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
# set up as a linked_dir in config/deploy.rb
|
27
|
-
Rails.root.join(TMP_DIR, DATA_DUMPS_DIR)
|
28
|
-
end
|
23
|
+
# Expects Pathname
|
24
|
+
# https://ruby-doc.org/stdlib-2.5.1/libdoc/pathname/rdoc/Pathname.html
|
25
|
+
attr_reader :app_root
|
29
26
|
|
30
|
-
|
31
|
-
|
32
|
-
|
27
|
+
def initialize(app_root:)
|
28
|
+
@app_root = app_root
|
29
|
+
end
|
33
30
|
|
34
|
-
|
35
|
-
|
36
|
-
|
31
|
+
def dump_dir
|
32
|
+
# Capistrano note:
|
33
|
+
# set up as a linked_dir in config/deploy.rb
|
34
|
+
app_root.join(TMP_DIR, DATA_DUMPS_DIR)
|
35
|
+
end
|
37
36
|
|
38
|
-
|
39
|
-
|
40
|
-
|
37
|
+
def seed_file_dir
|
38
|
+
app_root.join(TMP_DIR, SEED_FILE_DIR)
|
39
|
+
end
|
41
40
|
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
41
|
+
def get_filename(table_name:)
|
42
|
+
File.join(seed_file_dir, "#{SEEDSTER}-#{table_name}.csv")
|
43
|
+
end
|
44
|
+
|
45
|
+
def create_dump_dir
|
46
|
+
puts "Creating directory: '#{dump_dir}'"
|
47
|
+
FileUtils.mkdir_p(dump_dir)
|
48
|
+
end
|
49
|
+
|
50
|
+
def create_seed_file_dir
|
51
|
+
puts "Creating directory: '#{seed_file_dir}'"
|
52
|
+
FileUtils.mkdir_p(seed_file_dir)
|
53
|
+
end
|
46
54
|
|
47
|
-
|
48
|
-
|
49
|
-
|
55
|
+
def consolidated_dump_file_name
|
56
|
+
app_root.join(dump_dir, self.class.dump_file_name)
|
57
|
+
end
|
58
|
+
|
59
|
+
class << self
|
60
|
+
def dump_file_name
|
61
|
+
"#{SEEDSTER}-dump-latest.tar.gz"
|
50
62
|
end
|
51
63
|
end
|
52
64
|
end
|
@@ -18,8 +18,6 @@ require 'seedster'
|
|
18
18
|
namespace :seedster do
|
19
19
|
desc "Dump application seed data to a data dump file"
|
20
20
|
task dump: :environment do
|
21
|
-
puts "Seedster loading..."
|
22
|
-
|
23
21
|
Seedster::DataDumper.new(
|
24
22
|
dump_host: Seedster.configuration.dump_host,
|
25
23
|
dump_database: Seedster.configuration.dump_database,
|
data/lib/seedster/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: seedster
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.7
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Groupon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-08-
|
11
|
+
date: 2019-08-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -100,7 +100,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
100
100
|
version: '0'
|
101
101
|
requirements: []
|
102
102
|
rubyforge_project:
|
103
|
-
rubygems_version: 2.6.
|
103
|
+
rubygems_version: 2.7.6.2
|
104
104
|
signing_key:
|
105
105
|
specification_version: 4
|
106
106
|
summary: Work with real content in development
|