dynamocli 0.1.5 → 0.1.6

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ff41269be712ae8e16cab262ee39da16da6eba5adc8efaa2d6316a357133bc91
4
- data.tar.gz: 94f3af39b5c2a3373a7adb4fae76967302170a3a74b01068184671896dbb9f0b
3
+ metadata.gz: d472e7b32f0ee64582e39cb00bcab25ab4070b025806793fd8595e2acba9bedf
4
+ data.tar.gz: e7838ac7811278a88629e17fd521fcce36b60b96b37c3ad8349a3ccbdcafc4d1
5
5
  SHA512:
6
- metadata.gz: a4040a8df21120f3b5e41baf4a3d5b50afe8f95983fc35fcad3f468afc4142ecebff8629e6478d53dc9c7fd420b222fd5531eeb664b3ca7b1e931cd31adbf382
7
- data.tar.gz: 1ab22e6e4abc793cfdfc25dcee422530e13209f6b9de8dbca5118d35c09842876f26da1b05eddd75cf90f995924d668f34479e74a2ad93865ec62e15c36ee5d0
6
+ metadata.gz: 47efb4fe6a7a2325f2517c2686c66e5945e7c1953f1477e2a4947fd9bb3b452ae95d5edf1b4fae5555089a460353df0cbf2de84b5e3b2712bac6ff0465aa3e75
7
+ data.tar.gz: 56a7caa27afe9f7289bf3389d17403d0c98f4d7a93ce9d3ea0d0bba831518bb524309938e9da5cc85872fde57bac73ae5ce6b123833d6507e50d9ed82112bef1
@@ -2,6 +2,13 @@
2
2
 
3
3
  All notable changes to this project will be documented in this file.
4
4
 
5
+ ## [0.1.5] - 2020-01-13
6
+ ### Added
7
+ - Option to import data from a CSV exported from AWS.
8
+
9
+ ### Changed
10
+ - Big refactoring in erase functionality.
11
+
5
12
  ## [0.1.5] - 2019-08-08
6
13
  ### Fixed
7
14
  - Fix erase table without GSIs.
@@ -29,6 +36,7 @@ All notable changes to this project will be documented in this file.
29
36
  ### Added
30
37
  - Command to import data from a CSV file to a DynamoDB table.
31
38
 
39
+ [0.1.6]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.5...v0.1.6
32
40
  [0.1.5]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.4...v0.1.5
33
41
  [0.1.4]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.3...v0.1.4
34
42
  [0.1.3]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.2...v0.1.3
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- dynamocli (0.1.5)
4
+ dynamocli (0.1.6)
5
5
  aws-sdk-cloudformation (~> 1.23)
6
6
  aws-sdk-dynamodb (~> 1.28)
7
7
  thor (~> 0.20)
@@ -11,17 +11,17 @@ GEM
11
11
  remote: https://rubygems.org/
12
12
  specs:
13
13
  aws-eventstream (1.0.3)
14
- aws-partitions (1.197.0)
15
- aws-sdk-cloudformation (1.25.0)
16
- aws-sdk-core (~> 3, >= 3.61.1)
14
+ aws-partitions (1.263.0)
15
+ aws-sdk-cloudformation (1.29.0)
16
+ aws-sdk-core (~> 3, >= 3.71.0)
17
17
  aws-sigv4 (~> 1.1)
18
- aws-sdk-core (3.62.0)
18
+ aws-sdk-core (3.89.0)
19
19
  aws-eventstream (~> 1.0, >= 1.0.2)
20
- aws-partitions (~> 1.0)
20
+ aws-partitions (~> 1, >= 1.239.0)
21
21
  aws-sigv4 (~> 1.1)
22
22
  jmespath (~> 1.0)
23
- aws-sdk-dynamodb (1.34.0)
24
- aws-sdk-core (~> 3, >= 3.61.1)
23
+ aws-sdk-dynamodb (1.41.0)
24
+ aws-sdk-core (~> 3, >= 3.71.0)
25
25
  aws-sigv4 (~> 1.1)
26
26
  aws-sigv4 (1.1.0)
27
27
  aws-eventstream (~> 1.0, >= 1.0.2)
@@ -33,19 +33,19 @@ GEM
33
33
  equatable (~> 0.6)
34
34
  tty-color (~> 0.5)
35
35
  rake (10.5.0)
36
- rspec (3.8.0)
37
- rspec-core (~> 3.8.0)
38
- rspec-expectations (~> 3.8.0)
39
- rspec-mocks (~> 3.8.0)
40
- rspec-core (3.8.0)
41
- rspec-support (~> 3.8.0)
42
- rspec-expectations (3.8.3)
36
+ rspec (3.9.0)
37
+ rspec-core (~> 3.9.0)
38
+ rspec-expectations (~> 3.9.0)
39
+ rspec-mocks (~> 3.9.0)
40
+ rspec-core (3.9.1)
41
+ rspec-support (~> 3.9.1)
42
+ rspec-expectations (3.9.0)
43
43
  diff-lcs (>= 1.2.0, < 2.0)
44
- rspec-support (~> 3.8.0)
45
- rspec-mocks (3.8.0)
44
+ rspec-support (~> 3.9.0)
45
+ rspec-mocks (3.9.1)
46
46
  diff-lcs (>= 1.2.0, < 2.0)
47
- rspec-support (~> 3.8.0)
48
- rspec-support (3.8.0)
47
+ rspec-support (~> 3.9.0)
48
+ rspec-support (3.9.2)
49
49
  thor (0.20.3)
50
50
  tty-color (0.5.0)
51
51
  tty-logger (0.1.0)
data/README.md CHANGED
@@ -15,12 +15,15 @@ You have to configure AWS in your computer first. The program will use the AWS c
15
15
 
16
16
  - Import data from a CSV file to a DynamoDB table
17
17
 
18
+ If you have exported the CSV file you want to import from AWS DynamoDB console, you probaly want to modify the headers before importing the CSV file, because AWS exports the CSV file with a symbol indicating the type of the field in the header. You can pass the option `--exported-from-aws` to do that, the default is false.
19
+
18
20
  ```
19
21
  Usage:
20
22
  dynamocli import FILE -t, --table, --to=TABLE
21
23
 
22
24
  Options:
23
- -t, --table, --to=TABLE # table you want to import the data
25
+ -t, --table, --to=TABLE # table you want to import the data
26
+ [--exported-from-aws], [--no-exported-from-aws] # modify the headers before importing the csv
24
27
 
25
28
  Description:
26
29
  `dynamocli import` will import the data in from a file to a table specified.
@@ -56,6 +59,16 @@ From the DynamoDB Guidelines for Working with Tables documentation:
56
59
 
57
60
  > Deleting an entire table is significantly more efficient than removing items one-by-one, which essentially doubles the write throughput as you do as many delete operations as put operations.
58
61
 
62
+ ## Known Issues
63
+
64
+ ### Importing a CSV file with arrays and objects as values in it
65
+
66
+ Unfortunately, at this moment, this library cannot properly import array and objects. These values will appear as strings in the DynamoDB table.
67
+
68
+ ## Cross account or multiple profiles usage
69
+
70
+ You can run `dynamocli` passing the `AWS_PROFILE` environment variable with the profile you want to use, for example: `AWS_PROFILE=nondefaultprofile dynamocli erase users`.
71
+
59
72
  ## Development
60
73
 
61
74
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
18
18
  `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
19
19
  end
20
20
  spec.bindir = "bin"
21
- spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
21
+ spec.executables = spec.files.grep(%r{^bin/dynamocli}) { |f| File.basename(f) }
22
22
  spec.require_paths = ["lib"]
23
23
 
24
24
  spec.add_dependency "thor", "~> 0.20"
@@ -13,8 +13,9 @@ module Dynamocli
13
13
  > $ dynamo import users.csv --to users
14
14
  LONGDESC
15
15
  option :to, required: true, desc: "table you want to import the data", banner: "TABLE", aliases: ["-t", "--table"]
16
+ option "exported-from-aws", desc: "modify the headers before importing the csv", type: :boolean
16
17
  def import(file)
17
- Dynamocli::Import.new(file: file, table: options[:to]).start
18
+ Dynamocli::Import.new(file: file, table: options[:to], exported_from_aws: options["exported-from-aws"]).start
18
19
  end
19
20
 
20
21
  desc "erase TABLE", "erase all the data from the DynamoDB TABLE"
@@ -0,0 +1,98 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "forwardable"
4
+ require "json"
5
+ require "yaml"
6
+ require "aws-sdk-cloudformation"
7
+ require "tty-logger"
8
+
9
+ module Dynamocli::AWS
10
+ class Stack
11
+ attr_reader :name, :resources, :template_body, :original_template, :template_without_table, :policy_body
12
+
13
+ extend Forwardable
14
+ def_delegators :stack_on_aws,
15
+ :parameters, :capabilities, :role_arn, :rollback_configuration, :notification_arns, :tags
16
+
17
+ def initialize(table_name:, table_resource:, cloudformation: nil, logger: nil)
18
+ @table_name = table_name
19
+ @table_resource = table_resource
20
+ @cloudformation = cloudformation || CLOUDFORMARTION.new
21
+ @logger = logger || LOGGER.new
22
+
23
+ set_attributes_now_because_they_will_change
24
+ end
25
+
26
+ def deploying?
27
+ current_status != DEPLOY_COMPLETED_KEY
28
+ end
29
+
30
+ private
31
+
32
+ CLOUDFORMARTION = Aws::CloudFormation::Client
33
+ LOGGER = TTY::Logger
34
+ DEPLOY_COMPLETED_KEY = "UPDATE_COMPLETE"
35
+ private_constant :CLOUDFORMARTION, :LOGGER, :DEPLOY_COMPLETED_KEY
36
+
37
+ attr_reader :table_name, :table_resource, :cloudformation, :logger, :stack_on_aws
38
+
39
+ def set_attributes_now_because_they_will_change
40
+ set_name
41
+ set_stack_on_aws
42
+ set_resources
43
+ set_template_body
44
+ set_original_template
45
+ set_template_without_table
46
+ set_policy_body
47
+ end
48
+
49
+ def set_name
50
+ @name ||= table_resource[:stack_name]
51
+ end
52
+
53
+ def set_stack_on_aws
54
+ @stack_on_aws ||= cloudformation.describe_stacks(stack_name: name)[0][0]
55
+ end
56
+
57
+ def set_resources
58
+ @resources ||= cloudformation.describe_stack_resources(physical_resource_id: table_name).to_h
59
+ end
60
+
61
+ def set_template_body
62
+ @template_body ||= cloudformation.get_template(stack_name: name).to_h[:template_body]
63
+ end
64
+
65
+ def set_original_template
66
+ @original_template ||= parse_template(template_body)
67
+ end
68
+
69
+ def set_template_without_table
70
+ @template_without_table ||= parse_template(template_body).tap do |template_without_table|
71
+ tables = original_template["Resources"].select { |_, v| v["Type"] == "AWS::DynamoDB::Table" }
72
+ table = tables.find { |_, v| v["Properties"]["TableName"] == table_name }
73
+
74
+ if tables.nil?
75
+ logger.error("table #{table_name} not found in the #{@name} stack")
76
+ exit(42)
77
+ end
78
+
79
+ logical_resource_id = table.first
80
+ template_without_table["Resources"].delete(logical_resource_id)
81
+ end
82
+ end
83
+
84
+ def parse_template(template)
85
+ JSON.parse(template)
86
+ rescue JSON::ParserError
87
+ YAML.load(template)
88
+ end
89
+
90
+ def set_policy_body
91
+ @policy_body ||= cloudformation.get_stack_policy(stack_name: name).stack_policy_body
92
+ end
93
+
94
+ def current_status
95
+ cloudformation.describe_stacks(stack_name: name)[0][0].stack_status
96
+ end
97
+ end
98
+ end
@@ -0,0 +1,64 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "forwardable"
4
+ require "aws-sdk-dynamodb"
5
+
6
+ module Dynamocli::AWS
7
+ class Table
8
+ attr_reader :schema
9
+
10
+ extend Forwardable
11
+ def_delegators :table_on_aws, :delete
12
+
13
+ def initialize(table_name:, table_on_aws:, dynamodb: nil)
14
+ @table_name = table_name
15
+ @table_on_aws = table_on_aws
16
+ @dynamodb = dynamodb || DYNAMODB.new
17
+
18
+ set_schema_before_we_delete_the_table
19
+ end
20
+
21
+ def deleting?
22
+ status == DELETION_IN_PROCESSING_KEY
23
+ rescue Aws::DynamoDB::Errors::ResourceNotFoundException
24
+ false
25
+ end
26
+
27
+ private
28
+
29
+ DYNAMODB = Aws::DynamoDB::Client
30
+ DELETION_IN_PROCESSING_KEY = "DELETING"
31
+ private_constant :DYNAMODB, :DELETION_IN_PROCESSING_KEY
32
+
33
+ attr_reader :table_name, :table_on_aws, :dynamodb
34
+
35
+ def status
36
+ dynamodb.describe_table(table_name: table_name).table.table_status
37
+ end
38
+
39
+ def set_schema_before_we_delete_the_table
40
+ @schema ||= dynamodb.describe_table(table_name: table_name).to_h[:table].tap do |schema|
41
+ schema.delete(:table_status)
42
+ schema.delete(:creation_date_time)
43
+ schema.delete(:table_size_bytes)
44
+ schema.delete(:item_count)
45
+ schema.delete(:table_arn)
46
+ schema.delete(:table_id)
47
+ schema[:provisioned_throughput]&.delete(:number_of_decreases_today)
48
+ schema[:local_secondary_indexes]&.each do |lsi|
49
+ lsi.delete(:index_status)
50
+ lsi.delete(:index_size_bytes)
51
+ lsi.delete(:item_count)
52
+ lsi.delete(:index_arn)
53
+ end
54
+ schema[:global_secondary_indexes]&.each do |gsi|
55
+ gsi.delete(:index_status)
56
+ gsi.delete(:index_size_bytes)
57
+ gsi.delete(:item_count)
58
+ gsi.delete(:index_arn)
59
+ gsi[:provisioned_throughput].delete(:number_of_decreases_today)
60
+ end
61
+ end
62
+ end
63
+ end
64
+ end
@@ -1,127 +1,52 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require "json"
4
- require "yaml"
5
3
  require "tty-logger"
6
4
  require "aws-sdk-dynamodb"
7
5
  require "aws-sdk-cloudformation"
6
+ require "dynamocli/table/cloudformation_table"
7
+ require "dynamocli/table/standalone_table"
8
+ require "dynamocli/aws/stack"
9
+ require "dynamocli/aws/table"
8
10
 
9
11
  class Dynamocli::Erase
10
- LOGGER = TTY::Logger.new
11
-
12
12
  def initialize(table_name:, with_drift: false)
13
13
  @with_drift = with_drift
14
14
  @table_name = table_name
15
15
 
16
16
  @dynamodb = Aws::DynamoDB::Client.new
17
17
  @cloudformation = Aws::CloudFormation::Client.new
18
- @table = Aws::DynamoDB::Table.new(@table_name)
19
-
20
- set_schema
18
+ @table_on_aws = Aws::DynamoDB::Table.new(@table_name)
21
19
 
22
20
  @stack_resources = @cloudformation.describe_stack_resources(physical_resource_id: @table_name).to_h
23
-
24
- set_stack_information
25
- rescue Aws::DynamoDB::Errors::ResourceNotFoundException => e
26
- LOGGER.error(e.message)
27
- exit(42)
28
21
  rescue Aws::CloudFormation::Errors::ValidationError
29
22
  @stack_resources = nil
30
23
  end
31
24
 
32
25
  def start
33
26
  erase_table
34
- rescue Aws::CloudFormation::Errors::ValidationError => e
27
+ rescue Aws::CloudFormation::Errors::ValidationError,
28
+ Aws::DynamoDB::Errors::ValidationException,
29
+ Aws::DynamoDB::Errors::ResourceNotFoundException => e
35
30
  LOGGER.error(e.message)
36
31
  exit(42)
37
32
  end
38
33
 
39
34
  private
40
35
 
41
- def set_schema
42
- @schema = @dynamodb.describe_table(table_name: @table_name).to_h[:table].tap do |schema|
43
- schema.delete(:table_status)
44
- schema.delete(:creation_date_time)
45
- schema.delete(:table_size_bytes)
46
- schema.delete(:item_count)
47
- schema.delete(:table_arn)
48
- schema.delete(:table_id)
49
- schema[:provisioned_throughput].delete(:number_of_decreases_today)
50
- schema[:local_secondary_indexes]&.each do |lsi|
51
- lsi.delete(:index_status)
52
- lsi.delete(:index_size_bytes)
53
- lsi.delete(:item_count)
54
- lsi.delete(:index_arn)
55
- end
56
- schema[:global_secondary_indexes]&.each do |gsi|
57
- gsi.delete(:index_status)
58
- gsi.delete(:index_size_bytes)
59
- gsi.delete(:item_count)
60
- gsi.delete(:index_arn)
61
- gsi[:provisioned_throughput].delete(:number_of_decreases_today)
62
- end
63
- end
64
- end
65
-
66
- def set_stack_information
67
- return if @stack_resources.nil?
68
-
69
- set_stack_name
70
- set_stack
71
- set_templates
72
- rescue Aws::CloudFormation::Errors::ValidationError => e
73
- LOGGER.error(e.message)
74
- exit(42)
75
- end
76
-
77
- def set_stack_name
78
- table_resource = @stack_resources[:stack_resources].find do |resource|
79
- resource[:physical_resource_id] == @table_name
80
- end
81
- @stack_name = table_resource[:stack_name]
82
- end
83
-
84
- def set_stack
85
- @stack = @cloudformation.describe_stacks(stack_name: @stack_name)[0][0]
86
- end
87
-
88
- def set_templates
89
- template_body = @cloudformation.get_template(stack_name: @stack_name).to_h[:template_body]
90
- @original_template = parse_template(template_body)
91
- @template_without_table = parse_template(template_body)
92
-
93
- tables = @original_template["Resources"].select { |_, v| v["Type"] == "AWS::DynamoDB::Table" }
94
- table = tables.find { |_, v| v["Properties"]["TableName"] == @table_name }
95
-
96
- if tables.nil?
97
- LOGGER.error("table #{@table_name} not found in the #{@stack_name} stack")
98
- exit(42)
99
- end
100
-
101
- logical_resource_id = table.first
102
- @template_without_table["Resources"].delete(logical_resource_id)
103
- end
36
+ LOGGER = TTY::Logger.new
37
+ private_constant :LOGGER
104
38
 
105
- def parse_template(template)
106
- JSON.parse(template)
107
- rescue JSON::ParserError
108
- YAML.load(template)
109
- end
39
+ attr_reader :table_name, :table_on_aws, :stack_resources
110
40
 
111
41
  def erase_table
112
- if @stack_resources.nil? || @with_drift
113
- check_if_user_wants_to_continue_with_recreation
114
- delete_and_recreate_the_table
115
- else
116
- check_if_user_wants_to_continue_with_deployment
117
- erase_table_through_cloudformation
118
- end
42
+ check_if_user_wants_to_continue
43
+ dynamocli_table.erase
119
44
  end
120
45
 
121
- def check_if_user_wants_to_continue_with_recreation
46
+ def check_if_user_wants_to_continue
122
47
  LOGGER.warn(
123
- "You're going to drop and recreate your #{@table_name} table,\n" \
124
- "do you really want to continue?"
48
+ "#{dynamocli_table.alert_message_before_continue} " \
49
+ "Do you really want to continue?"
125
50
  )
126
51
  STDOUT.print("(anything other than 'y' will cancel) > ")
127
52
 
@@ -136,111 +61,24 @@ class Dynamocli::Erase
136
61
  "Erase of #{@table_name} table canceled"
137
62
  end
138
63
 
139
- def delete_and_recreate_the_table
140
- delete_table
141
- wait_for_deletion_to_complete
142
- create_table
143
- end
144
-
145
- def delete_table
146
- LOGGER.info("Deleting the #{@table_name} table")
147
-
148
- @table.delete
149
-
150
- LOGGER.success("#{@table_name} table deleted")
151
- end
152
-
153
- def wait_for_deletion_to_complete
154
- waiting_seconds = 0
155
- while get_table_status == "DELETING"
156
- LOGGER.info("Waiting for deletion to complete")
157
- sleep waiting_seconds += 1
158
- end
159
- rescue Aws::DynamoDB::Errors::ResourceNotFoundException
160
- true
161
- end
162
-
163
- def get_table_status
164
- @dynamodb.describe_table(table_name: @table_name).table.table_status
165
- end
166
-
167
- def create_table
168
- LOGGER.info("Creating the #{@table_name} table")
169
-
170
- @dynamodb.create_table(@schema)
171
-
172
- LOGGER.success("#{@table_name} table created")
173
- end
174
-
175
- def check_if_user_wants_to_continue_with_deployment
176
- LOGGER.warn(
177
- "You are going to deploy and redeploy your #{@stack_name} stack\n" \
178
- "to drop and recreate the #{@table_name} table, do you really want to continue?"
179
- )
180
- STDOUT.print("(anything other than 'y' will cancel) > ")
181
-
182
- confirmation = STDIN.gets.strip
183
- return if confirmation == "y"
184
-
185
- LOGGER.info(abort_message)
186
- exit(0)
187
- end
188
-
189
- def erase_table_through_cloudformation
190
- deploy_stack_without_the_table
191
- wait_for_deployment_to_complete
192
- deploy_stack_with_the_original_template
193
- end
194
-
195
- def deploy_stack_without_the_table
196
- LOGGER.info("Deploying the stack without the #{@table_name} table")
197
-
198
- @cloudformation.update_stack(
199
- stack_name: @stack_name,
200
- template_body: @template_without_table.to_json,
201
- parameters: @stack.parameters.map(&:to_h),
202
- capabilities: @stack.capabilities,
203
- role_arn: @stack.role_arn,
204
- rollback_configuration: @stack.rollback_configuration.to_h,
205
- stack_policy_body: get_stack_policy_body,
206
- notification_arns: @stack.notification_arns,
207
- tags: @stack.tags.map(&:to_h)
208
- )
209
-
210
- LOGGER.success("Stack deployed without the #{@table_name} table")
64
+ def dynamocli_table
65
+ @dynamocli_table ||=
66
+ if stack_resources.nil? || with_drift?
67
+ table = Dynamocli::AWS::Table.new(table_name: table_name, table_on_aws: table_on_aws)
68
+ Dynamocli::Table::StandaloneTable.new(table_name: table_name, table: table)
69
+ else
70
+ stack = Dynamocli::AWS::Stack.new(table_name: table_name, table_resource: table_resource)
71
+ Dynamocli::Table::CloudformationTable.new(table_name: table_name, stack: stack)
72
+ end
211
73
  end
212
74
 
213
- def get_stack_policy_body
214
- @cloudformation.get_stack_policy(stack_name: @stack_name).stack_policy_body
75
+ def with_drift?
76
+ @with_drift
215
77
  end
216
78
 
217
- def wait_for_deployment_to_complete
218
- waiting_seconds = 0
219
- while get_stack_status != "UPDATE_COMPLETE"
220
- LOGGER.info("Waiting for deployment to complete")
221
- sleep waiting_seconds += 1
79
+ def table_resource
80
+ @table_resource ||= stack_resources[:stack_resources].find do |resource|
81
+ resource[:physical_resource_id] == @table_name
222
82
  end
223
83
  end
224
-
225
- def get_stack_status
226
- @cloudformation.describe_stacks(stack_name: @stack_name)[0][0].stack_status
227
- end
228
-
229
- def deploy_stack_with_the_original_template
230
- LOGGER.info("Deploying the stack with the #{@table_name} table")
231
-
232
- @cloudformation.update_stack(
233
- stack_name: @stack_name,
234
- template_body: @original_template.to_json,
235
- parameters: @stack.parameters.map(&:to_h),
236
- capabilities: @stack.capabilities,
237
- role_arn: @stack.role_arn,
238
- rollback_configuration: @stack.rollback_configuration.to_h,
239
- stack_policy_body: get_stack_policy_body,
240
- notification_arns: @stack.notification_arns,
241
- tags: @stack.tags.map(&:to_h)
242
- )
243
-
244
- LOGGER.success("Stack deployed with the #{@table_name} table")
245
- end
246
84
  end
@@ -8,25 +8,32 @@ class Dynamocli::Import
8
8
  LOGGER = TTY::Logger.new
9
9
  SUPPORTED_FILE_FORMATS = ["CSV"]
10
10
 
11
- def initialize(file:, table:)
11
+ def initialize(file:, table:, exported_from_aws: false)
12
12
  @file = file
13
13
  @table = table
14
+ @exported_from_aws = exported_from_aws
14
15
  @dynamodb = Aws::DynamoDB::Client.new
15
16
  end
16
17
 
17
18
  def start
18
19
  records = get_records
19
20
  write_records_to_dynamodb_table(records)
21
+ LOGGER.success("#{records.size} record#{"s" if records.size != 1} imported to #{table}")
22
+ rescue Aws::DynamoDB::Errors::ValidationException => e
23
+ LOGGER.error(e.message)
24
+ exit(42)
20
25
  end
21
26
 
22
27
  private
23
28
 
29
+ attr_reader :file, :table, :dynamodb
30
+
24
31
  def get_records
25
- extension = File.extname(@file)
32
+ extension = File.extname(file)
26
33
 
27
34
  case extension
28
35
  when ".csv"
29
- records_from_csv(@file)
36
+ records_from_csv(file)
30
37
  else
31
38
  LOGGER.error("Not supported file format. Only supported file formats are: #{SUPPORTED_FILE_FORMATS}")
32
39
  exit(42)
@@ -37,7 +44,30 @@ class Dynamocli::Import
37
44
  set_custom_converter_for_csv
38
45
  csv_options = { encoding: "UTF-8", headers: true, converters: :attribute_definitions }
39
46
  records_csv = CSV.read(csv, csv_options)
40
- records_csv.map(&:to_hash)
47
+ if exported_from_aws?
48
+ transform_records_csv_from_aws(records_csv)
49
+ else
50
+ records_csv.map(&:to_hash)
51
+ end
52
+ end
53
+
54
+ def exported_from_aws?
55
+ @exported_from_aws
56
+ end
57
+
58
+ # When the CSV comes from AWS, the header of the CSV is like this: "email (S)".
59
+ # However, we cannot import the CSV with the header like this, we have to remove
60
+ # the part that specifies the type of the field before importing it.
61
+ RANGE_TO_REMOVE_TYPE_FROM_HEADER = (0..-5)
62
+ private_constant :RANGE_TO_REMOVE_TYPE_FROM_HEADER
63
+ def transform_records_csv_from_aws(records_csv)
64
+ records_csv.map do |record_csv|
65
+ record = record_csv.to_h
66
+
67
+ record.each_with_object({}) do |(key, value), records|
68
+ records[key[RANGE_TO_REMOVE_TYPE_FROM_HEADER]] = value
69
+ end
70
+ end
41
71
  end
42
72
 
43
73
  ATTRIBUTE_TYPES_CONVERTERS = {
@@ -46,7 +76,7 @@ class Dynamocli::Import
46
76
  "B" => Proc.new(&StringIO.method(:new))
47
77
  }
48
78
  def set_custom_converter_for_csv
49
- attribute_definitions = @dynamodb.describe_table(table_name: @table).table.attribute_definitions
79
+ attribute_definitions = dynamodb.describe_table(table_name: table).table.attribute_definitions
50
80
  CSV::Converters[:attribute_definitions] = lambda do |value, info|
51
81
  attribute_definition = attribute_definitions.find { |it| it.attribute_name == info.header }
52
82
  return value if attribute_definition.nil?
@@ -56,7 +86,7 @@ class Dynamocli::Import
56
86
 
57
87
  def write_records_to_dynamodb_table(records)
58
88
  slice_items_to_attend_batch_write_limit(records).each do |items|
59
- @dynamodb.batch_write_item(request_items: format_request_items(items))
89
+ dynamodb.batch_write_item(request_items: format_request_items(items))
60
90
  end
61
91
  end
62
92
 
@@ -66,6 +96,6 @@ class Dynamocli::Import
66
96
  end
67
97
 
68
98
  def format_request_items(items)
69
- { @table => items.map { |item| { put_request: { item: item } } } }
99
+ { table => items.map { |item| { put_request: { item: item } } } }
70
100
  end
71
101
  end
@@ -0,0 +1,77 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "aws-sdk-cloudformation"
4
+ require "tty-logger"
5
+
6
+ module Dynamocli::Table
7
+ class CloudformationTable
8
+ def initialize(table_name:, stack:, cloudformation: nil, logger: nil)
9
+ @table_name = table_name
10
+ @stack = stack
11
+ @cloudformation = cloudformation || CLOUDFORMARTION.new
12
+ @logger = logger || LOGGER.new
13
+ end
14
+
15
+ def alert_message_before_continue
16
+ "You're going to deploy and redeploy your #{stack.name} stack to drop and recreate the #{@table_name} table!"
17
+ end
18
+
19
+ def erase
20
+ deploy_stack_without_the_table
21
+ wait_for_deployment_to_complete
22
+ deploy_stack_with_the_original_template
23
+ end
24
+
25
+ private
26
+
27
+ CLOUDFORMARTION = Aws::CloudFormation::Client
28
+ LOGGER = TTY::Logger
29
+ private_constant :CLOUDFORMARTION, :LOGGER
30
+
31
+ attr_reader :table_name, :stack, :cloudformation, :logger
32
+
33
+ def deploy_stack_without_the_table
34
+ logger.info("Deploying the stack without the #{table_name} table")
35
+
36
+ cloudformation.update_stack(
37
+ stack_name: stack.name,
38
+ template_body: stack.template_without_table.to_json,
39
+ parameters: stack.parameters.map(&:to_h),
40
+ capabilities: stack.capabilities,
41
+ role_arn: stack.role_arn,
42
+ rollback_configuration: stack.rollback_configuration.to_h,
43
+ stack_policy_body: stack.policy_body,
44
+ notification_arns: stack.notification_arns,
45
+ tags: stack.tags.map(&:to_h)
46
+ )
47
+
48
+ logger.success("Stack deployed without the #{table_name} table")
49
+ end
50
+
51
+ def wait_for_deployment_to_complete
52
+ waiting_seconds = 0
53
+ while stack.deploying?
54
+ logger.info("Waiting for deployment to complete")
55
+ sleep waiting_seconds += 1
56
+ end
57
+ end
58
+
59
+ def deploy_stack_with_the_original_template
60
+ logger.info("Deploying the stack with the #{table_name} table")
61
+
62
+ cloudformation.update_stack(
63
+ stack_name: stack.name,
64
+ template_body: stack.original_template.to_json,
65
+ parameters: stack.parameters.map(&:to_h),
66
+ capabilities: stack.capabilities,
67
+ role_arn: stack.role_arn,
68
+ rollback_configuration: stack.rollback_configuration.to_h,
69
+ stack_policy_body: stack.policy_body,
70
+ notification_arns: stack.notification_arns,
71
+ tags: stack.tags.map(&:to_h)
72
+ )
73
+
74
+ logger.success("Stack deployed with the #{table_name} table")
75
+ end
76
+ end
77
+ end
@@ -0,0 +1,57 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "aws-sdk-dynamodb"
4
+ require "tty-logger"
5
+
6
+ module Dynamocli::Table
7
+ class StandaloneTable
8
+ def initialize(table_name:, table:, dynamodb: nil, logger: nil)
9
+ @table_name = table_name
10
+ @table = table
11
+ @dynamodb = dynamodb || DYNAMODB.new
12
+ @logger = logger || LOGGER.new
13
+ end
14
+
15
+ def alert_message_before_continue
16
+ "You're going to drop and recreate your #{@table_name} table!"
17
+ end
18
+
19
+ def erase
20
+ delete_table
21
+ wait_for_deletion_to_complete
22
+ create_table
23
+ end
24
+
25
+ private
26
+
27
+ LOGGER = TTY::Logger
28
+ DYNAMODB = Aws::DynamoDB::Client
29
+ private_constant :LOGGER, :DYNAMODB
30
+
31
+ attr_reader :table_name, :table, :dynamodb, :logger
32
+
33
+ def delete_table
34
+ logger.info("Deleting the #{table_name} table")
35
+
36
+ table.delete
37
+
38
+ logger.success("#{table_name} table deleted")
39
+ end
40
+
41
+ def wait_for_deletion_to_complete
42
+ waiting_seconds = 0
43
+ while table.deleting?
44
+ logger.info("Waiting for deletion to complete")
45
+ sleep waiting_seconds += 1
46
+ end
47
+ end
48
+
49
+ def create_table
50
+ logger.info("Creating the #{table_name} table")
51
+
52
+ dynamodb.create_table(table.schema)
53
+
54
+ logger.success("#{table_name} table created")
55
+ end
56
+ end
57
+ end
@@ -1,3 +1,5 @@
1
+ # frozen_string_literal: true
2
+
1
3
  module Dynamocli
2
- VERSION = "0.1.5"
4
+ VERSION = "0.1.6"
3
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: dynamocli
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.5
4
+ version: 0.1.6
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matheus Silva Santos de Oliveira
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-08-08 00:00:00.000000000 Z
11
+ date: 2020-01-13 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: thor
@@ -126,9 +126,7 @@ description:
126
126
  email:
127
127
  - oliveira.matheussilvasantos@gmail.com
128
128
  executables:
129
- - console
130
129
  - dynamocli
131
- - setup
132
130
  extensions: []
133
131
  extra_rdoc_files: []
134
132
  files:
@@ -147,8 +145,12 @@ files:
147
145
  - bin/setup
148
146
  - dynamocli.gemspec
149
147
  - lib/dynamocli.rb
148
+ - lib/dynamocli/aws/stack.rb
149
+ - lib/dynamocli/aws/table.rb
150
150
  - lib/dynamocli/erase.rb
151
151
  - lib/dynamocli/import.rb
152
+ - lib/dynamocli/table/cloudformation_table.rb
153
+ - lib/dynamocli/table/standalone_table.rb
152
154
  - lib/dynamocli/version.rb
153
155
  homepage: https://github.com/matheussilvasantos/dynamocli
154
156
  licenses:
@@ -169,7 +171,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
169
171
  - !ruby/object:Gem::Version
170
172
  version: '0'
171
173
  requirements: []
172
- rubygems_version: 3.0.3
174
+ rubygems_version: 3.0.6
173
175
  signing_key:
174
176
  specification_version: 4
175
177
  summary: Utilities for interaction with AWS DynamoDB