dynamocli 0.1.5 → 0.1.6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/Gemfile.lock +19 -19
- data/README.md +14 -1
- data/dynamocli.gemspec +1 -1
- data/lib/dynamocli.rb +2 -1
- data/lib/dynamocli/aws/stack.rb +98 -0
- data/lib/dynamocli/aws/table.rb +64 -0
- data/lib/dynamocli/erase.rb +30 -192
- data/lib/dynamocli/import.rb +37 -7
- data/lib/dynamocli/table/cloudformation_table.rb +77 -0
- data/lib/dynamocli/table/standalone_table.rb +57 -0
- data/lib/dynamocli/version.rb +3 -1
- metadata +7 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d472e7b32f0ee64582e39cb00bcab25ab4070b025806793fd8595e2acba9bedf
|
4
|
+
data.tar.gz: e7838ac7811278a88629e17fd521fcce36b60b96b37c3ad8349a3ccbdcafc4d1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 47efb4fe6a7a2325f2517c2686c66e5945e7c1953f1477e2a4947fd9bb3b452ae95d5edf1b4fae5555089a460353df0cbf2de84b5e3b2712bac6ff0465aa3e75
|
7
|
+
data.tar.gz: 56a7caa27afe9f7289bf3389d17403d0c98f4d7a93ce9d3ea0d0bba831518bb524309938e9da5cc85872fde57bac73ae5ce6b123833d6507e50d9ed82112bef1
|
data/CHANGELOG.md
CHANGED
@@ -2,6 +2,13 @@
|
|
2
2
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
4
4
|
|
5
|
+
## [0.1.5] - 2020-01-13
|
6
|
+
### Added
|
7
|
+
- Option to import data from a CSV exported from AWS.
|
8
|
+
|
9
|
+
### Changed
|
10
|
+
- Big refactoring in erase functionality.
|
11
|
+
|
5
12
|
## [0.1.5] - 2019-08-08
|
6
13
|
### Fixed
|
7
14
|
- Fix erase table without GSIs.
|
@@ -29,6 +36,7 @@ All notable changes to this project will be documented in this file.
|
|
29
36
|
### Added
|
30
37
|
- Command to import data from a CSV file to a DynamoDB table.
|
31
38
|
|
39
|
+
[0.1.6]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.5...v0.1.6
|
32
40
|
[0.1.5]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.4...v0.1.5
|
33
41
|
[0.1.4]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.3...v0.1.4
|
34
42
|
[0.1.3]: https://github.com/matheussilvasantos/dynamocli/compare/v0.1.2...v0.1.3
|
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
dynamocli (0.1.
|
4
|
+
dynamocli (0.1.6)
|
5
5
|
aws-sdk-cloudformation (~> 1.23)
|
6
6
|
aws-sdk-dynamodb (~> 1.28)
|
7
7
|
thor (~> 0.20)
|
@@ -11,17 +11,17 @@ GEM
|
|
11
11
|
remote: https://rubygems.org/
|
12
12
|
specs:
|
13
13
|
aws-eventstream (1.0.3)
|
14
|
-
aws-partitions (1.
|
15
|
-
aws-sdk-cloudformation (1.
|
16
|
-
aws-sdk-core (~> 3, >= 3.
|
14
|
+
aws-partitions (1.263.0)
|
15
|
+
aws-sdk-cloudformation (1.29.0)
|
16
|
+
aws-sdk-core (~> 3, >= 3.71.0)
|
17
17
|
aws-sigv4 (~> 1.1)
|
18
|
-
aws-sdk-core (3.
|
18
|
+
aws-sdk-core (3.89.0)
|
19
19
|
aws-eventstream (~> 1.0, >= 1.0.2)
|
20
|
-
aws-partitions (~> 1.0)
|
20
|
+
aws-partitions (~> 1, >= 1.239.0)
|
21
21
|
aws-sigv4 (~> 1.1)
|
22
22
|
jmespath (~> 1.0)
|
23
|
-
aws-sdk-dynamodb (1.
|
24
|
-
aws-sdk-core (~> 3, >= 3.
|
23
|
+
aws-sdk-dynamodb (1.41.0)
|
24
|
+
aws-sdk-core (~> 3, >= 3.71.0)
|
25
25
|
aws-sigv4 (~> 1.1)
|
26
26
|
aws-sigv4 (1.1.0)
|
27
27
|
aws-eventstream (~> 1.0, >= 1.0.2)
|
@@ -33,19 +33,19 @@ GEM
|
|
33
33
|
equatable (~> 0.6)
|
34
34
|
tty-color (~> 0.5)
|
35
35
|
rake (10.5.0)
|
36
|
-
rspec (3.
|
37
|
-
rspec-core (~> 3.
|
38
|
-
rspec-expectations (~> 3.
|
39
|
-
rspec-mocks (~> 3.
|
40
|
-
rspec-core (3.
|
41
|
-
rspec-support (~> 3.
|
42
|
-
rspec-expectations (3.
|
36
|
+
rspec (3.9.0)
|
37
|
+
rspec-core (~> 3.9.0)
|
38
|
+
rspec-expectations (~> 3.9.0)
|
39
|
+
rspec-mocks (~> 3.9.0)
|
40
|
+
rspec-core (3.9.1)
|
41
|
+
rspec-support (~> 3.9.1)
|
42
|
+
rspec-expectations (3.9.0)
|
43
43
|
diff-lcs (>= 1.2.0, < 2.0)
|
44
|
-
rspec-support (~> 3.
|
45
|
-
rspec-mocks (3.
|
44
|
+
rspec-support (~> 3.9.0)
|
45
|
+
rspec-mocks (3.9.1)
|
46
46
|
diff-lcs (>= 1.2.0, < 2.0)
|
47
|
-
rspec-support (~> 3.
|
48
|
-
rspec-support (3.
|
47
|
+
rspec-support (~> 3.9.0)
|
48
|
+
rspec-support (3.9.2)
|
49
49
|
thor (0.20.3)
|
50
50
|
tty-color (0.5.0)
|
51
51
|
tty-logger (0.1.0)
|
data/README.md
CHANGED
@@ -15,12 +15,15 @@ You have to configure AWS in your computer first. The program will use the AWS c
|
|
15
15
|
|
16
16
|
- Import data from a CSV file to a DynamoDB table
|
17
17
|
|
18
|
+
If you have exported the CSV file you want to import from AWS DynamoDB console, you probaly want to modify the headers before importing the CSV file, because AWS exports the CSV file with a symbol indicating the type of the field in the header. You can pass the option `--exported-from-aws` to do that, the default is false.
|
19
|
+
|
18
20
|
```
|
19
21
|
Usage:
|
20
22
|
dynamocli import FILE -t, --table, --to=TABLE
|
21
23
|
|
22
24
|
Options:
|
23
|
-
-t, --table, --to=TABLE
|
25
|
+
-t, --table, --to=TABLE # table you want to import the data
|
26
|
+
[--exported-from-aws], [--no-exported-from-aws] # modify the headers before importing the csv
|
24
27
|
|
25
28
|
Description:
|
26
29
|
`dynamocli import` will import the data in from a file to a table specified.
|
@@ -56,6 +59,16 @@ From the DynamoDB Guidelines for Working with Tables documentation:
|
|
56
59
|
|
57
60
|
> Deleting an entire table is significantly more efficient than removing items one-by-one, which essentially doubles the write throughput as you do as many delete operations as put operations.
|
58
61
|
|
62
|
+
## Known Issues
|
63
|
+
|
64
|
+
### Importing a CSV file with arrays and objects as values in it
|
65
|
+
|
66
|
+
Unfortunately, at this moment, this library cannot properly import array and objects. These values will appear as strings in the DynamoDB table.
|
67
|
+
|
68
|
+
## Cross account or multiple profiles usage
|
69
|
+
|
70
|
+
You can run `dynamocli` passing the `AWS_PROFILE` environment variable with the profile you want to use, for example: `AWS_PROFILE=nondefaultprofile dynamocli erase users`.
|
71
|
+
|
59
72
|
## Development
|
60
73
|
|
61
74
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
data/dynamocli.gemspec
CHANGED
@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
|
|
18
18
|
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
19
19
|
end
|
20
20
|
spec.bindir = "bin"
|
21
|
-
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
21
|
+
spec.executables = spec.files.grep(%r{^bin/dynamocli}) { |f| File.basename(f) }
|
22
22
|
spec.require_paths = ["lib"]
|
23
23
|
|
24
24
|
spec.add_dependency "thor", "~> 0.20"
|
data/lib/dynamocli.rb
CHANGED
@@ -13,8 +13,9 @@ module Dynamocli
|
|
13
13
|
> $ dynamo import users.csv --to users
|
14
14
|
LONGDESC
|
15
15
|
option :to, required: true, desc: "table you want to import the data", banner: "TABLE", aliases: ["-t", "--table"]
|
16
|
+
option "exported-from-aws", desc: "modify the headers before importing the csv", type: :boolean
|
16
17
|
def import(file)
|
17
|
-
Dynamocli::Import.new(file: file, table: options[:to]).start
|
18
|
+
Dynamocli::Import.new(file: file, table: options[:to], exported_from_aws: options["exported-from-aws"]).start
|
18
19
|
end
|
19
20
|
|
20
21
|
desc "erase TABLE", "erase all the data from the DynamoDB TABLE"
|
@@ -0,0 +1,98 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "forwardable"
|
4
|
+
require "json"
|
5
|
+
require "yaml"
|
6
|
+
require "aws-sdk-cloudformation"
|
7
|
+
require "tty-logger"
|
8
|
+
|
9
|
+
module Dynamocli::AWS
|
10
|
+
class Stack
|
11
|
+
attr_reader :name, :resources, :template_body, :original_template, :template_without_table, :policy_body
|
12
|
+
|
13
|
+
extend Forwardable
|
14
|
+
def_delegators :stack_on_aws,
|
15
|
+
:parameters, :capabilities, :role_arn, :rollback_configuration, :notification_arns, :tags
|
16
|
+
|
17
|
+
def initialize(table_name:, table_resource:, cloudformation: nil, logger: nil)
|
18
|
+
@table_name = table_name
|
19
|
+
@table_resource = table_resource
|
20
|
+
@cloudformation = cloudformation || CLOUDFORMARTION.new
|
21
|
+
@logger = logger || LOGGER.new
|
22
|
+
|
23
|
+
set_attributes_now_because_they_will_change
|
24
|
+
end
|
25
|
+
|
26
|
+
def deploying?
|
27
|
+
current_status != DEPLOY_COMPLETED_KEY
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
CLOUDFORMARTION = Aws::CloudFormation::Client
|
33
|
+
LOGGER = TTY::Logger
|
34
|
+
DEPLOY_COMPLETED_KEY = "UPDATE_COMPLETE"
|
35
|
+
private_constant :CLOUDFORMARTION, :LOGGER, :DEPLOY_COMPLETED_KEY
|
36
|
+
|
37
|
+
attr_reader :table_name, :table_resource, :cloudformation, :logger, :stack_on_aws
|
38
|
+
|
39
|
+
def set_attributes_now_because_they_will_change
|
40
|
+
set_name
|
41
|
+
set_stack_on_aws
|
42
|
+
set_resources
|
43
|
+
set_template_body
|
44
|
+
set_original_template
|
45
|
+
set_template_without_table
|
46
|
+
set_policy_body
|
47
|
+
end
|
48
|
+
|
49
|
+
def set_name
|
50
|
+
@name ||= table_resource[:stack_name]
|
51
|
+
end
|
52
|
+
|
53
|
+
def set_stack_on_aws
|
54
|
+
@stack_on_aws ||= cloudformation.describe_stacks(stack_name: name)[0][0]
|
55
|
+
end
|
56
|
+
|
57
|
+
def set_resources
|
58
|
+
@resources ||= cloudformation.describe_stack_resources(physical_resource_id: table_name).to_h
|
59
|
+
end
|
60
|
+
|
61
|
+
def set_template_body
|
62
|
+
@template_body ||= cloudformation.get_template(stack_name: name).to_h[:template_body]
|
63
|
+
end
|
64
|
+
|
65
|
+
def set_original_template
|
66
|
+
@original_template ||= parse_template(template_body)
|
67
|
+
end
|
68
|
+
|
69
|
+
def set_template_without_table
|
70
|
+
@template_without_table ||= parse_template(template_body).tap do |template_without_table|
|
71
|
+
tables = original_template["Resources"].select { |_, v| v["Type"] == "AWS::DynamoDB::Table" }
|
72
|
+
table = tables.find { |_, v| v["Properties"]["TableName"] == table_name }
|
73
|
+
|
74
|
+
if tables.nil?
|
75
|
+
logger.error("table #{table_name} not found in the #{@name} stack")
|
76
|
+
exit(42)
|
77
|
+
end
|
78
|
+
|
79
|
+
logical_resource_id = table.first
|
80
|
+
template_without_table["Resources"].delete(logical_resource_id)
|
81
|
+
end
|
82
|
+
end
|
83
|
+
|
84
|
+
def parse_template(template)
|
85
|
+
JSON.parse(template)
|
86
|
+
rescue JSON::ParserError
|
87
|
+
YAML.load(template)
|
88
|
+
end
|
89
|
+
|
90
|
+
def set_policy_body
|
91
|
+
@policy_body ||= cloudformation.get_stack_policy(stack_name: name).stack_policy_body
|
92
|
+
end
|
93
|
+
|
94
|
+
def current_status
|
95
|
+
cloudformation.describe_stacks(stack_name: name)[0][0].stack_status
|
96
|
+
end
|
97
|
+
end
|
98
|
+
end
|
@@ -0,0 +1,64 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "forwardable"
|
4
|
+
require "aws-sdk-dynamodb"
|
5
|
+
|
6
|
+
module Dynamocli::AWS
|
7
|
+
class Table
|
8
|
+
attr_reader :schema
|
9
|
+
|
10
|
+
extend Forwardable
|
11
|
+
def_delegators :table_on_aws, :delete
|
12
|
+
|
13
|
+
def initialize(table_name:, table_on_aws:, dynamodb: nil)
|
14
|
+
@table_name = table_name
|
15
|
+
@table_on_aws = table_on_aws
|
16
|
+
@dynamodb = dynamodb || DYNAMODB.new
|
17
|
+
|
18
|
+
set_schema_before_we_delete_the_table
|
19
|
+
end
|
20
|
+
|
21
|
+
def deleting?
|
22
|
+
status == DELETION_IN_PROCESSING_KEY
|
23
|
+
rescue Aws::DynamoDB::Errors::ResourceNotFoundException
|
24
|
+
false
|
25
|
+
end
|
26
|
+
|
27
|
+
private
|
28
|
+
|
29
|
+
DYNAMODB = Aws::DynamoDB::Client
|
30
|
+
DELETION_IN_PROCESSING_KEY = "DELETING"
|
31
|
+
private_constant :DYNAMODB, :DELETION_IN_PROCESSING_KEY
|
32
|
+
|
33
|
+
attr_reader :table_name, :table_on_aws, :dynamodb
|
34
|
+
|
35
|
+
def status
|
36
|
+
dynamodb.describe_table(table_name: table_name).table.table_status
|
37
|
+
end
|
38
|
+
|
39
|
+
def set_schema_before_we_delete_the_table
|
40
|
+
@schema ||= dynamodb.describe_table(table_name: table_name).to_h[:table].tap do |schema|
|
41
|
+
schema.delete(:table_status)
|
42
|
+
schema.delete(:creation_date_time)
|
43
|
+
schema.delete(:table_size_bytes)
|
44
|
+
schema.delete(:item_count)
|
45
|
+
schema.delete(:table_arn)
|
46
|
+
schema.delete(:table_id)
|
47
|
+
schema[:provisioned_throughput]&.delete(:number_of_decreases_today)
|
48
|
+
schema[:local_secondary_indexes]&.each do |lsi|
|
49
|
+
lsi.delete(:index_status)
|
50
|
+
lsi.delete(:index_size_bytes)
|
51
|
+
lsi.delete(:item_count)
|
52
|
+
lsi.delete(:index_arn)
|
53
|
+
end
|
54
|
+
schema[:global_secondary_indexes]&.each do |gsi|
|
55
|
+
gsi.delete(:index_status)
|
56
|
+
gsi.delete(:index_size_bytes)
|
57
|
+
gsi.delete(:item_count)
|
58
|
+
gsi.delete(:index_arn)
|
59
|
+
gsi[:provisioned_throughput].delete(:number_of_decreases_today)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
data/lib/dynamocli/erase.rb
CHANGED
@@ -1,127 +1,52 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require "json"
|
4
|
-
require "yaml"
|
5
3
|
require "tty-logger"
|
6
4
|
require "aws-sdk-dynamodb"
|
7
5
|
require "aws-sdk-cloudformation"
|
6
|
+
require "dynamocli/table/cloudformation_table"
|
7
|
+
require "dynamocli/table/standalone_table"
|
8
|
+
require "dynamocli/aws/stack"
|
9
|
+
require "dynamocli/aws/table"
|
8
10
|
|
9
11
|
class Dynamocli::Erase
|
10
|
-
LOGGER = TTY::Logger.new
|
11
|
-
|
12
12
|
def initialize(table_name:, with_drift: false)
|
13
13
|
@with_drift = with_drift
|
14
14
|
@table_name = table_name
|
15
15
|
|
16
16
|
@dynamodb = Aws::DynamoDB::Client.new
|
17
17
|
@cloudformation = Aws::CloudFormation::Client.new
|
18
|
-
@
|
19
|
-
|
20
|
-
set_schema
|
18
|
+
@table_on_aws = Aws::DynamoDB::Table.new(@table_name)
|
21
19
|
|
22
20
|
@stack_resources = @cloudformation.describe_stack_resources(physical_resource_id: @table_name).to_h
|
23
|
-
|
24
|
-
set_stack_information
|
25
|
-
rescue Aws::DynamoDB::Errors::ResourceNotFoundException => e
|
26
|
-
LOGGER.error(e.message)
|
27
|
-
exit(42)
|
28
21
|
rescue Aws::CloudFormation::Errors::ValidationError
|
29
22
|
@stack_resources = nil
|
30
23
|
end
|
31
24
|
|
32
25
|
def start
|
33
26
|
erase_table
|
34
|
-
rescue Aws::CloudFormation::Errors::ValidationError
|
27
|
+
rescue Aws::CloudFormation::Errors::ValidationError,
|
28
|
+
Aws::DynamoDB::Errors::ValidationException,
|
29
|
+
Aws::DynamoDB::Errors::ResourceNotFoundException => e
|
35
30
|
LOGGER.error(e.message)
|
36
31
|
exit(42)
|
37
32
|
end
|
38
33
|
|
39
34
|
private
|
40
35
|
|
41
|
-
|
42
|
-
|
43
|
-
schema.delete(:table_status)
|
44
|
-
schema.delete(:creation_date_time)
|
45
|
-
schema.delete(:table_size_bytes)
|
46
|
-
schema.delete(:item_count)
|
47
|
-
schema.delete(:table_arn)
|
48
|
-
schema.delete(:table_id)
|
49
|
-
schema[:provisioned_throughput].delete(:number_of_decreases_today)
|
50
|
-
schema[:local_secondary_indexes]&.each do |lsi|
|
51
|
-
lsi.delete(:index_status)
|
52
|
-
lsi.delete(:index_size_bytes)
|
53
|
-
lsi.delete(:item_count)
|
54
|
-
lsi.delete(:index_arn)
|
55
|
-
end
|
56
|
-
schema[:global_secondary_indexes]&.each do |gsi|
|
57
|
-
gsi.delete(:index_status)
|
58
|
-
gsi.delete(:index_size_bytes)
|
59
|
-
gsi.delete(:item_count)
|
60
|
-
gsi.delete(:index_arn)
|
61
|
-
gsi[:provisioned_throughput].delete(:number_of_decreases_today)
|
62
|
-
end
|
63
|
-
end
|
64
|
-
end
|
65
|
-
|
66
|
-
def set_stack_information
|
67
|
-
return if @stack_resources.nil?
|
68
|
-
|
69
|
-
set_stack_name
|
70
|
-
set_stack
|
71
|
-
set_templates
|
72
|
-
rescue Aws::CloudFormation::Errors::ValidationError => e
|
73
|
-
LOGGER.error(e.message)
|
74
|
-
exit(42)
|
75
|
-
end
|
76
|
-
|
77
|
-
def set_stack_name
|
78
|
-
table_resource = @stack_resources[:stack_resources].find do |resource|
|
79
|
-
resource[:physical_resource_id] == @table_name
|
80
|
-
end
|
81
|
-
@stack_name = table_resource[:stack_name]
|
82
|
-
end
|
83
|
-
|
84
|
-
def set_stack
|
85
|
-
@stack = @cloudformation.describe_stacks(stack_name: @stack_name)[0][0]
|
86
|
-
end
|
87
|
-
|
88
|
-
def set_templates
|
89
|
-
template_body = @cloudformation.get_template(stack_name: @stack_name).to_h[:template_body]
|
90
|
-
@original_template = parse_template(template_body)
|
91
|
-
@template_without_table = parse_template(template_body)
|
92
|
-
|
93
|
-
tables = @original_template["Resources"].select { |_, v| v["Type"] == "AWS::DynamoDB::Table" }
|
94
|
-
table = tables.find { |_, v| v["Properties"]["TableName"] == @table_name }
|
95
|
-
|
96
|
-
if tables.nil?
|
97
|
-
LOGGER.error("table #{@table_name} not found in the #{@stack_name} stack")
|
98
|
-
exit(42)
|
99
|
-
end
|
100
|
-
|
101
|
-
logical_resource_id = table.first
|
102
|
-
@template_without_table["Resources"].delete(logical_resource_id)
|
103
|
-
end
|
36
|
+
LOGGER = TTY::Logger.new
|
37
|
+
private_constant :LOGGER
|
104
38
|
|
105
|
-
|
106
|
-
JSON.parse(template)
|
107
|
-
rescue JSON::ParserError
|
108
|
-
YAML.load(template)
|
109
|
-
end
|
39
|
+
attr_reader :table_name, :table_on_aws, :stack_resources
|
110
40
|
|
111
41
|
def erase_table
|
112
|
-
|
113
|
-
|
114
|
-
delete_and_recreate_the_table
|
115
|
-
else
|
116
|
-
check_if_user_wants_to_continue_with_deployment
|
117
|
-
erase_table_through_cloudformation
|
118
|
-
end
|
42
|
+
check_if_user_wants_to_continue
|
43
|
+
dynamocli_table.erase
|
119
44
|
end
|
120
45
|
|
121
|
-
def
|
46
|
+
def check_if_user_wants_to_continue
|
122
47
|
LOGGER.warn(
|
123
|
-
"
|
124
|
-
"
|
48
|
+
"#{dynamocli_table.alert_message_before_continue} " \
|
49
|
+
"Do you really want to continue?"
|
125
50
|
)
|
126
51
|
STDOUT.print("(anything other than 'y' will cancel) > ")
|
127
52
|
|
@@ -136,111 +61,24 @@ class Dynamocli::Erase
|
|
136
61
|
"Erase of #{@table_name} table canceled"
|
137
62
|
end
|
138
63
|
|
139
|
-
def
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
|
145
|
-
|
146
|
-
|
147
|
-
|
148
|
-
@table.delete
|
149
|
-
|
150
|
-
LOGGER.success("#{@table_name} table deleted")
|
151
|
-
end
|
152
|
-
|
153
|
-
def wait_for_deletion_to_complete
|
154
|
-
waiting_seconds = 0
|
155
|
-
while get_table_status == "DELETING"
|
156
|
-
LOGGER.info("Waiting for deletion to complete")
|
157
|
-
sleep waiting_seconds += 1
|
158
|
-
end
|
159
|
-
rescue Aws::DynamoDB::Errors::ResourceNotFoundException
|
160
|
-
true
|
161
|
-
end
|
162
|
-
|
163
|
-
def get_table_status
|
164
|
-
@dynamodb.describe_table(table_name: @table_name).table.table_status
|
165
|
-
end
|
166
|
-
|
167
|
-
def create_table
|
168
|
-
LOGGER.info("Creating the #{@table_name} table")
|
169
|
-
|
170
|
-
@dynamodb.create_table(@schema)
|
171
|
-
|
172
|
-
LOGGER.success("#{@table_name} table created")
|
173
|
-
end
|
174
|
-
|
175
|
-
def check_if_user_wants_to_continue_with_deployment
|
176
|
-
LOGGER.warn(
|
177
|
-
"You are going to deploy and redeploy your #{@stack_name} stack\n" \
|
178
|
-
"to drop and recreate the #{@table_name} table, do you really want to continue?"
|
179
|
-
)
|
180
|
-
STDOUT.print("(anything other than 'y' will cancel) > ")
|
181
|
-
|
182
|
-
confirmation = STDIN.gets.strip
|
183
|
-
return if confirmation == "y"
|
184
|
-
|
185
|
-
LOGGER.info(abort_message)
|
186
|
-
exit(0)
|
187
|
-
end
|
188
|
-
|
189
|
-
def erase_table_through_cloudformation
|
190
|
-
deploy_stack_without_the_table
|
191
|
-
wait_for_deployment_to_complete
|
192
|
-
deploy_stack_with_the_original_template
|
193
|
-
end
|
194
|
-
|
195
|
-
def deploy_stack_without_the_table
|
196
|
-
LOGGER.info("Deploying the stack without the #{@table_name} table")
|
197
|
-
|
198
|
-
@cloudformation.update_stack(
|
199
|
-
stack_name: @stack_name,
|
200
|
-
template_body: @template_without_table.to_json,
|
201
|
-
parameters: @stack.parameters.map(&:to_h),
|
202
|
-
capabilities: @stack.capabilities,
|
203
|
-
role_arn: @stack.role_arn,
|
204
|
-
rollback_configuration: @stack.rollback_configuration.to_h,
|
205
|
-
stack_policy_body: get_stack_policy_body,
|
206
|
-
notification_arns: @stack.notification_arns,
|
207
|
-
tags: @stack.tags.map(&:to_h)
|
208
|
-
)
|
209
|
-
|
210
|
-
LOGGER.success("Stack deployed without the #{@table_name} table")
|
64
|
+
def dynamocli_table
|
65
|
+
@dynamocli_table ||=
|
66
|
+
if stack_resources.nil? || with_drift?
|
67
|
+
table = Dynamocli::AWS::Table.new(table_name: table_name, table_on_aws: table_on_aws)
|
68
|
+
Dynamocli::Table::StandaloneTable.new(table_name: table_name, table: table)
|
69
|
+
else
|
70
|
+
stack = Dynamocli::AWS::Stack.new(table_name: table_name, table_resource: table_resource)
|
71
|
+
Dynamocli::Table::CloudformationTable.new(table_name: table_name, stack: stack)
|
72
|
+
end
|
211
73
|
end
|
212
74
|
|
213
|
-
def
|
214
|
-
@
|
75
|
+
def with_drift?
|
76
|
+
@with_drift
|
215
77
|
end
|
216
78
|
|
217
|
-
def
|
218
|
-
|
219
|
-
|
220
|
-
LOGGER.info("Waiting for deployment to complete")
|
221
|
-
sleep waiting_seconds += 1
|
79
|
+
def table_resource
|
80
|
+
@table_resource ||= stack_resources[:stack_resources].find do |resource|
|
81
|
+
resource[:physical_resource_id] == @table_name
|
222
82
|
end
|
223
83
|
end
|
224
|
-
|
225
|
-
def get_stack_status
|
226
|
-
@cloudformation.describe_stacks(stack_name: @stack_name)[0][0].stack_status
|
227
|
-
end
|
228
|
-
|
229
|
-
def deploy_stack_with_the_original_template
|
230
|
-
LOGGER.info("Deploying the stack with the #{@table_name} table")
|
231
|
-
|
232
|
-
@cloudformation.update_stack(
|
233
|
-
stack_name: @stack_name,
|
234
|
-
template_body: @original_template.to_json,
|
235
|
-
parameters: @stack.parameters.map(&:to_h),
|
236
|
-
capabilities: @stack.capabilities,
|
237
|
-
role_arn: @stack.role_arn,
|
238
|
-
rollback_configuration: @stack.rollback_configuration.to_h,
|
239
|
-
stack_policy_body: get_stack_policy_body,
|
240
|
-
notification_arns: @stack.notification_arns,
|
241
|
-
tags: @stack.tags.map(&:to_h)
|
242
|
-
)
|
243
|
-
|
244
|
-
LOGGER.success("Stack deployed with the #{@table_name} table")
|
245
|
-
end
|
246
84
|
end
|
data/lib/dynamocli/import.rb
CHANGED
@@ -8,25 +8,32 @@ class Dynamocli::Import
|
|
8
8
|
LOGGER = TTY::Logger.new
|
9
9
|
SUPPORTED_FILE_FORMATS = ["CSV"]
|
10
10
|
|
11
|
-
def initialize(file:, table:)
|
11
|
+
def initialize(file:, table:, exported_from_aws: false)
|
12
12
|
@file = file
|
13
13
|
@table = table
|
14
|
+
@exported_from_aws = exported_from_aws
|
14
15
|
@dynamodb = Aws::DynamoDB::Client.new
|
15
16
|
end
|
16
17
|
|
17
18
|
def start
|
18
19
|
records = get_records
|
19
20
|
write_records_to_dynamodb_table(records)
|
21
|
+
LOGGER.success("#{records.size} record#{"s" if records.size != 1} imported to #{table}")
|
22
|
+
rescue Aws::DynamoDB::Errors::ValidationException => e
|
23
|
+
LOGGER.error(e.message)
|
24
|
+
exit(42)
|
20
25
|
end
|
21
26
|
|
22
27
|
private
|
23
28
|
|
29
|
+
attr_reader :file, :table, :dynamodb
|
30
|
+
|
24
31
|
def get_records
|
25
|
-
extension = File.extname(
|
32
|
+
extension = File.extname(file)
|
26
33
|
|
27
34
|
case extension
|
28
35
|
when ".csv"
|
29
|
-
records_from_csv(
|
36
|
+
records_from_csv(file)
|
30
37
|
else
|
31
38
|
LOGGER.error("Not supported file format. Only supported file formats are: #{SUPPORTED_FILE_FORMATS}")
|
32
39
|
exit(42)
|
@@ -37,7 +44,30 @@ class Dynamocli::Import
|
|
37
44
|
set_custom_converter_for_csv
|
38
45
|
csv_options = { encoding: "UTF-8", headers: true, converters: :attribute_definitions }
|
39
46
|
records_csv = CSV.read(csv, csv_options)
|
40
|
-
|
47
|
+
if exported_from_aws?
|
48
|
+
transform_records_csv_from_aws(records_csv)
|
49
|
+
else
|
50
|
+
records_csv.map(&:to_hash)
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
def exported_from_aws?
|
55
|
+
@exported_from_aws
|
56
|
+
end
|
57
|
+
|
58
|
+
# When the CSV comes from AWS, the header of the CSV is like this: "email (S)".
|
59
|
+
# However, we cannot import the CSV with the header like this, we have to remove
|
60
|
+
# the part that specifies the type of the field before importing it.
|
61
|
+
RANGE_TO_REMOVE_TYPE_FROM_HEADER = (0..-5)
|
62
|
+
private_constant :RANGE_TO_REMOVE_TYPE_FROM_HEADER
|
63
|
+
def transform_records_csv_from_aws(records_csv)
|
64
|
+
records_csv.map do |record_csv|
|
65
|
+
record = record_csv.to_h
|
66
|
+
|
67
|
+
record.each_with_object({}) do |(key, value), records|
|
68
|
+
records[key[RANGE_TO_REMOVE_TYPE_FROM_HEADER]] = value
|
69
|
+
end
|
70
|
+
end
|
41
71
|
end
|
42
72
|
|
43
73
|
ATTRIBUTE_TYPES_CONVERTERS = {
|
@@ -46,7 +76,7 @@ class Dynamocli::Import
|
|
46
76
|
"B" => Proc.new(&StringIO.method(:new))
|
47
77
|
}
|
48
78
|
def set_custom_converter_for_csv
|
49
|
-
attribute_definitions =
|
79
|
+
attribute_definitions = dynamodb.describe_table(table_name: table).table.attribute_definitions
|
50
80
|
CSV::Converters[:attribute_definitions] = lambda do |value, info|
|
51
81
|
attribute_definition = attribute_definitions.find { |it| it.attribute_name == info.header }
|
52
82
|
return value if attribute_definition.nil?
|
@@ -56,7 +86,7 @@ class Dynamocli::Import
|
|
56
86
|
|
57
87
|
def write_records_to_dynamodb_table(records)
|
58
88
|
slice_items_to_attend_batch_write_limit(records).each do |items|
|
59
|
-
|
89
|
+
dynamodb.batch_write_item(request_items: format_request_items(items))
|
60
90
|
end
|
61
91
|
end
|
62
92
|
|
@@ -66,6 +96,6 @@ class Dynamocli::Import
|
|
66
96
|
end
|
67
97
|
|
68
98
|
def format_request_items(items)
|
69
|
-
{
|
99
|
+
{ table => items.map { |item| { put_request: { item: item } } } }
|
70
100
|
end
|
71
101
|
end
|
@@ -0,0 +1,77 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "aws-sdk-cloudformation"
|
4
|
+
require "tty-logger"
|
5
|
+
|
6
|
+
module Dynamocli::Table
|
7
|
+
class CloudformationTable
|
8
|
+
def initialize(table_name:, stack:, cloudformation: nil, logger: nil)
|
9
|
+
@table_name = table_name
|
10
|
+
@stack = stack
|
11
|
+
@cloudformation = cloudformation || CLOUDFORMARTION.new
|
12
|
+
@logger = logger || LOGGER.new
|
13
|
+
end
|
14
|
+
|
15
|
+
def alert_message_before_continue
|
16
|
+
"You're going to deploy and redeploy your #{stack.name} stack to drop and recreate the #{@table_name} table!"
|
17
|
+
end
|
18
|
+
|
19
|
+
def erase
|
20
|
+
deploy_stack_without_the_table
|
21
|
+
wait_for_deployment_to_complete
|
22
|
+
deploy_stack_with_the_original_template
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
CLOUDFORMARTION = Aws::CloudFormation::Client
|
28
|
+
LOGGER = TTY::Logger
|
29
|
+
private_constant :CLOUDFORMARTION, :LOGGER
|
30
|
+
|
31
|
+
attr_reader :table_name, :stack, :cloudformation, :logger
|
32
|
+
|
33
|
+
def deploy_stack_without_the_table
|
34
|
+
logger.info("Deploying the stack without the #{table_name} table")
|
35
|
+
|
36
|
+
cloudformation.update_stack(
|
37
|
+
stack_name: stack.name,
|
38
|
+
template_body: stack.template_without_table.to_json,
|
39
|
+
parameters: stack.parameters.map(&:to_h),
|
40
|
+
capabilities: stack.capabilities,
|
41
|
+
role_arn: stack.role_arn,
|
42
|
+
rollback_configuration: stack.rollback_configuration.to_h,
|
43
|
+
stack_policy_body: stack.policy_body,
|
44
|
+
notification_arns: stack.notification_arns,
|
45
|
+
tags: stack.tags.map(&:to_h)
|
46
|
+
)
|
47
|
+
|
48
|
+
logger.success("Stack deployed without the #{table_name} table")
|
49
|
+
end
|
50
|
+
|
51
|
+
def wait_for_deployment_to_complete
|
52
|
+
waiting_seconds = 0
|
53
|
+
while stack.deploying?
|
54
|
+
logger.info("Waiting for deployment to complete")
|
55
|
+
sleep waiting_seconds += 1
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
def deploy_stack_with_the_original_template
|
60
|
+
logger.info("Deploying the stack with the #{table_name} table")
|
61
|
+
|
62
|
+
cloudformation.update_stack(
|
63
|
+
stack_name: stack.name,
|
64
|
+
template_body: stack.original_template.to_json,
|
65
|
+
parameters: stack.parameters.map(&:to_h),
|
66
|
+
capabilities: stack.capabilities,
|
67
|
+
role_arn: stack.role_arn,
|
68
|
+
rollback_configuration: stack.rollback_configuration.to_h,
|
69
|
+
stack_policy_body: stack.policy_body,
|
70
|
+
notification_arns: stack.notification_arns,
|
71
|
+
tags: stack.tags.map(&:to_h)
|
72
|
+
)
|
73
|
+
|
74
|
+
logger.success("Stack deployed with the #{table_name} table")
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "aws-sdk-dynamodb"
|
4
|
+
require "tty-logger"
|
5
|
+
|
6
|
+
module Dynamocli::Table
|
7
|
+
class StandaloneTable
|
8
|
+
def initialize(table_name:, table:, dynamodb: nil, logger: nil)
|
9
|
+
@table_name = table_name
|
10
|
+
@table = table
|
11
|
+
@dynamodb = dynamodb || DYNAMODB.new
|
12
|
+
@logger = logger || LOGGER.new
|
13
|
+
end
|
14
|
+
|
15
|
+
def alert_message_before_continue
|
16
|
+
"You're going to drop and recreate your #{@table_name} table!"
|
17
|
+
end
|
18
|
+
|
19
|
+
def erase
|
20
|
+
delete_table
|
21
|
+
wait_for_deletion_to_complete
|
22
|
+
create_table
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
LOGGER = TTY::Logger
|
28
|
+
DYNAMODB = Aws::DynamoDB::Client
|
29
|
+
private_constant :LOGGER, :DYNAMODB
|
30
|
+
|
31
|
+
attr_reader :table_name, :table, :dynamodb, :logger
|
32
|
+
|
33
|
+
def delete_table
|
34
|
+
logger.info("Deleting the #{table_name} table")
|
35
|
+
|
36
|
+
table.delete
|
37
|
+
|
38
|
+
logger.success("#{table_name} table deleted")
|
39
|
+
end
|
40
|
+
|
41
|
+
def wait_for_deletion_to_complete
|
42
|
+
waiting_seconds = 0
|
43
|
+
while table.deleting?
|
44
|
+
logger.info("Waiting for deletion to complete")
|
45
|
+
sleep waiting_seconds += 1
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
def create_table
|
50
|
+
logger.info("Creating the #{table_name} table")
|
51
|
+
|
52
|
+
dynamodb.create_table(table.schema)
|
53
|
+
|
54
|
+
logger.success("#{table_name} table created")
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
data/lib/dynamocli/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: dynamocli
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matheus Silva Santos de Oliveira
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-01-13 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: thor
|
@@ -126,9 +126,7 @@ description:
|
|
126
126
|
email:
|
127
127
|
- oliveira.matheussilvasantos@gmail.com
|
128
128
|
executables:
|
129
|
-
- console
|
130
129
|
- dynamocli
|
131
|
-
- setup
|
132
130
|
extensions: []
|
133
131
|
extra_rdoc_files: []
|
134
132
|
files:
|
@@ -147,8 +145,12 @@ files:
|
|
147
145
|
- bin/setup
|
148
146
|
- dynamocli.gemspec
|
149
147
|
- lib/dynamocli.rb
|
148
|
+
- lib/dynamocli/aws/stack.rb
|
149
|
+
- lib/dynamocli/aws/table.rb
|
150
150
|
- lib/dynamocli/erase.rb
|
151
151
|
- lib/dynamocli/import.rb
|
152
|
+
- lib/dynamocli/table/cloudformation_table.rb
|
153
|
+
- lib/dynamocli/table/standalone_table.rb
|
152
154
|
- lib/dynamocli/version.rb
|
153
155
|
homepage: https://github.com/matheussilvasantos/dynamocli
|
154
156
|
licenses:
|
@@ -169,7 +171,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
169
171
|
- !ruby/object:Gem::Version
|
170
172
|
version: '0'
|
171
173
|
requirements: []
|
172
|
-
rubygems_version: 3.0.
|
174
|
+
rubygems_version: 3.0.6
|
173
175
|
signing_key:
|
174
176
|
specification_version: 4
|
175
177
|
summary: Utilities for interaction with AWS DynamoDB
|