s3twin 0.0.2 → 0.0.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +1 -1
- data/README.md +29 -16
- data/bin/s3twin +22 -8
- data/lib/s3twin.rb +1 -1
- data/lib/s3twin/helpers.rb +19 -8
- data/lib/s3twin/twin.rb +27 -28
- data/s3twin.gemspec +3 -3
- metadata +8 -9
- data/.env.sample +0 -10
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: e0d4ae42cce139cc7fd2957f965f68c10e386661
|
4
|
+
data.tar.gz: c17b586aeef609c1e88458f20c5bdf2855210140
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: eab5b8a228593e2c15a27ae7ad334903132c613f3cbd2c01526d2ca227859355425f33fcfbdf124fc989f1e3d6091bdadffb085d73951ed17632a4ec5cda865b
|
7
|
+
data.tar.gz: a863873bdd7499318b688d79dcc490535b2b5beb2fdf0b748f3483d1f967ffebd461944a4cad50746bc07b2e69c60233496bcb375d0cd1ca382c56caf03f581c
|
data/.gitignore
CHANGED
data/README.md
CHANGED
@@ -1,43 +1,56 @@
|
|
1
1
|
# S3 Twin
|
2
2
|
|
3
|
-
Take a mirror of a S3 bucket. It leverages [aws-sdk's](http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html#copy_to-instance_method) `copy_to` function to do the heavy lifting at Amazon, similar to [s3cmd](http://s3tools.org/s3cmd-sync).
|
4
|
-
Supports local usage or remote workers
|
3
|
+
Take a mirror of a S3 bucket. It leverages [aws-sdk's](http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html#copy_to-instance_method) `copy_to` function to do the heavy lifting at Amazon, similar to [s3cmd](http://s3tools.org/s3cmd-sync).
|
4
|
+
Supports local usage or remote workers some of which may support scheduling - to keep a bucket's twin up to date.
|
5
5
|
|
6
|
-
One way copy only i.e. will not keep two buckets in sync.
|
7
|
-
Carries across Public ACLs only
|
6
|
+
One way copy only i.e. will not keep two buckets in sync.
|
7
|
+
Carries across Public ACLs only; there is no support for S3 features like server side encryption.
|
8
8
|
|
9
|
-
If an object exists in the source bucket and does not exist in the destiantion bucket
|
10
|
-
If an object exists in both
|
11
|
-
Does not remove
|
9
|
+
If an object exists in the source bucket and does not exist in the destiantion bucket then the object is copied.
|
10
|
+
If an object exists in both buckets and the [etags](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html) are identical the object is not copied, if the etags do not match the object is copied.
|
11
|
+
Does not remove objects from the destination bucket that have been deleted from the source bucket.
|
12
12
|
|
13
13
|
Please note; while this works it's the first release so the API may change drastically.
|
14
14
|
|
15
|
+
|
15
16
|
## Installation
|
16
17
|
|
17
18
|
`$ gem install s3twin`
|
18
19
|
|
20
|
+
|
19
21
|
## Usage
|
20
22
|
|
21
23
|
`$ s3twin help`
|
22
24
|
|
23
|
-
`$ s3twin go`
|
25
|
+
`$ s3twin go # run twin now locally`
|
26
|
+
|
27
|
+
#### Configuration
|
28
|
+
|
29
|
+
`$ s3twin init # initialise a .env file`
|
24
30
|
|
25
|
-
|
26
|
-
|
31
|
+
The configuration payload can be set via an argument `--payload=key:value key2:value2` or `.env` file.
|
32
|
+
If neither of those are present the user will be prompted for input.
|
33
|
+
|
34
|
+
_Example with payload:_
|
35
|
+
```bash
|
36
|
+
$ s3twin go --payload=source_s3_bucket:bar source_aws_access_key_id:foo source_aws_secret_access_key:world destination_s3_bucket:bar destination_aws_access_key_id:foo destination_aws_secret_access_key:world
|
37
|
+
```
|
27
38
|
|
28
|
-
The payload can be set via an argument `--payload=key:value key2:value2` or `.env` file, if neither of those are present the user will be prompted for input.
|
29
39
|
|
30
40
|
### Remote Workers
|
31
41
|
|
32
|
-
Currently only [iron.io] IronWorker tasks (including scheduled) are supported.
|
42
|
+
Currently only [iron.io](https://iron.io/) IronWorker tasks (including scheduled) are supported.
|
33
43
|
|
34
44
|
#### IronWorker Usage
|
35
45
|
|
36
|
-
Create a new project at [https://hud.iron.io](https://hud.iron.io), collect your Iron Token and Project ID.
|
46
|
+
Create a new project at [https://hud.iron.io](https://hud.iron.io/), collect your Iron Token and Project ID.
|
47
|
+
|
48
|
+
`$ s3twin ironworker help`
|
49
|
+
`$ s3twin ironworker upload # upload worker code`
|
50
|
+
`$ s3twin ironworker go # run worker once now`
|
51
|
+
`$ s3twin ironworker schedule --time # schedule worker for the future`
|
37
52
|
|
38
|
-
|
39
|
-
`$ s3twin ironworker go`
|
40
|
-
`$ s3twin ironworker schedule --time `
|
53
|
+
Configure `--payload=iron_token:foo iron_project_id:bar`
|
41
54
|
|
42
55
|
Optionally create, or append, your `.env` file:
|
43
56
|
|
data/bin/s3twin
CHANGED
@@ -9,20 +9,34 @@ require 's3twin'
|
|
9
9
|
module S3twin
|
10
10
|
class S3twinCLI < Thor
|
11
11
|
include Thor::Actions
|
12
|
-
class_option :payload, :type => :hash, :default => {
|
13
|
-
'
|
14
|
-
'
|
15
|
-
'
|
16
|
-
'
|
17
|
-
'
|
18
|
-
'
|
12
|
+
class_option :payload, :type => :hash, :default => {
|
13
|
+
'source_s3_bucket' => ENV['SOURCE_S3_BUCKET'],
|
14
|
+
'source_aws_access_key_id' => ENV['SOURCE_AWS_ACCESS_KEY_ID'],
|
15
|
+
'source_aws_secret_access_key' => ENV['SOURCE_AWS_SECRET_ACCESS_KEY'],
|
16
|
+
'destination_s3_bucket' => ENV['DESTINATION_S3_BUCKET'],
|
17
|
+
'destination_aws_access_key_id' => ENV['DESTINATION_AWS_ACCESS_KEY_ID'],
|
18
|
+
'destination_aws_secret_access_key' => ENV['DESTINATION_AWS_SECRET_ACCESS_KEY']
|
19
19
|
}
|
20
20
|
|
21
21
|
desc 'go', 'Make a twin of a S3 Bucket'
|
22
22
|
def go
|
23
23
|
Twin.go(S3twin::Helpers.prompt_payload(options['payload']))
|
24
24
|
end
|
25
|
+
|
26
|
+
desc 'init', 'Initalise a .env file'
|
27
|
+
def init
|
28
|
+
filename = '.env'
|
29
|
+
env = S3twin::Helpers.generate_env(options['payload'])
|
30
|
+
if File.exist?(filename)
|
31
|
+
puts "#{filename} already exists; manually append the below:\n"
|
32
|
+
puts env
|
33
|
+
else
|
34
|
+
File.open(filename, 'w') {|f| f.write(env)}
|
35
|
+
puts "#{filename} created."
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
25
39
|
# worker sub-commands live in ../lib/s3twin/workers
|
26
40
|
end
|
27
41
|
S3twinCLI.start
|
28
|
-
end
|
42
|
+
end
|
data/lib/s3twin.rb
CHANGED
data/lib/s3twin/helpers.rb
CHANGED
@@ -1,16 +1,27 @@
|
|
1
1
|
module S3twin::Helpers
|
2
2
|
class << self
|
3
|
-
|
4
3
|
def prompt_payload(payload)
|
5
|
-
payload['
|
6
|
-
payload['
|
7
|
-
payload['
|
8
|
-
payload['
|
9
|
-
payload['
|
10
|
-
payload['
|
4
|
+
payload['source_s3_bucket'] ||= ask('Source Bucket:')
|
5
|
+
payload['source_aws_access_key_id'] ||= ask('Source Access Key:')
|
6
|
+
payload['source_aws_secret_access_key'] ||= ask('Source Secret Key:')
|
7
|
+
payload['destination_s3_bucket'] ||= ask('Destination Bucket:')
|
8
|
+
payload['destination_aws_access_key_id'] ||= ask('Destination Access Key:')
|
9
|
+
payload['destination_aws_secret_access_key'] ||= ask('Destination Secret Key:')
|
11
10
|
return payload
|
12
11
|
end
|
13
12
|
|
13
|
+
def generate_env(payload)
|
14
|
+
env = "IRON_TOKEN='#{payload['iron_token']}'
|
15
|
+
IRON_PROJECT_ID='#{payload['iron_project_id']}'
|
16
|
+
|
17
|
+
SOURCE_S3_BUCKET='#{payload['source_s3_bucket']}'
|
18
|
+
SOURCE_AWS_ACCESS_KEY_ID='#{payload['source_aws_access_key_id']}'
|
19
|
+
SOURCE_AWS_SECRET_ACCESS_KEY='#{payload['source_aws_secret_access_key']}'
|
20
|
+
|
21
|
+
DESTINATION_S3_BUCKET='#{payload['destination_s3_bucket']}'
|
22
|
+
DESTINATION_AWS_ACCESS_KEY_ID='#{payload['destination_aws_access_key_id']}'
|
23
|
+
DESTINATION_AWS_SECRET_ACCESS_KEY='#{payload['destination_aws_secret_access_key']}'"
|
24
|
+
end
|
25
|
+
|
14
26
|
end
|
15
27
|
end
|
16
|
-
|
data/lib/s3twin/twin.rb
CHANGED
@@ -2,15 +2,15 @@ require 'aws-sdk'
|
|
2
2
|
|
3
3
|
class Twin
|
4
4
|
# these exist for ironworker compatibility. It should probaby be seperated out to a wrapper.
|
5
|
-
attr_accessor :
|
5
|
+
attr_accessor :source_s3_bucket, :source_aws_access_key_id, :source_aws_secret_access_key, :destination_s3_bucket, :destination_aws_access_key_id, :destination_aws_secret_access_key
|
6
6
|
def run
|
7
7
|
params = {
|
8
|
-
'
|
9
|
-
'
|
10
|
-
'
|
11
|
-
'
|
12
|
-
'
|
13
|
-
'
|
8
|
+
'source_s3_bucket' => source_s3_bucket,
|
9
|
+
'source_aws_access_key_id' => source_aws_access_key_id,
|
10
|
+
'source_aws_secret_access_key' => source_aws_secret_access_key,
|
11
|
+
'destination_s3_bucket' => destination_s3_bucket,
|
12
|
+
'destination_aws_access_key_id' => destination_aws_access_key_id,
|
13
|
+
'destination_aws_secret_access_key' => destination_aws_secret_access_key
|
14
14
|
}
|
15
15
|
Twin.go(params)
|
16
16
|
end
|
@@ -18,36 +18,33 @@ class Twin
|
|
18
18
|
# Real stuff starts here
|
19
19
|
class << self
|
20
20
|
def go(params)
|
21
|
-
source_bucket = params['source_bucket']
|
22
|
-
source_access_key = params['source_access_key']
|
23
|
-
source_secret_key = params['source_secret_key']
|
24
|
-
destination_bucket = params['destination_bucket']
|
25
|
-
destination_access_key = params['destination_access_key']
|
26
|
-
destination_secret_key = params['destination_secret_key']
|
27
|
-
|
28
21
|
puts 'Starting S3Twin run'
|
29
22
|
@source = AWS::S3.new(
|
30
|
-
:access_key_id =>
|
31
|
-
:secret_access_key =>
|
32
|
-
@source_bucket = @source.buckets[
|
23
|
+
:access_key_id => params['source_aws_access_key_id'],
|
24
|
+
:secret_access_key => params['source_aws_secret_access_key'])
|
25
|
+
@source_bucket = @source.buckets[params['source_s3_bucket']]
|
33
26
|
|
34
27
|
@destination = AWS::S3.new(
|
35
|
-
:access_key_id =>
|
36
|
-
:secret_access_key =>
|
37
|
-
@destination_bucket = @destination.buckets[
|
28
|
+
:access_key_id => params['destination_aws_access_key_id'],
|
29
|
+
:secret_access_key => params['destination_aws_secret_access_key'])
|
30
|
+
@destination_bucket = @destination.buckets[params['destination_s3_bucket']]
|
38
31
|
|
39
32
|
@source_bucket.objects.each do |obj|
|
40
33
|
dest_key = @destination_bucket.objects[obj.key]
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
else
|
45
|
-
unless etag_match(obj.key,obj.etag)
|
46
|
-
puts "Updating: #{obj.key}"
|
34
|
+
begin
|
35
|
+
unless dest_key.exists?
|
36
|
+
puts "Creating: #{obj.key}"
|
47
37
|
obj.copy_to(dest_key, :acl => public_acl(obj))
|
48
38
|
else
|
49
|
-
|
50
|
-
|
39
|
+
unless etag_match(obj.key,obj.etag)
|
40
|
+
puts "Updating: #{obj.key}"
|
41
|
+
obj.copy_to(dest_key, :acl => public_acl(obj))
|
42
|
+
else
|
43
|
+
puts "Skipping: #{obj.key}"
|
44
|
+
end
|
45
|
+
end
|
46
|
+
rescue AWS::S3::Errors::AccessDenied
|
47
|
+
puts "Access Denied: #{obj.key}"
|
51
48
|
end
|
52
49
|
end
|
53
50
|
puts 'Completed S3Twin run'
|
@@ -65,6 +62,8 @@ class Twin
|
|
65
62
|
else
|
66
63
|
return 'private'
|
67
64
|
end
|
65
|
+
else
|
66
|
+
return nil
|
68
67
|
end
|
69
68
|
end
|
70
69
|
end
|
data/s3twin.gemspec
CHANGED
@@ -18,9 +18,9 @@ Gem::Specification.new do |spec|
|
|
18
18
|
spec.require_paths = ['lib']
|
19
19
|
|
20
20
|
spec.add_dependency 'thor', '~> 0.18'
|
21
|
-
spec.add_dependency 'dotenv', '~> 0.
|
22
|
-
spec.add_dependency 'aws-sdk', '~> 1.
|
23
|
-
spec.add_dependency 'iron_worker_ng', '~> 0
|
21
|
+
spec.add_dependency 'dotenv', '~> 0.9'
|
22
|
+
spec.add_dependency 'aws-sdk', '~> 1.19'
|
23
|
+
spec.add_dependency 'iron_worker_ng', '~> 1.0'
|
24
24
|
|
25
25
|
spec.add_development_dependency 'bundler', '~> 1.3'
|
26
26
|
spec.add_development_dependency 'rake'
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: s3twin
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Robert Coleman
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2013-
|
11
|
+
date: 2013-09-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: thor
|
@@ -30,42 +30,42 @@ dependencies:
|
|
30
30
|
requirements:
|
31
31
|
- - ~>
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '0.
|
33
|
+
version: '0.9'
|
34
34
|
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - ~>
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '0.
|
40
|
+
version: '0.9'
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: aws-sdk
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
45
|
- - ~>
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '1.
|
47
|
+
version: '1.19'
|
48
48
|
type: :runtime
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
52
|
- - ~>
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '1.
|
54
|
+
version: '1.19'
|
55
55
|
- !ruby/object:Gem::Dependency
|
56
56
|
name: iron_worker_ng
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
58
58
|
requirements:
|
59
59
|
- - ~>
|
60
60
|
- !ruby/object:Gem::Version
|
61
|
-
version: '0
|
61
|
+
version: '1.0'
|
62
62
|
type: :runtime
|
63
63
|
prerelease: false
|
64
64
|
version_requirements: !ruby/object:Gem::Requirement
|
65
65
|
requirements:
|
66
66
|
- - ~>
|
67
67
|
- !ruby/object:Gem::Version
|
68
|
-
version: '0
|
68
|
+
version: '1.0'
|
69
69
|
- !ruby/object:Gem::Dependency
|
70
70
|
name: bundler
|
71
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -131,7 +131,6 @@ executables:
|
|
131
131
|
extensions: []
|
132
132
|
extra_rdoc_files: []
|
133
133
|
files:
|
134
|
-
- .env.sample
|
135
134
|
- .gitignore
|
136
135
|
- Gemfile
|
137
136
|
- LICENSE
|