s3cp 1.1.0.pre.1 → 1.1.0.pre.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/History.txt +55 -38
- data/lib/s3cp/s3cp.rb +5 -0
- data/lib/s3cp/utils.rb +1 -7
- data/lib/s3cp/version.rb +1 -1
- metadata +4 -4
data/History.txt
CHANGED
@@ -1,19 +1,36 @@
|
|
1
|
-
=== 1.1.0 / (Pending)
|
1
|
+
=== 1.1.0.pre.2 / (Pending)
|
2
|
+
|
3
|
+
* Added: 's3cp --version' now reports version information.
|
4
|
+
|
5
|
+
* Changed: S3CP uses the default credential provider from aws-sdk which makes
|
6
|
+
a best effort to locate your AWS credentials. It checks a variety of
|
7
|
+
locations in the following order:
|
8
|
+
|
9
|
+
* Static credentials from AWS.config (e.g. AWS.config.access_key_id,
|
10
|
+
AWS.config.secret_access_key)
|
11
|
+
|
12
|
+
* The environment (e.g. ENV['AWS_ACCESS_KEY_ID'] or
|
13
|
+
ENV['AMAZON_ACCESS_KEY_ID'])
|
14
|
+
|
15
|
+
* EC2 metadata service (checks for credentials provided by
|
16
|
+
roles for instances).
|
17
|
+
|
18
|
+
=== 1.1.0.pre.1 / (2012-07-23)
|
2
19
|
|
3
20
|
* Changed: Underlying AWS library has been changed from 'right-aws' to the official
|
4
|
-
|
5
|
-
|
6
|
-
|
21
|
+
AWS SDK ('aws-sdk'). The main driver for this is to stay current on the
|
22
|
+
latest features, including instance-specific roles, temporary credentials,
|
23
|
+
etc.
|
7
24
|
|
8
25
|
* Added: New 's3buckets' command displays all S3 buckets associated with the
|
9
|
-
|
26
|
+
account.
|
10
27
|
|
11
28
|
* Added: New '--acl PERMISSION' option to 's3cp' and 's3up' commands.
|
12
29
|
|
13
30
|
=== 1.0.6 / (2012-07-23)
|
14
31
|
|
15
32
|
* Fixed: s3cp would exit with errorlevel 0 despite errors when running in
|
16
|
-
|
33
|
+
interactive mode.
|
17
34
|
|
18
35
|
=== 1.0.5 / (2012-06-25)
|
19
36
|
|
@@ -27,15 +44,15 @@
|
|
27
44
|
|
28
45
|
* Added: Exponential backoff for `s3cp` command based on the formula:
|
29
46
|
|
30
|
-
|
47
|
+
delay = initial_retry_delay * (factor ^ retries)
|
31
48
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
49
|
+
where:
|
50
|
+
- `initial_retry_delay` is 1 second by default.
|
51
|
+
- `factor` is 1.4142 by default
|
52
|
+
(retry delay doubles every two retries)
|
53
|
+
- max. number of retries is now 20 by default.
|
37
54
|
|
38
|
-
|
55
|
+
(which means s3cp retries for roughly 58 minutes by default)
|
39
56
|
|
40
57
|
* Fixed: `s3ls` and `s3du` now properly handle Errno::EPIPE
|
41
58
|
|
@@ -45,18 +62,18 @@
|
|
45
62
|
|
46
63
|
* Added: `s3up` now outputs uploaded file size to STDERR, e.g.,
|
47
64
|
|
48
|
-
|
49
|
-
|
65
|
+
% ls | s3up s3://bucket/path/to/file
|
66
|
+
s3://bucket/path/to/file => 214B
|
50
67
|
|
51
68
|
=== 1.0.1 / 2012-03-9
|
52
69
|
|
53
70
|
* Added: New command `s3up` to upload STDIN to S3
|
54
|
-
|
71
|
+
e.g. some_cmd | s3up s3://bucket/path/to/destination
|
55
72
|
|
56
73
|
Note: `s3up` does not yet directly stream data to S3 since right_aws gem
|
57
|
-
|
58
|
-
|
59
|
-
|
74
|
+
requires the upload size to be known in advance. `s3up` currently
|
75
|
+
persists STDIN into a temp file although this may change in the
|
76
|
+
future.
|
60
77
|
|
61
78
|
=== 1.0.0 / 2012-03-9
|
62
79
|
|
@@ -70,27 +87,27 @@ bugs -- but at least all known bugs have been addressed.
|
|
70
87
|
Probably worth mentioning: there are two outstanding experimental features,
|
71
88
|
|
72
89
|
1) s3cp --sync: Does not handle missing files. It's basically
|
73
|
-
|
90
|
+
a faster copy; not a true "sync" at this point.
|
74
91
|
|
75
92
|
2) Bash command-line completion is still very rough.
|
76
93
|
|
77
94
|
=== 0.2.7 / 2012-03-09
|
78
95
|
|
79
96
|
* Fixed #3: s3cp adds extra slash when copying to the root of a bucket.
|
80
|
-
|
97
|
+
e.g. s3://myBucket//someFile or s3://myBucket//someFolderToCopy/someFile
|
81
98
|
|
82
99
|
=== 0.2.6 / 2012-03-06
|
83
100
|
|
84
101
|
* Fixed: Possible division-by-zero error in s3du if it encounters
|
85
|
-
|
102
|
+
zero-length files.
|
86
103
|
|
87
104
|
=== 0.2.5 / 2012-03-02
|
88
105
|
|
89
106
|
* Added: "s3du" command to calculate disk usage.
|
90
|
-
|
107
|
+
Supports --depth, --regex, --unit parameters and more!
|
91
108
|
|
92
109
|
* Changed: "s3ls -l" command now accepts --unit and --precision to configure
|
93
|
-
|
110
|
+
file size display. Uses "smart" unit by default.
|
94
111
|
|
95
112
|
* Changed: "s3ls -l" will now use S3CP_DATE_FORMAT environment if set.
|
96
113
|
|
@@ -132,12 +149,12 @@ Probably worth mentioning: there are two outstanding experimental features,
|
|
132
149
|
* Added: Progress bars during upload/download if $stdout.isatty
|
133
150
|
|
134
151
|
* Fixed: s3cat now handles broken pipes properly
|
135
|
-
|
152
|
+
e.g. "s3cat bucket:some/file | head" will now terminate early.
|
136
153
|
|
137
154
|
=== 0.2.1 / (2012-02-20)
|
138
155
|
|
139
156
|
* Added: Bash completion now supports exclusions through S3CP_EXCLUDES
|
140
|
-
|
157
|
+
and defaults to excluding keys containing "_$folder$".
|
141
158
|
|
142
159
|
* Changed: s3dir and s3ls --delimiter now display both directories and files
|
143
160
|
|
@@ -146,15 +163,15 @@ Probably worth mentioning: there are two outstanding experimental features,
|
|
146
163
|
* Added: s3stat command to display S3 object properties
|
147
164
|
|
148
165
|
* Added: s3dir as a shortcut for "s3ls --delimiter / ..."
|
149
|
-
|
166
|
+
(default to character in S3CP_DELIMITER environment variable or "/" if not defined)
|
150
167
|
|
151
168
|
* Added: s3cp defaults can now be set using environment variables
|
152
|
-
|
169
|
+
S3CP_OVERWRITE, S3CP_CHECKSUM, S3CP_RETRIES, S3CP_RETRY_DELAY
|
153
170
|
|
154
171
|
* Added: Support for Bash command-line completion of S3 URLs (see below).
|
155
172
|
|
156
173
|
* Fixed: Skip checksum verification for S3 objects with invalid MD5's
|
157
|
-
|
174
|
+
(details @ https://forums.aws.amazon.com/message.jspa?messageID=234538)
|
158
175
|
|
159
176
|
To install Bash completion for S3 URLs, add the following to ~/.bashrc:
|
160
177
|
|
@@ -165,10 +182,10 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
|
|
165
182
|
=== 0.1.15 / (2012-02-17)
|
166
183
|
|
167
184
|
* Added: s3cp now automatically checks MD5 checksums during download/upload
|
168
|
-
|
169
|
-
|
170
|
-
|
171
|
-
|
185
|
+
and retries up to 5 times by default if the checksum fails.
|
186
|
+
The number of attempts may be configured using --max-attempts,
|
187
|
+
the retry delay may be changed with --retry-delay and the check
|
188
|
+
may be disabled completely using --no-checksum.
|
172
189
|
|
173
190
|
=== 0.1.14 / (2012-02-09)
|
174
191
|
|
@@ -190,7 +207,7 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
|
|
190
207
|
|
191
208
|
* Fixed: --max-keys now works correctly with --delimiter
|
192
209
|
* Fixed: do not display any keys if there are no common-prefix
|
193
|
-
|
210
|
+
delimiter-matching keys
|
194
211
|
|
195
212
|
=== 0.1.9 / (2012-01-23)
|
196
213
|
|
@@ -199,18 +216,18 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
|
|
199
216
|
=== 0.1.8 / (2011-12-29)
|
200
217
|
|
201
218
|
* Fixed: Apparently, File.new(path, File::CREAT|File::WRONLY) does not
|
202
|
-
|
219
|
+
truncate existing files; use File.new(path, "wb") instead.
|
203
220
|
|
204
221
|
=== 0.1.7 / (2011-12-29)
|
205
222
|
|
206
223
|
* Fixed: s3cp would not truncate existing files when overwriting, possibly
|
207
|
-
|
224
|
+
resulting in corrupted files.
|
208
225
|
|
209
226
|
=== 0.1.6 / (2011-12-16)
|
210
227
|
|
211
228
|
* Changed: s3rm now uses multi-object delete operation for faster deletes
|
212
229
|
* Changed: dependency on 'aboisvert-aws' instead of 'right-aws' until
|
213
|
-
|
230
|
+
it supports multi-object delete.
|
214
231
|
|
215
232
|
=== 0.1.5 / 2011-10-17
|
216
233
|
|
@@ -223,7 +240,7 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
|
|
223
240
|
=== 0.1.3 / 2011-09-29
|
224
241
|
|
225
242
|
* Fixed: s3cp --headers names are now converted to lowercase since underlying
|
226
|
-
|
243
|
+
RightAWS gem expects lowercase header names.
|
227
244
|
|
228
245
|
=== 0.1.2 / 2011-09-29
|
229
246
|
|
data/lib/s3cp/s3cp.rb
CHANGED
@@ -119,6 +119,11 @@ op = OptionParser.new do |opts|
|
|
119
119
|
options[:debug] = true
|
120
120
|
end
|
121
121
|
|
122
|
+
opts.on("--version", "Display version information") do
|
123
|
+
puts S3CP::VERSION
|
124
|
+
exit
|
125
|
+
end
|
126
|
+
|
122
127
|
opts.on_tail("-h", "--help", "Show this message") do
|
123
128
|
puts op
|
124
129
|
exit
|
data/lib/s3cp/utils.rb
CHANGED
@@ -39,13 +39,7 @@ module S3CP
|
|
39
39
|
|
40
40
|
# Connect to AWS S3
|
41
41
|
def connect()
|
42
|
-
|
43
|
-
secret_key = ENV["AWS_SECRET_ACCESS_KEY"] || raise("Missing environment variable AWS_SECRET_ACCESS_KEY")
|
44
|
-
|
45
|
-
::AWS::S3.new(
|
46
|
-
:access_key_id => access_key,
|
47
|
-
:secret_access_key => secret_key
|
48
|
-
)
|
42
|
+
::AWS::S3.new()
|
49
43
|
end
|
50
44
|
|
51
45
|
# Parse URL and return bucket and key.
|
data/lib/s3cp/version.rb
CHANGED
metadata
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: s3cp
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash: -
|
4
|
+
hash: -3585171968
|
5
5
|
prerelease: 6
|
6
6
|
segments:
|
7
7
|
- 1
|
8
8
|
- 1
|
9
9
|
- 0
|
10
10
|
- pre
|
11
|
-
-
|
12
|
-
version: 1.1.0.pre.
|
11
|
+
- 2
|
12
|
+
version: 1.1.0.pre.2
|
13
13
|
platform: ruby
|
14
14
|
authors:
|
15
15
|
- Alex Boisvert
|
@@ -17,7 +17,7 @@ autorequire:
|
|
17
17
|
bindir: bin
|
18
18
|
cert_chain: []
|
19
19
|
|
20
|
-
date: 2012-
|
20
|
+
date: 2012-08-09 00:00:00 Z
|
21
21
|
dependencies:
|
22
22
|
- !ruby/object:Gem::Dependency
|
23
23
|
prerelease: false
|