filestack 2.6.3 → 2.7.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/CHANGELOG.md +15 -0
- data/README.md +28 -12
- data/VERSION +1 -1
- data/docs/TransformConfig.html +2 -1
- data/filestack-ruby.gemspec +1 -1
- data/lib/filestack/config.rb +21 -7
- data/lib/filestack/models/filestack_client.rb +8 -15
- data/lib/filestack/ruby/version.rb +1 -1
- data/lib/filestack/utils/multipart_upload_utils.rb +43 -75
- data/lib/filestack/utils/utils.rb +51 -39
- metadata +12 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 36cd3251f64e64ffcdb9d42c73cbf6d49bcda8a2533cb8053e0087229a077bf2
|
4
|
+
data.tar.gz: f2e38e698557c0b336c332be5b68775fe2624e046a56e1c521717554f23b6495
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 26a1b21b1ad69ae42708e94323e28d7656881824e668677b0dc27a4bb13a035ed6f5b104f90615a0e53bf3b454621fa3a3e129a6a412eed15dd7271f33100f84
|
7
|
+
data.tar.gz: 4d2c6c3e96da48df28e067f6990e3313669ffb2cf0f33f62964db5f708b42b0818c9992cec713c0b70acc6db72425bb727545726a1757c9c64c9dc7addcf6090
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,20 @@
|
|
1
1
|
# Filestack-Ruby Changelog
|
2
2
|
|
3
|
+
## 2.7.0 (September 28, 2020)
|
4
|
+
- Add workflows
|
5
|
+
|
6
|
+
## 2.6.7 (August 4, 2020)
|
7
|
+
- Add content disposition task
|
8
|
+
|
9
|
+
## 2.6.6 (June 19, 2020)
|
10
|
+
- Add compress task to transformations
|
11
|
+
|
12
|
+
## 2.6.5 (January 9, 2020)
|
13
|
+
- Updated version of the parallel gem
|
14
|
+
|
15
|
+
## 2.6.4 (November 4, 2019)
|
16
|
+
- Fix issue with file upload
|
17
|
+
|
3
18
|
## 2.6.3 (August 12, 2019)
|
4
19
|
- Add animate to transformations
|
5
20
|
|
data/README.md
CHANGED
@@ -44,41 +44,50 @@ Or install it yourself as:
|
|
44
44
|
require 'filestack'
|
45
45
|
```
|
46
46
|
Intialize the client using your API key, and security if you are using it.
|
47
|
+
|
47
48
|
```ruby
|
48
49
|
client = FilestackClient.new('YOUR_API_KEY', security: security_object)
|
49
50
|
```
|
50
51
|
### Uploading
|
51
|
-
Filestack uses multipart uploading by default, which is faster for larger files. This can be turned off by passing in ```multipart: false```. Multipart is disabled when uploading external URLs.
|
52
52
|
```ruby
|
53
|
-
filelink = client.upload(filepath: '/path/to/
|
54
|
-
|
55
|
-
filelink = client.upload(filepath: '/path/to/file', multipart: false)
|
53
|
+
filelink = client.upload(filepath: '/path/to/localfile')
|
56
54
|
|
57
55
|
# OR
|
58
56
|
|
59
|
-
filelink = client.upload(external_url: 'http://
|
57
|
+
filelink = client.upload(external_url: 'http://domain.com/image.png')
|
60
58
|
```
|
61
59
|
|
62
60
|
To upload a local and an external file with query parameters:
|
63
61
|
```ruby
|
64
|
-
filelink = client.upload(filepath: '/path/to/
|
62
|
+
filelink = client.upload(filepath: '/path/to/localfile', options: { mimetype: 'image/png' })
|
65
63
|
|
66
|
-
filelink = client.upload(external_url: 'http://
|
64
|
+
filelink = client.upload(external_url: 'http://domain.com/image.png', options: { mimetype: 'image/jpeg' })
|
67
65
|
```
|
68
66
|
|
69
67
|
To store file on `dropbox`, `azure`, `gcs` or `rackspace`, you must have the chosen provider configured in the developer portal to enable this feature. By default the file is stored on `s3`. You can add more details of the storage in `options`.
|
70
68
|
|
71
69
|
```ruby
|
72
|
-
filelink = client.upload(filepath: '/path/to/file', storage: '
|
70
|
+
filelink = client.upload(filepath: '/path/to/file', storage: 's3', options: { path: 'folder_name/', container: 'container_name', location: 's3', region: 'region_name' })
|
71
|
+
|
72
|
+
filelink = client.upload(external_url: 'http://someurl.com/image.png', options: { location: 'dropbox', path: 'folder_name' })
|
73
|
+
```
|
74
|
+
|
75
|
+
### Workflows
|
76
|
+
Workflows allow you to wire up conditional logic and image processing to enforce business processes, automate ingest, and save valuable development time. In order to trigger the workflow job for each upload:
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
filelink = client.upload(filepath: '/path/to/file', options: { workflows: ["workflow_id_1", "workflow_id_2"] })
|
80
|
+
|
81
|
+
#OR
|
73
82
|
|
74
|
-
filelink = client.upload(external_url: 'http://someurl.com/image.png',
|
83
|
+
filelink = client.upload(external_url: 'http://someurl.com/image.png', options: { workflows: ["workflow_id_1"] })
|
75
84
|
```
|
76
85
|
|
77
86
|
### Security
|
78
87
|
If security is enabled on your account, or if you are using certain actions that require security (delete, overwrite and certain transformations), you will need to create a security object and pass it into the client on instantiation.
|
79
88
|
|
80
89
|
```ruby
|
81
|
-
security = FilestackSecurity.new('YOUR_APP_SECRET', options: {call: %w[read store pick]})
|
90
|
+
security = FilestackSecurity.new('YOUR_APP_SECRET', options: {call: %w[read store pick runWorkflow]})
|
82
91
|
client = FilestackClient.new('YOUR_API_KEY', security: security)
|
83
92
|
```
|
84
93
|
|
@@ -123,13 +132,20 @@ Return `default` file if the source of the transformation does not work or the t
|
|
123
132
|
To use fallback, you should provide `handle` of the file that should be returned. Optionally, you can add `cache`, which means number of seconds fallback response should be cached in CDN.
|
124
133
|
|
125
134
|
```ruby
|
126
|
-
transform = client.transform_external('https://someurl.com').fallback(
|
135
|
+
transform = client.transform_external('https://someurl.com/file.png').fallback(file: 'DEFAULT_HANDLE_OR_FILEPATH')
|
127
136
|
```
|
128
137
|
|
129
138
|
If you are using fallback handle that belongs to different application than the one which runs transformation (APIKEY) and it is secured with security policy, appropriate signature and policy with read call should be used:
|
130
139
|
|
131
140
|
```ruby
|
132
|
-
transform = client.transform_external('https://someurl.com').fallback(
|
141
|
+
transform = client.transform_external('https://someurl.com/file.png').fallback(file: 'DEFAULT_HANDLE_OR_FILEPATH?policy=HANDLE_APIKEY_POLICY&signature=HANDLE_APIKEY_SIGNATURE', cache: 10)
|
142
|
+
```
|
143
|
+
|
144
|
+
### Content
|
145
|
+
Sets `Content-Disposition` header for given file.
|
146
|
+
|
147
|
+
```ruby
|
148
|
+
transform = filelink.transform.content(filename: 'DEFAULT_FILENAME', type: 'TYPE')
|
133
149
|
```
|
134
150
|
|
135
151
|
### Tagging
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
2.
|
1
|
+
2.7.0
|
data/docs/TransformConfig.html
CHANGED
@@ -114,7 +114,8 @@
|
|
114
114
|
</span><span class='tstring_content'> partial_pixelate</span><span class='words_sep'> </span><span class='tstring_content'>partial_blur</span><span class='words_sep'> </span><span class='tstring_content'>collage</span><span class='words_sep'> </span><span class='tstring_content'>upscale</span><span class='words_sep'> </span><span class='tstring_content'>enhance</span><span class='words_sep'>
|
115
115
|
</span><span class='tstring_content'> redeye</span><span class='words_sep'> </span><span class='tstring_content'>ascii</span><span class='words_sep'> </span><span class='tstring_content'>filetype_conversion</span><span class='words_sep'> </span><span class='tstring_content'>quality</span><span class='words_sep'> </span><span class='tstring_content'>urlscreenshot</span><span class='words_sep'>
|
116
116
|
</span><span class='tstring_content'> no_metadata</span><span class='words_sep'></span><span class='tstring_content'> fallback</span><span class='words_sep'></span><span class='tstring_content'> pdfinfo</span><span class='words_sep'></span><span class='tstring_content'> pdfconvert</span><span class='words_sep'></span><span class='tstring_content'> cache</span><span class='words_sep'></span><span class='tstring_content'> auto_image</span><span class='words_sep'>
|
117
|
-
</span><span class='tstring_content'> minify_js</span><span class='words_sep'></span><span class='tstring_content'> minify_css</span><span class='words_sep'></span><span class='tstring_content'> animate</span><span class='words_sep'></span>
|
117
|
+
</span><span class='tstring_content'> minify_js</span><span class='words_sep'></span><span class='tstring_content'> minify_css</span><span class='words_sep'></span><span class='tstring_content'> animate</span><span class='words_sep'></span><span class='tstring_content'> video_convert</span><span class='words_sep'></span><span class='tstring_content'> video_playlist</span><span class='words_sep'>
|
118
|
+
</span><span class='tstring_content'> compress</span><span class='words_sep'></span><span class='tstring_content'> content</span><span class='words_sep'></span>
|
118
119
|
<span class='tstring_end'>]</span></span><span class='period'>.</span><span class='id identifier rubyid_freeze'>freeze</span></pre></dd>
|
119
120
|
|
120
121
|
</dl>
|
data/filestack-ruby.gemspec
CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
|
|
23
23
|
spec.require_paths = ["lib"]
|
24
24
|
|
25
25
|
spec.add_dependency "typhoeus", "~> 1.1"
|
26
|
-
spec.add_dependency "parallel", "~> 1.11.2"
|
26
|
+
spec.add_dependency "parallel", "~> 1.11", ">= 1.11.2"
|
27
27
|
spec.add_dependency "mimemagic", "~> 0.3.2"
|
28
28
|
spec.add_dependency "progress_bar"
|
29
29
|
|
data/lib/filestack/config.rb
CHANGED
@@ -7,11 +7,6 @@ class FilestackConfig
|
|
7
7
|
CDN_URL = 'https://cdn.filestackcontent.com'.freeze
|
8
8
|
PROCESS_URL = 'https://process.filestackapi.com'.freeze
|
9
9
|
|
10
|
-
MULTIPART_START_URL = 'https://upload.filestackapi.com/multipart/start'.freeze
|
11
|
-
MULTIPART_UPLOAD_URL = 'https://upload.filestackapi.com/multipart/upload'.freeze
|
12
|
-
MULTIPART_COMMIT_URL = 'https://upload.filestackapi.com/multipart/commit'.freeze
|
13
|
-
MULTIPART_COMPLETE_URL = 'https://upload.filestackapi.com/multipart/complete'.freeze
|
14
|
-
|
15
10
|
MULTIPART_PARAMS = %w[
|
16
11
|
store_location store_region store_container
|
17
12
|
store_path store_access
|
@@ -22,10 +17,28 @@ class FilestackConfig
|
|
22
17
|
VERSION = Filestack::Ruby::VERSION
|
23
18
|
HEADERS = {
|
24
19
|
'User-Agent' => "filestack-ruby #{VERSION}",
|
25
|
-
'Filestack-Source' => "Ruby-#{VERSION}"
|
20
|
+
'Filestack-Source' => "Ruby-#{VERSION}",
|
21
|
+
'Content-Type' => "application/json",
|
22
|
+
'Accept-Encoding' => "application/json"
|
26
23
|
}.freeze
|
27
24
|
|
28
25
|
INTELLIGENT_ERROR_MESSAGES = ['BACKEND_SERVER', 'BACKEND_NETWORK', 'S3_SERVER', 'S3_NETWORK']
|
26
|
+
|
27
|
+
def self.multipart_start_url
|
28
|
+
"https://upload.filestackapi.com/multipart/start"
|
29
|
+
end
|
30
|
+
|
31
|
+
def self.multipart_upload_url(base_url)
|
32
|
+
"https://#{base_url}/multipart/upload"
|
33
|
+
end
|
34
|
+
|
35
|
+
def self.multipart_commit_url(base_url)
|
36
|
+
"https://#{base_url}/multipart/commit"
|
37
|
+
end
|
38
|
+
|
39
|
+
def self.multipart_complete_url(base_url)
|
40
|
+
"https://#{base_url}/multipart/complete"
|
41
|
+
end
|
29
42
|
end
|
30
43
|
|
31
44
|
class TransformConfig
|
@@ -37,6 +50,7 @@ class TransformConfig
|
|
37
50
|
partial_pixelate partial_blur collage upscale enhance
|
38
51
|
redeye ascii filetype_conversion quality urlscreenshot
|
39
52
|
no_metadata fallback pdfinfo pdfconvert cache auto_image
|
40
|
-
minify_js minify_css animate
|
53
|
+
minify_js minify_css animate video_convert video_playlist
|
54
|
+
compress content
|
41
55
|
].freeze
|
42
56
|
end
|
@@ -26,26 +26,19 @@ class FilestackClient
|
|
26
26
|
# Upload a local file or external url
|
27
27
|
# @param [String] filepath The path of a local file
|
28
28
|
# @param [String] external_url An external URL
|
29
|
-
# @param [Bool] multipart Switch for miltipart
|
30
|
-
# (Default: true)
|
31
29
|
# @param [Hash] options User-supplied upload options
|
32
30
|
#
|
33
31
|
# return [Filestack::FilestackFilelink]
|
34
|
-
def upload(filepath: nil, external_url: nil,
|
35
|
-
if filepath && external_url
|
36
|
-
|
37
|
-
|
38
|
-
response = if filepath && multipart
|
32
|
+
def upload(filepath: nil, external_url: nil, options: {}, intelligent: false, timeout: 60, storage: 'S3')
|
33
|
+
return 'You cannot upload a URL and file at the same time' if filepath && external_url
|
34
|
+
|
35
|
+
response = if filepath
|
39
36
|
multipart_upload(@apikey, filepath, @security, options, timeout, storage, intelligent: intelligent)
|
40
37
|
else
|
41
|
-
send_upload(
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
options: options,
|
46
|
-
security: @security,
|
47
|
-
storage: storage
|
48
|
-
)
|
38
|
+
send_upload(@apikey,
|
39
|
+
external_url: external_url,
|
40
|
+
options: options,
|
41
|
+
security: @security)
|
49
42
|
end
|
50
43
|
FilestackFilelink.new(response['handle'], security: @security, apikey: @apikey)
|
51
44
|
end
|
@@ -23,16 +23,6 @@ module MultipartUploadUtils
|
|
23
23
|
[filename, filesize, mimetype.to_s]
|
24
24
|
end
|
25
25
|
|
26
|
-
def multipart_options(options)
|
27
|
-
[:region, :container, :path, :access].each do |key|
|
28
|
-
if options.has_key?(key)
|
29
|
-
options[:"store_#{key}"] = options[key]
|
30
|
-
options.delete(key)
|
31
|
-
end
|
32
|
-
end
|
33
|
-
return options
|
34
|
-
end
|
35
|
-
|
36
26
|
# Send start response to multipart endpoint
|
37
27
|
#
|
38
28
|
# @param [String] apikey Filestack API key
|
@@ -45,28 +35,27 @@ module MultipartUploadUtils
|
|
45
35
|
# multipart uploads
|
46
36
|
#
|
47
37
|
# @return [Typhoeus::Response]
|
48
|
-
def multipart_start(apikey, filename, filesize, mimetype, security, storage, options = {})
|
38
|
+
def multipart_start(apikey, filename, filesize, mimetype, security, storage, options = {}, intelligent)
|
49
39
|
params = {
|
50
40
|
apikey: apikey,
|
51
41
|
filename: filename,
|
52
42
|
mimetype: mimetype,
|
53
43
|
size: filesize,
|
54
|
-
|
55
|
-
|
56
|
-
'multipart' => 'true'
|
44
|
+
store: { location: storage },
|
45
|
+
fii: intelligent
|
57
46
|
}
|
58
|
-
|
59
|
-
params
|
47
|
+
|
48
|
+
params[:store].merge!(options) if options
|
60
49
|
|
61
50
|
unless security.nil?
|
62
51
|
params[:policy] = security.policy
|
63
52
|
params[:signature] = security.signature
|
64
53
|
end
|
65
54
|
|
66
|
-
response = Typhoeus.post(
|
67
|
-
|
68
|
-
|
69
|
-
|
55
|
+
response = Typhoeus.post(FilestackConfig.multipart_start_url,
|
56
|
+
body: params.to_json,
|
57
|
+
headers: FilestackConfig::HEADERS)
|
58
|
+
|
70
59
|
if response.code == 200
|
71
60
|
JSON.parse(response.body)
|
72
61
|
else
|
@@ -92,7 +81,7 @@ module MultipartUploadUtils
|
|
92
81
|
seek_point = 0
|
93
82
|
while seek_point < filesize
|
94
83
|
part_info = {
|
95
|
-
|
84
|
+
seek_point: seek_point,
|
96
85
|
filepath: filepath,
|
97
86
|
filename: filename,
|
98
87
|
apikey: apikey,
|
@@ -103,10 +92,10 @@ module MultipartUploadUtils
|
|
103
92
|
upload_id: start_response['upload_id'],
|
104
93
|
location_url: start_response['location_url'],
|
105
94
|
start_response: start_response,
|
106
|
-
|
95
|
+
store: { location: storage }
|
107
96
|
}
|
108
|
-
|
109
|
-
part_info
|
97
|
+
|
98
|
+
part_info[:store].merge!(options) if options
|
110
99
|
|
111
100
|
if seek_point + FilestackConfig::DEFAULT_CHUNK_SIZE > filesize
|
112
101
|
size = filesize - (seek_point)
|
@@ -134,9 +123,9 @@ module MultipartUploadUtils
|
|
134
123
|
# multipart uploads
|
135
124
|
#
|
136
125
|
# @return [Typhoeus::Response]
|
137
|
-
def upload_chunk(job, apikey, filepath, options)
|
126
|
+
def upload_chunk(job, apikey, filepath, options, storage)
|
138
127
|
file = File.open(filepath)
|
139
|
-
file.seek(job[:
|
128
|
+
file.seek(job[:seek_point])
|
140
129
|
chunk = file.read(FilestackConfig::DEFAULT_CHUNK_SIZE)
|
141
130
|
|
142
131
|
md5 = Digest::MD5.new
|
@@ -149,14 +138,14 @@ module MultipartUploadUtils
|
|
149
138
|
uri: job[:uri],
|
150
139
|
region: job[:region],
|
151
140
|
upload_id: job[:upload_id],
|
152
|
-
|
141
|
+
store: { location: storage },
|
153
142
|
file: Tempfile.new(job[:filename])
|
154
143
|
}
|
155
144
|
data = data.merge!(options) if options
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
145
|
+
|
146
|
+
fs_response = Typhoeus.post(FilestackConfig.multipart_upload_url(job[:location_url]),
|
147
|
+
body: data.to_json,
|
148
|
+
headers: FilestackConfig::HEADERS).body
|
160
149
|
fs_response = JSON.parse(fs_response)
|
161
150
|
Typhoeus.put(
|
162
151
|
fs_response['url'], headers: fs_response['headers'], body: chunk
|
@@ -171,17 +160,17 @@ module MultipartUploadUtils
|
|
171
160
|
# multipart uploads
|
172
161
|
#
|
173
162
|
# @return [Array] Array of parts/etags strings
|
174
|
-
def run_uploads(jobs, apikey, filepath, options)
|
163
|
+
def run_uploads(jobs, apikey, filepath, options, storage)
|
175
164
|
bar = ProgressBar.new(jobs.length)
|
176
165
|
results = Parallel.map(jobs, in_threads: 4) do |job|
|
177
166
|
response = upload_chunk(
|
178
|
-
job, apikey, filepath, options
|
167
|
+
job, apikey, filepath, options, storage
|
179
168
|
)
|
180
169
|
if response.code == 200
|
181
170
|
bar.increment!
|
182
171
|
part = job[:part]
|
183
172
|
etag = response.headers[:etag]
|
184
|
-
|
173
|
+
{ part_number: part, etag: etag }
|
185
174
|
end
|
186
175
|
end
|
187
176
|
results
|
@@ -204,40 +193,22 @@ module MultipartUploadUtils
|
|
204
193
|
#
|
205
194
|
# @return [Typhoeus::Response]
|
206
195
|
def multipart_complete(apikey, filename, filesize, mimetype, start_response, parts_and_etags, options, storage, intelligent = false)
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
}
|
220
|
-
else
|
221
|
-
data = {
|
222
|
-
apikey: apikey,
|
223
|
-
uri: start_response['uri'],
|
224
|
-
region: start_response['region'],
|
225
|
-
upload_id: start_response['upload_id'],
|
226
|
-
filename: filename,
|
227
|
-
size: filesize,
|
228
|
-
mimetype: mimetype,
|
229
|
-
store_location: storage,
|
230
|
-
file: Tempfile.new(filename),
|
231
|
-
'multipart' => 'true'
|
232
|
-
}
|
233
|
-
end
|
234
|
-
options = multipart_options(options)
|
235
|
-
data = data.merge!(options) if options
|
196
|
+
data = {
|
197
|
+
apikey: apikey,
|
198
|
+
uri: start_response['uri'],
|
199
|
+
region: start_response['region'],
|
200
|
+
upload_id: start_response['upload_id'],
|
201
|
+
filename: filename,
|
202
|
+
size: filesize,
|
203
|
+
mimetype: mimetype,
|
204
|
+
store: { location: storage },
|
205
|
+
}
|
206
|
+
data[:store].merge!(options) if options
|
207
|
+
data.merge!(intelligent ? { fii: intelligent } : { parts: parts_and_etags })
|
236
208
|
|
237
|
-
Typhoeus.post(
|
238
|
-
|
239
|
-
|
240
|
-
)
|
209
|
+
Typhoeus.post(FilestackConfig.multipart_complete_url(start_response['location_url']),
|
210
|
+
body: data.to_json,
|
211
|
+
headers: FilestackConfig::HEADERS)
|
241
212
|
end
|
242
213
|
|
243
214
|
# Run entire multipart process through with file and options
|
@@ -252,27 +223,24 @@ module MultipartUploadUtils
|
|
252
223
|
# @return [Hash]
|
253
224
|
def multipart_upload(apikey, filepath, security, options, timeout, storage, intelligent: false)
|
254
225
|
filename, filesize, mimetype = get_file_info(filepath)
|
226
|
+
|
255
227
|
start_response = multipart_start(
|
256
|
-
apikey, filename, filesize, mimetype, security, storage, options
|
228
|
+
apikey, filename, filesize, mimetype, security, storage, options, intelligent
|
257
229
|
)
|
258
230
|
|
259
|
-
unless start_response['upload_type'].nil?
|
260
|
-
intelligent_enabled = ((start_response['upload_type'].include? 'intelligent_ingestion')) && intelligent
|
261
|
-
end
|
262
|
-
|
263
231
|
jobs = create_upload_jobs(
|
264
232
|
apikey, filename, filepath, filesize, start_response, storage, options
|
265
233
|
)
|
266
234
|
|
267
|
-
if
|
235
|
+
if intelligent
|
268
236
|
state = IntelligentState.new
|
269
|
-
run_intelligent_upload_flow(jobs, state)
|
237
|
+
run_intelligent_upload_flow(jobs, state, storage)
|
270
238
|
response_complete = multipart_complete(
|
271
239
|
apikey, filename, filesize, mimetype,
|
272
240
|
start_response, nil, options, storage, intelligent
|
273
241
|
)
|
274
242
|
else
|
275
|
-
parts_and_etags = run_uploads(jobs, apikey, filepath, options)
|
243
|
+
parts_and_etags = run_uploads(jobs, apikey, filepath, options, storage)
|
276
244
|
response_complete = multipart_complete(
|
277
245
|
apikey, filename, filesize, mimetype,
|
278
246
|
start_response, parts_and_etags, options, storage
|
@@ -62,36 +62,43 @@ module UploadUtils
|
|
62
62
|
)
|
63
63
|
end
|
64
64
|
|
65
|
+
def build_store_task(options = {})
|
66
|
+
return 'store' if options.empty?
|
67
|
+
tasks = []
|
68
|
+
options.each do |key, value|
|
69
|
+
value = case key
|
70
|
+
when :workflows
|
71
|
+
[value.join('","')]
|
72
|
+
when :path
|
73
|
+
"\"#{value}\""
|
74
|
+
else
|
75
|
+
value
|
76
|
+
end
|
77
|
+
tasks.push("#{key}:#{value.to_s.downcase}")
|
78
|
+
end
|
79
|
+
"store=#{tasks.join(',')}"
|
80
|
+
end
|
81
|
+
|
65
82
|
# Uploads to v1 REST API (for external URLs or if multipart is turned off)
|
66
83
|
#
|
67
84
|
# @param [String] apikey Filestack API key
|
68
|
-
# @param [String] filepath Local path to file
|
69
85
|
# @param [String] external_url External URL to be uploaded
|
70
86
|
# @param [FilestackSecurity] security Security object with
|
71
87
|
# policy/signature
|
72
88
|
# @param [Hash] options User-defined options for
|
73
89
|
# multipart uploads
|
74
|
-
# @param [String] storage Storage destination
|
75
|
-
# (s3, rackspace, etc)
|
76
90
|
# @return [Hash]
|
77
|
-
def send_upload(apikey,
|
78
|
-
|
79
|
-
{ fileUpload: File.open(filepath) }
|
80
|
-
else
|
81
|
-
{ url: external_url }
|
82
|
-
end
|
83
|
-
|
84
|
-
# adds any user-defined upload options to request payload
|
85
|
-
data = data.merge!(options) unless options.nil?
|
86
|
-
base = "#{FilestackConfig::API_URL}/store/#{storage}?key=#{apikey}"
|
91
|
+
def send_upload(apikey, external_url: nil, security: nil, options: nil)
|
92
|
+
base = "#{FilestackConfig::CDN_URL}/#{apikey}/#{build_store_task(options)}"
|
87
93
|
|
88
94
|
if security
|
89
95
|
policy = security.policy
|
90
96
|
signature = security.signature
|
91
|
-
base = "#{base}
|
97
|
+
base = "#{base}/security=s:#{signature},p:#{policy}"
|
92
98
|
end
|
93
99
|
|
94
|
-
response =
|
100
|
+
response = Typhoeus.post("#{base}/#{external_url}", headers: FilestackConfig::HEADERS)
|
101
|
+
|
95
102
|
if response.code == 200
|
96
103
|
response_body = JSON.parse(response.body)
|
97
104
|
handle = response_body['url'].split('/').last
|
@@ -192,7 +199,7 @@ module IntelligentUtils
|
|
192
199
|
# @param [IntelligentState] state An IntelligentState object
|
193
200
|
#
|
194
201
|
# @return [Array]
|
195
|
-
def run_intelligent_upload_flow(jobs, state)
|
202
|
+
def run_intelligent_upload_flow(jobs, state, storage)
|
196
203
|
bar = ProgressBar.new(jobs.length)
|
197
204
|
generator = create_intelligent_generator(jobs)
|
198
205
|
working_offset = FilestackConfig::DEFAULT_OFFSET_SIZE
|
@@ -200,7 +207,7 @@ module IntelligentUtils
|
|
200
207
|
batch = get_generator_batch(generator)
|
201
208
|
# run parts
|
202
209
|
Parallel.map(batch, in_threads: 4) do |part|
|
203
|
-
state = run_intelligent_uploads(part, state)
|
210
|
+
state = run_intelligent_uploads(part, state, storage)
|
204
211
|
# condition: a chunk has failed but we have not reached the maximum retries
|
205
212
|
while bad_state(state)
|
206
213
|
# condition: timeout to S3, requiring offset size to be changed
|
@@ -212,7 +219,7 @@ module IntelligentUtils
|
|
212
219
|
sleep(state.backoff)
|
213
220
|
end
|
214
221
|
state.add_retry
|
215
|
-
state = run_intelligent_uploads(part, state)
|
222
|
+
state = run_intelligent_uploads(part, state, storage)
|
216
223
|
end
|
217
224
|
raise "Upload has failed. Please try again later." unless state.ok
|
218
225
|
bar.increment!
|
@@ -268,13 +275,14 @@ module IntelligentUtils
|
|
268
275
|
# multipart_start
|
269
276
|
#
|
270
277
|
# @return [Dict]
|
271
|
-
def chunk_job(job, state, apikey, filename, filepath, filesize, start_response)
|
278
|
+
def chunk_job(job, state, apikey, filename, filepath, filesize, start_response, storage)
|
272
279
|
offset = 0
|
273
|
-
seek_point = job[:
|
280
|
+
seek_point = job[:seek_point]
|
274
281
|
chunk_list = []
|
282
|
+
|
275
283
|
while (offset < FilestackConfig::DEFAULT_CHUNK_SIZE) && (seek_point + offset) < filesize
|
276
284
|
chunk_list.push(
|
277
|
-
|
285
|
+
seek_point: seek_point,
|
278
286
|
filepath: filepath,
|
279
287
|
filename: filename,
|
280
288
|
apikey: apikey,
|
@@ -284,7 +292,7 @@ module IntelligentUtils
|
|
284
292
|
region: start_response['region'],
|
285
293
|
upload_id: start_response['upload_id'],
|
286
294
|
location_url: start_response['location_url'],
|
287
|
-
|
295
|
+
store: { location: storage },
|
288
296
|
offset: offset
|
289
297
|
)
|
290
298
|
offset += state.offset
|
@@ -299,15 +307,15 @@ module IntelligentUtils
|
|
299
307
|
# @param [IntelligentState] state An IntelligentState object
|
300
308
|
#
|
301
309
|
# @return [IntelligentState]
|
302
|
-
def run_intelligent_uploads(part, state)
|
310
|
+
def run_intelligent_uploads(part, state, storage)
|
303
311
|
failed = false
|
304
312
|
chunks = chunk_job(
|
305
313
|
part, state, part[:apikey], part[:filename], part[:filepath],
|
306
|
-
part[:filesize], part[:start_response]
|
314
|
+
part[:filesize], part[:start_response], storage
|
307
315
|
)
|
308
316
|
Parallel.map(chunks, in_threads: 3) do |chunk|
|
309
317
|
begin
|
310
|
-
upload_chunk_intelligently(chunk, state, part[:apikey], part[:filepath], part[:options])
|
318
|
+
upload_chunk_intelligently(chunk, state, part[:apikey], part[:filepath], part[:options], storage)
|
311
319
|
rescue => e
|
312
320
|
state.error_type = e.message
|
313
321
|
failed = true
|
@@ -321,6 +329,7 @@ module IntelligentUtils
|
|
321
329
|
else
|
322
330
|
state.ok = true
|
323
331
|
end
|
332
|
+
|
324
333
|
commit_params = {
|
325
334
|
apikey: part[:apikey],
|
326
335
|
uri: part[:uri],
|
@@ -328,12 +337,14 @@ module IntelligentUtils
|
|
328
337
|
upload_id: part[:upload_id],
|
329
338
|
size: part[:filesize],
|
330
339
|
part: part[:part],
|
331
|
-
location_url: part[:location_url],
|
332
|
-
|
333
|
-
file: Tempfile.new(part[:filename])
|
340
|
+
location_url: part[:start_response]['location_url'],
|
341
|
+
store: { location: storage }
|
334
342
|
}
|
335
|
-
|
336
|
-
|
343
|
+
|
344
|
+
response = Typhoeus.post(FilestackConfig.multipart_commit_url(commit_params[:location_url]),
|
345
|
+
body: commit_params.to_json,
|
346
|
+
headers: FilestackConfig::HEADERS)
|
347
|
+
|
337
348
|
if response.code == 200
|
338
349
|
state.reset
|
339
350
|
else
|
@@ -353,9 +364,10 @@ module IntelligentUtils
|
|
353
364
|
# multipart uploads
|
354
365
|
#
|
355
366
|
# @return [Typhoeus::Response]
|
356
|
-
def upload_chunk_intelligently(job, state, apikey, filepath, options)
|
367
|
+
def upload_chunk_intelligently(job, state, apikey, filepath, options, storage)
|
357
368
|
file = File.open(filepath)
|
358
|
-
file.seek(job[:
|
369
|
+
file.seek(job[:seek_point] + job[:offset])
|
370
|
+
|
359
371
|
chunk = file.read(state.offset)
|
360
372
|
md5 = Digest::MD5.new
|
361
373
|
md5 << chunk
|
@@ -367,17 +379,17 @@ module IntelligentUtils
|
|
367
379
|
uri: job[:uri],
|
368
380
|
region: job[:region],
|
369
381
|
upload_id: job[:upload_id],
|
370
|
-
|
382
|
+
store: { location: storage },
|
371
383
|
offset: job[:offset],
|
372
|
-
|
373
|
-
'multipart' => 'true'
|
384
|
+
fii: true
|
374
385
|
}
|
375
386
|
|
376
387
|
data = data.merge!(options) if options
|
377
|
-
|
378
|
-
|
379
|
-
|
380
|
-
|
388
|
+
|
389
|
+
fs_response = Typhoeus.post(FilestackConfig.multipart_upload_url(job[:location_url]),
|
390
|
+
body: data.to_json,
|
391
|
+
headers: FilestackConfig::HEADERS)
|
392
|
+
|
381
393
|
# POST to multipart/upload
|
382
394
|
begin
|
383
395
|
unless fs_response.code == 200
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: filestack
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.7.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Filestack
|
8
|
-
autorequire:
|
8
|
+
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-09-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: typhoeus
|
@@ -29,6 +29,9 @@ dependencies:
|
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '1.11'
|
34
|
+
- - ">="
|
32
35
|
- !ruby/object:Gem::Version
|
33
36
|
version: 1.11.2
|
34
37
|
type: :runtime
|
@@ -36,6 +39,9 @@ dependencies:
|
|
36
39
|
version_requirements: !ruby/object:Gem::Requirement
|
37
40
|
requirements:
|
38
41
|
- - "~>"
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: '1.11'
|
44
|
+
- - ">="
|
39
45
|
- !ruby/object:Gem::Version
|
40
46
|
version: 1.11.2
|
41
47
|
- !ruby/object:Gem::Dependency
|
@@ -195,7 +201,7 @@ homepage: https://github.com/filestack/filestack-ruby
|
|
195
201
|
licenses:
|
196
202
|
- MIT
|
197
203
|
metadata: {}
|
198
|
-
post_install_message:
|
204
|
+
post_install_message:
|
199
205
|
rdoc_options: []
|
200
206
|
require_paths:
|
201
207
|
- lib
|
@@ -210,9 +216,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
210
216
|
- !ruby/object:Gem::Version
|
211
217
|
version: '0'
|
212
218
|
requirements: []
|
213
|
-
|
214
|
-
|
215
|
-
signing_key:
|
219
|
+
rubygems_version: 3.0.8
|
220
|
+
signing_key:
|
216
221
|
specification_version: 4
|
217
222
|
summary: Official Ruby SDK for the Filestack API
|
218
223
|
test_files: []
|