aspera-cli 4.0.0 → 4.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +843 -304
- data/bin/dascli +13 -0
- data/docs/Makefile +4 -4
- data/docs/README.erb.md +805 -172
- data/docs/test_env.conf +22 -3
- data/examples/aoc.rb +14 -3
- data/examples/faspex4.rb +89 -0
- data/lib/aspera/aoc.rb +87 -108
- data/lib/aspera/cli/formater.rb +2 -0
- data/lib/aspera/cli/main.rb +89 -49
- data/lib/aspera/cli/plugin.rb +9 -4
- data/lib/aspera/cli/plugins/alee.rb +1 -1
- data/lib/aspera/cli/plugins/aoc.rb +188 -173
- data/lib/aspera/cli/plugins/ats.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +218 -145
- data/lib/aspera/cli/plugins/console.rb +2 -2
- data/lib/aspera/cli/plugins/faspex.rb +114 -61
- data/lib/aspera/cli/plugins/faspex5.rb +85 -43
- data/lib/aspera/cli/plugins/node.rb +3 -3
- data/lib/aspera/cli/plugins/preview.rb +59 -45
- data/lib/aspera/cli/plugins/server.rb +23 -8
- data/lib/aspera/cli/transfer_agent.rb +77 -49
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +49 -31
- data/lib/aspera/cos_node.rb +33 -28
- data/lib/aspera/environment.rb +2 -2
- data/lib/aspera/fasp/connect.rb +28 -21
- data/lib/aspera/fasp/http_gw.rb +140 -28
- data/lib/aspera/fasp/installation.rb +93 -46
- data/lib/aspera/fasp/local.rb +88 -45
- data/lib/aspera/fasp/manager.rb +15 -0
- data/lib/aspera/fasp/node.rb +4 -4
- data/lib/aspera/fasp/parameters.rb +59 -101
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/resume_policy.rb +13 -12
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/log.rb +1 -1
- data/lib/aspera/node.rb +61 -1
- data/lib/aspera/oauth.rb +49 -46
- data/lib/aspera/persistency_folder.rb +9 -4
- data/lib/aspera/preview/file_types.rb +53 -21
- data/lib/aspera/preview/generator.rb +3 -3
- data/lib/aspera/rest.rb +29 -18
- data/lib/aspera/secrets.rb +20 -0
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/temp_file_manager.rb +19 -0
- data/lib/aspera/web_auth.rb +105 -0
- metadata +54 -20
- data/docs/transfer_spec.html +0 -99
data/docs/README.erb.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
2
|
<%
|
3
3
|
# check that required env vars exist, and files
|
4
|
-
%w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION
|
4
|
+
%w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_DIR_GEM}.each do |e|
|
5
5
|
raise "missing env var #{e}" unless ENV.has_key?(e)
|
6
6
|
raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
|
7
7
|
end
|
@@ -14,28 +14,102 @@ prsts='['+opprst+'s](#lprt)'
|
|
14
14
|
prstt=opprst.capitalize # in title
|
15
15
|
gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
|
16
16
|
geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
|
17
|
+
$LOAD_PATH.unshift(ENV["INCL_DIR_GEM"])
|
18
|
+
require 'aspera/fasp/parameters'
|
19
|
+
def spec_table
|
20
|
+
r='<table><tr><th>Field</th><th>Type</th>'
|
21
|
+
Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
|
22
|
+
r << '<th>'<<c.to_s.upcase<<'</th>'
|
23
|
+
end
|
24
|
+
r << '<th>Description</th></tr>'
|
25
|
+
Aspera::Fasp::Parameters.man_table.each do |p|
|
26
|
+
p[:description] << (p[:description].empty? ? '' : "\n") << "(" << p[:cli] << ")" unless p[:cli].to_s.empty?
|
27
|
+
p.delete(:cli)
|
28
|
+
p.keys.each{|c|p[c]=' ' if p[c].to_s.empty?}
|
29
|
+
p[:description].gsub!("\n",'<br/>')
|
30
|
+
p[:type].gsub!(',','<br/>')
|
31
|
+
r << '<tr><td>'<<p[:name]<<'</td><td>'<<p[:type]<<'</td>'
|
32
|
+
Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
|
33
|
+
r << '<td>'<<p[c]<<'</td>'
|
34
|
+
end
|
35
|
+
r << '<td>'<<p[:description]<<'</td></tr>'
|
36
|
+
end
|
37
|
+
r << '</table>'
|
38
|
+
return r
|
39
|
+
end
|
17
40
|
-%>
|
18
|
-
# <%=tool%> :
|
41
|
+
# <%=tool%> : Command Line Interface for IBM Aspera products
|
19
42
|
|
20
43
|
Version : <%= gemspec.version.to_s %>
|
21
44
|
|
22
45
|
_Laurent/2016-<%=Time.new.year%>_
|
23
46
|
|
24
|
-
This gem provides a command line interface to Aspera Applications.
|
47
|
+
This gem provides <%=tool%>: a command line interface to Aspera Applications.
|
48
|
+
|
49
|
+
<%=tool%> is a also great tool to learn Aspera APIs.
|
50
|
+
|
51
|
+
Ruby Gem: [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubygems_uri'] %>)
|
52
|
+
|
53
|
+
Ruby Doc: [<%= gemspec.metadata['documentation_uri'] %>](<%= gemspec.metadata['documentation_uri'] %>)
|
54
|
+
|
55
|
+
Ruby version must be >= <%= gemspec.required_ruby_version %>
|
56
|
+
|
57
|
+
# <a name="when_to_use"></a>When to use and when not to use
|
58
|
+
|
59
|
+
<%=tool%> is designed to be used as a command line tool to:
|
60
|
+
|
61
|
+
* execute commands on Aspera products
|
62
|
+
* transfer to/from Aspera products
|
63
|
+
|
64
|
+
So it is designed for:
|
65
|
+
|
66
|
+
* Interactive operations on a text terminal (typically, VT100 compatible)
|
67
|
+
* Batch operations in (shell) scripts (e.g. cron job)
|
25
68
|
|
26
|
-
|
27
|
-
[<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubygems_uri'] %>)
|
69
|
+
<%=tool%> can be seen as a command line tool integrating:
|
28
70
|
|
29
|
-
|
71
|
+
* a configuration file (config.yaml) and advanced command line options
|
72
|
+
* cURL (for REST calls)
|
73
|
+
* Aspera transfer (ascp)
|
30
74
|
|
31
|
-
|
75
|
+
One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea.
|
76
|
+
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
32
77
|
|
33
|
-
|
78
|
+
* Product APIs (REST) : e.g. AoC, Faspex, node
|
79
|
+
* Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
80
|
+
|
81
|
+
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
82
|
+
|
83
|
+
For scripting and ad'hoc command line operations, <%=tool%> is perfect.
|
84
|
+
|
85
|
+
# Notations
|
34
86
|
|
35
87
|
In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
|
88
|
+
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
36
89
|
|
37
90
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
38
91
|
|
92
|
+
# <a name="parsing"></a>Shell and Command line parsing
|
93
|
+
|
94
|
+
<%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
|
95
|
+
|
96
|
+
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
97
|
+
|
98
|
+
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
99
|
+
|
100
|
+
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
101
|
+
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
102
|
+
|
103
|
+
In case of doubt of argument values after parsing test like this:
|
104
|
+
|
105
|
+
```
|
106
|
+
$ <%=cmd%> conf echo "Hello World" arg2 3
|
107
|
+
"Hello World"
|
108
|
+
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
109
|
+
```
|
110
|
+
|
111
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
|
112
|
+
|
39
113
|
# Quick Start
|
40
114
|
|
41
115
|
This section guides you from installation, first use and advanced use.
|
@@ -55,9 +129,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
|
|
55
129
|
|
56
130
|
If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
57
131
|
|
58
|
-
|
132
|
+
To test with Aspera demo transfer server, setup the environment and then test:
|
59
133
|
|
60
134
|
```
|
135
|
+
$ <%=cmd%> config initdemo
|
61
136
|
$ <%=cmd%> server browse /
|
62
137
|
:............:...........:......:........:...........................:.......................:
|
63
138
|
: zmode : zuid : zgid : size : mtime : name :
|
@@ -77,7 +152,7 @@ If you want to use <%=tool%> with another server, and in order to make further c
|
|
77
152
|
* download a file
|
78
153
|
|
79
154
|
```
|
80
|
-
$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
155
|
+
$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
81
156
|
updated: myserver
|
82
157
|
$ <%=cmd%> config id default set server myserver
|
83
158
|
updated: default→server to myserver
|
@@ -111,77 +186,159 @@ Then, follow the section relative to the product you want to interact with ( Asp
|
|
111
186
|
|
112
187
|
# <a name="installation"></a>Installation
|
113
188
|
|
114
|
-
|
189
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
|
115
190
|
|
116
|
-
|
191
|
+
The direct installation is recommended and consists in installing:
|
192
|
+
|
193
|
+
* [Ruby](#ruby) version >= <%= gemspec.required_ruby_version %>
|
117
194
|
* [<%= gemspec.name %>](#the_gem)
|
118
|
-
* [
|
195
|
+
* [Aspera SDK (ascp)](#fasp_prot)
|
196
|
+
|
197
|
+
The following sections provide information on the various installation methods.
|
119
198
|
|
120
|
-
|
199
|
+
An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
200
|
+
|
201
|
+
## Docker container
|
202
|
+
|
203
|
+
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
204
|
+
|
205
|
+
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
206
|
+
|
207
|
+
Ensure that you have Docker installed.
|
208
|
+
|
209
|
+
```
|
210
|
+
$ docker --version
|
211
|
+
```
|
212
|
+
|
213
|
+
Download the wrapping script:
|
214
|
+
|
215
|
+
```
|
216
|
+
$ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
217
|
+
$ chmod a+x <%=cmd%>
|
218
|
+
```
|
219
|
+
|
220
|
+
Install the container image:
|
221
|
+
|
222
|
+
```
|
223
|
+
$ ./<%=cmd%> install
|
224
|
+
```
|
225
|
+
|
226
|
+
Start using it !
|
227
|
+
|
228
|
+
Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
|
229
|
+
|
230
|
+
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
|
231
|
+
|
232
|
+
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
233
|
+
To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
|
121
234
|
|
122
235
|
## <a name="ruby"></a>Ruby
|
123
236
|
|
124
|
-
|
237
|
+
Use this method to install on the native host.
|
125
238
|
|
126
|
-
|
239
|
+
A ruby interpreter is required to run the tool or to use the gem and tool.
|
127
240
|
|
128
|
-
Ruby 3 is also supported.
|
241
|
+
Ruby minimum version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
|
129
242
|
|
130
|
-
|
243
|
+
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
131
244
|
|
132
245
|
Refer to the following sections for a proposed method for specific operating systems.
|
133
246
|
|
134
|
-
|
247
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
|
248
|
+
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
249
|
+
If you have a simpler better way to install Ruby version >= <%= gemspec.required_ruby_version %> : use it !
|
250
|
+
|
251
|
+
### Generic: RVM: single user installation (not root)
|
252
|
+
|
253
|
+
Use this method which provides more flexibility.
|
254
|
+
|
255
|
+
Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
135
256
|
|
136
|
-
|
257
|
+
Install the 2 keys
|
137
258
|
|
138
259
|
```
|
139
|
-
$
|
260
|
+
$ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
140
261
|
```
|
141
262
|
|
142
|
-
|
263
|
+
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
143
264
|
|
144
265
|
```
|
145
|
-
$
|
266
|
+
$ \curl -sSL https://get.rvm.io | bash -s stable
|
267
|
+
```
|
268
|
+
|
269
|
+
If you keep the same terminal (ont needed if re-login):
|
270
|
+
|
271
|
+
```
|
272
|
+
$ source ~/.rvm/scripts/rvm
|
273
|
+
```
|
274
|
+
|
275
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
276
|
+
|
277
|
+
```
|
278
|
+
$ rvm list --remote
|
146
279
|
```
|
147
280
|
|
148
|
-
|
281
|
+
Install the chosen pre-compiled Ruby version:
|
149
282
|
|
150
|
-
|
283
|
+
```
|
284
|
+
$ rvm install 2.7.2 --binary
|
285
|
+
```
|
151
286
|
|
152
|
-
|
287
|
+
Ruby is now installed for the user, go on to Gem installation.
|
153
288
|
|
154
|
-
|
289
|
+
### Generic: RVM: global installation (as root)
|
155
290
|
|
156
|
-
|
291
|
+
Follow the same method as single user install, but execute as "root".
|
157
292
|
|
158
|
-
|
293
|
+
As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
|
294
|
+
One can install in another location with :
|
159
295
|
|
160
|
-
|
296
|
+
```
|
297
|
+
# curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
298
|
+
```
|
161
299
|
|
162
|
-
|
163
|
-
|
300
|
+
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
301
|
+
If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok`.
|
302
|
+
To activate ruby (and <%=cmd%>) later, source it:
|
164
303
|
|
165
304
|
```
|
166
|
-
|
305
|
+
# source /etc/profile.d/rvm.sh.ok
|
306
|
+
# rvm version
|
167
307
|
```
|
168
308
|
|
169
|
-
|
309
|
+
### Windows: Installer
|
310
|
+
|
311
|
+
Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/) :
|
312
|
+
|
313
|
+
* Go to "Downloads".
|
314
|
+
* Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
|
315
|
+
* At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.
|
316
|
+
|
317
|
+
### macOS: pre-installed or `brew`
|
318
|
+
|
319
|
+
MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%= gemspec.name %> using `sudo` :
|
170
320
|
|
171
321
|
```
|
172
|
-
|
322
|
+
$ sudo gem install <%= gemspec.name %><%=geminstadd%>
|
173
323
|
```
|
174
324
|
|
175
|
-
|
176
|
-
rename the file: `/etc/profile.d/rvm.sh` with another extension, and source it to get rvm.
|
325
|
+
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
177
326
|
|
178
|
-
|
327
|
+
```
|
328
|
+
$ brew install ruby
|
329
|
+
```
|
330
|
+
|
331
|
+
### Linux: package
|
332
|
+
|
333
|
+
If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
|
334
|
+
|
335
|
+
Example:
|
179
336
|
|
180
337
|
```
|
181
338
|
# yum install -y ruby rubygems ruby-json
|
182
339
|
```
|
183
340
|
|
184
|
-
One can cleanup
|
341
|
+
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
185
342
|
|
186
343
|
```
|
187
344
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
@@ -206,6 +363,50 @@ For instance to build from source, and install in `/opt/ruby` :
|
|
206
363
|
# make install
|
207
364
|
```
|
208
365
|
|
366
|
+
### <a name="offline_install"></a>Installation without internet access
|
367
|
+
|
368
|
+
Note that currently no pre-packaged version exist yet.
|
369
|
+
A method to build one provided here:
|
370
|
+
|
371
|
+
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
372
|
+
|
373
|
+
Then create an archive:
|
374
|
+
|
375
|
+
```
|
376
|
+
$ cd
|
377
|
+
$ tar zcvf rvm-<%=cmd%>.tgz .rvm
|
378
|
+
```
|
379
|
+
|
380
|
+
Get the Aspera SDK. Execute:
|
381
|
+
|
382
|
+
```
|
383
|
+
$ <%=cmd%> conf --show-config|grep sdk_url
|
384
|
+
```
|
385
|
+
|
386
|
+
Then download the SDK archive from that URL.
|
387
|
+
|
388
|
+
Another method for the SDK is to install the SDK (`<%=cmd%> conf ascp install`) on the first system, and archive `$HOME/.aspera`.
|
389
|
+
|
390
|
+
Transfer those 2 archives to the target system without internet access.
|
391
|
+
|
392
|
+
On the target system:
|
393
|
+
|
394
|
+
* Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
|
395
|
+
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
396
|
+
|
397
|
+
```
|
398
|
+
source path_to_rvm_root/scripts/rvm
|
399
|
+
rvm use 2.7.2
|
400
|
+
```
|
401
|
+
|
402
|
+
For the SDK, either install from archive:
|
403
|
+
|
404
|
+
```
|
405
|
+
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
406
|
+
```
|
407
|
+
|
408
|
+
or restore the `$HOME/.aspera` folder for the user.
|
409
|
+
|
209
410
|
## <a name="the_gem"></a>`<%= gemspec.name %>` gem
|
210
411
|
|
211
412
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
@@ -220,21 +421,40 @@ To upgrade to the latest version:
|
|
220
421
|
# gem update <%= gemspec.name %>
|
221
422
|
```
|
222
423
|
|
424
|
+
<%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
425
|
+
|
426
|
+
To check manually:
|
427
|
+
|
428
|
+
```
|
429
|
+
# <%=cmd%> conf check_update
|
430
|
+
```
|
431
|
+
|
432
|
+
|
433
|
+
|
223
434
|
## <a name="fasp_prot"></a>FASP Protocol
|
224
435
|
|
225
|
-
Most file transfers will be done using the FASP protocol
|
226
|
-
an Aspera Transfer:
|
436
|
+
Most file transfers will be done using the FASP protocol, using `ascp`.
|
437
|
+
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
227
438
|
|
228
439
|
* ascp
|
229
440
|
* aspera-license (in same folder, or ../etc)
|
230
441
|
|
231
|
-
This can be installed
|
442
|
+
This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
|
232
443
|
|
233
444
|
```
|
234
445
|
$ <%=cmd%> conf ascp install
|
235
446
|
```
|
236
447
|
|
237
|
-
|
448
|
+
If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
|
449
|
+
|
450
|
+
```
|
451
|
+
$ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
452
|
+
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
453
|
+
```
|
454
|
+
|
455
|
+
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
456
|
+
|
457
|
+
If the embedded method is not used, the following packages are also suitable:
|
238
458
|
|
239
459
|
* IBM Aspera Connect Client (Free)
|
240
460
|
* IBM Aspera Desktop Client (Free)
|
@@ -243,7 +463,7 @@ Those can be found in one of IBM Aspera transfer server or client with its licen
|
|
243
463
|
* IBM Aspera High Speed Transfer EndPoint (Licensed)
|
244
464
|
|
245
465
|
For instance, Aspera Connect Client can be installed
|
246
|
-
by visiting the page: [
|
466
|
+
by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
247
467
|
|
248
468
|
<%=tool%> will detect most of Aspera transfer products in standard locations and use the first one found.
|
249
469
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
@@ -251,6 +471,33 @@ Refer to section [FASP](#client) for details on how to select a client or set pa
|
|
251
471
|
Several methods are provided on how to start a transfer. Use of a local client is one of them, but
|
252
472
|
other methods are available. Refer to section: [Transfer Agents](#agents)
|
253
473
|
|
474
|
+
## <a name="offline_install"></a>Offline Installation (without internet)
|
475
|
+
|
476
|
+
The procedure consists in:
|
477
|
+
|
478
|
+
* Follow the non-root installation procedure with RVM, including gem
|
479
|
+
* archive (zip, tar) the main RVM folder (includes <%=cmd%>):
|
480
|
+
|
481
|
+
```
|
482
|
+
$ cd ~
|
483
|
+
$ tar zcvf rvm_<%=cmd%>.tgz .rvm
|
484
|
+
```
|
485
|
+
|
486
|
+
* retrieve the SDK:
|
487
|
+
|
488
|
+
```
|
489
|
+
$ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
490
|
+
```
|
491
|
+
|
492
|
+
* on the system without internet access:
|
493
|
+
|
494
|
+
```
|
495
|
+
$ cd ~
|
496
|
+
$ tar zxvf rvm_<%=cmd%>.tgz
|
497
|
+
$ source ~/.rvm/scripts/rvm
|
498
|
+
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
499
|
+
```
|
500
|
+
|
254
501
|
# <a name="cli"></a>Command Line Interface: <%=tool%>
|
255
502
|
|
256
503
|
The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
@@ -431,6 +678,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
431
678
|
* a,b,c : the list of attributes specified by the comma separated list
|
432
679
|
* Array extended value: for instance, @json:'["a","b","c"]' same as above
|
433
680
|
* +a,b,c : add selected properties to the default selection.
|
681
|
+
* -a,b,c : remove selected properties from the default selection.
|
434
682
|
|
435
683
|
## <a name="extended"></a>Extended Value Syntax
|
436
684
|
|
@@ -444,25 +692,25 @@ The extended value syntax is:
|
|
444
692
|
|
445
693
|
The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
|
446
694
|
|
447
|
-
The following "readers" are supported:
|
695
|
+
The following "readers" are supported (returns value in []):
|
448
696
|
|
449
|
-
* @val:VALUE
|
450
|
-
* @file:PATH
|
451
|
-
* @path:PATH
|
452
|
-
* @env:ENVVAR
|
453
|
-
* @stdin:
|
454
|
-
* @preset:NAME
|
697
|
+
* @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
|
698
|
+
* @file:PATH : [String] read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
|
699
|
+
* @path:PATH : [String] performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
|
700
|
+
* @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
701
|
+
* @stdin: : [String] read from stdin (no value on right)
|
702
|
+
* @preset:NAME : [Hash] get whole <%=opprst%> value by name
|
455
703
|
|
456
704
|
In addition it is possible to decode a value, using one or multiple decoders :
|
457
705
|
|
458
|
-
* @base64: decode a base64 encoded string
|
459
|
-
* @json: decode JSON values (convenient to provide complex structures)
|
460
|
-
* @zlib: uncompress data
|
461
|
-
* @ruby: execute ruby code
|
462
|
-
* @csvt: decode a titled CSV value
|
463
|
-
* @lines: split a string in multiple lines and return an array
|
464
|
-
* @list: split a string in multiple items taking first character as separator and return an array
|
465
|
-
* @incps: include values of presets specified by key
|
706
|
+
* @base64: [String] decode a base64 encoded string
|
707
|
+
* @json: [any] decode JSON values (convenient to provide complex structures)
|
708
|
+
* @zlib: [String] uncompress data
|
709
|
+
* @ruby: [any] execute ruby code
|
710
|
+
* @csvt: [Array] decode a titled CSV value
|
711
|
+
* @lines: [Array] split a string in multiple lines and return an array
|
712
|
+
* @list: [Array] split a string in multiple items taking first character as separator and return an array
|
713
|
+
* @incps: [Hash] include values of presets specified by key `incps` in input hash
|
466
714
|
|
467
715
|
To display the result of an extended value, use the `config echo` command.
|
468
716
|
|
@@ -494,7 +742,7 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
|
|
494
742
|
:......:.....................:
|
495
743
|
```
|
496
744
|
|
497
|
-
Example: create a hash and include values from preset named "config" of config file
|
745
|
+
Example: create a hash and include values from preset named "config" of config file in this hash
|
498
746
|
|
499
747
|
```
|
500
748
|
$ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
@@ -533,9 +781,9 @@ It can be overriden using the envinonment variable `<%=evp%>HOME`.
|
|
533
781
|
Example (Windows):
|
534
782
|
|
535
783
|
```
|
536
|
-
$ set <%=evp%>HOME=C:\Users\Kenji\.aspera
|
784
|
+
$ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
|
537
785
|
$ <%=cmd%> config folder
|
538
|
-
C:\Users\Kenji\.aspera
|
786
|
+
C:\Users\Kenji\.aspera\<%=cmd%>
|
539
787
|
```
|
540
788
|
|
541
789
|
## <a name="configfile"></a>Configuration file
|
@@ -570,7 +818,7 @@ $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
|
|
570
818
|
The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
|
571
819
|
|
572
820
|
```
|
573
|
-
$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
821
|
+
$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
574
822
|
```
|
575
823
|
|
576
824
|
* This creates a <%=prst%> `demo_server` with all provided options.
|
@@ -578,13 +826,13 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
|
|
578
826
|
The command `set` allows setting individual options in a <%=prst%>.
|
579
827
|
|
580
828
|
```
|
581
|
-
$ <%=cmd%> config id demo_server set password
|
829
|
+
$ <%=cmd%> config id demo_server set password _demo_pass_
|
582
830
|
```
|
583
831
|
|
584
832
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
585
833
|
|
586
834
|
```
|
587
|
-
$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"
|
835
|
+
$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
588
836
|
```
|
589
837
|
|
590
838
|
A good practice is to not manually edit the configuration file and use modification commands instead.
|
@@ -621,11 +869,19 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
621
869
|
Operations on this preset are done using regular `config` operations:
|
622
870
|
|
623
871
|
```
|
624
|
-
$ <%=cmd%> config id default set _plugin_name_
|
872
|
+
$ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
|
625
873
|
$ <%=cmd%> config id default get _plugin_name_
|
626
|
-
"
|
874
|
+
"_default_preset_for_plugin_"
|
627
875
|
```
|
628
876
|
|
877
|
+
### <a name="lprtdef"></a>Special Plugin: config
|
878
|
+
|
879
|
+
Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
|
880
|
+
|
881
|
+
When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
882
|
+
|
883
|
+
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
884
|
+
|
629
885
|
### Format of file
|
630
886
|
|
631
887
|
The configuration file is a hash in a YAML file. Example:
|
@@ -641,7 +897,7 @@ cli_default:
|
|
641
897
|
demo_server:
|
642
898
|
url: ssh://demo.asperasoft.com:33001
|
643
899
|
username: asperaweb
|
644
|
-
password:
|
900
|
+
password: _demo_pass_
|
645
901
|
```
|
646
902
|
|
647
903
|
We can see here:
|
@@ -710,7 +966,6 @@ A <%=prst%> value can be removed with `unset`:
|
|
710
966
|
$ <%=cmd%> config id cli_default unset interactive
|
711
967
|
```
|
712
968
|
|
713
|
-
|
714
969
|
### Examples
|
715
970
|
|
716
971
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
@@ -830,21 +1085,42 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
830
1085
|
|
831
1086
|
```
|
832
1087
|
$ <%=cmd%> config ascp show
|
833
|
-
/Users/laurent/
|
1088
|
+
/Users/laurent/.aspera/ascli/sdk/ascp
|
1089
|
+
$ <%=cmd%> config ascp info
|
1090
|
+
+--------------------+-----------------------------------------------------------+
|
1091
|
+
| key | value |
|
1092
|
+
+--------------------+-----------------------------------------------------------+
|
1093
|
+
| ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
|
1094
|
+
...
|
834
1095
|
```
|
835
1096
|
|
836
1097
|
### Selection of local `ascp`
|
837
1098
|
|
1099
|
+
By default, <%=tool%> uses any found local product with ascp, including SDK.
|
1100
|
+
|
838
1101
|
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
839
1102
|
|
840
|
-
|
1103
|
+
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
1104
|
+
|
1105
|
+
Using a POSIX shell:
|
841
1106
|
|
842
1107
|
```
|
843
1108
|
$ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
844
|
-
|
1109
|
+
ascp version: 4.0.0.182279
|
1110
|
+
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
1111
|
+
Saved to default global preset global_common_defaults
|
1112
|
+
```
|
1113
|
+
|
1114
|
+
Windows:
|
1115
|
+
|
1116
|
+
```
|
1117
|
+
$ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1118
|
+
ascp version: 4.0.0.182279
|
1119
|
+
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1120
|
+
Saved to default global preset global_common_defaults
|
845
1121
|
```
|
846
1122
|
|
847
|
-
|
1123
|
+
If the path has spaces, read section: [Shell and Command line parsing](#parsing).
|
848
1124
|
|
849
1125
|
### List locally installed Aspera Transfer products
|
850
1126
|
|
@@ -931,7 +1207,7 @@ will effectively push files to the related server from the agent node.
|
|
931
1207
|
|
932
1208
|
### <a name="direct"></a>Direct (local ascp using FASPManager API)
|
933
1209
|
|
934
|
-
By default
|
1210
|
+
By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
|
935
1211
|
<%=tool%> will detect locally installed Aspera products.
|
936
1212
|
Refer to section [FASP](#client).
|
937
1213
|
|
@@ -941,17 +1217,33 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
|
|
941
1217
|
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
942
1218
|
* `EX_ascp_args`
|
943
1219
|
|
944
|
-
The `transfer-info`
|
1220
|
+
The `transfer-info` accepts the following optional parameters:
|
945
1221
|
|
946
1222
|
<table>
|
947
|
-
<tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
948
|
-
<tr><td>
|
949
|
-
<tr><td>
|
950
|
-
<tr><td>
|
951
|
-
<tr><td>
|
952
|
-
<tr><td>wss</td> <td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
1223
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1224
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
|
1225
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
|
1226
|
+
<tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
1227
|
+
<tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
|
953
1228
|
</table>
|
954
1229
|
|
1230
|
+
Resume parameters:
|
1231
|
+
|
1232
|
+
<table>
|
1233
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1234
|
+
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
1235
|
+
<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
1236
|
+
<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
|
1237
|
+
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
1238
|
+
</table>
|
1239
|
+
|
1240
|
+
Examples:
|
1241
|
+
|
1242
|
+
```
|
1243
|
+
$ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
1244
|
+
$ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
|
1245
|
+
```
|
1246
|
+
|
955
1247
|
### IBM Aspera Connect Client GUI
|
956
1248
|
|
957
1249
|
By specifying option: `--transfer=connect`, <%=tool%> will start transfers
|
@@ -983,9 +1275,11 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
983
1275
|
Example:
|
984
1276
|
|
985
1277
|
```
|
986
|
-
$ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://
|
1278
|
+
$ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
987
1279
|
```
|
988
1280
|
|
1281
|
+
Note that the gateway only supports transfers authorized with a token.
|
1282
|
+
|
989
1283
|
## <a name="transferspec"></a>Transfer Specification
|
990
1284
|
|
991
1285
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
@@ -999,7 +1293,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
999
1293
|
|
1000
1294
|
<%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1001
1295
|
|
1002
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
|
1296
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
1003
1297
|
|
1004
1298
|
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
1005
1299
|
|
@@ -1012,11 +1306,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
|
|
1012
1306
|
|
1013
1307
|
## <a name="transferparams"></a>Transfer Parameters
|
1014
1308
|
|
1015
|
-
All standard _transfer-spec_ parameters can be
|
1016
|
-
|
1017
|
-
|
1309
|
+
All standard _transfer-spec_ parameters can be speficied.
|
1310
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
1311
|
+
|
1312
|
+
References:
|
1313
|
+
|
1314
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
1315
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
1316
|
+
|
1317
|
+
Parameters can be displayed with commands:
|
1018
1318
|
|
1019
|
-
|
1319
|
+
```
|
1320
|
+
$ <%=cmd%> config ascp spec
|
1321
|
+
$ <%=cmd%> config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
|
1322
|
+
```
|
1323
|
+
|
1324
|
+
Columns:
|
1325
|
+
|
1326
|
+
* D=Direct (local `ascp` execution)
|
1327
|
+
* N=Node API
|
1328
|
+
* C=Connect Client
|
1329
|
+
* arg=`ascp` argument or environment variable
|
1330
|
+
|
1331
|
+
Fields with EX_ prefix are extensions to transfer agent `direct`. (only in <%=tool%>).
|
1332
|
+
|
1333
|
+
<%= spec_table %>
|
1020
1334
|
|
1021
1335
|
### Destination folder for transfers
|
1022
1336
|
|
@@ -1032,8 +1346,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1032
1346
|
|
1033
1347
|
### List of files for transfers
|
1034
1348
|
|
1035
|
-
When uploading, downloading or sending files, the user must specify
|
1036
|
-
|
1349
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
1350
|
+
Most of the time, the list of files to transfer will be simply specified on the command line:
|
1037
1351
|
|
1038
1352
|
```
|
1039
1353
|
$ <%=cmd%> server upload ~/mysample.file secondfile
|
@@ -1047,7 +1361,8 @@ $ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
|
|
1047
1361
|
|
1048
1362
|
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1049
1363
|
|
1050
|
-
Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
|
1364
|
+
Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
|
1365
|
+
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
1051
1366
|
|
1052
1367
|
For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. Accepted values are:
|
1053
1368
|
|
@@ -1097,7 +1412,7 @@ Source files are located on "Aspera on cloud", when :
|
|
1097
1412
|
|
1098
1413
|
### <a name="multisession"></a>Support of multi-session
|
1099
1414
|
|
1100
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
1415
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
1101
1416
|
|
1102
1417
|
* when agent=node :
|
1103
1418
|
|
@@ -1118,6 +1433,7 @@ shall be preferred.
|
|
1118
1433
|
|
1119
1434
|
Multi-session spawn is done by <%=tool%>.
|
1120
1435
|
|
1436
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
1121
1437
|
|
1122
1438
|
### Examples
|
1123
1439
|
|
@@ -1290,7 +1606,7 @@ updated: my_aoc_org
|
|
1290
1606
|
Define this <%=prst%> as default configuration for the `aspera` plugin:
|
1291
1607
|
|
1292
1608
|
```
|
1293
|
-
$ <%=cmd%> config id default set
|
1609
|
+
$ <%=cmd%> config id default set aoc my_aoc_org
|
1294
1610
|
```
|
1295
1611
|
|
1296
1612
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
@@ -1393,7 +1709,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
|
|
1393
1709
|
|
1394
1710
|
* change auth method to JWT
|
1395
1711
|
* provide location of private key
|
1396
|
-
* provide username to login as (
|
1712
|
+
* provide username to login as (OAuth "subject")
|
1397
1713
|
|
1398
1714
|
Execute:
|
1399
1715
|
|
@@ -1664,7 +1980,7 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
1664
1980
|
Then, transfer between those:
|
1665
1981
|
|
1666
1982
|
```
|
1667
|
-
$ <%=cmd%> -Paoc_show
|
1983
|
+
$ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
1668
1984
|
```
|
1669
1985
|
|
1670
1986
|
* create registration key to register a node
|
@@ -1744,10 +2060,10 @@ $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
1744
2060
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
1745
2061
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
1746
2062
|
|
1747
|
-
Typically, one would regularly execute this command on a regular basis, using the method
|
2063
|
+
Typically, one would regularly execute this command on a regular basis, using the method of your choice:
|
1748
2064
|
|
1749
|
-
* Windows scheduler
|
1750
|
-
* cron
|
2065
|
+
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2066
|
+
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
1751
2067
|
* etc...
|
1752
2068
|
|
1753
2069
|
## Download Files
|
@@ -1804,19 +2120,40 @@ The activity app can be queried with:
|
|
1804
2120
|
$ <%=cmd%> aoc admin analytics transfers
|
1805
2121
|
```
|
1806
2122
|
|
1807
|
-
It can also support filters and send notification
|
2123
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
2124
|
+
|
2125
|
+
`mytemplate.erb`:
|
2126
|
+
|
2127
|
+
```
|
2128
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
2129
|
+
To: <<%='<'%>%=ev['user_email']%>>
|
2130
|
+
Subject: <%='<'%>%=ev['files_completed']%> files received
|
2131
|
+
|
2132
|
+
Dear <%='<'%>%=ev[:user_email.to_s]%>,
|
2133
|
+
We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
|
2134
|
+
<%='<'%>%=ev['content']%>
|
1808
2135
|
|
2136
|
+
Thank you.
|
1809
2137
|
```
|
1810
|
-
|
2138
|
+
The environment provided contains the following additional variable:
|
2139
|
+
|
2140
|
+
* ev : all details on the transfer event
|
2141
|
+
|
2142
|
+
Example:
|
2143
|
+
|
2144
|
+
```
|
2145
|
+
$ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
1811
2146
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
1812
|
-
--
|
2147
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
1813
2148
|
```
|
1814
2149
|
|
2150
|
+
Options:
|
2151
|
+
|
1815
2152
|
* `once_only` keep track of last date it was called, so next call will get only new events
|
1816
2153
|
* `query` filter (on API call)
|
1817
2154
|
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
1818
2155
|
|
1819
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months.
|
2156
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
1820
2157
|
|
1821
2158
|
## Using specific transfer ports
|
1822
2159
|
|
@@ -1967,7 +2304,7 @@ This can also be set as default using a preset
|
|
1967
2304
|
One can test the "server" application using the well known demo server:
|
1968
2305
|
|
1969
2306
|
```
|
1970
|
-
$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
2307
|
+
$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
1971
2308
|
$ <%=cmd%> config id default set server aspera_demo_server
|
1972
2309
|
$ <%=cmd%> server browse /aspera-test-dir-large
|
1973
2310
|
$ <%=cmd%> server download /aspera-test-dir-large/200MB
|
@@ -2063,27 +2400,130 @@ to download files.
|
|
2063
2400
|
$ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
2064
2401
|
```
|
2065
2402
|
|
2066
|
-
# Plugin: IBM Aspera
|
2403
|
+
# Plugin: IBM Aspera Faspex5
|
2404
|
+
|
2405
|
+
3 authentication methods are supported:
|
2406
|
+
|
2407
|
+
* jwt
|
2408
|
+
* web
|
2409
|
+
* boot
|
2410
|
+
|
2411
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
2412
|
+
|
2413
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
2414
|
+
|
2415
|
+
For boot method: (will be removed in future)
|
2416
|
+
|
2417
|
+
* open a browser
|
2418
|
+
* start developer mode
|
2419
|
+
* login to faspex 5
|
2420
|
+
* find the first API call with `Authorization` token, and copy it (kind of base64 long string)
|
2421
|
+
|
2422
|
+
Use it as password and use `--auth=boot`.
|
2423
|
+
|
2424
|
+
```
|
2425
|
+
$ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
2426
|
+
```
|
2427
|
+
|
2428
|
+
Ready to use Faspex5 with CLI.
|
2429
|
+
|
2430
|
+
# Plugin: IBM Aspera Faspex (4.x)
|
2431
|
+
|
2432
|
+
Notes:
|
2433
|
+
|
2434
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
2435
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
2436
|
+
|
2437
|
+
## Listing Packages
|
2438
|
+
|
2439
|
+
Command: `faspex package list`
|
2440
|
+
|
2441
|
+
### Option `box`
|
2442
|
+
|
2443
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
2444
|
+
|
2445
|
+
### Option `recipient`
|
2446
|
+
|
2447
|
+
A user can receive a package because the recipient is:
|
2448
|
+
|
2449
|
+
* the user himself (default)
|
2450
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
2451
|
+
|
2452
|
+
### Option `query`
|
2067
2453
|
|
2068
|
-
|
2454
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
2455
|
+
|
2456
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
2457
|
+
* `page` : (native) id of page in call (default=0)
|
2458
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
2459
|
+
* `max` : maximum number of items
|
2460
|
+
* `pmax` : maximum number of pages
|
2461
|
+
|
2462
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
2463
|
+
|
2464
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
2465
|
+
|
2466
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
2467
|
+
|
2468
|
+
### Example
|
2469
|
+
|
2470
|
+
```
|
2471
|
+
$ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
2472
|
+
```
|
2473
|
+
|
2474
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
2475
|
+
|
2476
|
+
## Receiving a Package
|
2477
|
+
|
2478
|
+
The command is `package recv`, possible methods are:
|
2479
|
+
|
2480
|
+
* provide a package id with option `id`
|
2481
|
+
* provide a public link with option `link`
|
2482
|
+
* provide a `faspe:` URI with option `link`
|
2483
|
+
|
2484
|
+
```
|
2485
|
+
$ <%=cmd%> faspex package recv --id=12345
|
2486
|
+
$ <%=cmd%> faspex package recv --link=faspe://...
|
2487
|
+
```
|
2488
|
+
|
2489
|
+
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
2490
|
+
|
2491
|
+
```
|
2492
|
+
$ <%=cmd%> faspex package list --recipient='*thedropboxname'
|
2493
|
+
```
|
2494
|
+
|
2495
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
2069
2496
|
|
2070
2497
|
## Sending a Package
|
2071
2498
|
|
2072
|
-
|
2499
|
+
The command is `faspex package send`. Package information (title, note, metadata, options) is provided in option `delivery_info`. (Refer to Faspex API).
|
2500
|
+
|
2501
|
+
Example:
|
2073
2502
|
|
2074
2503
|
```
|
2075
|
-
--delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}'
|
2504
|
+
$ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
2076
2505
|
```
|
2077
2506
|
|
2078
|
-
a
|
2507
|
+
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
2508
|
+
|
2509
|
+
Additional optional parameters in `delivery_info`:
|
2510
|
+
|
2511
|
+
* Package Note: : `"note":"note this and that"`
|
2512
|
+
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
2079
2513
|
|
2080
|
-
|
2514
|
+
## Email notification on transfer
|
2081
2515
|
|
2516
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
2517
|
+
|
2518
|
+
Example:
|
2519
|
+
|
2520
|
+
```
|
2521
|
+
$ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
2522
|
+
```
|
2082
2523
|
|
2083
|
-
|
2084
|
-
[Reference on Developer Site](https://developer.asperasoft.com/web/faspex/sending)
|
2524
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
2085
2525
|
|
2086
|
-
##
|
2526
|
+
## Operation on dropboxes
|
2087
2527
|
|
2088
2528
|
Example:
|
2089
2529
|
|
@@ -2144,6 +2584,8 @@ $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p
|
|
2144
2584
|
# Plugin: IBM Cloud Object Storage
|
2145
2585
|
|
2146
2586
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
2587
|
+
It uses the same transfer service as Aspera on Cloud.
|
2588
|
+
see [https://status.aspera.io](https://status.aspera.io)
|
2147
2589
|
|
2148
2590
|
Required options are either:
|
2149
2591
|
|
@@ -2200,7 +2642,14 @@ Endpoints for regions can be found by querying the `endpoints` URL.
|
|
2200
2642
|
For convenience, let us create a default configuration, for example:
|
2201
2643
|
|
2202
2644
|
```
|
2203
|
-
$ <%=cmd%> conf id mycos update --service-credentials=@val:@json:@file
|
2645
|
+
$ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
2646
|
+
$ <%=cmd%> conf id default set cos mycos
|
2647
|
+
```
|
2648
|
+
|
2649
|
+
or using direct parameters:
|
2650
|
+
|
2651
|
+
```
|
2652
|
+
$ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
2204
2653
|
$ <%=cmd%> conf id default set cos mycos
|
2205
2654
|
```
|
2206
2655
|
|
@@ -2218,7 +2667,7 @@ to start from ma configuration file, using <%=tool%> standard options.
|
|
2218
2667
|
|
2219
2668
|
# Plugin: Preview
|
2220
2669
|
|
2221
|
-
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on
|
2670
|
+
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
2222
2671
|
This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
|
2223
2672
|
Several parameters can be used to tune several aspects:
|
2224
2673
|
|
@@ -2266,6 +2715,7 @@ The tool requires the following external tools available in the `PATH`:
|
|
2266
2715
|
* OptiPNG : `optipng`
|
2267
2716
|
* FFmpeg : `ffmpeg` `ffprobe`
|
2268
2717
|
* Libreoffice : `libreoffice`
|
2718
|
+
* ruby gem `mimemagic`
|
2269
2719
|
|
2270
2720
|
Here shown on Redhat/CentOS.
|
2271
2721
|
|
@@ -2277,6 +2727,26 @@ To check if all tools are found properly, execute:
|
|
2277
2727
|
$ <%=cmd%> preview check
|
2278
2728
|
```
|
2279
2729
|
|
2730
|
+
### mimemagic
|
2731
|
+
|
2732
|
+
To benefit from extra mime type detection install gem mimemagic:
|
2733
|
+
|
2734
|
+
```
|
2735
|
+
# gem install mimemagic
|
2736
|
+
```
|
2737
|
+
|
2738
|
+
or to install an earlier version if any problem:
|
2739
|
+
|
2740
|
+
```
|
2741
|
+
# gem install mimemagic -v '~> 0.3.0'
|
2742
|
+
```
|
2743
|
+
|
2744
|
+
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
2745
|
+
|
2746
|
+
If not used, Mime type used for conversion is the one provided by the node API.
|
2747
|
+
|
2748
|
+
If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
|
2749
|
+
|
2280
2750
|
### Image: Imagemagick and optipng
|
2281
2751
|
|
2282
2752
|
```
|
@@ -2301,90 +2771,111 @@ The generation of preview in based on the use of `unoconv` and `libreoffice`
|
|
2301
2771
|
# dnf install unoconv
|
2302
2772
|
```
|
2303
2773
|
|
2774
|
+
* Amazon Linux
|
2775
|
+
|
2776
|
+
```
|
2777
|
+
# amazon-linux-extras enable libreoffice
|
2778
|
+
# yum clean metadata
|
2779
|
+
# yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
2780
|
+
# wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
2781
|
+
# mv unoconv /usr/bin
|
2782
|
+
# chmod a+x /usr/bin/unoconv
|
2783
|
+
```
|
2304
2784
|
|
2305
2785
|
## Configuration
|
2306
2786
|
|
2307
|
-
|
2787
|
+
The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)
|
2788
|
+
|
2789
|
+
Like any <%=tool%> commands, parameters can be passed on command line or using a configuration <%=prst%>. The configuration file must be created with the same user used to run so that it is properly used on runtime.
|
2790
|
+
|
2791
|
+
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
2308
2792
|
|
2309
2793
|
```
|
2310
2794
|
# su -s /bin/bash - xfer
|
2311
|
-
$ <%=cmd%> config id
|
2312
|
-
$ <%=cmd%> config id default set preview
|
2795
|
+
$ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
2796
|
+
$ <%=cmd%> config id default set preview previewconf
|
2313
2797
|
```
|
2314
2798
|
|
2315
|
-
Here we assume that Office file generation is disabled, else remove
|
2799
|
+
Here we assume that Office file generation is disabled, else remove this option.
|
2800
|
+
`lock_port` prevents concurrent execution of generation when using a scheduler.
|
2316
2801
|
|
2317
|
-
|
2802
|
+
One can check if the access key is well configured using:
|
2318
2803
|
|
2319
2804
|
```
|
2320
|
-
$ <%=cmd%> -
|
2805
|
+
$ <%=cmd%> -Ppreviewconf node browse /
|
2321
2806
|
```
|
2322
2807
|
|
2323
2808
|
This shall list the contents of the storage root of the access key.
|
2324
2809
|
|
2325
2810
|
## Execution
|
2326
2811
|
|
2327
|
-
The tool intentionally supports only a "one shot" mode in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
2328
|
-
It needs to be run
|
2812
|
+
The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
2813
|
+
It needs to be run on a regular basis to create or update preview files. For that use your best
|
2329
2814
|
reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.
|
2330
2815
|
|
2331
|
-
Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be
|
2332
|
-
should be run as user `xfer`.
|
2816
|
+
Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
|
2333
2817
|
|
2334
2818
|
Lets do a one shot test, using the configuration previously created:
|
2335
2819
|
|
2336
2820
|
```
|
2337
2821
|
# su -s /bin/bash - xfer
|
2338
|
-
$ <%=cmd%> preview scan --overwrite=always
|
2822
|
+
xfer$ <%=cmd%> preview scan --overwrite=always
|
2339
2823
|
```
|
2340
2824
|
|
2341
2825
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
2342
|
-
which contains the access key used.
|
2826
|
+
in the previews folder which contains the access key used.
|
2343
2827
|
On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
|
2344
2828
|
|
2345
2829
|
## Configuration for Execution in scheduler
|
2346
2830
|
|
2347
|
-
Here is an example of configuration for use with cron on Linux.
|
2831
|
+
Here is an example of configuration for use with cron on Linux.
|
2832
|
+
Adapt the scripts to your own preference.
|
2348
2833
|
|
2349
2834
|
We assume here that a configuration preset was created as shown previously.
|
2350
2835
|
|
2351
|
-
|
2836
|
+
Lets first setup a script that will be used in the sceduler and sets up the environment.
|
2352
2837
|
|
2838
|
+
Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
|
2839
|
+
|
2840
|
+
```
|
2841
|
+
#!/bin/bash
|
2842
|
+
# set a timeout protection, just in case
|
2843
|
+
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
2844
|
+
. /etc/profile.d/rvm.sh
|
2845
|
+
rvm use 2.6 --quiet
|
2846
|
+
exec timeout ${tmout} <%=cmd%> "${@}"
|
2353
2847
|
```
|
2354
|
-
|
2355
|
-
|
2356
|
-
|
2848
|
+
|
2849
|
+
Here the cronjob is created for user `xfer`.
|
2850
|
+
|
2851
|
+
```
|
2852
|
+
xfer$ crontab<<EOF
|
2853
|
+
0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
|
2854
|
+
2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
|
2357
2855
|
EOF
|
2358
2856
|
```
|
2359
2857
|
|
2360
|
-
|
2858
|
+
Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
2361
2859
|
|
2362
2860
|
## Candidate detection for creation or update (or deletion)
|
2363
2861
|
|
2364
|
-
The tool
|
2862
|
+
The tool generates preview files using those commands:
|
2365
2863
|
|
2366
2864
|
* `trevents` : only recently uploaded files will be tested (transfer events)
|
2367
2865
|
* `events` : only recently uploaded files will be tested (file events: not working)
|
2368
|
-
* `scan` :
|
2369
|
-
* `
|
2370
|
-
* `file` : for an individual file generation
|
2371
|
-
|
2372
|
-
Note that for the `event`, the option `iteration_file` should be specified so that
|
2373
|
-
successive calls only process new events. This file will hold an identifier
|
2374
|
-
telling from where to get new events.
|
2375
|
-
|
2376
|
-
It is also possible to test a local file, using the `test` command.
|
2866
|
+
* `scan` : recursively scan all files under the access key's "storage root"
|
2867
|
+
* `test` : test using a local file
|
2377
2868
|
|
2378
2869
|
Once candidate are selected, once candidates are selected,
|
2379
2870
|
a preview is always generated if it does not exist already,
|
2380
2871
|
else if a preview already exist, it will be generated
|
2381
|
-
using one of three overwrite
|
2872
|
+
using one of three values for the `overwrite` option:
|
2382
2873
|
|
2383
2874
|
* `always` : preview is always generated, even if it already exists and is newer than original
|
2384
2875
|
* `never` : preview is generated only if it does not exist already
|
2385
2876
|
* `mtime` : preview is generated only if the original file is newer than the existing
|
2386
2877
|
|
2387
|
-
Deletion of preview for deleted source files: not implemented yet.
|
2878
|
+
Deletion of preview for deleted source files: not implemented yet (TODO).
|
2388
2879
|
|
2389
2880
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
2390
2881
|
|
@@ -2394,6 +2885,14 @@ $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
|
|
2394
2885
|
|
2395
2886
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
2396
2887
|
|
2888
|
+
When scanning the option `value` has the same behaviour as for the `node find` command.
|
2889
|
+
|
2890
|
+
For instance to filter out files beginning with `._` do:
|
2891
|
+
|
2892
|
+
```
|
2893
|
+
... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
|
2894
|
+
```
|
2895
|
+
|
2397
2896
|
## Preview File types
|
2398
2897
|
|
2399
2898
|
Two types of preview can be generated:
|
@@ -2442,14 +2941,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
2442
2941
|
The `smtp` option is a hash table (extended value) with the following fields:
|
2443
2942
|
<table>
|
2444
2943
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
2445
|
-
<tr><td
|
2446
|
-
<tr><td
|
2447
|
-
<tr><td
|
2448
|
-
<tr><td
|
2449
|
-
<tr><td
|
2450
|
-
<tr><td
|
2451
|
-
<tr><td
|
2452
|
-
<tr><td
|
2944
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
2945
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
2946
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
2947
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
2948
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
2949
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
2950
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
2951
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
2453
2952
|
</table>
|
2454
2953
|
|
2455
2954
|
## Example of configuration:
|
@@ -2479,13 +2978,52 @@ $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
|
|
2479
2978
|
$ <%=cmd%> config id default set config cli_default
|
2480
2979
|
```
|
2481
2980
|
|
2981
|
+
## Email templates
|
2982
|
+
|
2983
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
2984
|
+
|
2985
|
+
The template is the full SMTP message, including headers.
|
2986
|
+
|
2987
|
+
The following variables are defined by default:
|
2988
|
+
|
2989
|
+
* from_name
|
2990
|
+
* from_email
|
2991
|
+
* to
|
2992
|
+
|
2993
|
+
Other variables are defined depending on context.
|
2994
|
+
|
2482
2995
|
## Test
|
2483
2996
|
|
2484
2997
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
2485
2998
|
|
2486
2999
|
```
|
2487
3000
|
$ <%=cmd%> config --smtp=@preset:smtp_google smtp
|
2488
|
-
$ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
|
3001
|
+
$ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
3002
|
+
```
|
3003
|
+
|
3004
|
+
## Notifications for transfer status
|
3005
|
+
|
3006
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
3007
|
+
|
3008
|
+
To activate, use option `notif_to`.
|
3009
|
+
|
3010
|
+
A default e-mail template is used, but it can be overriden with option `notif_template`.
|
3011
|
+
|
3012
|
+
The environment provided contains the following additional variables:
|
3013
|
+
|
3014
|
+
* subject
|
3015
|
+
* body
|
3016
|
+
* global_transfer_status
|
3017
|
+
* ts
|
3018
|
+
|
3019
|
+
Example of template:
|
3020
|
+
|
3021
|
+
```
|
3022
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
3023
|
+
To: <<%='<'%>%=to%>>
|
3024
|
+
Subject: <%='<'%>%=subject%>
|
3025
|
+
|
3026
|
+
Transfer is: <%='<'%>%=global_transfer_status%>
|
2489
3027
|
```
|
2490
3028
|
|
2491
3029
|
# Tool: `asession`
|
@@ -2531,7 +3069,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
|
|
2531
3069
|
## Simple session
|
2532
3070
|
|
2533
3071
|
```
|
2534
|
-
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"
|
3072
|
+
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
|
2535
3073
|
|
2536
3074
|
echo "${MY_TSPEC}"|asession
|
2537
3075
|
```
|
@@ -2544,7 +3082,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
|
|
2544
3082
|
|
2545
3083
|
```
|
2546
3084
|
$ asession
|
2547
|
-
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"
|
3085
|
+
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
2548
3086
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
2549
3087
|
{"type":"DONE"}
|
2550
3088
|
```
|
@@ -2596,7 +3134,7 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
2596
3134
|
Note that:
|
2597
3135
|
|
2598
3136
|
* <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
2599
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
3137
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
2600
3138
|
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
|
2601
3139
|
|
2602
3140
|
### server side and configuration
|
@@ -2618,6 +3156,34 @@ $ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=1
|
|
2618
3156
|
|
2619
3157
|
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
2620
3158
|
|
3159
|
+
# Aspera Health check and Nagios
|
3160
|
+
|
3161
|
+
Each plugin provide a `health` command that will check the health status of the application. Example:
|
3162
|
+
|
3163
|
+
```
|
3164
|
+
$ <%=cmd%> console health
|
3165
|
+
+--------+-------------+------------+
|
3166
|
+
| status | component | message |
|
3167
|
+
+--------+-------------+------------+
|
3168
|
+
| ok | console api | accessible |
|
3169
|
+
+--------+-------------+------------+
|
3170
|
+
```
|
3171
|
+
|
3172
|
+
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
3173
|
+
|
3174
|
+
* issuing a transfer to the server
|
3175
|
+
* checking web app status with `asctl all:status`
|
3176
|
+
* checking daemons process status
|
3177
|
+
|
3178
|
+
<%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
3179
|
+
|
3180
|
+
```
|
3181
|
+
$ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
|
3182
|
+
OK - [transfer:ok]
|
3183
|
+
$ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
3184
|
+
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
3185
|
+
```
|
3186
|
+
|
2621
3187
|
# Module: `Aspera`
|
2622
3188
|
|
2623
3189
|
Main components:
|
@@ -2637,6 +3203,16 @@ This sample code shows some example of use of the API as well as
|
|
2637
3203
|
REST API.
|
2638
3204
|
Note: although nice, it's probably a good idea to use RestClient for REST.
|
2639
3205
|
|
3206
|
+
Example of use of the API of Aspera on Cloud:
|
3207
|
+
|
3208
|
+
```
|
3209
|
+
require 'aspera/aoc'
|
3210
|
+
|
3211
|
+
aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/<%=cmd%>/aspera_on_cloud_key')),username: 'laurent.martin.aspera@fr.ibm.com',subpath: 'api/v1')
|
3212
|
+
|
3213
|
+
aoc.read('self')
|
3214
|
+
```
|
3215
|
+
|
2640
3216
|
# History
|
2641
3217
|
|
2642
3218
|
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
@@ -2653,11 +3229,55 @@ So, it evolved into <%=tool%>:
|
|
2653
3229
|
* supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from ascp command line to [_transfer-spec_](#transferspec) (more reliable , more standard)
|
2654
3230
|
* `ruby` is consistent with other Aspera products
|
2655
3231
|
|
2656
|
-
|
2657
|
-
|
2658
|
-
|
2659
|
-
|
2660
|
-
*
|
3232
|
+
# Changes (Release notes)
|
3233
|
+
|
3234
|
+
* <%= gemspec.version.to_s %>
|
3235
|
+
|
3236
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
3237
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
3238
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
3239
|
+
* new: `conf ascp spec` displays supported transfer spec
|
3240
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
3241
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
3242
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
3243
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
|
3244
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
3245
|
+
|
3246
|
+
* 4.2.1
|
3247
|
+
|
3248
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
3249
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
3250
|
+
|
3251
|
+
* 4.2.0
|
3252
|
+
|
3253
|
+
* new: command `aoc remind` to receive organization membership by email
|
3254
|
+
* new: in `preview` option `value` to filter out on file name
|
3255
|
+
* new: `initdemo` to initialize for demo server
|
3256
|
+
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3257
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
3258
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
3259
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3260
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3261
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3262
|
+
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
3263
|
+
|
3264
|
+
* 4.1.0
|
3265
|
+
|
3266
|
+
* fix: remove keys from transfer spec and command line when not needed
|
3267
|
+
* fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
3268
|
+
* new: update documentation with regard to offline and docker installation
|
3269
|
+
* new: renamed command `nagios_check` to `health`
|
3270
|
+
* new: agent `http_gw` now supports upload
|
3271
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
3272
|
+
* new: check new gem version periodically
|
3273
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
3274
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
3275
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
3276
|
+
* new: AoC a password can be provided for a public link
|
3277
|
+
* new: `conf doc` take an optional parameter to go to a section
|
3278
|
+
* new: initial support for Faspex 5 Beta 1
|
3279
|
+
|
3280
|
+
* 4.0.0
|
2661
3281
|
|
2662
3282
|
* now available as open source at [<%= gemspec.homepage %>](<%= gemspec.homepage %>) with general cleanup
|
2663
3283
|
* changed default tool name from `mlia` to `ascli`
|
@@ -2785,7 +3405,7 @@ So, it evolved into <%=tool%>:
|
|
2785
3405
|
|
2786
3406
|
* 0.10.6
|
2787
3407
|
|
2788
|
-
* FaspManager: transfer spec `authentication` no more needed for local tranfer to use
|
3408
|
+
* FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
2789
3409
|
* gem version requirements made more open
|
2790
3410
|
|
2791
3411
|
* 0.10.5
|
@@ -3012,11 +3632,15 @@ Breaking change:
|
|
3012
3632
|
|
3013
3633
|
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
|
3014
3634
|
|
3015
|
-
# BUGS
|
3635
|
+
# BUGS, FEATURES, CONTRIBUTION
|
3016
3636
|
|
3017
|
-
|
3637
|
+
For issues or feature requests use the Github repository and issues.
|
3018
3638
|
|
3019
|
-
|
3639
|
+
You can also contribute to this open source project.
|
3640
|
+
|
3641
|
+
One can also create one's own command nplugin.
|
3642
|
+
|
3643
|
+
## Only one value for any option
|
3020
3644
|
|
3021
3645
|
Some commands and sub commands may ask for the same option name.
|
3022
3646
|
Currently, since option definition is position independant (last one wins), it is not possible
|
@@ -3030,7 +3654,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
|
|
3030
3654
|
|
3031
3655
|
## ED255519 key not supported
|
3032
3656
|
|
3033
|
-
|
3657
|
+
ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
|
3658
|
+
|
3034
3659
|
Without this deactivation, if such key was present the following error was generated:
|
3035
3660
|
|
3036
3661
|
```
|
@@ -3040,7 +3665,21 @@ OpenSSH keys only supported if ED25519 is available
|
|
3040
3665
|
Which meant that you do not have ruby support for ED25519 SSH keys.
|
3041
3666
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
3042
3667
|
|
3043
|
-
|
3668
|
+
## Error "Remote host is not who we expected"
|
3669
|
+
|
3670
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
3671
|
+
|
3672
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
3673
|
+
|
3674
|
+
```
|
3675
|
+
--ts=@json:'{"sshfp":null}'
|
3676
|
+
```
|
3677
|
+
|
3678
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
3679
|
+
|
3680
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
3681
|
+
|
3682
|
+
## Miscelaneous
|
3044
3683
|
|
3045
3684
|
* remove rest and oauth classes and use ruby standard gems:
|
3046
3685
|
|
@@ -3056,9 +3695,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
3056
3695
|
* Going through proxy: use env var http_proxy and https_proxy, no_proxy
|
3057
3696
|
|
3058
3697
|
* easier use with https://github.com/pmq20/ruby-packer
|
3059
|
-
|
3060
|
-
# Contribution
|
3061
|
-
|
3062
|
-
Send comments !
|
3063
|
-
|
3064
|
-
Create your own plugin !
|