aspera-cli 4.0.0 → 4.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +843 -304
- data/bin/dascli +13 -0
- data/docs/Makefile +4 -4
- data/docs/README.erb.md +805 -172
- data/docs/test_env.conf +22 -3
- data/examples/aoc.rb +14 -3
- data/examples/faspex4.rb +89 -0
- data/lib/aspera/aoc.rb +87 -108
- data/lib/aspera/cli/formater.rb +2 -0
- data/lib/aspera/cli/main.rb +89 -49
- data/lib/aspera/cli/plugin.rb +9 -4
- data/lib/aspera/cli/plugins/alee.rb +1 -1
- data/lib/aspera/cli/plugins/aoc.rb +188 -173
- data/lib/aspera/cli/plugins/ats.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +218 -145
- data/lib/aspera/cli/plugins/console.rb +2 -2
- data/lib/aspera/cli/plugins/faspex.rb +114 -61
- data/lib/aspera/cli/plugins/faspex5.rb +85 -43
- data/lib/aspera/cli/plugins/node.rb +3 -3
- data/lib/aspera/cli/plugins/preview.rb +59 -45
- data/lib/aspera/cli/plugins/server.rb +23 -8
- data/lib/aspera/cli/transfer_agent.rb +77 -49
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +49 -31
- data/lib/aspera/cos_node.rb +33 -28
- data/lib/aspera/environment.rb +2 -2
- data/lib/aspera/fasp/connect.rb +28 -21
- data/lib/aspera/fasp/http_gw.rb +140 -28
- data/lib/aspera/fasp/installation.rb +93 -46
- data/lib/aspera/fasp/local.rb +88 -45
- data/lib/aspera/fasp/manager.rb +15 -0
- data/lib/aspera/fasp/node.rb +4 -4
- data/lib/aspera/fasp/parameters.rb +59 -101
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/resume_policy.rb +13 -12
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/log.rb +1 -1
- data/lib/aspera/node.rb +61 -1
- data/lib/aspera/oauth.rb +49 -46
- data/lib/aspera/persistency_folder.rb +9 -4
- data/lib/aspera/preview/file_types.rb +53 -21
- data/lib/aspera/preview/generator.rb +3 -3
- data/lib/aspera/rest.rb +29 -18
- data/lib/aspera/secrets.rb +20 -0
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/temp_file_manager.rb +19 -0
- data/lib/aspera/web_auth.rb +105 -0
- metadata +54 -20
- data/docs/transfer_spec.html +0 -99
data/README.md
CHANGED
@@ -1,25 +1,76 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
|
-
# `ascli` :
|
2
|
+
# `ascli` : Command Line Interface for IBM Aspera products
|
3
3
|
|
4
|
-
Version : 4.
|
4
|
+
Version : 4.2.2
|
5
5
|
|
6
6
|
_Laurent/2016-2021_
|
7
7
|
|
8
|
-
This gem provides a command line interface to Aspera Applications.
|
8
|
+
This gem provides `ascli`: a command line interface to Aspera Applications.
|
9
9
|
|
10
|
-
|
11
|
-
[https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
|
10
|
+
`ascli` is a also great tool to learn Aspera APIs.
|
12
11
|
|
13
|
-
|
12
|
+
Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
|
14
13
|
|
15
|
-
|
14
|
+
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
16
15
|
|
17
|
-
|
16
|
+
Ruby version must be >= > 2.4
|
17
|
+
|
18
|
+
# <a name="when_to_use"></a>When to use and when not to use
|
19
|
+
|
20
|
+
`ascli` is designed to be used as a command line tool to:
|
21
|
+
|
22
|
+
* execute commands on Aspera products
|
23
|
+
* transfer to/from Aspera products
|
24
|
+
|
25
|
+
So it is designed for:
|
26
|
+
|
27
|
+
* Interactive operations on a text terminal (typically, VT100 compatible)
|
28
|
+
* Batch operations in (shell) scripts (e.g. cron job)
|
29
|
+
|
30
|
+
`ascli` can be seen as a command line tool integrating:
|
31
|
+
|
32
|
+
* a configuration file (config.yaml) and advanced command line options
|
33
|
+
* cURL (for REST calls)
|
34
|
+
* Aspera transfer (ascp)
|
35
|
+
|
36
|
+
One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea.
|
37
|
+
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
38
|
+
|
39
|
+
* Product APIs (REST) : e.g. AoC, Faspex, node
|
40
|
+
* Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
41
|
+
|
42
|
+
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
43
|
+
|
44
|
+
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
45
|
+
|
46
|
+
# Notations
|
18
47
|
|
19
48
|
In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
|
49
|
+
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
20
50
|
|
21
51
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
22
52
|
|
53
|
+
# <a name="parsing"></a>Shell and Command line parsing
|
54
|
+
|
55
|
+
`ascli` is typically executed in a shell, either interactively or in a script. `ascli` receives its arguments from this shell.
|
56
|
+
|
57
|
+
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
58
|
+
|
59
|
+
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
60
|
+
|
61
|
+
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
62
|
+
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
63
|
+
|
64
|
+
In case of doubt of argument values after parsing test like this:
|
65
|
+
|
66
|
+
```
|
67
|
+
$ ascli conf echo "Hello World" arg2 3
|
68
|
+
"Hello World"
|
69
|
+
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
70
|
+
```
|
71
|
+
|
72
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
|
73
|
+
|
23
74
|
# Quick Start
|
24
75
|
|
25
76
|
This section guides you from installation, first use and advanced use.
|
@@ -30,7 +81,7 @@ Once the gem is installed, `ascli` shall be accessible:
|
|
30
81
|
|
31
82
|
```
|
32
83
|
$ ascli --version
|
33
|
-
4.
|
84
|
+
4.2.2
|
34
85
|
```
|
35
86
|
|
36
87
|
## First use
|
@@ -39,9 +90,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
|
|
39
90
|
|
40
91
|
If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
41
92
|
|
42
|
-
|
93
|
+
To test with Aspera demo transfer server, setup the environment and then test:
|
43
94
|
|
44
95
|
```
|
96
|
+
$ ascli config initdemo
|
45
97
|
$ ascli server browse /
|
46
98
|
:............:...........:......:........:...........................:.......................:
|
47
99
|
: zmode : zuid : zgid : size : mtime : name :
|
@@ -61,7 +113,7 @@ If you want to use `ascli` with another server, and in order to make further cal
|
|
61
113
|
* download a file
|
62
114
|
|
63
115
|
```
|
64
|
-
$ ascli config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
116
|
+
$ ascli config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
65
117
|
updated: myserver
|
66
118
|
$ ascli config id default set server myserver
|
67
119
|
updated: default→server to myserver
|
@@ -95,77 +147,159 @@ Then, follow the section relative to the product you want to interact with ( Asp
|
|
95
147
|
|
96
148
|
# <a name="installation"></a>Installation
|
97
149
|
|
98
|
-
|
150
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
|
151
|
+
|
152
|
+
The direct installation is recommended and consists in installing:
|
99
153
|
|
100
|
-
* [Ruby](#ruby)
|
154
|
+
* [Ruby](#ruby) version >= > 2.4
|
101
155
|
* [aspera-cli](#the_gem)
|
102
|
-
* [
|
156
|
+
* [Aspera SDK (ascp)](#fasp_prot)
|
157
|
+
|
158
|
+
The following sections provide information on the various installation methods.
|
159
|
+
|
160
|
+
An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
161
|
+
|
162
|
+
## Docker container
|
163
|
+
|
164
|
+
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
165
|
+
|
166
|
+
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
167
|
+
|
168
|
+
Ensure that you have Docker installed.
|
169
|
+
|
170
|
+
```
|
171
|
+
$ docker --version
|
172
|
+
```
|
173
|
+
|
174
|
+
Download the wrapping script:
|
175
|
+
|
176
|
+
```
|
177
|
+
$ curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
178
|
+
$ chmod a+x ascli
|
179
|
+
```
|
180
|
+
|
181
|
+
Install the container image:
|
182
|
+
|
183
|
+
```
|
184
|
+
$ ./ascli install
|
185
|
+
```
|
186
|
+
|
187
|
+
Start using it !
|
103
188
|
|
104
|
-
|
189
|
+
Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
|
190
|
+
|
191
|
+
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/ascli` on host.
|
192
|
+
|
193
|
+
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
194
|
+
To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
|
105
195
|
|
106
196
|
## <a name="ruby"></a>Ruby
|
107
197
|
|
108
|
-
|
198
|
+
Use this method to install on the native host.
|
109
199
|
|
110
|
-
|
200
|
+
A ruby interpreter is required to run the tool or to use the gem and tool.
|
111
201
|
|
112
|
-
Ruby 3 is also supported.
|
202
|
+
Ruby minimum version: > 2.4. Ruby version 3 is also supported.
|
113
203
|
|
114
|
-
|
204
|
+
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
115
205
|
|
116
206
|
Refer to the following sections for a proposed method for specific operating systems.
|
117
207
|
|
118
|
-
|
208
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
|
209
|
+
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
210
|
+
If you have a simpler better way to install Ruby version >= > 2.4 : use it !
|
211
|
+
|
212
|
+
### Generic: RVM: single user installation (not root)
|
119
213
|
|
120
|
-
|
214
|
+
Use this method which provides more flexibility.
|
215
|
+
|
216
|
+
Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
217
|
+
|
218
|
+
Install the 2 keys
|
121
219
|
|
122
220
|
```
|
123
|
-
$
|
221
|
+
$ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
124
222
|
```
|
125
223
|
|
126
|
-
|
224
|
+
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
127
225
|
|
128
226
|
```
|
129
|
-
$
|
227
|
+
$ \curl -sSL https://get.rvm.io | bash -s stable
|
228
|
+
```
|
229
|
+
|
230
|
+
If you keep the same terminal (ont needed if re-login):
|
231
|
+
|
232
|
+
```
|
233
|
+
$ source ~/.rvm/scripts/rvm
|
234
|
+
```
|
235
|
+
|
236
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
237
|
+
|
238
|
+
```
|
239
|
+
$ rvm list --remote
|
240
|
+
```
|
241
|
+
|
242
|
+
Install the chosen pre-compiled Ruby version:
|
243
|
+
|
244
|
+
```
|
245
|
+
$ rvm install 2.7.2 --binary
|
130
246
|
```
|
131
247
|
|
132
|
-
|
248
|
+
Ruby is now installed for the user, go on to Gem installation.
|
249
|
+
|
250
|
+
### Generic: RVM: global installation (as root)
|
251
|
+
|
252
|
+
Follow the same method as single user install, but execute as "root".
|
253
|
+
|
254
|
+
As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
|
255
|
+
One can install in another location with :
|
133
256
|
|
134
|
-
|
257
|
+
```
|
258
|
+
# curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
259
|
+
```
|
260
|
+
|
261
|
+
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
262
|
+
If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok`.
|
263
|
+
To activate ruby (and ascli) later, source it:
|
135
264
|
|
136
|
-
|
265
|
+
```
|
266
|
+
# source /etc/profile.d/rvm.sh.ok
|
267
|
+
# rvm version
|
268
|
+
```
|
137
269
|
|
138
|
-
|
270
|
+
### Windows: Installer
|
139
271
|
|
140
|
-
|
272
|
+
Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/) :
|
141
273
|
|
142
|
-
|
274
|
+
* Go to "Downloads".
|
275
|
+
* Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
|
276
|
+
* At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.
|
143
277
|
|
144
|
-
###
|
278
|
+
### macOS: pre-installed or `brew`
|
145
279
|
|
146
|
-
|
147
|
-
It installs by default in /usr/local/rvm , but you can install in another location:
|
280
|
+
MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
148
281
|
|
149
282
|
```
|
150
|
-
|
283
|
+
$ sudo gem install aspera-cli
|
151
284
|
```
|
152
285
|
|
153
|
-
|
286
|
+
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
154
287
|
|
155
288
|
```
|
156
|
-
|
289
|
+
$ brew install ruby
|
157
290
|
```
|
158
291
|
|
159
|
-
|
160
|
-
rename the file: `/etc/profile.d/rvm.sh` with another extension, and source it to get rvm.
|
292
|
+
### Linux: package
|
161
293
|
|
162
|
-
|
294
|
+
If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
|
295
|
+
|
296
|
+
Example:
|
163
297
|
|
164
298
|
```
|
165
299
|
# yum install -y ruby rubygems ruby-json
|
166
300
|
```
|
167
301
|
|
168
|
-
One can cleanup
|
302
|
+
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
169
303
|
|
170
304
|
```
|
171
305
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
@@ -190,6 +324,50 @@ For instance to build from source, and install in `/opt/ruby` :
|
|
190
324
|
# make install
|
191
325
|
```
|
192
326
|
|
327
|
+
### <a name="offline_install"></a>Installation without internet access
|
328
|
+
|
329
|
+
Note that currently no pre-packaged version exist yet.
|
330
|
+
A method to build one provided here:
|
331
|
+
|
332
|
+
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
333
|
+
|
334
|
+
Then create an archive:
|
335
|
+
|
336
|
+
```
|
337
|
+
$ cd
|
338
|
+
$ tar zcvf rvm-ascli.tgz .rvm
|
339
|
+
```
|
340
|
+
|
341
|
+
Get the Aspera SDK. Execute:
|
342
|
+
|
343
|
+
```
|
344
|
+
$ ascli conf --show-config|grep sdk_url
|
345
|
+
```
|
346
|
+
|
347
|
+
Then download the SDK archive from that URL.
|
348
|
+
|
349
|
+
Another method for the SDK is to install the SDK (`ascli conf ascp install`) on the first system, and archive `$HOME/.aspera`.
|
350
|
+
|
351
|
+
Transfer those 2 archives to the target system without internet access.
|
352
|
+
|
353
|
+
On the target system:
|
354
|
+
|
355
|
+
* Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
|
356
|
+
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
357
|
+
|
358
|
+
```
|
359
|
+
source path_to_rvm_root/scripts/rvm
|
360
|
+
rvm use 2.7.2
|
361
|
+
```
|
362
|
+
|
363
|
+
For the SDK, either install from archive:
|
364
|
+
|
365
|
+
```
|
366
|
+
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
367
|
+
```
|
368
|
+
|
369
|
+
or restore the `$HOME/.aspera` folder for the user.
|
370
|
+
|
193
371
|
## <a name="the_gem"></a>`aspera-cli` gem
|
194
372
|
|
195
373
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
@@ -204,21 +382,40 @@ To upgrade to the latest version:
|
|
204
382
|
# gem update aspera-cli
|
205
383
|
```
|
206
384
|
|
385
|
+
`ascli` checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
386
|
+
|
387
|
+
To check manually:
|
388
|
+
|
389
|
+
```
|
390
|
+
# ascli conf check_update
|
391
|
+
```
|
392
|
+
|
393
|
+
|
394
|
+
|
207
395
|
## <a name="fasp_prot"></a>FASP Protocol
|
208
396
|
|
209
|
-
Most file transfers will be done using the FASP protocol
|
210
|
-
an Aspera Transfer:
|
397
|
+
Most file transfers will be done using the FASP protocol, using `ascp`.
|
398
|
+
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
211
399
|
|
212
400
|
* ascp
|
213
401
|
* aspera-license (in same folder, or ../etc)
|
214
402
|
|
215
|
-
This can be installed
|
403
|
+
This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
|
216
404
|
|
217
405
|
```
|
218
406
|
$ ascli conf ascp install
|
219
407
|
```
|
220
408
|
|
221
|
-
|
409
|
+
If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
|
410
|
+
|
411
|
+
```
|
412
|
+
$ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
413
|
+
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
414
|
+
```
|
415
|
+
|
416
|
+
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
417
|
+
|
418
|
+
If the embedded method is not used, the following packages are also suitable:
|
222
419
|
|
223
420
|
* IBM Aspera Connect Client (Free)
|
224
421
|
* IBM Aspera Desktop Client (Free)
|
@@ -227,7 +424,7 @@ Those can be found in one of IBM Aspera transfer server or client with its licen
|
|
227
424
|
* IBM Aspera High Speed Transfer EndPoint (Licensed)
|
228
425
|
|
229
426
|
For instance, Aspera Connect Client can be installed
|
230
|
-
by visiting the page: [
|
427
|
+
by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
231
428
|
|
232
429
|
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
233
430
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
@@ -235,6 +432,33 @@ Refer to section [FASP](#client) for details on how to select a client or set pa
|
|
235
432
|
Several methods are provided on how to start a transfer. Use of a local client is one of them, but
|
236
433
|
other methods are available. Refer to section: [Transfer Agents](#agents)
|
237
434
|
|
435
|
+
## <a name="offline_install"></a>Offline Installation (without internet)
|
436
|
+
|
437
|
+
The procedure consists in:
|
438
|
+
|
439
|
+
* Follow the non-root installation procedure with RVM, including gem
|
440
|
+
* archive (zip, tar) the main RVM folder (includes ascli):
|
441
|
+
|
442
|
+
```
|
443
|
+
$ cd ~
|
444
|
+
$ tar zcvf rvm_ascli.tgz .rvm
|
445
|
+
```
|
446
|
+
|
447
|
+
* retrieve the SDK:
|
448
|
+
|
449
|
+
```
|
450
|
+
$ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
451
|
+
```
|
452
|
+
|
453
|
+
* on the system without internet access:
|
454
|
+
|
455
|
+
```
|
456
|
+
$ cd ~
|
457
|
+
$ tar zxvf rvm_ascli.tgz
|
458
|
+
$ source ~/.rvm/scripts/rvm
|
459
|
+
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
460
|
+
```
|
461
|
+
|
238
462
|
# <a name="cli"></a>Command Line Interface: `ascli`
|
239
463
|
|
240
464
|
The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
@@ -415,6 +639,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
415
639
|
* a,b,c : the list of attributes specified by the comma separated list
|
416
640
|
* Array extended value: for instance, @json:'["a","b","c"]' same as above
|
417
641
|
* +a,b,c : add selected properties to the default selection.
|
642
|
+
* -a,b,c : remove selected properties from the default selection.
|
418
643
|
|
419
644
|
## <a name="extended"></a>Extended Value Syntax
|
420
645
|
|
@@ -428,25 +653,25 @@ The extended value syntax is:
|
|
428
653
|
|
429
654
|
The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
|
430
655
|
|
431
|
-
The following "readers" are supported:
|
656
|
+
The following "readers" are supported (returns value in []):
|
432
657
|
|
433
|
-
* @val:VALUE
|
434
|
-
* @file:PATH
|
435
|
-
* @path:PATH
|
436
|
-
* @env:ENVVAR
|
437
|
-
* @stdin:
|
438
|
-
* @preset:NAME
|
658
|
+
* @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
|
659
|
+
* @file:PATH : [String] read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
|
660
|
+
* @path:PATH : [String] performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
|
661
|
+
* @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
662
|
+
* @stdin: : [String] read from stdin (no value on right)
|
663
|
+
* @preset:NAME : [Hash] get whole option preset value by name
|
439
664
|
|
440
665
|
In addition it is possible to decode a value, using one or multiple decoders :
|
441
666
|
|
442
|
-
* @base64: decode a base64 encoded string
|
443
|
-
* @json: decode JSON values (convenient to provide complex structures)
|
444
|
-
* @zlib: uncompress data
|
445
|
-
* @ruby: execute ruby code
|
446
|
-
* @csvt: decode a titled CSV value
|
447
|
-
* @lines: split a string in multiple lines and return an array
|
448
|
-
* @list: split a string in multiple items taking first character as separator and return an array
|
449
|
-
* @incps: include values of presets specified by key
|
667
|
+
* @base64: [String] decode a base64 encoded string
|
668
|
+
* @json: [any] decode JSON values (convenient to provide complex structures)
|
669
|
+
* @zlib: [String] uncompress data
|
670
|
+
* @ruby: [any] execute ruby code
|
671
|
+
* @csvt: [Array] decode a titled CSV value
|
672
|
+
* @lines: [Array] split a string in multiple lines and return an array
|
673
|
+
* @list: [Array] split a string in multiple items taking first character as separator and return an array
|
674
|
+
* @incps: [Hash] include values of presets specified by key `incps` in input hash
|
450
675
|
|
451
676
|
To display the result of an extended value, use the `config echo` command.
|
452
677
|
|
@@ -478,7 +703,7 @@ $ ascli config echo @csvt:@file:test.csv
|
|
478
703
|
:......:.....................:
|
479
704
|
```
|
480
705
|
|
481
|
-
Example: create a hash and include values from preset named "config" of config file
|
706
|
+
Example: create a hash and include values from preset named "config" of config file in this hash
|
482
707
|
|
483
708
|
```
|
484
709
|
$ ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
@@ -554,7 +779,7 @@ $ ascli config id <option preset> set|delete|show|initialize|update
|
|
554
779
|
The command `update` allows the easy creation of [option preset](#lprt) by simply providing the options in their command line format, e.g. :
|
555
780
|
|
556
781
|
```
|
557
|
-
$ ascli config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
782
|
+
$ ascli config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
558
783
|
```
|
559
784
|
|
560
785
|
* This creates a [option preset](#lprt) `demo_server` with all provided options.
|
@@ -562,13 +787,13 @@ $ ascli config id demo_server update --url=ssh://demo.asperasoft.com:33001 --use
|
|
562
787
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
563
788
|
|
564
789
|
```
|
565
|
-
$ ascli config id demo_server set password
|
790
|
+
$ ascli config id demo_server set password _demo_pass_
|
566
791
|
```
|
567
792
|
|
568
793
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
569
794
|
|
570
795
|
```
|
571
|
-
$ ascli config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"
|
796
|
+
$ ascli config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
572
797
|
```
|
573
798
|
|
574
799
|
A good practice is to not manually edit the configuration file and use modification commands instead.
|
@@ -605,11 +830,19 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
605
830
|
Operations on this preset are done using regular `config` operations:
|
606
831
|
|
607
832
|
```
|
608
|
-
$ ascli config id default set _plugin_name_
|
833
|
+
$ ascli config id default set _plugin_name_ _default_preset_for_plugin_
|
609
834
|
$ ascli config id default get _plugin_name_
|
610
|
-
"
|
835
|
+
"_default_preset_for_plugin_"
|
611
836
|
```
|
612
837
|
|
838
|
+
### <a name="lprtdef"></a>Special Plugin: config
|
839
|
+
|
840
|
+
Plugin `config` (not to be confused with Option preset config) is used to configure `ascli` but it also contains global options.
|
841
|
+
|
842
|
+
When `ascli` starts, it lookjs for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
843
|
+
|
844
|
+
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
845
|
+
|
613
846
|
### Format of file
|
614
847
|
|
615
848
|
The configuration file is a hash in a YAML file. Example:
|
@@ -625,7 +858,7 @@ cli_default:
|
|
625
858
|
demo_server:
|
626
859
|
url: ssh://demo.asperasoft.com:33001
|
627
860
|
username: asperaweb
|
628
|
-
password:
|
861
|
+
password: _demo_pass_
|
629
862
|
```
|
630
863
|
|
631
864
|
We can see here:
|
@@ -694,7 +927,6 @@ A [option preset](#lprt) value can be removed with `unset`:
|
|
694
927
|
$ ascli config id cli_default unset interactive
|
695
928
|
```
|
696
929
|
|
697
|
-
|
698
930
|
### Examples
|
699
931
|
|
700
932
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
@@ -814,21 +1046,42 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
814
1046
|
|
815
1047
|
```
|
816
1048
|
$ ascli config ascp show
|
817
|
-
/Users/laurent/
|
1049
|
+
/Users/laurent/.aspera/ascli/sdk/ascp
|
1050
|
+
$ ascli config ascp info
|
1051
|
+
+--------------------+-----------------------------------------------------------+
|
1052
|
+
| key | value |
|
1053
|
+
+--------------------+-----------------------------------------------------------+
|
1054
|
+
| ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
|
1055
|
+
...
|
818
1056
|
```
|
819
1057
|
|
820
1058
|
### Selection of local `ascp`
|
821
1059
|
|
1060
|
+
By default, `ascli` uses any found local product with ascp, including SDK.
|
1061
|
+
|
822
1062
|
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
823
1063
|
|
824
|
-
|
1064
|
+
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
1065
|
+
|
1066
|
+
Using a POSIX shell:
|
825
1067
|
|
826
1068
|
```
|
827
1069
|
$ ascli config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
828
|
-
|
1070
|
+
ascp version: 4.0.0.182279
|
1071
|
+
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
1072
|
+
Saved to default global preset global_common_defaults
|
1073
|
+
```
|
1074
|
+
|
1075
|
+
Windows:
|
1076
|
+
|
1077
|
+
```
|
1078
|
+
$ ascli config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1079
|
+
ascp version: 4.0.0.182279
|
1080
|
+
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1081
|
+
Saved to default global preset global_common_defaults
|
829
1082
|
```
|
830
1083
|
|
831
|
-
|
1084
|
+
If the path has spaces, read section: [Shell and Command line parsing](#parsing).
|
832
1085
|
|
833
1086
|
### List locally installed Aspera Transfer products
|
834
1087
|
|
@@ -915,7 +1168,7 @@ will effectively push files to the related server from the agent node.
|
|
915
1168
|
|
916
1169
|
### <a name="direct"></a>Direct (local ascp using FASPManager API)
|
917
1170
|
|
918
|
-
By default
|
1171
|
+
By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
|
919
1172
|
`ascli` will detect locally installed Aspera products.
|
920
1173
|
Refer to section [FASP](#client).
|
921
1174
|
|
@@ -925,17 +1178,33 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
|
|
925
1178
|
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
926
1179
|
* `EX_ascp_args`
|
927
1180
|
|
928
|
-
The `transfer-info`
|
1181
|
+
The `transfer-info` accepts the following optional parameters:
|
1182
|
+
|
1183
|
+
<table>
|
1184
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1185
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
|
1186
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
|
1187
|
+
<tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
1188
|
+
<tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
|
1189
|
+
</table>
|
1190
|
+
|
1191
|
+
Resume parameters:
|
929
1192
|
|
930
1193
|
<table>
|
931
|
-
<tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
932
|
-
<tr><td>iter_max</td
|
933
|
-
<tr><td>sleep_initial</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
934
|
-
<tr><td>sleep_factor</td>
|
935
|
-
<tr><td>sleep_max</td
|
936
|
-
<tr><td>wss</td> <td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
1194
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1195
|
+
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
1196
|
+
<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
1197
|
+
<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
|
1198
|
+
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
937
1199
|
</table>
|
938
1200
|
|
1201
|
+
Examples:
|
1202
|
+
|
1203
|
+
```
|
1204
|
+
$ ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
1205
|
+
$ ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
|
1206
|
+
```
|
1207
|
+
|
939
1208
|
### IBM Aspera Connect Client GUI
|
940
1209
|
|
941
1210
|
By specifying option: `--transfer=connect`, `ascli` will start transfers
|
@@ -967,9 +1236,11 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
967
1236
|
Example:
|
968
1237
|
|
969
1238
|
```
|
970
|
-
$ ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://
|
1239
|
+
$ ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
971
1240
|
```
|
972
1241
|
|
1242
|
+
Note that the gateway only supports transfers authorized with a token.
|
1243
|
+
|
973
1244
|
## <a name="transferspec"></a>Transfer Specification
|
974
1245
|
|
975
1246
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
@@ -983,7 +1254,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
983
1254
|
|
984
1255
|
`ascli` builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
985
1256
|
|
986
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
|
1257
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
987
1258
|
|
988
1259
|
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
989
1260
|
|
@@ -996,97 +1267,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
|
|
996
1267
|
|
997
1268
|
## <a name="transferparams"></a>Transfer Parameters
|
998
1269
|
|
999
|
-
All standard _transfer-spec_ parameters can be
|
1000
|
-
|
1001
|
-
|
1270
|
+
All standard _transfer-spec_ parameters can be speficied.
|
1271
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
1272
|
+
|
1273
|
+
References:
|
1274
|
+
|
1275
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
1276
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
1002
1277
|
|
1278
|
+
Parameters can be displayed with commands:
|
1279
|
+
|
1280
|
+
```
|
1281
|
+
$ ascli config ascp spec
|
1282
|
+
$ ascli config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
|
1283
|
+
```
|
1003
1284
|
|
1004
|
-
<p>
|
1005
1285
|
Columns:
|
1006
|
-
<ul>
|
1007
|
-
<li>F=Fasp Manager(local FASP execution)</li>
|
1008
|
-
<li>N=remote node(node API)</li>
|
1009
|
-
<li>C=Connect Client(web plugin)</li>
|
1010
|
-
</ul>
|
1011
|
-
</p>
|
1012
|
-
<p>
|
1013
|
-
Req/Def : Required or default value (- means emty)
|
1014
|
-
</p>
|
1015
|
-
<p>
|
1016
|
-
Fields with EX_ prefix are specific extensions to local mode.
|
1017
|
-
</p>
|
1018
|
-
<p>
|
1019
|
-
arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)
|
1020
|
-
</p>
|
1021
|
-
<p>
|
1022
|
-
UNDER CONSTRUCTION<br/>
|
1023
|
-
<a href="https://developer.ibm.com/api/view/aspera-prod:ibm-aspera:title-IBM_Aspera#id90944">Documentation→Node API→/opt/transfers</a><br/>
|
1024
|
-
</p>
|
1025
1286
|
|
1026
|
-
|
1027
|
-
|
1028
|
-
|
1029
|
-
|
1030
|
-
<tr><td>remote_user</td><td>Required</td><td>string</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--user</td><td>Remote user. Default value is "xfer" on node or connect.</td></tr>
|
1031
|
-
<tr><td>destination_root</td><td>Required</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>last arg</td><td>Destination root directory.</td></tr>
|
1032
|
-
<tr><td>title</td><td>-</td><td>string</td><td class="no">N</td><td class="yes">Y</td><td class="yes">Y</td><td>-</td><td>Title of the transfer</td></tr>
|
1033
|
-
<tr><td>tags</td><td>-</td><td>hash</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--tags<br>--tags64</td><td>Metadata for transfer</td></tr>
|
1034
|
-
<tr><td>token</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>TOKEN<br/>-W</td><td>Authorization token: Bearer, Basic or ATM</td></tr>
|
1035
|
-
<tr><td>cookie</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>COOKIE</td><td>Metadata for transfer (older,string)</td></tr>
|
1036
|
-
<tr><td>remote_access_key</td><td>TODO</td><td>string</td><td></td><td></td><td></td><td>?</td><td>Node only?</td></tr>
|
1037
|
-
<tr><td>source_root</td><td>-</td><td>string</td><td></td><td></td><td></td><td>--source-prefix<br/>--source-prefix64</td><td>Source root directory.(TODO: verify option)</td></tr>
|
1038
|
-
<tr><td>fasp_port</td><td>33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-O</td><td>Specifies fasp (UDP) port.</td></tr>
|
1039
|
-
<tr><td>ssh_port</td><td>22 or 33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-P</td><td>Specifies ssh (TCP) port.</td></tr>
|
1040
|
-
<tr><td>rate_policy</td><td>server config</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--policy</td><td>Valid literals include "low","fair","high" and "fixed".</td></tr>
|
1041
|
-
<tr><td>symlink_policy</td><td>follow</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--symbolic-links</td><td>copy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).</td></tr>
|
1042
|
-
<tr><td>target_rate_kbps</td><td>-</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-l</td><td>Specifies desired speed for the transfer.</td></tr>
|
1043
|
-
<tr><td>min_rate_kbps</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-m</td><td>Set the minimum transfer rate in kilobits per second.</td></tr>
|
1044
|
-
<tr><td>cipher</td><td>none</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-c</td><td>in transit encryption type.<br/>none, aes-128, aes-256</td></tr>
|
1045
|
-
<tr><td>content_protection</td><td>encrypt<br/>decrypt</td><td>string</td><td></td><td></td><td></td><td>--file-crypt=</td><td>encryption at rest</td></tr>
|
1046
|
-
<tr><td>content_protection_password</td><td>-</td><td>string</td><td></td><td></td><td></td><td>PASS</td><td>Specifies a string password.</td></tr>
|
1047
|
-
<tr><td>overwrite</td><td>diff</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--overwrite</td><td>Overwrite destination files with the source files of the same name.<br/>never, always, diff, older, or diff+older</td></tr>
|
1048
|
-
<tr><td>retry_duration</td><td></td><td>string</td><td></td><td></td><td></td><td>TODO</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr>
|
1049
|
-
<tr><td>http_fallback</td><td></td><td>bool (node), integer</td><td></td><td></td><td></td><td>-y<br/>TODO</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.</td></tr>
|
1050
|
-
<tr><td>create_dir</td><td></td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-d</td><td>Specifies whether to create new directories.</td></tr>
|
1051
|
-
<tr><td>precalculate_job_size</td><td>srv. def.</td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--precalculate-job-size</td><td>Specifies whether to precalculate the job size.</td></tr>
|
1052
|
-
<tr><td>delete_source</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>?</td></tr>
|
1053
|
-
<tr><td>remove_after_transfer</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove file after transfer.</td></tr>
|
1054
|
-
<tr><td>remove_empty_directories</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove empty directories.</td></tr>
|
1055
|
-
<tr><td>multi_session</td><td>1</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-C</td><td>Specifies how many parts the transfer is in.</td></tr>
|
1056
|
-
<tr><td>multi_session_threshold</td><td>null</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-</td><td>in bytes</td></tr>
|
1057
|
-
<tr><td>exclude_newer_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-newer-than</td><td>-</td></tr>
|
1058
|
-
<tr><td>exclude_older_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-older-than</td><td>-</td></tr>
|
1059
|
-
<tr><td>preserve_acls</td><td></td><td>string</td><td class="yes">Y</td><td></td><td></td><td>--preserve-acls</td><td>-</td></tr>
|
1060
|
-
<tr><td>dgram_size</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-Z</td><td>in bytes</td></tr>
|
1061
|
-
<tr><td>compression</td><td></td><td>integer</td><td></td><td></td><td></td><td></td><td>ascp4 only, 0 / 1?</td></tr>
|
1062
|
-
<tr><td>read_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
1063
|
-
<tr><td>write_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
1064
|
-
<tr><td>use_ascp4</td><td>false</td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>-</td><td>specify version of protocol</td></tr>
|
1065
|
-
<tr><td>paths</td><td>source files (dest)</td><td>array</td><td></td><td></td><td></td><td>positional<br/>--file-list<br/>--file-pair-list</td><td>Contains a path to the source (required) and a path to the destination.</td></tr>
|
1066
|
-
<tr><td>http_fallback_port</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-t</td><td>Specifies http port.</td></tr>
|
1067
|
-
<tr><td>https_fallback_port</td><td></td><td>integer</td><td></td><td></td><td></td><td>todo</td><td>Specifies https port.</td></tr>
|
1068
|
-
<tr><td>cipher_allowed</td><td></td><td>string</td><td></td><td></td><td></td><td>-</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr>
|
1069
|
-
<tr><td>target_rate_cap_kbps</td><td></td><td></td><td class="no">N</td><td class="no">?</td><td class="yes">?</td><td>-</td><td>Returned by upload/download_setup node api.</td></tr>
|
1070
|
-
<tr><td>rate_policy_allowed</td><td></td><td></td><td></td><td></td><td></td><td>-</td><td>returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".</td></tr>
|
1071
|
-
<tr><td>ssh_private_key</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>KEY</td><td>Private key used for SSH authentication, Shall look like: `-----BEGIN RSA PRIVATE KEY-----\nMII`<br/>Note the JSON encoding `\` + `n` for newlines.</td></tr>
|
1072
|
-
<tr><td>remote_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>PASS</td><td>SSH session password</td></tr>
|
1073
|
-
<tr><td>resume_policy</td><td>faspmgr:<br/>none<br/>other:<br/>sparse_csum</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-k</td><td>none,attrs,sparse_csum,full_csum</td></tr>
|
1074
|
-
<tr><td>authentication</td><td>-</td><td class="no">N</td><td class="no">N</td><td class="yes">Y</td><td>-</td><td>token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.</td></tr>
|
1075
|
-
<tr><td>EX_ssh_key_paths</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-i</td><td>Use public key authentication and specify the private key file</td></tr>
|
1076
|
-
<tr><td>EX_at_rest_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>FILEPASS</td><td>Passphrase used for at rest encryption or decryption</td></tr>
|
1077
|
-
<tr><td>EX_proxy_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>PROXY_PASS</td><td>TODO</td></tr>
|
1078
|
-
<tr><td>EX_fasp_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--proxy</td><td>Specify the address of the Aspera high-speed proxy server</td></tr>
|
1079
|
-
<tr><td>EX_http_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-x</td><td>Specify the proxy server address used by HTTP Fallback</td></tr>
|
1080
|
-
<tr><td>EX_ascp_args</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>same</td><td>Add command line arguments to ascp</td></tr>
|
1081
|
-
<tr><td>EX_http_transfer_jpeg</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-j</td><td>HTTP transfers as JPEG file</td></tr>
|
1082
|
-
<tr><td>EX_license_text</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>LICENSE</td><td>license file text</td></tr>
|
1083
|
-
<tr><td>EX_file_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-list</td><td>source file list</td></tr>
|
1084
|
-
<tr><td>EX_file_pair_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-pair-list</td><td>source file pair list</td></tr>
|
1085
|
-
<tr><td>EX_multi_session_part</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-C</td><td>part for multisession</td></tr>
|
1086
|
-
<tr><td>EX_no_read</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-read</td><td>no read source</td></tr>
|
1087
|
-
<tr><td>EX_no_write</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-write</td><td>no write estination</td></tr>
|
1088
|
-
</table>
|
1287
|
+
* D=Direct (local `ascp` execution)
|
1288
|
+
* N=Node API
|
1289
|
+
* C=Connect Client
|
1290
|
+
* arg=`ascp` argument or environment variable
|
1089
1291
|
|
1292
|
+
Fields with EX_ prefix are extensions to transfer agent `direct`. (only in `ascli`).
|
1293
|
+
|
1294
|
+
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
1090
1295
|
|
1091
1296
|
### Destination folder for transfers
|
1092
1297
|
|
@@ -1102,8 +1307,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1102
1307
|
|
1103
1308
|
### List of files for transfers
|
1104
1309
|
|
1105
|
-
When uploading, downloading or sending files, the user must specify
|
1106
|
-
|
1310
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
1311
|
+
Most of the time, the list of files to transfer will be simply specified on the command line:
|
1107
1312
|
|
1108
1313
|
```
|
1109
1314
|
$ ascli server upload ~/mysample.file secondfile
|
@@ -1117,7 +1322,8 @@ $ ascli server upload --sources=@args ~/mysample.file secondfile
|
|
1117
1322
|
|
1118
1323
|
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1119
1324
|
|
1120
|
-
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
1325
|
+
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
1326
|
+
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
1121
1327
|
|
1122
1328
|
For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. Accepted values are:
|
1123
1329
|
|
@@ -1167,7 +1373,7 @@ Source files are located on "Aspera on cloud", when :
|
|
1167
1373
|
|
1168
1374
|
### <a name="multisession"></a>Support of multi-session
|
1169
1375
|
|
1170
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
1376
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
1171
1377
|
|
1172
1378
|
* when agent=node :
|
1173
1379
|
|
@@ -1188,6 +1394,7 @@ shall be preferred.
|
|
1188
1394
|
|
1189
1395
|
Multi-session spawn is done by `ascli`.
|
1190
1396
|
|
1397
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
1191
1398
|
|
1192
1399
|
### Examples
|
1193
1400
|
|
@@ -1234,7 +1441,9 @@ A non complete list of commands used in unit tests:
|
|
1234
1441
|
```
|
1235
1442
|
ascli
|
1236
1443
|
ascli -h
|
1237
|
-
ascli aoc
|
1444
|
+
ascli aoc -N remind --username=my_aoc_user_email
|
1445
|
+
ascli aoc -N servers
|
1446
|
+
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
1238
1447
|
ascli aoc admin ats access_key --id=akibmcloud --secret=somesecret node browse /
|
1239
1448
|
ascli aoc admin ats access_key --id=akibmcloud delete
|
1240
1449
|
ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
@@ -1247,6 +1456,8 @@ ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
|
|
1247
1456
|
ascli aoc admin res apps_new list
|
1248
1457
|
ascli aoc admin res client list
|
1249
1458
|
ascli aoc admin res client_access_key list
|
1459
|
+
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
1460
|
+
ascli aoc admin res client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}'
|
1250
1461
|
ascli aoc admin res client_registration_token list
|
1251
1462
|
ascli aoc admin res contact list
|
1252
1463
|
ascli aoc admin res dropbox list
|
@@ -1270,7 +1481,6 @@ ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4
|
|
1270
1481
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 mkdir /folder1
|
1271
1482
|
ascli aoc admin resource workspace list
|
1272
1483
|
ascli aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
1273
|
-
ascli aoc apiinfo
|
1274
1484
|
ascli aoc automation workflow --id="my_wf_id" action create --value=@json:'{"name":"toto"}' | tee action.info
|
1275
1485
|
ascli aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
1276
1486
|
ascli aoc automation workflow delete --id="my_wf_id"
|
@@ -1284,7 +1494,7 @@ ascli aoc files browse /
|
|
1284
1494
|
ascli aoc files browse / -N --link=my_aoc_publink_folder
|
1285
1495
|
ascli aoc files delete /testsrc
|
1286
1496
|
ascli aoc files download --transfer=connect /200KB.1
|
1287
|
-
ascli aoc files file
|
1497
|
+
ascli aoc files file --id=my_file_id show
|
1288
1498
|
ascli aoc files find / --value='\.partial$'
|
1289
1499
|
ascli aoc files http_node_download --to-folder=. /200KB.1
|
1290
1500
|
ascli aoc files mkdir /testsrc
|
@@ -1305,7 +1515,7 @@ ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
|
|
1305
1515
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
1306
1516
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["internal.user@example.com"],"note":"my note"}' testfile.bin
|
1307
1517
|
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
1308
|
-
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user
|
1518
|
+
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
1309
1519
|
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
1310
1520
|
ascli aoc user info modify @json:'{"name":"dummy change"}'
|
1311
1521
|
ascli aoc user info show
|
@@ -1325,8 +1535,8 @@ ascli ats cluster list
|
|
1325
1535
|
ascli ats cluster show --cloud=aws --region=eu-west-1
|
1326
1536
|
ascli ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
|
1327
1537
|
ascli conf flush_tokens
|
1328
|
-
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=
|
1329
|
-
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=
|
1538
|
+
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes
|
1539
|
+
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes --use-generic-client=yes
|
1330
1540
|
ascli config ascp connect id 'Aspera Connect for Windows' info
|
1331
1541
|
ascli config ascp connect id 'Aspera Connect for Windows' links id 'Windows Installer' download --to-folder=.
|
1332
1542
|
ascli config ascp connect id 'Aspera Connect for Windows' links list
|
@@ -1335,35 +1545,39 @@ ascli config ascp info
|
|
1335
1545
|
ascli config ascp install
|
1336
1546
|
ascli config ascp products list
|
1337
1547
|
ascli config ascp show
|
1338
|
-
ascli config
|
1548
|
+
ascli config check_update
|
1549
|
+
ascli config doc
|
1550
|
+
ascli config doc transfer-parameters
|
1551
|
+
ascli config email_test --notif-to=my_recipient_email
|
1339
1552
|
ascli config export
|
1340
1553
|
ascli config genkey mykey
|
1341
1554
|
ascli config plugins
|
1342
1555
|
ascli config proxy_check --fpac=file:///examples/proxy.pac https://eudemo.asperademo.com
|
1343
1556
|
ascli console transfer current list
|
1344
1557
|
ascli console transfer smart list
|
1345
|
-
ascli console transfer smart sub
|
1558
|
+
ascli console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
1346
1559
|
ascli cos -N --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
1347
1560
|
ascli cos -N --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
1348
1561
|
ascli cos node access_key --id=self show
|
1349
1562
|
ascli cos node download testfile.bin --to-folder=.
|
1350
1563
|
ascli cos node info
|
1351
1564
|
ascli cos node upload testfile.bin
|
1352
|
-
ascli faspex
|
1565
|
+
ascli faspex health
|
1353
1566
|
ascli faspex package list
|
1354
1567
|
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data|tail -n 1);\
|
1355
|
-
ascli faspex package
|
1568
|
+
ascli faspex package list --fields=package_id --format=csv --display=data|tail -n 1);\
|
1569
|
+
ascli faspex package recv --to-folder=. --box=sent --id="my_package_id"
|
1356
1570
|
ascli faspex package recv --to-folder=. --id="my_package_id"
|
1357
1571
|
ascli faspex package recv --to-folder=. --id=ALL --once-only=yes
|
1358
1572
|
ascli faspex package recv --to-folder=. --link="my_faspex_publink_recv_from_fxuser"
|
1359
|
-
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com"]}' testfile.bin
|
1573
|
+
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com","FASPEX_USERNAME"]}' testfile.bin
|
1360
1574
|
ascli faspex package send --link="my_faspex_publink_send_to_dropbox" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
1361
1575
|
ascli faspex package send --link="my_faspex_publink_send_to_fxuser" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
1362
1576
|
ascli faspex source name "Server Files" node br /
|
1363
1577
|
ascli faspex5 node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
1364
|
-
ascli faspex5 package list --value=@json:'{"state":["released"]}'
|
1578
|
+
ascli faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
1365
1579
|
ascli faspex5 package receive --id="my_package_id" --to-folder=.
|
1366
|
-
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":["
|
1580
|
+
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"${f5_user}"}]}' testfile.bin
|
1367
1581
|
ascli node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
1368
1582
|
ascli node -N -Ptst_node_preview access_key delete --id=aoc_1
|
1369
1583
|
ascli node async --id=1 bandwidth
|
@@ -1377,8 +1591,8 @@ ascli node browse / -r
|
|
1377
1591
|
ascli node delete folder_1/10MB.1
|
1378
1592
|
ascli node delete folder_1/testfile.bin
|
1379
1593
|
ascli node download --to-folder=. folder_1/testfile.bin
|
1594
|
+
ascli node health
|
1380
1595
|
ascli node info
|
1381
|
-
ascli node nagios_check
|
1382
1596
|
ascli node search / --value=@json:'{"sort":"mtime"}'
|
1383
1597
|
ascli node service --id=service1 delete
|
1384
1598
|
ascli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -1408,7 +1622,8 @@ ascli preview test --case=test png "TSTFILE_MXF" --video-png-conv=fixed --log-le
|
|
1408
1622
|
ascli preview test --case=test png "TSTFILE_PDF" --log-level=debug
|
1409
1623
|
ascli preview trevents --once-only=yes --skip-types=office --log-level=info
|
1410
1624
|
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user ctl all:status
|
1411
|
-
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user
|
1625
|
+
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health app_services --format=nagios
|
1626
|
+
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health asctlstatus --format=nagios --cmd-prefix='sudo '
|
1412
1627
|
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user nodeadmin -- -l
|
1413
1628
|
ascli server -N -Ptst_server_bykey -Plocal_user br /
|
1414
1629
|
ascli server browse /
|
@@ -1418,24 +1633,27 @@ ascli server delete NEW_SERVER_FOLDER
|
|
1418
1633
|
ascli server delete folder_1/target_hot
|
1419
1634
|
ascli server delete folder_1/to.delete
|
1420
1635
|
ascli server df
|
1421
|
-
ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=.
|
1636
|
+
ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=. --transfer-info=@json:'{"wss":false,"resume":{"iter_max":1}}'
|
1422
1637
|
ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=folder_1 --transfer=node
|
1423
1638
|
ascli server du /
|
1639
|
+
ascli server health transfer --to-folder=folder_1 --format=nagios
|
1424
1640
|
ascli server info
|
1425
1641
|
ascli server md5sum NEW_SERVER_FOLDER/testfile.bin
|
1426
1642
|
ascli server mkdir NEW_SERVER_FOLDER --logger=stdout
|
1427
1643
|
ascli server mkdir folder_1/target_hot
|
1428
1644
|
ascli server mv folder_1/200KB.2 folder_1/to.delete
|
1429
|
-
ascli server nagios transfer --to-folder=folder_1 --format=nagios
|
1430
1645
|
ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
1431
1646
|
ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
1432
|
-
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername
|
1647
|
+
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
1433
1648
|
ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
1434
|
-
ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER
|
1649
|
+
ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
1650
|
+
ascli shares admin share list
|
1435
1651
|
ascli shares repository browse /
|
1436
1652
|
ascli shares repository delete /SHARES_UPLOAD/testfile.bin
|
1437
1653
|
ascli shares repository download --to-folder=. /SHARES_UPLOAD/testfile.bin
|
1654
|
+
ascli shares repository download --to-folder=. /SHARES_UPLOAD/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://HTTP_GW_FQDN/aspera/http-gwy/v1"}'
|
1438
1655
|
ascli shares repository upload --to-folder=/SHARES_UPLOAD testfile.bin
|
1656
|
+
ascli shares repository upload --to-folder=/SHARES_UPLOAD testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://HTTP_GW_FQDN/aspera/http-gwy/v1"}'
|
1439
1657
|
ascli shares2 appinfo
|
1440
1658
|
ascli shares2 organization list
|
1441
1659
|
ascli shares2 project list --organization=Sport
|
@@ -1451,7 +1669,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
1451
1669
|
```
|
1452
1670
|
$ ascli -h
|
1453
1671
|
NAME
|
1454
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
1672
|
+
ascli -- a command line tool for Aspera Applications (v4.2.2)
|
1455
1673
|
|
1456
1674
|
SYNOPSIS
|
1457
1675
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -1462,6 +1680,10 @@ DESCRIPTION
|
|
1462
1680
|
execute: ascli conf doc
|
1463
1681
|
or visit: http://www.rubydoc.info/gems/aspera-cli
|
1464
1682
|
|
1683
|
+
ENVIRONMENT VARIABLES
|
1684
|
+
ASCLI_HOME config folder, default: $HOME/.aspera/ascli
|
1685
|
+
#any option can be set as an environment variable, refer to the manual
|
1686
|
+
|
1465
1687
|
COMMANDS
|
1466
1688
|
To list first level commands, execute: ascli
|
1467
1689
|
Note that commands can be written shortened (provided it is unique).
|
@@ -1490,7 +1712,7 @@ OPTIONS: global
|
|
1490
1712
|
-v, --version display version
|
1491
1713
|
-w, --warnings check for language warnings
|
1492
1714
|
--ui=ENUM method to start browser: text, graphical
|
1493
|
-
--log-level=ENUM Log level: debug, info,
|
1715
|
+
--log-level=ENUM Log level: debug, info, warn, error, fatal, unknown
|
1494
1716
|
--logger=ENUM log method: stderr, stdout, syslog
|
1495
1717
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
1496
1718
|
--query=VALUE additional filter for API calls (extended value) (some commands)
|
@@ -1498,7 +1720,7 @@ OPTIONS: global
|
|
1498
1720
|
--once-only=ENUM process only new items (some commands): yes, no
|
1499
1721
|
|
1500
1722
|
COMMAND: config
|
1501
|
-
SUBCOMMANDS: gem_path genkey plugins flush_tokens list overview open echo id documentation wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file
|
1723
|
+
SUBCOMMANDS: gem_path genkey plugins flush_tokens list overview open echo id documentation wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update initdemo
|
1502
1724
|
OPTIONS:
|
1503
1725
|
--value=VALUE extended value for create, update, list filter
|
1504
1726
|
--property=VALUE name of property to set
|
@@ -1514,10 +1736,15 @@ OPTIONS:
|
|
1514
1736
|
--fpac=VALUE proxy auto configuration URL
|
1515
1737
|
-P, --presetVALUE load the named option preset from current config file
|
1516
1738
|
--default=VALUE set as default configuration for specified plugin
|
1517
|
-
--secret=VALUE
|
1518
|
-
--secrets=VALUE
|
1739
|
+
--secret=VALUE default secret
|
1740
|
+
--secrets=VALUE secret repository (Hash)
|
1741
|
+
--sdk-url=VALUE URL to get SDK
|
1742
|
+
--sdk-folder=VALUE SDK folder location
|
1743
|
+
--notif-to=VALUE email recipient for notification of transfers
|
1744
|
+
--notif-template=VALUE email ERB template for notification of transfers
|
1519
1745
|
--test-mode=ENUM skip user validation in wizard mode: yes, no
|
1520
|
-
--
|
1746
|
+
--version-check-days=VALUE period to check neew version in days (zero to disable)
|
1747
|
+
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
1521
1748
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
1522
1749
|
--to-folder=VALUE destination folder for downloaded files
|
1523
1750
|
--sources=VALUE list of source files (see doc)
|
@@ -1536,7 +1763,7 @@ OPTIONS:
|
|
1536
1763
|
|
1537
1764
|
|
1538
1765
|
COMMAND: node
|
1539
|
-
SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details
|
1766
|
+
SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details health events space info license mkdir mklink mkfile rename delete search
|
1540
1767
|
OPTIONS:
|
1541
1768
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1542
1769
|
--username=VALUE username to log in
|
@@ -1589,11 +1816,16 @@ OPTIONS:
|
|
1589
1816
|
|
1590
1817
|
|
1591
1818
|
COMMAND: faspex5
|
1592
|
-
SUBCOMMANDS: node package
|
1819
|
+
SUBCOMMANDS: node package auth_client jobs
|
1593
1820
|
OPTIONS:
|
1594
1821
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1595
1822
|
--username=VALUE username to log in
|
1596
1823
|
--password=VALUE user's password
|
1824
|
+
--client-id=VALUE API client identifier in application
|
1825
|
+
--client-secret=VALUE API client secret in application
|
1826
|
+
--redirect-uri=VALUE API client redirect URI
|
1827
|
+
--auth=ENUM type of Oauth authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey, boot
|
1828
|
+
--private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
|
1597
1829
|
|
1598
1830
|
|
1599
1831
|
COMMAND: cos
|
@@ -1608,7 +1840,7 @@ OPTIONS:
|
|
1608
1840
|
|
1609
1841
|
|
1610
1842
|
COMMAND: faspex
|
1611
|
-
SUBCOMMANDS:
|
1843
|
+
SUBCOMMANDS: health package source me dropbox v4 address_book login_methods
|
1612
1844
|
OPTIONS:
|
1613
1845
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1614
1846
|
--username=VALUE username to log in
|
@@ -1617,7 +1849,8 @@ OPTIONS:
|
|
1617
1849
|
--delivery-info=VALUE package delivery information (extended value)
|
1618
1850
|
--source-name=VALUE create package from remote source (by name)
|
1619
1851
|
--storage=VALUE Faspex local storage definition
|
1620
|
-
--
|
1852
|
+
--recipient=VALUE use if recipient is a dropbox (with *)
|
1853
|
+
--box=ENUM package box: inbox, archive, sent
|
1621
1854
|
|
1622
1855
|
|
1623
1856
|
COMMAND: shares2
|
@@ -1646,6 +1879,7 @@ OPTIONS:
|
|
1646
1879
|
--case=VALUE basename of output for for test
|
1647
1880
|
--scan-path=VALUE subpath in folder id to start scan in (default=/)
|
1648
1881
|
--scan-id=VALUE forder id in storage to start scan in, default is access key main folder id
|
1882
|
+
--mimemagic=ENUM use Mime type detection of gem mimemagic: yes, no
|
1649
1883
|
--overwrite=ENUM when to overwrite result file: always, never, mtime
|
1650
1884
|
--file-access=ENUM how to read and write files in repository: local, remote
|
1651
1885
|
--max-size=VALUE maximum size (in bytes) of preview file
|
@@ -1672,42 +1906,30 @@ OPTIONS:
|
|
1672
1906
|
|
1673
1907
|
|
1674
1908
|
COMMAND: aoc
|
1675
|
-
SUBCOMMANDS:
|
1909
|
+
SUBCOMMANDS: reminder bearer_token organization tier_restrictions user workspace packages files gateway admin automation servers
|
1676
1910
|
OPTIONS:
|
1677
1911
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1678
1912
|
--username=VALUE username to log in
|
1679
1913
|
--password=VALUE user's password
|
1680
|
-
|
1681
|
-
COMMAND: ats
|
1682
|
-
SUBCOMMANDS: cluster access_key api_key aws_trust_policy
|
1683
|
-
OPTIONS:
|
1684
|
-
--ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
|
1685
|
-
--instance=VALUE ATS instance in ibm cloud
|
1686
|
-
--ats-key=VALUE ATS key identifier (ats_xxx)
|
1687
|
-
--ats-secret=VALUE ATS key secret
|
1688
|
-
--params=VALUE Parameters access key creation (@json:)
|
1689
|
-
--cloud=VALUE Cloud provider
|
1690
|
-
--region=VALUE Cloud region
|
1691
|
-
--auth=ENUM type of Oauth authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey
|
1914
|
+
--auth=ENUM OAuth type of authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey
|
1692
1915
|
--operation=ENUM client operation for transfers: push, pull
|
1693
|
-
--client-id=VALUE API client identifier in application
|
1694
|
-
--client-secret=VALUE API client passcode
|
1695
|
-
--redirect-uri=VALUE API client redirect URI
|
1696
|
-
--private-key=VALUE RSA private key PEM value
|
1916
|
+
--client-id=VALUE OAuth API client identifier in application
|
1917
|
+
--client-secret=VALUE OAuth API client passcode
|
1918
|
+
--redirect-uri=VALUE OAuth API client redirect URI
|
1919
|
+
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @val:@file:)
|
1697
1920
|
--workspace=VALUE name of workspace
|
1698
|
-
--eid=VALUE identifier
|
1699
1921
|
--name=VALUE resource name
|
1922
|
+
--path=VALUE file or folder path
|
1700
1923
|
--link=VALUE public link to shared resource
|
1701
1924
|
--new-user-option=VALUE new user creation option
|
1702
1925
|
--from-folder=VALUE share to share source folder
|
1703
|
-
--scope=VALUE scope for AoC API calls
|
1704
|
-
--notify=VALUE notify users that file was received
|
1926
|
+
--scope=VALUE OAuth scope for AoC API calls
|
1705
1927
|
--bulk=ENUM bulk operation: yes, no
|
1706
1928
|
--default-ports=ENUM use standard FASP ports or get from node api: yes, no
|
1707
1929
|
|
1708
1930
|
|
1709
1931
|
COMMAND: server
|
1710
|
-
SUBCOMMANDS:
|
1932
|
+
SUBCOMMANDS: health nodeadmin userdata configurator ctl download upload browse delete rename ls rm mv du info mkdir cp df md5sum
|
1711
1933
|
OPTIONS:
|
1712
1934
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1713
1935
|
--username=VALUE username to log in
|
@@ -1718,7 +1940,7 @@ OPTIONS:
|
|
1718
1940
|
|
1719
1941
|
|
1720
1942
|
COMMAND: console
|
1721
|
-
SUBCOMMANDS: transfer
|
1943
|
+
SUBCOMMANDS: transfer health
|
1722
1944
|
OPTIONS:
|
1723
1945
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1724
1946
|
--username=VALUE username to log in
|
@@ -1845,7 +2067,7 @@ updated: my_aoc_org
|
|
1845
2067
|
Define this [option preset](#lprt) as default configuration for the `aspera` plugin:
|
1846
2068
|
|
1847
2069
|
```
|
1848
|
-
$ ascli config id default set
|
2070
|
+
$ ascli config id default set aoc my_aoc_org
|
1849
2071
|
```
|
1850
2072
|
|
1851
2073
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
@@ -1948,7 +2170,7 @@ To activate default use of JWT authentication for `ascli` using the [option pres
|
|
1948
2170
|
|
1949
2171
|
* change auth method to JWT
|
1950
2172
|
* provide location of private key
|
1951
|
-
* provide username to login as (
|
2173
|
+
* provide username to login as (OAuth "subject")
|
1952
2174
|
|
1953
2175
|
Execute:
|
1954
2176
|
|
@@ -2219,7 +2441,7 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
2219
2441
|
Then, transfer between those:
|
2220
2442
|
|
2221
2443
|
```
|
2222
|
-
$ ascli -Paoc_show
|
2444
|
+
$ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
2223
2445
|
```
|
2224
2446
|
|
2225
2447
|
* create registration key to register a node
|
@@ -2299,10 +2521,10 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
2299
2521
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
2300
2522
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
2301
2523
|
|
2302
|
-
Typically, one would regularly execute this command on a regular basis, using the method
|
2524
|
+
Typically, one would regularly execute this command on a regular basis, using the method of your choice:
|
2303
2525
|
|
2304
|
-
* Windows scheduler
|
2305
|
-
* cron
|
2526
|
+
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2527
|
+
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2306
2528
|
* etc...
|
2307
2529
|
|
2308
2530
|
## Download Files
|
@@ -2359,19 +2581,40 @@ The activity app can be queried with:
|
|
2359
2581
|
$ ascli aoc admin analytics transfers
|
2360
2582
|
```
|
2361
2583
|
|
2362
|
-
It can also support filters and send notification
|
2584
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
2585
|
+
|
2586
|
+
`mytemplate.erb`:
|
2587
|
+
|
2588
|
+
```
|
2589
|
+
From: <%=from_name%> <<%=from_email%>>
|
2590
|
+
To: <<%=ev['user_email']%>>
|
2591
|
+
Subject: <%=ev['files_completed']%> files received
|
2592
|
+
|
2593
|
+
Dear <%=ev[:user_email.to_s]%>,
|
2594
|
+
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
|
2595
|
+
<%=ev['content']%>
|
2363
2596
|
|
2597
|
+
Thank you.
|
2364
2598
|
```
|
2365
|
-
|
2599
|
+
The environment provided contains the following additional variable:
|
2600
|
+
|
2601
|
+
* ev : all details on the transfer event
|
2602
|
+
|
2603
|
+
Example:
|
2604
|
+
|
2605
|
+
```
|
2606
|
+
$ ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
2366
2607
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2367
|
-
--
|
2608
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
2368
2609
|
```
|
2369
2610
|
|
2611
|
+
Options:
|
2612
|
+
|
2370
2613
|
* `once_only` keep track of last date it was called, so next call will get only new events
|
2371
2614
|
* `query` filter (on API call)
|
2372
2615
|
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
2373
2616
|
|
2374
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months.
|
2617
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
2375
2618
|
|
2376
2619
|
## Using specific transfer ports
|
2377
2620
|
|
@@ -2522,7 +2765,7 @@ This can also be set as default using a preset
|
|
2522
2765
|
One can test the "server" application using the well known demo server:
|
2523
2766
|
|
2524
2767
|
```
|
2525
|
-
$ ascli config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
2768
|
+
$ ascli config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
2526
2769
|
$ ascli config id default set server aspera_demo_server
|
2527
2770
|
$ ascli server browse /aspera-test-dir-large
|
2528
2771
|
$ ascli server download /aspera-test-dir-large/200MB
|
@@ -2618,27 +2861,130 @@ to download files.
|
|
2618
2861
|
$ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
2619
2862
|
```
|
2620
2863
|
|
2621
|
-
# Plugin: IBM Aspera
|
2864
|
+
# Plugin: IBM Aspera Faspex5
|
2865
|
+
|
2866
|
+
3 authentication methods are supported:
|
2867
|
+
|
2868
|
+
* jwt
|
2869
|
+
* web
|
2870
|
+
* boot
|
2871
|
+
|
2872
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
2873
|
+
|
2874
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
2875
|
+
|
2876
|
+
For boot method: (will be removed in future)
|
2877
|
+
|
2878
|
+
* open a browser
|
2879
|
+
* start developer mode
|
2880
|
+
* login to faspex 5
|
2881
|
+
* find the first API call with `Authorization` token, and copy it (kind of base64 long string)
|
2882
|
+
|
2883
|
+
Use it as password and use `--auth=boot`.
|
2884
|
+
|
2885
|
+
```
|
2886
|
+
$ ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
2887
|
+
```
|
2888
|
+
|
2889
|
+
Ready to use Faspex5 with CLI.
|
2890
|
+
|
2891
|
+
# Plugin: IBM Aspera Faspex (4.x)
|
2892
|
+
|
2893
|
+
Notes:
|
2894
|
+
|
2895
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
2896
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
2897
|
+
|
2898
|
+
## Listing Packages
|
2899
|
+
|
2900
|
+
Command: `faspex package list`
|
2622
2901
|
|
2623
|
-
|
2902
|
+
### Option `box`
|
2903
|
+
|
2904
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
2905
|
+
|
2906
|
+
### Option `recipient`
|
2907
|
+
|
2908
|
+
A user can receive a package because the recipient is:
|
2909
|
+
|
2910
|
+
* the user himself (default)
|
2911
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
2912
|
+
|
2913
|
+
### Option `query`
|
2914
|
+
|
2915
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
2916
|
+
|
2917
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
2918
|
+
* `page` : (native) id of page in call (default=0)
|
2919
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
2920
|
+
* `max` : maximum number of items
|
2921
|
+
* `pmax` : maximum number of pages
|
2922
|
+
|
2923
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
2924
|
+
|
2925
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
2926
|
+
|
2927
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
2928
|
+
|
2929
|
+
### Example
|
2930
|
+
|
2931
|
+
```
|
2932
|
+
$ ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
2933
|
+
```
|
2934
|
+
|
2935
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
2936
|
+
|
2937
|
+
## Receiving a Package
|
2938
|
+
|
2939
|
+
The command is `package recv`, possible methods are:
|
2940
|
+
|
2941
|
+
* provide a package id with option `id`
|
2942
|
+
* provide a public link with option `link`
|
2943
|
+
* provide a `faspe:` URI with option `link`
|
2944
|
+
|
2945
|
+
```
|
2946
|
+
$ ascli faspex package recv --id=12345
|
2947
|
+
$ ascli faspex package recv --link=faspe://...
|
2948
|
+
```
|
2949
|
+
|
2950
|
+
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
2951
|
+
|
2952
|
+
```
|
2953
|
+
$ ascli faspex package list --recipient='*thedropboxname'
|
2954
|
+
```
|
2955
|
+
|
2956
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
2624
2957
|
|
2625
2958
|
## Sending a Package
|
2626
2959
|
|
2627
|
-
|
2960
|
+
The command is `faspex package send`. Package information (title, note, metadata, options) is provided in option `delivery_info`. (Refer to Faspex API).
|
2961
|
+
|
2962
|
+
Example:
|
2628
2963
|
|
2629
2964
|
```
|
2630
|
-
--delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}'
|
2965
|
+
$ ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
2631
2966
|
```
|
2632
2967
|
|
2633
|
-
a
|
2968
|
+
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
2969
|
+
|
2970
|
+
Additional optional parameters in `delivery_info`:
|
2971
|
+
|
2972
|
+
* Package Note: : `"note":"note this and that"`
|
2973
|
+
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
2634
2974
|
|
2635
|
-
|
2975
|
+
## Email notification on transfer
|
2636
2976
|
|
2977
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
2637
2978
|
|
2638
|
-
|
2639
|
-
|
2979
|
+
Example:
|
2980
|
+
|
2981
|
+
```
|
2982
|
+
$ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
2983
|
+
```
|
2984
|
+
|
2985
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
2640
2986
|
|
2641
|
-
##
|
2987
|
+
## Operation on dropboxes
|
2642
2988
|
|
2643
2989
|
Example:
|
2644
2990
|
|
@@ -2699,6 +3045,8 @@ $ for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}'
|
|
2699
3045
|
# Plugin: IBM Cloud Object Storage
|
2700
3046
|
|
2701
3047
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
3048
|
+
It uses the same transfer service as Aspera on Cloud.
|
3049
|
+
see [https://status.aspera.io](https://status.aspera.io)
|
2702
3050
|
|
2703
3051
|
Required options are either:
|
2704
3052
|
|
@@ -2755,7 +3103,14 @@ Endpoints for regions can be found by querying the `endpoints` URL.
|
|
2755
3103
|
For convenience, let us create a default configuration, for example:
|
2756
3104
|
|
2757
3105
|
```
|
2758
|
-
$ ascli conf id mycos update --service-credentials=@val:@json:@file
|
3106
|
+
$ ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
3107
|
+
$ ascli conf id default set cos mycos
|
3108
|
+
```
|
3109
|
+
|
3110
|
+
or using direct parameters:
|
3111
|
+
|
3112
|
+
```
|
3113
|
+
$ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
2759
3114
|
$ ascli conf id default set cos mycos
|
2760
3115
|
```
|
2761
3116
|
|
@@ -2773,7 +3128,7 @@ to start from ma configuration file, using `ascli` standard options.
|
|
2773
3128
|
|
2774
3129
|
# Plugin: Preview
|
2775
3130
|
|
2776
|
-
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on
|
3131
|
+
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
2777
3132
|
This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
|
2778
3133
|
Several parameters can be used to tune several aspects:
|
2779
3134
|
|
@@ -2821,6 +3176,7 @@ The tool requires the following external tools available in the `PATH`:
|
|
2821
3176
|
* OptiPNG : `optipng`
|
2822
3177
|
* FFmpeg : `ffmpeg` `ffprobe`
|
2823
3178
|
* Libreoffice : `libreoffice`
|
3179
|
+
* ruby gem `mimemagic`
|
2824
3180
|
|
2825
3181
|
Here shown on Redhat/CentOS.
|
2826
3182
|
|
@@ -2832,6 +3188,26 @@ To check if all tools are found properly, execute:
|
|
2832
3188
|
$ ascli preview check
|
2833
3189
|
```
|
2834
3190
|
|
3191
|
+
### mimemagic
|
3192
|
+
|
3193
|
+
To benefit from extra mime type detection install gem mimemagic:
|
3194
|
+
|
3195
|
+
```
|
3196
|
+
# gem install mimemagic
|
3197
|
+
```
|
3198
|
+
|
3199
|
+
or to install an earlier version if any problem:
|
3200
|
+
|
3201
|
+
```
|
3202
|
+
# gem install mimemagic -v '~> 0.3.0'
|
3203
|
+
```
|
3204
|
+
|
3205
|
+
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
3206
|
+
|
3207
|
+
If not used, Mime type used for conversion is the one provided by the node API.
|
3208
|
+
|
3209
|
+
If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
|
3210
|
+
|
2835
3211
|
### Image: Imagemagick and optipng
|
2836
3212
|
|
2837
3213
|
```
|
@@ -2856,90 +3232,111 @@ The generation of preview in based on the use of `unoconv` and `libreoffice`
|
|
2856
3232
|
# dnf install unoconv
|
2857
3233
|
```
|
2858
3234
|
|
3235
|
+
* Amazon Linux
|
3236
|
+
|
3237
|
+
```
|
3238
|
+
# amazon-linux-extras enable libreoffice
|
3239
|
+
# yum clean metadata
|
3240
|
+
# yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
3241
|
+
# wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
3242
|
+
# mv unoconv /usr/bin
|
3243
|
+
# chmod a+x /usr/bin/unoconv
|
3244
|
+
```
|
2859
3245
|
|
2860
3246
|
## Configuration
|
2861
3247
|
|
2862
|
-
|
3248
|
+
The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)
|
3249
|
+
|
3250
|
+
Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). The configuration file must be created with the same user used to run so that it is properly used on runtime.
|
3251
|
+
|
3252
|
+
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
2863
3253
|
|
2864
3254
|
```
|
2865
3255
|
# su -s /bin/bash - xfer
|
2866
|
-
$ ascli config id
|
2867
|
-
$ ascli config id default set preview
|
3256
|
+
$ ascli config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
3257
|
+
$ ascli config id default set preview previewconf
|
2868
3258
|
```
|
2869
3259
|
|
2870
|
-
Here we assume that Office file generation is disabled, else remove
|
3260
|
+
Here we assume that Office file generation is disabled, else remove this option.
|
3261
|
+
`lock_port` prevents concurrent execution of generation when using a scheduler.
|
2871
3262
|
|
2872
|
-
|
3263
|
+
One can check if the access key is well configured using:
|
2873
3264
|
|
2874
3265
|
```
|
2875
|
-
$ ascli -
|
3266
|
+
$ ascli -Ppreviewconf node browse /
|
2876
3267
|
```
|
2877
3268
|
|
2878
3269
|
This shall list the contents of the storage root of the access key.
|
2879
3270
|
|
2880
3271
|
## Execution
|
2881
3272
|
|
2882
|
-
The tool intentionally supports only a "one shot" mode in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
2883
|
-
It needs to be run
|
3273
|
+
The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
3274
|
+
It needs to be run on a regular basis to create or update preview files. For that use your best
|
2884
3275
|
reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.
|
2885
3276
|
|
2886
|
-
Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be
|
2887
|
-
should be run as user `xfer`.
|
3277
|
+
Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
|
2888
3278
|
|
2889
3279
|
Lets do a one shot test, using the configuration previously created:
|
2890
3280
|
|
2891
3281
|
```
|
2892
3282
|
# su -s /bin/bash - xfer
|
2893
|
-
$ ascli preview scan --overwrite=always
|
3283
|
+
xfer$ ascli preview scan --overwrite=always
|
2894
3284
|
```
|
2895
3285
|
|
2896
3286
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
2897
|
-
which contains the access key used.
|
3287
|
+
in the previews folder which contains the access key used.
|
2898
3288
|
On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
|
2899
3289
|
|
2900
3290
|
## Configuration for Execution in scheduler
|
2901
3291
|
|
2902
|
-
Here is an example of configuration for use with cron on Linux.
|
3292
|
+
Here is an example of configuration for use with cron on Linux.
|
3293
|
+
Adapt the scripts to your own preference.
|
2903
3294
|
|
2904
3295
|
We assume here that a configuration preset was created as shown previously.
|
2905
3296
|
|
2906
|
-
|
3297
|
+
Lets first setup a script that will be used in the sceduler and sets up the environment.
|
3298
|
+
|
3299
|
+
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
3300
|
+
|
3301
|
+
```
|
3302
|
+
#!/bin/bash
|
3303
|
+
# set a timeout protection, just in case
|
3304
|
+
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
3305
|
+
. /etc/profile.d/rvm.sh
|
3306
|
+
rvm use 2.6 --quiet
|
3307
|
+
exec timeout ${tmout} ascli "${@}"
|
3308
|
+
```
|
3309
|
+
|
3310
|
+
Here the cronjob is created for user `xfer`.
|
2907
3311
|
|
2908
3312
|
```
|
2909
|
-
|
2910
|
-
|
2911
|
-
|
3313
|
+
xfer$ crontab<<EOF
|
3314
|
+
0 * * * * /home/xfer/cron_ascli preview scan --logger=syslog --display=error
|
3315
|
+
2-59 * * * * /home/xfer/cron_ascli preview trev --logger=syslog --display=error
|
2912
3316
|
EOF
|
2913
3317
|
```
|
2914
3318
|
|
2915
|
-
|
3319
|
+
Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
2916
3320
|
|
2917
3321
|
## Candidate detection for creation or update (or deletion)
|
2918
3322
|
|
2919
|
-
The tool
|
3323
|
+
The tool generates preview files using those commands:
|
2920
3324
|
|
2921
3325
|
* `trevents` : only recently uploaded files will be tested (transfer events)
|
2922
3326
|
* `events` : only recently uploaded files will be tested (file events: not working)
|
2923
|
-
* `scan` :
|
2924
|
-
* `
|
2925
|
-
* `file` : for an individual file generation
|
2926
|
-
|
2927
|
-
Note that for the `event`, the option `iteration_file` should be specified so that
|
2928
|
-
successive calls only process new events. This file will hold an identifier
|
2929
|
-
telling from where to get new events.
|
2930
|
-
|
2931
|
-
It is also possible to test a local file, using the `test` command.
|
3327
|
+
* `scan` : recursively scan all files under the access key's "storage root"
|
3328
|
+
* `test` : test using a local file
|
2932
3329
|
|
2933
3330
|
Once candidate are selected, once candidates are selected,
|
2934
3331
|
a preview is always generated if it does not exist already,
|
2935
3332
|
else if a preview already exist, it will be generated
|
2936
|
-
using one of three overwrite
|
3333
|
+
using one of three values for the `overwrite` option:
|
2937
3334
|
|
2938
3335
|
* `always` : preview is always generated, even if it already exists and is newer than original
|
2939
3336
|
* `never` : preview is generated only if it does not exist already
|
2940
3337
|
* `mtime` : preview is generated only if the original file is newer than the existing
|
2941
3338
|
|
2942
|
-
Deletion of preview for deleted source files: not implemented yet.
|
3339
|
+
Deletion of preview for deleted source files: not implemented yet (TODO).
|
2943
3340
|
|
2944
3341
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
2945
3342
|
|
@@ -2949,6 +3346,14 @@ $ ascli preview scan --skip-folders=@json:'["/not_here"]'
|
|
2949
3346
|
|
2950
3347
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
2951
3348
|
|
3349
|
+
When scanning the option `value` has the same behaviour as for the `node find` command.
|
3350
|
+
|
3351
|
+
For instance to filter out files beginning with `._` do:
|
3352
|
+
|
3353
|
+
```
|
3354
|
+
... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
|
3355
|
+
```
|
3356
|
+
|
2952
3357
|
## Preview File types
|
2953
3358
|
|
2954
3359
|
Two types of preview can be generated:
|
@@ -2997,14 +3402,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
2997
3402
|
The `smtp` option is a hash table (extended value) with the following fields:
|
2998
3403
|
<table>
|
2999
3404
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
3000
|
-
<tr><td
|
3001
|
-
<tr><td
|
3002
|
-
<tr><td
|
3003
|
-
<tr><td
|
3004
|
-
<tr><td
|
3005
|
-
<tr><td
|
3006
|
-
<tr><td
|
3007
|
-
<tr><td
|
3405
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
3406
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
3407
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
3408
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
3409
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
3410
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
3411
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
3412
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
3008
3413
|
</table>
|
3009
3414
|
|
3010
3415
|
## Example of configuration:
|
@@ -3034,13 +3439,52 @@ $ ascli config id cli_default set smtp @val:@preset:smtp_google
|
|
3034
3439
|
$ ascli config id default set config cli_default
|
3035
3440
|
```
|
3036
3441
|
|
3442
|
+
## Email templates
|
3443
|
+
|
3444
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
3445
|
+
|
3446
|
+
The template is the full SMTP message, including headers.
|
3447
|
+
|
3448
|
+
The following variables are defined by default:
|
3449
|
+
|
3450
|
+
* from_name
|
3451
|
+
* from_email
|
3452
|
+
* to
|
3453
|
+
|
3454
|
+
Other variables are defined depending on context.
|
3455
|
+
|
3037
3456
|
## Test
|
3038
3457
|
|
3039
3458
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
3040
3459
|
|
3041
3460
|
```
|
3042
3461
|
$ ascli config --smtp=@preset:smtp_google smtp
|
3043
|
-
$ ascli config --smtp=@preset:smtp_google email sample.dest@example.com
|
3462
|
+
$ ascli config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
3463
|
+
```
|
3464
|
+
|
3465
|
+
## Notifications for transfer status
|
3466
|
+
|
3467
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
3468
|
+
|
3469
|
+
To activate, use option `notif_to`.
|
3470
|
+
|
3471
|
+
A default e-mail template is used, but it can be overriden with option `notif_template`.
|
3472
|
+
|
3473
|
+
The environment provided contains the following additional variables:
|
3474
|
+
|
3475
|
+
* subject
|
3476
|
+
* body
|
3477
|
+
* global_transfer_status
|
3478
|
+
* ts
|
3479
|
+
|
3480
|
+
Example of template:
|
3481
|
+
|
3482
|
+
```
|
3483
|
+
From: <%=from_name%> <<%=from_email%>>
|
3484
|
+
To: <<%=to%>>
|
3485
|
+
Subject: <%=subject%>
|
3486
|
+
|
3487
|
+
Transfer is: <%=global_transfer_status%>
|
3044
3488
|
```
|
3045
3489
|
|
3046
3490
|
# Tool: `asession`
|
@@ -3086,7 +3530,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
|
|
3086
3530
|
## Simple session
|
3087
3531
|
|
3088
3532
|
```
|
3089
|
-
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"
|
3533
|
+
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
|
3090
3534
|
|
3091
3535
|
echo "${MY_TSPEC}"|asession
|
3092
3536
|
```
|
@@ -3099,7 +3543,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
|
|
3099
3543
|
|
3100
3544
|
```
|
3101
3545
|
$ asession
|
3102
|
-
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"
|
3546
|
+
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
3103
3547
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
3104
3548
|
{"type":"DONE"}
|
3105
3549
|
```
|
@@ -3171,7 +3615,7 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
3171
3615
|
Note that:
|
3172
3616
|
|
3173
3617
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
3174
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
3618
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3175
3619
|
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
|
3176
3620
|
|
3177
3621
|
### server side and configuration
|
@@ -3193,6 +3637,34 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
|
|
3193
3637
|
|
3194
3638
|
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3195
3639
|
|
3640
|
+
# Aspera Health check and Nagios
|
3641
|
+
|
3642
|
+
Each plugin provide a `health` command that will check the health status of the application. Example:
|
3643
|
+
|
3644
|
+
```
|
3645
|
+
$ ascli console health
|
3646
|
+
+--------+-------------+------------+
|
3647
|
+
| status | component | message |
|
3648
|
+
+--------+-------------+------------+
|
3649
|
+
| ok | console api | accessible |
|
3650
|
+
+--------+-------------+------------+
|
3651
|
+
```
|
3652
|
+
|
3653
|
+
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
3654
|
+
|
3655
|
+
* issuing a transfer to the server
|
3656
|
+
* checking web app status with `asctl all:status`
|
3657
|
+
* checking daemons process status
|
3658
|
+
|
3659
|
+
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
3660
|
+
|
3661
|
+
```
|
3662
|
+
$ ascli server health transfer --to-folder=/Upload --format=nagios --progress=none
|
3663
|
+
OK - [transfer:ok]
|
3664
|
+
$ ascli server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
3665
|
+
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
3666
|
+
```
|
3667
|
+
|
3196
3668
|
# Module: `Aspera`
|
3197
3669
|
|
3198
3670
|
Main components:
|
@@ -3212,6 +3684,16 @@ This sample code shows some example of use of the API as well as
|
|
3212
3684
|
REST API.
|
3213
3685
|
Note: although nice, it's probably a good idea to use RestClient for REST.
|
3214
3686
|
|
3687
|
+
Example of use of the API of Aspera on Cloud:
|
3688
|
+
|
3689
|
+
```
|
3690
|
+
require 'aspera/aoc'
|
3691
|
+
|
3692
|
+
aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/ascli/aspera_on_cloud_key')),username: 'laurent.martin.aspera@fr.ibm.com',subpath: 'api/v1')
|
3693
|
+
|
3694
|
+
aoc.read('self')
|
3695
|
+
```
|
3696
|
+
|
3215
3697
|
# History
|
3216
3698
|
|
3217
3699
|
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
@@ -3228,11 +3710,55 @@ So, it evolved into `ascli`:
|
|
3228
3710
|
* supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from ascp command line to [_transfer-spec_](#transferspec) (more reliable , more standard)
|
3229
3711
|
* `ruby` is consistent with other Aspera products
|
3230
3712
|
|
3231
|
-
|
3232
|
-
|
3233
|
-
|
3234
|
-
|
3235
|
-
*
|
3713
|
+
# Changes (Release notes)
|
3714
|
+
|
3715
|
+
* 4.2.2
|
3716
|
+
|
3717
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
3718
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
3719
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
3720
|
+
* new: `conf ascp spec` displays supported transfer spec
|
3721
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
3722
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
3723
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
3724
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then `ascli` exits with code 1 (instead of zero=success)
|
3725
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
3726
|
+
|
3727
|
+
* 4.2.1
|
3728
|
+
|
3729
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
3730
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
3731
|
+
|
3732
|
+
* 4.2.0
|
3733
|
+
|
3734
|
+
* new: command `aoc remind` to receive organization membership by email
|
3735
|
+
* new: in `preview` option `value` to filter out on file name
|
3736
|
+
* new: `initdemo` to initialize for demo server
|
3737
|
+
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3738
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
3739
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
3740
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3741
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3742
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3743
|
+
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
3744
|
+
|
3745
|
+
* 4.1.0
|
3746
|
+
|
3747
|
+
* fix: remove keys from transfer spec and command line when not needed
|
3748
|
+
* fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
3749
|
+
* new: update documentation with regard to offline and docker installation
|
3750
|
+
* new: renamed command `nagios_check` to `health`
|
3751
|
+
* new: agent `http_gw` now supports upload
|
3752
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
3753
|
+
* new: check new gem version periodically
|
3754
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
3755
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
3756
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
3757
|
+
* new: AoC a password can be provided for a public link
|
3758
|
+
* new: `conf doc` take an optional parameter to go to a section
|
3759
|
+
* new: initial support for Faspex 5 Beta 1
|
3760
|
+
|
3761
|
+
* 4.0.0
|
3236
3762
|
|
3237
3763
|
* now available as open source at [https://github.com/IBM/aspera-cli](https://github.com/IBM/aspera-cli) with general cleanup
|
3238
3764
|
* changed default tool name from `mlia` to `ascli`
|
@@ -3360,7 +3886,7 @@ So, it evolved into `ascli`:
|
|
3360
3886
|
|
3361
3887
|
* 0.10.6
|
3362
3888
|
|
3363
|
-
* FaspManager: transfer spec `authentication` no more needed for local tranfer to use
|
3889
|
+
* FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
3364
3890
|
* gem version requirements made more open
|
3365
3891
|
|
3366
3892
|
* 0.10.5
|
@@ -3587,11 +4113,15 @@ Breaking change:
|
|
3587
4113
|
|
3588
4114
|
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
|
3589
4115
|
|
3590
|
-
# BUGS
|
4116
|
+
# BUGS, FEATURES, CONTRIBUTION
|
4117
|
+
|
4118
|
+
For issues or feature requests use the Github repository and issues.
|
4119
|
+
|
4120
|
+
You can also contribute to this open source project.
|
3591
4121
|
|
3592
|
-
|
4122
|
+
One can also create one's own command nplugin.
|
3593
4123
|
|
3594
|
-
##
|
4124
|
+
## Only one value for any option
|
3595
4125
|
|
3596
4126
|
Some commands and sub commands may ask for the same option name.
|
3597
4127
|
Currently, since option definition is position independant (last one wins), it is not possible
|
@@ -3605,7 +4135,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
|
|
3605
4135
|
|
3606
4136
|
## ED255519 key not supported
|
3607
4137
|
|
3608
|
-
|
4138
|
+
ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
|
4139
|
+
|
3609
4140
|
Without this deactivation, if such key was present the following error was generated:
|
3610
4141
|
|
3611
4142
|
```
|
@@ -3615,7 +4146,21 @@ OpenSSH keys only supported if ED25519 is available
|
|
3615
4146
|
Which meant that you do not have ruby support for ED25519 SSH keys.
|
3616
4147
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
3617
4148
|
|
3618
|
-
|
4149
|
+
## Error "Remote host is not who we expected"
|
4150
|
+
|
4151
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
4152
|
+
|
4153
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
4154
|
+
|
4155
|
+
```
|
4156
|
+
--ts=@json:'{"sshfp":null}'
|
4157
|
+
```
|
4158
|
+
|
4159
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
4160
|
+
|
4161
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
4162
|
+
|
4163
|
+
## Miscelaneous
|
3619
4164
|
|
3620
4165
|
* remove rest and oauth classes and use ruby standard gems:
|
3621
4166
|
|
@@ -3631,9 +4176,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
3631
4176
|
* Going through proxy: use env var http_proxy and https_proxy, no_proxy
|
3632
4177
|
|
3633
4178
|
* easier use with https://github.com/pmq20/ruby-packer
|
3634
|
-
|
3635
|
-
# Contribution
|
3636
|
-
|
3637
|
-
Send comments !
|
3638
|
-
|
3639
|
-
Create your own plugin !
|