aspera-cli 4.9.0 → 4.11.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +20 -0
- data/CHANGELOG.md +509 -0
- data/CONTRIBUTING.md +118 -0
- data/README.md +1241 -916
- data/bin/ascli +4 -4
- data/bin/asession +11 -11
- data/docs/test_env.conf +32 -21
- data/examples/aoc.rb +4 -4
- data/examples/dascli +16 -9
- data/examples/faspex4.rb +8 -8
- data/examples/node.rb +12 -12
- data/examples/server.rb +10 -10
- data/lib/aspera/aoc.rb +273 -266
- data/lib/aspera/ascmd.rb +56 -54
- data/lib/aspera/ats_api.rb +4 -4
- data/lib/aspera/cli/basic_auth_plugin.rb +15 -12
- data/lib/aspera/cli/extended_value.rb +5 -5
- data/lib/aspera/cli/formater.rb +64 -64
- data/lib/aspera/cli/info.rb +2 -2
- data/lib/aspera/cli/listener/line_dump.rb +1 -1
- data/lib/aspera/cli/listener/logger.rb +1 -1
- data/lib/aspera/cli/listener/progress.rb +5 -6
- data/lib/aspera/cli/listener/progress_multi.rb +14 -19
- data/lib/aspera/cli/main.rb +66 -67
- data/lib/aspera/cli/manager.rb +112 -110
- data/lib/aspera/cli/plugin.rb +57 -36
- data/lib/aspera/cli/plugins/alee.rb +4 -4
- data/lib/aspera/cli/plugins/aoc.rb +309 -670
- data/lib/aspera/cli/plugins/ats.rb +44 -46
- data/lib/aspera/cli/plugins/bss.rb +10 -10
- data/lib/aspera/cli/plugins/config.rb +497 -378
- data/lib/aspera/cli/plugins/console.rb +12 -12
- data/lib/aspera/cli/plugins/cos.rb +18 -20
- data/lib/aspera/cli/plugins/faspex.rb +112 -114
- data/lib/aspera/cli/plugins/faspex5.rb +71 -46
- data/lib/aspera/cli/plugins/node.rb +379 -283
- data/lib/aspera/cli/plugins/orchestrator.rb +46 -46
- data/lib/aspera/cli/plugins/preview.rb +122 -114
- data/lib/aspera/cli/plugins/server.rb +137 -83
- data/lib/aspera/cli/plugins/shares.rb +30 -29
- data/lib/aspera/cli/plugins/sync.rb +13 -33
- data/lib/aspera/cli/transfer_agent.rb +60 -59
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/colors.rb +3 -3
- data/lib/aspera/command_line_builder.rb +27 -27
- data/lib/aspera/cos_node.rb +22 -20
- data/lib/aspera/data_repository.rb +1 -1
- data/lib/aspera/environment.rb +35 -15
- data/lib/aspera/fasp/agent_base.rb +15 -15
- data/lib/aspera/fasp/agent_connect.rb +23 -21
- data/lib/aspera/fasp/agent_direct.rb +66 -64
- data/lib/aspera/fasp/agent_httpgw.rb +141 -78
- data/lib/aspera/fasp/agent_node.rb +23 -21
- data/lib/aspera/fasp/agent_trsdk.rb +20 -20
- data/lib/aspera/fasp/error.rb +3 -2
- data/lib/aspera/fasp/error_info.rb +11 -8
- data/lib/aspera/fasp/installation.rb +79 -79
- data/lib/aspera/fasp/listener.rb +1 -1
- data/lib/aspera/fasp/parameters.rb +86 -71
- data/lib/aspera/fasp/parameters.yaml +7 -4
- data/lib/aspera/fasp/resume_policy.rb +8 -8
- data/lib/aspera/fasp/transfer_spec.rb +35 -2
- data/lib/aspera/fasp/uri.rb +7 -7
- data/lib/aspera/faspex_gw.rb +7 -5
- data/lib/aspera/hash_ext.rb +3 -3
- data/lib/aspera/id_generator.rb +5 -5
- data/lib/aspera/keychain/encrypted_hash.rb +38 -105
- data/lib/aspera/keychain/macos_security.rb +128 -57
- data/lib/aspera/log.rb +7 -7
- data/lib/aspera/nagios.rb +19 -18
- data/lib/aspera/node.rb +209 -35
- data/lib/aspera/oauth.rb +37 -36
- data/lib/aspera/open_application.rb +19 -11
- data/lib/aspera/persistency_action_once.rb +4 -4
- data/lib/aspera/persistency_folder.rb +16 -15
- data/lib/aspera/preview/file_types.rb +8 -8
- data/lib/aspera/preview/generator.rb +67 -67
- data/lib/aspera/preview/utils.rb +27 -27
- data/lib/aspera/proxy_auto_config.js +41 -41
- data/lib/aspera/proxy_auto_config.rb +21 -14
- data/lib/aspera/rest.rb +72 -67
- data/lib/aspera/rest_call_error.rb +2 -1
- data/lib/aspera/rest_error_analyzer.rb +18 -17
- data/lib/aspera/rest_errors_aspera.rb +16 -16
- data/lib/aspera/secret_hider.rb +15 -13
- data/lib/aspera/ssh.rb +11 -10
- data/lib/aspera/sync.rb +158 -44
- data/lib/aspera/temp_file_manager.rb +2 -2
- data/lib/aspera/uri_reader.rb +4 -4
- data/lib/aspera/web_auth.rb +14 -13
- data.tar.gz.sig +0 -0
- metadata +11 -36
- metadata.gz.sig +0 -0
data/README.md
CHANGED
@@ -1,10 +1,13 @@
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
2
|
+
<!-- markdownlint-disable MD033 MD003 MD053 -->
|
2
3
|
|
3
4
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
4
5
|
|
5
|
-
|
6
|
+
##
|
6
7
|
|
7
|
-
|
8
|
+
Version : 4.11.0
|
9
|
+
|
10
|
+
Laurent/2016-2023
|
8
11
|
|
9
12
|
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
10
13
|
|
@@ -14,9 +17,11 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
14
17
|
|
15
18
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
16
19
|
|
17
|
-
Minimum required Ruby version: >= 2.4.
|
20
|
+
Minimum required Ruby version: >= 2.4.
|
21
|
+
|
22
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
18
23
|
|
19
|
-
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
24
|
+
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs&sort=title_asc)
|
20
25
|
[Link 2](https://developer.ibm.com/apis/catalog/?search=aspera)
|
21
26
|
|
22
27
|
Release notes: see [CHANGELOG.md](CHANGELOG.md)
|
@@ -33,26 +38,25 @@ One can also [create one's own plugin](#createownplugin).
|
|
33
38
|
|
34
39
|
`ascli` is designed to be used as a command line tool to:
|
35
40
|
|
36
|
-
|
37
|
-
|
41
|
+
- Execute commands remotely on Aspera products
|
42
|
+
- Transfer to/from Aspera products
|
38
43
|
|
39
44
|
So it is designed for:
|
40
45
|
|
41
|
-
|
42
|
-
|
46
|
+
- Interactive operations on a text terminal (typically, VT100 compatible), e.g. for maintenance
|
47
|
+
- Scripting, e.g. batch operations in (shell) scripts (e.g. cron job)
|
43
48
|
|
44
49
|
`ascli` can be seen as a command line tool integrating:
|
45
50
|
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
51
|
+
- A configuration file (config.yaml)
|
52
|
+
- Advanced command line options
|
53
|
+
- cURL (for REST calls)
|
54
|
+
- Aspera transfer (`ascp`)
|
50
55
|
|
51
|
-
|
52
|
-
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
56
|
+
If the need is to perform operations programmatically in languages such as: C, Go, Python, nodejs, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
53
57
|
|
54
|
-
|
55
|
-
|
58
|
+
- Product APIs (REST) : e.g. AoC, Faspex, node
|
59
|
+
- Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
56
60
|
|
57
61
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
58
62
|
|
@@ -64,7 +68,6 @@ In examples, command line operations are shown using a shell such: `bash` or `zs
|
|
64
68
|
|
65
69
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
66
70
|
|
67
|
-
|
68
71
|
## Quick Start
|
69
72
|
|
70
73
|
This section guides you from installation, first use and advanced use.
|
@@ -78,7 +81,7 @@ ascli --version
|
|
78
81
|
```
|
79
82
|
|
80
83
|
```bash
|
81
|
-
4.
|
84
|
+
4.11.0
|
82
85
|
```
|
83
86
|
|
84
87
|
### First use
|
@@ -110,10 +113,10 @@ ascli server browse /
|
|
110
113
|
|
111
114
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
112
115
|
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
116
|
+
- create a [option preset](#lprt)
|
117
|
+
- define it as default for `server` plugin
|
118
|
+
- list files in a folder
|
119
|
+
- download a file
|
117
120
|
|
118
121
|
```bash
|
119
122
|
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_
|
@@ -128,7 +131,7 @@ ascli config preset set default server myserver
|
|
128
131
|
```
|
129
132
|
|
130
133
|
```output
|
131
|
-
updated: default→server to myserver
|
134
|
+
updated: default → server to myserver
|
132
135
|
```
|
133
136
|
|
134
137
|
```bash
|
@@ -159,7 +162,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
159
162
|
```
|
160
163
|
|
161
164
|
```output
|
162
|
-
Time: 00:00:02
|
165
|
+
Time: 00:00:02 =========================================================== 100% 100 Mbps Time: 00:00:00
|
163
166
|
complete
|
164
167
|
```
|
165
168
|
|
@@ -175,9 +178,13 @@ It is possible to install *either* directly on the host operating system (Linux,
|
|
175
178
|
|
176
179
|
The direct installation is recommended and consists in installing:
|
177
180
|
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
+
- [Ruby](#ruby)
|
182
|
+
- [aspera-cli](#the_gem)
|
183
|
+
- [Aspera SDK (`ascp`)](#fasp_prot)
|
184
|
+
|
185
|
+
Ruby version: >= 2.4.
|
186
|
+
|
187
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
181
188
|
|
182
189
|
The following sections provide information on the various installation methods.
|
183
190
|
|
@@ -185,45 +192,136 @@ An internet connection is required for the installation. If you don't have inter
|
|
185
192
|
|
186
193
|
### Docker container
|
187
194
|
|
188
|
-
|
195
|
+
The image is: [martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli).
|
196
|
+
The container contains: Ruby, `ascli` and the Aspera Transfer SDK.
|
197
|
+
To use the container, ensure that you have `docker` (or `podman`) installed.
|
189
198
|
|
190
|
-
|
199
|
+
```bash
|
200
|
+
docker --version
|
201
|
+
```
|
191
202
|
|
192
|
-
|
203
|
+
**Wanna start quickly ?** With an interactive shell ? Execute this:
|
193
204
|
|
194
205
|
```bash
|
195
|
-
docker --
|
206
|
+
docker run --tty --interactive --entrypoint bash martinlaurent/ascli:latest
|
196
207
|
```
|
197
208
|
|
198
|
-
|
209
|
+
Then, execute individual `ascli` commands such as:
|
199
210
|
|
200
211
|
```bash
|
201
|
-
|
212
|
+
ascli conf init
|
213
|
+
ascli conf preset overview
|
214
|
+
ascli conf ascp info
|
215
|
+
ascli server ls /
|
202
216
|
```
|
203
217
|
|
204
|
-
|
218
|
+
That is simple, but there are limitations:
|
219
|
+
|
220
|
+
- Everything happens in the container
|
221
|
+
- Any generated file in the container will be lost on container (shell) exit. Including configuration files and downloaded files.
|
222
|
+
- No possibility to upload files located on the host system
|
223
|
+
|
224
|
+
The container image is built from this [Dockerfile](Dockerfile): the entry point is `ascli` and the default command is `help`.
|
225
|
+
|
226
|
+
The container can also be execute for individual commands like this: (add `ascli` commands and options at the end of the command line, e.g. `-v` to display the version)
|
205
227
|
|
206
228
|
```bash
|
207
|
-
|
229
|
+
docker run --rm --tty --interactive martinlaurent/ascli:latest
|
208
230
|
```
|
209
231
|
|
232
|
+
For more convenience, you may define a shell alias:
|
233
|
+
|
210
234
|
```bash
|
211
|
-
|
235
|
+
alias ascli='docker run --rm --tty --interactive martinlaurent/ascli:latest'
|
236
|
+
```
|
237
|
+
|
238
|
+
Then, you can execute the container like a local command:
|
239
|
+
|
240
|
+
```bash
|
241
|
+
ascli -v
|
242
|
+
```
|
243
|
+
|
244
|
+
```text
|
245
|
+
4.11.0
|
246
|
+
```
|
247
|
+
|
248
|
+
In order to keep persistency of configuration on the host,
|
249
|
+
you should mount your user's config folder to the container.
|
250
|
+
To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`).
|
251
|
+
Add options:
|
252
|
+
|
253
|
+
```bash
|
254
|
+
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
255
|
+
```
|
256
|
+
|
257
|
+
> **Note:** if you are using a `podman machine`, e.g. on Macos , make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
258
|
+
|
259
|
+
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
260
|
+
you can change the entry point, add option:
|
261
|
+
|
262
|
+
```bash
|
263
|
+
--entrypoint bash
|
212
264
|
```
|
213
265
|
|
214
|
-
|
266
|
+
You may also probably want that files downloaded in the container are in fact placed on the host.
|
267
|
+
In this case you need also to mount the shared transfer folder:
|
215
268
|
|
216
269
|
```bash
|
217
|
-
|
270
|
+
--volume $HOME/xferdir:/xferfiles
|
218
271
|
```
|
219
272
|
|
220
|
-
Note
|
273
|
+
> **Note:** ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
221
274
|
|
222
|
-
|
223
|
-
|
275
|
+
And if you want all the above, simply use all the options:
|
276
|
+
|
277
|
+
```bash
|
278
|
+
alias asclish="docker run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash martinlaurent/ascli:latest"
|
279
|
+
```
|
280
|
+
|
281
|
+
```bash
|
282
|
+
export xferdir=$HOME/xferdir
|
283
|
+
mkdir -p $xferdir
|
284
|
+
chmod -R 777 $xferdir
|
285
|
+
mkdir -p $HOME/.aspera/ascli
|
286
|
+
asclish
|
287
|
+
```
|
224
288
|
|
225
|
-
|
226
|
-
|
289
|
+
A convenience sample script is also provided: download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
290
|
+
|
291
|
+
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
292
|
+
|
293
|
+
Some environment variables can be set for this script to adapt its behaviour:
|
294
|
+
|
295
|
+
| env var | description | default | example |
|
296
|
+
|--------------|------------------------------------|--------------------------|--------------------------|
|
297
|
+
| ASCLI_HOME | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascliconfig` |
|
298
|
+
| docker_args | additional options to `docker` | <empty> | `--volume /Users:/Users` |
|
299
|
+
| image | container image name | martinlaurent/ascli | |
|
300
|
+
| version | container image version | latest | `4.8.0.pre` |
|
301
|
+
|
302
|
+
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
303
|
+
(value expected in the container).
|
304
|
+
This allows having persistent configuration on the host.
|
305
|
+
|
306
|
+
To add local storage as a volume, you can use the env var `docker_args`:
|
307
|
+
|
308
|
+
Example of use:
|
309
|
+
|
310
|
+
```bash
|
311
|
+
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
312
|
+
chmod a+x ascli
|
313
|
+
export xferdir=$HOME/xferdir
|
314
|
+
mkdir -p $xferdir
|
315
|
+
chmod -R 777 $xferdir
|
316
|
+
export docker_args="--volume $xferdir:/xferfiles"
|
317
|
+
|
318
|
+
./ascli conf init
|
319
|
+
|
320
|
+
echo 'Local file to transfer' > $xferdir/samplefile.txt
|
321
|
+
./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload
|
322
|
+
```
|
323
|
+
|
324
|
+
> **Note:** The local file (`samplefile.txt`) is specified relative to storage view from container (`/xferfiles`) mapped to the host folder `$HOME/xferdir`
|
227
325
|
|
228
326
|
### <a id="ruby"></a>Ruby
|
229
327
|
|
@@ -231,7 +329,9 @@ Use this method to install on the native host.
|
|
231
329
|
|
232
330
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
233
331
|
|
234
|
-
Required Ruby version: >= 2.4.
|
332
|
+
Required Ruby version: >= 2.4.
|
333
|
+
|
334
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
235
335
|
|
236
336
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
237
337
|
|
@@ -239,7 +339,7 @@ Refer to the following sections for a proposed method for specific operating sys
|
|
239
339
|
|
240
340
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
241
341
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
242
|
-
If you have a simpler better way to install Ruby : use it !
|
342
|
+
If you have a simpler better way to install Ruby : use it !
|
243
343
|
|
244
344
|
#### Generic: RVM: single user installation (not root)
|
245
345
|
|
@@ -247,18 +347,14 @@ Use this method which provides more flexibility.
|
|
247
347
|
|
248
348
|
Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
249
349
|
|
250
|
-
Install the 2 keys
|
251
|
-
|
252
|
-
```bash
|
253
|
-
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
254
|
-
```
|
255
|
-
|
256
350
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
257
351
|
|
258
352
|
```bash
|
259
353
|
\curl -sSL https://get.rvm.io | bash -s stable
|
260
354
|
```
|
261
355
|
|
356
|
+
Follow on-screen instructions to install keys, and then re-execute the command.
|
357
|
+
|
262
358
|
If you keep the same terminal (not needed if re-login):
|
263
359
|
|
264
360
|
```bash
|
@@ -306,11 +402,10 @@ rvm version
|
|
306
402
|
|
307
403
|
Install Latest stable Ruby:
|
308
404
|
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
* Install the "mime info" file as specified in [this section](#mimeinfo).
|
405
|
+
- Navigate to [https://rubyinstaller.org/](https://rubyinstaller.org/) → **Downloads**.
|
406
|
+
- Download the latest Ruby installer **with devkit**. (Msys2 is needed to install some native extensions, such as `grpc`)
|
407
|
+
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
408
|
+
- At the end of the installation procedure, the Msys2 installer is automatically executed, select option 3 (msys and mingw)
|
314
409
|
|
315
410
|
#### macOS: pre-installed or `brew`
|
316
411
|
|
@@ -330,15 +425,20 @@ brew install ruby
|
|
330
425
|
|
331
426
|
If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
|
332
427
|
|
333
|
-
Example:
|
428
|
+
Example: RHEL 8 and 9: basic installation
|
429
|
+
|
430
|
+
```bash
|
431
|
+
yum module install ruby:3.1
|
432
|
+
```
|
433
|
+
|
434
|
+
Example: RHEL 8, Centos 8 Stream: with extensions to compile native gems
|
334
435
|
|
335
436
|
```bash
|
336
437
|
yum install make automake gcc gcc-c++ kernel-devel
|
337
438
|
yum install redhat-rpm-config
|
338
439
|
dnf module reset ruby
|
339
|
-
dnf module enable ruby:3.
|
340
|
-
dnf module -y install ruby:3.
|
341
|
-
gem install aspera-cli
|
440
|
+
dnf module enable ruby:3.1
|
441
|
+
dnf module -y install ruby:3.1/common
|
342
442
|
```
|
343
443
|
|
344
444
|
Other examples:
|
@@ -391,7 +491,7 @@ If you already have a Java JVM on your system (`java`), it is possible to use `j
|
|
391
491
|
|
392
492
|
<https://www.jruby.org/download>
|
393
493
|
|
394
|
-
Note
|
494
|
+
> **Note:** Using jruby the startup time is longer than the native ruby, but the transfer speed is not impacted (executed by `ascp` binary).
|
395
495
|
|
396
496
|
### <a id="the_gem"></a>`aspera-cli` gem
|
397
497
|
|
@@ -420,8 +520,8 @@ ascli conf check_update
|
|
420
520
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
421
521
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
422
522
|
|
423
|
-
|
424
|
-
|
523
|
+
- `ascp`
|
524
|
+
- aspera-license (in same folder, or ../etc)
|
425
525
|
|
426
526
|
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
427
527
|
|
@@ -443,11 +543,11 @@ The format is: `file:///<path>`, where `<path>` can be either a relative path (n
|
|
443
543
|
|
444
544
|
If the embedded method is not used, the following packages are also suitable:
|
445
545
|
|
446
|
-
|
447
|
-
|
448
|
-
|
449
|
-
|
450
|
-
|
546
|
+
- IBM Aspera Connect Client (Free)
|
547
|
+
- IBM Aspera Desktop Client (Free)
|
548
|
+
- IBM Aspera CLI (Free)
|
549
|
+
- IBM Aspera High Speed Transfer Server (Licensed)
|
550
|
+
- IBM Aspera High Speed Transfer EndPoint (Licensed)
|
451
551
|
|
452
552
|
For instance, Aspera Connect Client can be installed
|
453
553
|
by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
@@ -460,33 +560,34 @@ Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, b
|
|
460
560
|
|
461
561
|
### <a id="offline_install"></a>Installation in air gapped environment
|
462
562
|
|
463
|
-
Note
|
563
|
+
> **Note:** no pre-packaged version is provided.
|
564
|
+
|
464
565
|
A method to build one is provided here:
|
465
566
|
|
466
567
|
The procedure:
|
467
568
|
|
468
|
-
|
469
|
-
|
569
|
+
- Follow the non-root installation procedure with RVM, including gem
|
570
|
+
- Archive (zip, tar) the main RVM folder (includes ascli):
|
470
571
|
|
471
572
|
```bash
|
472
573
|
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
473
574
|
```
|
474
575
|
|
475
|
-
|
576
|
+
- Get the Aspera SDK.
|
476
577
|
|
477
578
|
```bash
|
478
579
|
ascli conf --show-config --fields=sdk_url
|
479
580
|
```
|
480
581
|
|
481
|
-
|
582
|
+
- Download the SDK archive from that URL.
|
482
583
|
|
483
584
|
```bash
|
484
585
|
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
485
586
|
```
|
486
587
|
|
487
|
-
|
588
|
+
- Transfer those 2 files to the target system
|
488
589
|
|
489
|
-
|
590
|
+
- On target system
|
490
591
|
|
491
592
|
```bash
|
492
593
|
cd $HOME
|
@@ -498,7 +599,7 @@ source ~/.rvm/scripts/rvm
|
|
498
599
|
ascli conf ascp install --sdk-url=file:///SDK.zip
|
499
600
|
```
|
500
601
|
|
501
|
-
|
602
|
+
- Add those lines to shell init (`.profile`)
|
502
603
|
|
503
604
|
```bash
|
504
605
|
source ~/.rvm/scripts/rvm
|
@@ -508,27 +609,27 @@ source ~/.rvm/scripts/rvm
|
|
508
609
|
|
509
610
|
The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
510
611
|
|
511
|
-
|
512
|
-
|
513
|
-
|
514
|
-
|
515
|
-
|
516
|
-
|
517
|
-
|
612
|
+
- IBM Aspera High Speed Transfer Server (FASP and Node)
|
613
|
+
- IBM Aspera on Cloud (including ATS)
|
614
|
+
- IBM Aspera Faspex
|
615
|
+
- IBM Aspera Shares
|
616
|
+
- IBM Aspera Console
|
617
|
+
- IBM Aspera Orchestrator
|
618
|
+
- and more...
|
518
619
|
|
519
620
|
`ascli` provides the following features:
|
520
621
|
|
521
|
-
|
522
|
-
|
523
|
-
|
524
|
-
|
525
|
-
|
526
|
-
|
527
|
-
|
528
|
-
|
529
|
-
|
530
|
-
|
531
|
-
|
622
|
+
- Supports most Aspera server products (on-premise and SaaS)
|
623
|
+
- Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
624
|
+
- Supports Commands, Option values and Parameters shortcuts
|
625
|
+
- FASP [Transfer Agents](#agents) can be: local `ascp`, or Connect Client, or any transfer node
|
626
|
+
- Transfer parameters can be altered by modification of [*transfer-spec*](#transferspec), this includes requiring multi-session
|
627
|
+
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
628
|
+
- Supports FaspStream creation (using Node API)
|
629
|
+
- Supports Watchfolder creation (using Node API)
|
630
|
+
- Additional command plugins can be written by the user
|
631
|
+
- Supports download of faspex and Aspera on Cloud "external" links
|
632
|
+
- Supports "legacy" ssh based FASP transfers and remote commands (ascmd)
|
532
633
|
|
533
634
|
Basic usage is displayed by executing:
|
534
635
|
|
@@ -544,12 +645,12 @@ Not all `ascli` features are fully documented here, the user may explore command
|
|
544
645
|
|
545
646
|
If you want to use `ascp` directly as a command line, refer to IBM Aspera documentation of either [Desktop Client](https://www.ibm.com/docs/en/asdc), [Endpoint](https://www.ibm.com/docs/en/ahte) or [Transfer Server](https://www.ibm.com/docs/en/ahts) where [a section on `ascp` can be found](https://www.ibm.com/docs/en/ahts/4.4?topic=linux-ascp-transferring-from-command-line).
|
546
647
|
|
547
|
-
Using `ascli` with plugin `server` for command line gives advantages over ascp
|
648
|
+
Using `ascli` with plugin `server` for command line gives advantages over `ascp`:
|
548
649
|
|
549
|
-
|
550
|
-
|
551
|
-
|
552
|
-
|
650
|
+
- automatic resume on error
|
651
|
+
- configuration file
|
652
|
+
- choice of transfer agents
|
653
|
+
- integrated support of multi-session
|
553
654
|
|
554
655
|
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`EX_ascp_args`).
|
555
656
|
|
@@ -573,8 +674,8 @@ On Windows, `cmd.exe` is typically used.
|
|
573
674
|
Windows process creation does not receive the list of arguments but just the whole line.
|
574
675
|
It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
575
676
|
|
576
|
-
|
577
|
-
|
677
|
+
- [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
678
|
+
- [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
578
679
|
|
579
680
|
#### Extended Values (JSON, Ruby, ...)
|
580
681
|
|
@@ -597,7 +698,8 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
597
698
|
```
|
598
699
|
|
599
700
|
`config echo` displays the value of the first argument using Ruby syntax: it surrounds a string with `"` and add `\` before special characters.
|
600
|
-
|
701
|
+
|
702
|
+
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
601
703
|
|
602
704
|
In the following examples (using a POSIX shell, such as `bash`), several sample commands are provided when equivalent.
|
603
705
|
For all example, most of special character handling is not specific to `ascli`: It depoends on the underlying syntax: shell , JSON, etc...
|
@@ -606,7 +708,7 @@ Depending on the case, a different `format` is used to display the actual value.
|
|
606
708
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
607
709
|
|
608
710
|
Double quotes are processed by the shell to create a single string argument.
|
609
|
-
For POSIX shells, single quotes can also be used in this case, or protext the special character ` ` (space) with a backslash.
|
711
|
+
For POSIX shells, single quotes can also be used in this case, or protext the special character ` ` (space) with a backslash. <!-- markdownlint-disable-line -->
|
610
712
|
|
611
713
|
```bash
|
612
714
|
ascli conf echo "Hello World" --format=text
|
@@ -621,8 +723,9 @@ Hello World
|
|
621
723
|
#### Using a shell variable, parsed by shell, in an extended value
|
622
724
|
|
623
725
|
To be evaluated by shell, the shell variable must not be in single quotes.
|
624
|
-
|
625
|
-
|
726
|
+
Even if the variable contains spaces it makes only one argument to `ascli` because word parsing is made before variable expansion by shell.
|
727
|
+
|
728
|
+
> **Note:** we use a simple variable here: the variable is not necessarily an environment variable.
|
626
729
|
|
627
730
|
```bash
|
628
731
|
MYVAR="Hello World"
|
@@ -666,9 +769,9 @@ Here a single quote or a backslash protects the double quote to avoid shell proc
|
|
666
769
|
|
667
770
|
Construction of values with special characters is done like this:
|
668
771
|
|
669
|
-
|
772
|
+
- First select a syntax to represent the extended value, e.g. JSON or Ruby
|
670
773
|
|
671
|
-
|
774
|
+
- Write the expression using this syntax, for example, using JSON:
|
672
775
|
|
673
776
|
```json
|
674
777
|
{"title":"Test \" ' & \\"}
|
@@ -683,7 +786,7 @@ or using Ruby:
|
|
683
786
|
|
684
787
|
Both `"` and `\` are special characters for JSON and Ruby and can be protected with `\` (unless Ruby's extended single quote notation `%q` is used).
|
685
788
|
|
686
|
-
|
789
|
+
- Then, since the value will be evaluated by shell, any shell special characters must be protected, either using preceding `\` for each character to protect, or by enclosing in single quote:
|
687
790
|
|
688
791
|
```bash
|
689
792
|
ascli conf echo @json:{\"title\":\"Test\ \\\"\ \'\ \&\ \\\\\"} --format=json
|
@@ -748,10 +851,10 @@ There are two types of command line arguments: Commands and Options. Example :
|
|
748
851
|
ascli command subcommand --option-name=VAL1 VAL2
|
749
852
|
```
|
750
853
|
|
751
|
-
|
752
|
-
|
753
|
-
|
754
|
-
|
854
|
+
- executes *command*: `command subcommand`
|
855
|
+
- with one *option*: `option_name`
|
856
|
+
- this option is given a *value* of: `VAL1`
|
857
|
+
- the command has one additional *argument*: `VAL2`
|
755
858
|
|
756
859
|
When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example `ascli conf ov` is the same as `ascli config overview`.
|
757
860
|
|
@@ -761,16 +864,16 @@ The value of options and arguments is evaluated with the [Extended Value Syntax]
|
|
761
864
|
|
762
865
|
All options, e.g. `--log-level=debug`, are command line arguments that:
|
763
866
|
|
764
|
-
|
765
|
-
|
766
|
-
|
767
|
-
|
867
|
+
- start with `--`
|
868
|
+
- have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
|
869
|
+
- have a value, separated from name with a `=`
|
870
|
+
- can be used by prefix, provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
|
768
871
|
|
769
872
|
Exceptions:
|
770
873
|
|
771
|
-
|
772
|
-
|
773
|
-
|
874
|
+
- some options accept a short form, e.g. `-Ptoto` is equivalent to `--preset=toto`, refer to the manual or `-h`.
|
875
|
+
- some options (flags) don't take a value, e.g. `-r`
|
876
|
+
- the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
774
877
|
|
775
878
|
```bash
|
776
879
|
ascli config echo -- --sample
|
@@ -780,15 +883,15 @@ ascli config echo -- --sample
|
|
780
883
|
"--sample"
|
781
884
|
```
|
782
885
|
|
783
|
-
Note
|
886
|
+
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
784
887
|
|
785
888
|
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
786
889
|
|
787
890
|
The value for *any* options can come from the following locations (in this order, last value evaluated overrides previous value):
|
788
891
|
|
789
|
-
|
790
|
-
|
791
|
-
|
892
|
+
- [Configuration file](#configfile).
|
893
|
+
- Environment variable
|
894
|
+
- Command line
|
792
895
|
|
793
896
|
Environment variable starting with prefix: ASCLI_ are taken as option values, e.g. `ASCLI_OPTION_NAME` is for `--option-name`.
|
794
897
|
|
@@ -804,11 +907,11 @@ Some options and parameters are mandatory and other optional. By default, the to
|
|
804
907
|
|
805
908
|
The behavior can be controlled with:
|
806
909
|
|
807
|
-
|
808
|
-
|
809
|
-
|
810
|
-
|
811
|
-
|
910
|
+
- --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
911
|
+
- yes : missing mandatory parameters/options are asked to the user
|
912
|
+
- no : missing mandatory parameters/options raise an error message
|
913
|
+
- --ask-options=<yes|no> (default=no)
|
914
|
+
- optional parameters/options are asked to user
|
812
915
|
|
813
916
|
### Output
|
814
917
|
|
@@ -819,12 +922,12 @@ The information displayed depends on the action.
|
|
819
922
|
|
820
923
|
Depending on action, the output will contain:
|
821
924
|
|
822
|
-
|
823
|
-
|
824
|
-
|
825
|
-
|
826
|
-
|
827
|
-
|
925
|
+
- `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
926
|
+
- `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
927
|
+
- `value_list` : a table with one column.
|
928
|
+
- `empty` : nothing
|
929
|
+
- `status` : a message
|
930
|
+
- `other_struct` : a complex structure that cannot be displayed as an array
|
828
931
|
|
829
932
|
#### Format of output
|
830
933
|
|
@@ -841,13 +944,13 @@ If transposition of single object is not desired, use option: `transpose_single`
|
|
841
944
|
|
842
945
|
The style of output can be set using the `format` parameter, supporting:
|
843
946
|
|
844
|
-
|
845
|
-
|
846
|
-
|
847
|
-
|
848
|
-
|
849
|
-
|
850
|
-
|
947
|
+
- `text` : Value as String
|
948
|
+
- `table` : Text table
|
949
|
+
- `ruby` : Ruby code
|
950
|
+
- `json` : JSON code
|
951
|
+
- `jsonpp` : JSON pretty printed
|
952
|
+
- `yaml` : YAML
|
953
|
+
- `csv` : Comma Separated Values
|
851
954
|
|
852
955
|
#### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
853
956
|
|
@@ -866,21 +969,21 @@ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sor
|
|
866
969
|
:...............................:..................................:...........:
|
867
970
|
```
|
868
971
|
|
869
|
-
Note
|
972
|
+
> **Note:** `select` filters selected elements from the result of API calls, while the `query` parameters gives filtering parameters to the API when listing elements.
|
870
973
|
|
871
974
|
#### Verbosity of output
|
872
975
|
|
873
976
|
Output messages are categorized in 3 types:
|
874
977
|
|
875
|
-
|
876
|
-
|
877
|
-
|
978
|
+
- `info` output contain additional information, such as number of elements in a table
|
979
|
+
- `data` output contain the actual output of the command (object, or list of objects)
|
980
|
+
- `error`output contain error messages
|
878
981
|
|
879
982
|
The option `display` controls the level of output:
|
880
983
|
|
881
|
-
|
882
|
-
|
883
|
-
|
984
|
+
- `info` displays all messages: `info`, `data`, and `error`
|
985
|
+
- `data` display `data` and `error` messages
|
986
|
+
- `error` display only error messages.
|
884
987
|
|
885
988
|
By default, secrets are removed from output: option `show_secrets` defaults to `no`, unless `display` is `data`, to allows piping results.
|
886
989
|
To hide secrets from output, set option `show_secrets` to `no`.
|
@@ -889,12 +992,12 @@ To hide secrets from output, set option `show_secrets` to `no`.
|
|
889
992
|
|
890
993
|
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
891
994
|
|
892
|
-
|
893
|
-
|
894
|
-
|
895
|
-
|
896
|
-
|
897
|
-
|
995
|
+
- DEF : default display of columns (that's the default, when not set)
|
996
|
+
- ALL : all columns available
|
997
|
+
- a,b,c : the list of attributes specified by the comma separated list
|
998
|
+
- Array extended value: for instance, @json:'["a","b","c"]' same as above
|
999
|
+
- +a,b,c : add selected properties to the default selection.
|
1000
|
+
- -a,b,c : remove selected properties from the default selection.
|
898
1001
|
|
899
1002
|
### <a id="extended"></a>Extended Value Syntax
|
900
1003
|
|
@@ -910,24 +1013,24 @@ The difference between reader and decoder is order and ordinality. Both act like
|
|
910
1013
|
|
911
1014
|
The following "readers" are supported (returns value in []):
|
912
1015
|
|
913
|
-
|
914
|
-
|
915
|
-
|
916
|
-
|
917
|
-
|
918
|
-
|
919
|
-
|
1016
|
+
- @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
|
1017
|
+
- @file:PATH : [String] read value from a URL, e.g. `--fpac=@uri:http://serv/f.pac`
|
1018
|
+
- @uri:URL : [String] read value from a file (prefix `~/` is replaced with the users home folder), e.g. `--key=@file:~/.ssh/mykey`
|
1019
|
+
- @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
|
1020
|
+
- @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
1021
|
+
- @stdin: : [String] read from stdin (no value on right)
|
1022
|
+
- @preset:NAME : [Hash] get whole option preset value by name. Subvalues can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]`
|
920
1023
|
|
921
1024
|
In addition it is possible to decode a value, using one or multiple decoders :
|
922
1025
|
|
923
|
-
|
924
|
-
|
925
|
-
|
926
|
-
|
927
|
-
|
928
|
-
|
929
|
-
|
930
|
-
|
1026
|
+
- @base64: [String] decode a base64 encoded string
|
1027
|
+
- @json: [any] decode JSON values (convenient to provide complex structures)
|
1028
|
+
- @zlib: [String] uncompress data
|
1029
|
+
- @ruby: [any] execute ruby code
|
1030
|
+
- @csvt: [Array] decode a titled CSV value
|
1031
|
+
- @lines: [Array] split a string in multiple lines and return an array
|
1032
|
+
- @list: [Array] split a string in multiple items taking first character as separator and return an array
|
1033
|
+
- @incps: [Hash] include values of presets specified by key `incps` in input hash
|
931
1034
|
|
932
1035
|
To display the result of an extended value, use the `config echo` command.
|
933
1036
|
|
@@ -978,7 +1081,7 @@ ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
|
978
1081
|
{"version"=>"0.9", "hello"=>true}
|
979
1082
|
```
|
980
1083
|
|
981
|
-
Note
|
1084
|
+
> **Note:** `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` are equivalent to: `@preset:config`
|
982
1085
|
|
983
1086
|
### <a id="native"></a>Structured Value
|
984
1087
|
|
@@ -1057,7 +1160,7 @@ The command `update` allows the easy creation of [option preset](#lprt) by simpl
|
|
1057
1160
|
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_ --ts=@json:'{"precalculate_job_size":true}'
|
1058
1161
|
```
|
1059
1162
|
|
1060
|
-
|
1163
|
+
- This creates a [option preset](#lprt) `demo_server` with all provided options.
|
1061
1164
|
|
1062
1165
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
1063
1166
|
|
@@ -1090,6 +1193,8 @@ If necessary, the configuration file can opened in a text editor with:
|
|
1090
1193
|
ascli config open
|
1091
1194
|
```
|
1092
1195
|
|
1196
|
+
> **Note:** this starts the editor specified by env var `EDITOR` if defined.
|
1197
|
+
|
1093
1198
|
Older format for commands are still supported:
|
1094
1199
|
|
1095
1200
|
```bash
|
@@ -1108,7 +1213,7 @@ This preset name is reserved and contains an array of key-value , where the key
|
|
1108
1213
|
|
1109
1214
|
When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
|
1110
1215
|
|
1111
|
-
Note
|
1216
|
+
> **Note:** Special plugin name: `config` can be associated with a preset that is loaded initially, typically used for default values.
|
1112
1217
|
|
1113
1218
|
Operations on this preset are done using regular `config` operations:
|
1114
1219
|
|
@@ -1124,9 +1229,9 @@ ascli config preset get default _plugin_name_
|
|
1124
1229
|
"_default_preset_for_plugin_"
|
1125
1230
|
```
|
1126
1231
|
|
1127
|
-
#### <a id="
|
1232
|
+
#### <a id="config"></a>Plugin: `config`: CLI Configuration
|
1128
1233
|
|
1129
|
-
Plugin `config`
|
1234
|
+
Plugin `config` is used to configure `ascli` and also contains global options.
|
1130
1235
|
|
1131
1236
|
When `ascli` starts, it looks for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
1132
1237
|
|
@@ -1195,16 +1300,16 @@ demo_server:
|
|
1195
1300
|
|
1196
1301
|
We can see here:
|
1197
1302
|
|
1198
|
-
|
1199
|
-
|
1200
|
-
|
1201
|
-
|
1303
|
+
- The configuration was created with CLI version 0.3.7
|
1304
|
+
- the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
1305
|
+
- the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
1306
|
+
- the default [option preset](#lprt) to load in any case is : `cli_default`
|
1202
1307
|
|
1203
1308
|
Two [option presets](#lprt) are reserved:
|
1204
1309
|
|
1205
|
-
|
1310
|
+
- `config` contains a single value: `version` showing the CLI
|
1206
1311
|
version used to create the configuration file. It is used to check compatibility.
|
1207
|
-
|
1312
|
+
- `default` is reserved to define the default [option preset](#lprt) name used for known plugins.
|
1208
1313
|
|
1209
1314
|
The user may create as many [option presets](#lprt) as needed. For instance, a particular [option preset](#lprt) can be created for a particular application instance and contain URL and credentials.
|
1210
1315
|
|
@@ -1233,11 +1338,11 @@ Some options are global, some options are available only for some plugins. (the
|
|
1233
1338
|
|
1234
1339
|
Options are loaded using this algorithm:
|
1235
1340
|
|
1236
|
-
|
1237
|
-
|
1238
|
-
|
1239
|
-
|
1240
|
-
|
1341
|
+
- If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
1342
|
+
- else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
1343
|
+
- If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
1344
|
+
- Environment variables are evaluated
|
1345
|
+
- Command line options are evaluated
|
1241
1346
|
|
1242
1347
|
Parameters are evaluated in the order of command line.
|
1243
1348
|
|
@@ -1286,7 +1391,7 @@ ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=_p
|
|
1286
1391
|
|
1287
1392
|
This can also be provisioned in a config file:
|
1288
1393
|
|
1289
|
-
|
1394
|
+
- Build [option preset](#lprt)
|
1290
1395
|
|
1291
1396
|
```bash
|
1292
1397
|
ascli config preset set shares06 url https://10.25.0.6
|
@@ -1306,19 +1411,19 @@ or
|
|
1306
1411
|
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=_pass_here_
|
1307
1412
|
```
|
1308
1413
|
|
1309
|
-
|
1414
|
+
- Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
1310
1415
|
|
1311
1416
|
```bash
|
1312
1417
|
ascli config preset set default shares shares06
|
1313
1418
|
```
|
1314
1419
|
|
1315
|
-
|
1420
|
+
- Display the content of configuration file in table format
|
1316
1421
|
|
1317
1422
|
```bash
|
1318
1423
|
ascli config overview
|
1319
1424
|
```
|
1320
1425
|
|
1321
|
-
|
1426
|
+
- Execute a command on the shares application using default parameters
|
1322
1427
|
|
1323
1428
|
```bash
|
1324
1429
|
ascli shares repo browse /
|
@@ -1326,67 +1431,86 @@ ascli shares repo browse /
|
|
1326
1431
|
|
1327
1432
|
### <a id="vault"></a>Secret Vault
|
1328
1433
|
|
1329
|
-
|
1434
|
+
Password and secrets are command options.
|
1435
|
+
They can be provided on command line, env vars, files etc.
|
1436
|
+
A more secure option is to retrieve values from a secret vault.
|
1330
1437
|
|
1331
|
-
|
1438
|
+
The vault is used with options `vault` and `vault_password`.
|
1332
1439
|
|
1333
|
-
|
1440
|
+
`vault` defines the vault to be used and shall be a Hash, example:
|
1334
1441
|
|
1335
|
-
|
1442
|
+
```json
|
1443
|
+
{"type":"system","name":"ascli"}
|
1444
|
+
```
|
1336
1445
|
|
1337
|
-
|
1446
|
+
`vault_password` specifies the password for the vault.
|
1447
|
+
Although it can be specified on command line, for security reason you can hide the value.
|
1448
|
+
For example it can be securely specified on command line like this:
|
1338
1449
|
|
1339
|
-
|
1450
|
+
```bash
|
1451
|
+
export ASCLI_VAULT_PASSWORD
|
1452
|
+
read -s ASCLI_VAULT_PASSWORD
|
1453
|
+
```
|
1340
1454
|
|
1341
|
-
####
|
1455
|
+
#### Vault: System keychain
|
1342
1456
|
|
1343
|
-
|
1344
|
-
For this use the `config vault` command.
|
1457
|
+
> **Note:** **macOS only**
|
1345
1458
|
|
1346
|
-
|
1459
|
+
It is possible to manage secrets in macOS keychain (only read supported currently).
|
1347
1460
|
|
1348
|
-
|
1461
|
+
```json
|
1462
|
+
--vault=@json:'{"type":"system","name":"ascli"}'
|
1463
|
+
```
|
1349
1464
|
|
1350
|
-
|
1351
|
-
* `get`
|
1352
|
-
* `list`
|
1353
|
-
* `delete`
|
1465
|
+
#### Vault: Encrypted file
|
1354
1466
|
|
1355
|
-
|
1467
|
+
It is possible to store and use secrets encrypted in a file.
|
1356
1468
|
|
1357
|
-
|
1469
|
+
```json
|
1470
|
+
--vault=@json:'{"type":"file","name":"vault.bin"}'
|
1471
|
+
```
|
1472
|
+
|
1473
|
+
`name` is the file path, absolute or relative to the config folder `ASCLI_HOME`.
|
1358
1474
|
|
1359
|
-
|
1475
|
+
#### Vault: Operations
|
1360
1476
|
|
1361
|
-
|
1477
|
+
For this use the `config vault` command.
|
1478
|
+
|
1479
|
+
Then secrets can be manipulated using commands:
|
1362
1480
|
|
1363
|
-
|
1481
|
+
- `create`
|
1482
|
+
- `show`
|
1483
|
+
- `list`
|
1484
|
+
- `delete`
|
1364
1485
|
|
1365
1486
|
```bash
|
1366
|
-
ascli conf
|
1487
|
+
ascli conf vault create mylabel @json:'{"password":"__value_here__","description":"for this account"}'
|
1488
|
+
```
|
1367
1489
|
|
1368
|
-
|
1490
|
+
#### <a id="config_finder"></a>Configuration Finder
|
1369
1491
|
|
1370
|
-
|
1492
|
+
When a secret is needed by a sub command, the command can search for existing configurations in the config file.
|
1371
1493
|
|
1372
|
-
|
1373
|
-
|
1494
|
+
The lookup is done by comparing the service URL and username (or access key).
|
1495
|
+
|
1496
|
+
#### Securing passwords and secrets
|
1374
1497
|
|
1375
|
-
|
1376
|
-
|
1498
|
+
A passwords can be saved in clear in a [option preset](#lprt) together with other account information (URL, username, etc...).
|
1499
|
+
Example:
|
1377
1500
|
|
1378
1501
|
```bash
|
1379
|
-
ascli conf
|
1502
|
+
`ascli` conf preset update myconf --url=... --username=... --password=...
|
1380
1503
|
```
|
1381
1504
|
|
1382
|
-
|
1383
|
-
|
1384
|
-
To test if a secret can be found use:
|
1505
|
+
For a more secure storage one can do:
|
1385
1506
|
|
1386
1507
|
```bash
|
1387
|
-
ascli conf
|
1508
|
+
`ascli` conf preset update myconf --url=... --username=... --password=@val:@vault:myconf.password
|
1509
|
+
`ascli` conf vault create myconf @json:'{"password":"__value_here__"}'
|
1388
1510
|
```
|
1389
1511
|
|
1512
|
+
> **Note:** use `@val:` in front of `@vault:` so that the extended value is not evaluated.
|
1513
|
+
|
1390
1514
|
### <a id="private_key"></a>Private Key
|
1391
1515
|
|
1392
1516
|
Some applications allow the user to be authenticated using a private key (Server, AoC, Faspex5, ...).
|
@@ -1397,15 +1521,17 @@ The private key can be protected by a passphrase or not.
|
|
1397
1521
|
If the key is protected by a passphrase, then it will be prompted.
|
1398
1522
|
(some plugins support option `passphrase`)
|
1399
1523
|
|
1400
|
-
|
1401
|
-
|
1402
|
-
|
1524
|
+
The following commands use the shell variable `PRIVKEYFILE`.
|
1525
|
+
Set it to the desired safe location of the private key.
|
1526
|
+
Typically, in `$HOME/.ssh` or `$HOME/.aspera/ascli`:
|
1403
1527
|
|
1404
1528
|
```bash
|
1405
1529
|
PRIVKEYFILE=~/.aspera/ascli/my_private_key
|
1406
1530
|
```
|
1407
1531
|
|
1408
|
-
|
1532
|
+
Several methods can be used to generate a key pair:
|
1533
|
+
|
1534
|
+
- `ascli`
|
1409
1535
|
|
1410
1536
|
The generated key is of type RSA, by default: 4096 bit.
|
1411
1537
|
For convenience, the public key is also extracted with extension `.pub`.
|
@@ -1415,23 +1541,30 @@ The key is not passphrase protected.
|
|
1415
1541
|
ascli config genkey ${PRIVKEYFILE} 4096
|
1416
1542
|
```
|
1417
1543
|
|
1418
|
-
|
1544
|
+
- `ssh-keygen`
|
1419
1545
|
|
1420
1546
|
Both private and public keys are generated, option `-N` is for passphrase.
|
1421
1547
|
|
1422
1548
|
```bash
|
1423
|
-
ssh-keygen -t rsa -b 4096 -N '' -f ${PRIVKEYFILE}
|
1549
|
+
ssh-keygen -t rsa -b 4096 -m PEM -N '' -f ${PRIVKEYFILE}
|
1424
1550
|
```
|
1425
1551
|
|
1426
|
-
|
1552
|
+
- `openssl`
|
1427
1553
|
|
1428
|
-
|
1429
|
-
To generate a private key pair without passphrase the following can be used on any system:
|
1554
|
+
To generate a private key pair with a passphrase the following can be used on any system:
|
1430
1555
|
|
1431
1556
|
```bash
|
1432
|
-
openssl genrsa -passout pass:
|
1433
|
-
openssl rsa -passin pass:dummypassword -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1557
|
+
openssl genrsa -passout pass:_passphrase_here_ -out ${PRIVKEYFILE}.protected 4096
|
1434
1558
|
openssl rsa -pubout -in ${PRIVKEYFILE} -out ${PRIVKEYFILE}.pub
|
1559
|
+
```
|
1560
|
+
|
1561
|
+
`openssl` is sometimes compiled to support option `-nodes` (no DES, i.e. no passphrase, e.g. on macOS).
|
1562
|
+
In that case, add option `-nodes` instead of `-passout pass:_passphrase_here_` to generate a key without passphrase.
|
1563
|
+
|
1564
|
+
If option `-nodes` is not available, the passphrase can be removed using this method:
|
1565
|
+
|
1566
|
+
```bash
|
1567
|
+
openssl rsa -passin pass:_passphrase_here_ -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1435
1568
|
rm -f ${PRIVKEYFILE}.protected
|
1436
1569
|
```
|
1437
1570
|
|
@@ -1505,9 +1638,9 @@ REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, O
|
|
1505
1638
|
|
1506
1639
|
Those are using options:
|
1507
1640
|
|
1508
|
-
|
1509
|
-
|
1510
|
-
|
1641
|
+
- url
|
1642
|
+
- username
|
1643
|
+
- password
|
1511
1644
|
|
1512
1645
|
Those can be provided using command line, parameter set, env var, see section above.
|
1513
1646
|
|
@@ -1528,17 +1661,17 @@ Available loggers: `stdout`, `stderr`, `syslog`.
|
|
1528
1661
|
|
1529
1662
|
Available levels: `debug`, `info`, `warn`, `error`.
|
1530
1663
|
|
1531
|
-
Note
|
1664
|
+
> **Note:** When using the `direct` agent (`ascp`), additional transfer logs can be activated using `ascp` option `EX_ascp_args`, see [`direct`](#agt_direct).
|
1532
1665
|
|
1533
1666
|
Examples:
|
1534
1667
|
|
1535
|
-
|
1668
|
+
- display debugging log on `stdout`:
|
1536
1669
|
|
1537
1670
|
```bash
|
1538
1671
|
ascli conf over --log-level=debug --logger=stdout
|
1539
1672
|
```
|
1540
1673
|
|
1541
|
-
|
1674
|
+
- log errors to `syslog`:
|
1542
1675
|
|
1543
1676
|
```bash
|
1544
1677
|
ascli conf over --log-level=error --logger=syslog
|
@@ -1582,76 +1715,100 @@ ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
|
1582
1715
|
|
1583
1716
|
Some actions may require the use of a graphical tool:
|
1584
1717
|
|
1585
|
-
|
1586
|
-
|
1718
|
+
- a browser for Aspera on Cloud authentication (web auth method)
|
1719
|
+
- a text editor for configuration file edition
|
1587
1720
|
|
1588
1721
|
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1589
1722
|
It is also possible to force the graphical mode with option --ui :
|
1590
1723
|
|
1591
|
-
|
1592
|
-
|
1724
|
+
- `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
1725
|
+
- `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
1593
1726
|
|
1594
1727
|
### Proxy
|
1595
1728
|
|
1596
|
-
There are several types of network connections, each of them use a different mechanism to define a
|
1729
|
+
There are several types of network connections, each of them use a different mechanism to define a (forward) **proxy**:
|
1597
1730
|
|
1598
|
-
|
1599
|
-
|
1600
|
-
|
1601
|
-
See [Ruby findproxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1602
|
-
|
1603
|
-
```bash
|
1604
|
-
export http_proxy=http://myproxy.org.net:3128
|
1605
|
-
```
|
1731
|
+
- Ruby HTTP: REST and HTTPGW client
|
1732
|
+
- Legacy Aspera HTTP/S Fallback
|
1733
|
+
- Aspera FASP
|
1606
1734
|
|
1607
|
-
|
1735
|
+
Refer to the following sections.
|
1608
1736
|
|
1609
|
-
|
1737
|
+
### Proxy for REST and HTTPGW
|
1610
1738
|
|
1611
|
-
|
1739
|
+
There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is used.
|
1612
1740
|
|
1613
|
-
|
1741
|
+
The `http_proxy` environment variable (**lower case**, preferred) can be set to the URL of the proxy, e.g. `http://myproxy.org.net:3128`.
|
1742
|
+
Refer to [Ruby findproxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1614
1743
|
|
1615
|
-
|
1616
|
-
(It is also possible to use `EX_ascp_args` and native options in `direct`)
|
1744
|
+
> **Note:** Ruby expects a URL and `myproxy.org.net:3128` alone is **not** accepted.
|
1617
1745
|
|
1618
|
-
|
1746
|
+
```bash
|
1747
|
+
export http_proxy=http://proxy.example.com:3128
|
1748
|
+
```
|
1619
1749
|
|
1620
|
-
The `fpac` option
|
1621
|
-
To read the script from a URL (`http:`, `https:` and `file:`), use: `@uri:`.
|
1622
|
-
A minimal script can be specified
|
1750
|
+
The `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) javascript value.
|
1751
|
+
To read the script from a URL (`http:`, `https:` and `file:`), use prefix: `@uri:`.
|
1752
|
+
A minimal script can be specified to define the use of a local proxy:
|
1623
1753
|
|
1624
1754
|
```bash
|
1625
|
-
|
1755
|
+
ascli --fpac='function FindProxyForURL(url, host){return "PROXY localhost:3128"}' ...
|
1626
1756
|
```
|
1627
1757
|
|
1628
|
-
The PAC file
|
1629
|
-
|
1630
|
-
The PAC file can be tested with command: `config proxy_check`. Example, using command line option:
|
1758
|
+
The result of a PAC file can be tested with command: `config proxy_check`.
|
1759
|
+
Example, using command line option:
|
1631
1760
|
|
1632
1761
|
```bash
|
1633
|
-
ascli conf proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:
|
1762
|
+
ascli conf proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:3128;DIRECT";}' http://example.com
|
1763
|
+
```
|
1764
|
+
|
1765
|
+
```text
|
1634
1766
|
PROXY proxy.example.com:1234;DIRECT
|
1635
1767
|
```
|
1636
1768
|
|
1637
1769
|
```bash
|
1638
1770
|
ascli config proxy_check --fpac=@file:./proxy.pac http://www.example.com
|
1771
|
+
```
|
1772
|
+
|
1773
|
+
```text
|
1639
1774
|
PROXY proxy.example.com:8080
|
1640
1775
|
```
|
1641
1776
|
|
1642
1777
|
```bash
|
1643
1778
|
ascli config proxy_check --fpac=@uri:http://server/proxy.pac http://www.example.com
|
1779
|
+
```
|
1780
|
+
|
1781
|
+
```text
|
1644
1782
|
PROXY proxy.example.com:8080
|
1645
1783
|
```
|
1646
1784
|
|
1785
|
+
If the proxy requires credentials, then use option `proxy_credentials` with username and password provided as an `Array`:
|
1786
|
+
|
1787
|
+
```bash
|
1788
|
+
ascli --proxy-credentials=@json:'["__username_here__","__password_here__"]' ...
|
1789
|
+
```
|
1790
|
+
|
1791
|
+
```bash
|
1792
|
+
ascli --proxy-credentials=@list::__username_here__:__password_here__ ...
|
1793
|
+
```
|
1794
|
+
|
1795
|
+
### Proxy for Legacy Aspera HTTP/S Fallback
|
1796
|
+
|
1797
|
+
To specify a proxy for legacy HTTP fallback, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url` (only supported with the `direct` agent).
|
1798
|
+
(It is also possible to use `EX_ascp_args` and native options in `direct`)
|
1799
|
+
|
1800
|
+
### FASP proxy (forward) for transfers
|
1801
|
+
|
1802
|
+
To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) parameter: `EX_fasp_proxy_url` (only supported with the `direct` agent).
|
1803
|
+
|
1647
1804
|
### <a id="client"></a>FASP configuration
|
1648
1805
|
|
1649
1806
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
1650
1807
|
|
1651
|
-
|
1652
|
-
|
1653
|
-
|
1654
|
-
|
1808
|
+
- `show` : shows the path of `ascp` used
|
1809
|
+
- `use` : list,download connect client versions available on internet
|
1810
|
+
- `products` : list Aspera transfer products available locally
|
1811
|
+
- `connect` : list,download connect client versions available on internet
|
1655
1812
|
|
1656
1813
|
#### Show path of currently used `ascp`
|
1657
1814
|
|
@@ -1677,9 +1834,9 @@ ascli config ascp info
|
|
1677
1834
|
|
1678
1835
|
#### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
1679
1836
|
|
1680
|
-
By default, `ascli` uses any found local product with ascp
|
1837
|
+
By default, `ascli` uses any found local product with `ascp`, including SDK.
|
1681
1838
|
|
1682
|
-
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
1839
|
+
To temporarily use an alternate `ascp` path use option `ascp_path` (`--ascp-path=`)
|
1683
1840
|
|
1684
1841
|
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
1685
1842
|
|
@@ -1730,11 +1887,11 @@ ascli config ascp products list
|
|
1730
1887
|
|
1731
1888
|
#### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
1732
1889
|
|
1733
|
-
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1890
|
+
If no `ascp` is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1734
1891
|
|
1735
|
-
Using the option use_product finds the ascp binary of the selected product.
|
1892
|
+
Using the option use_product finds the `ascp` binary of the selected product.
|
1736
1893
|
|
1737
|
-
To permanently use the ascp of a product:
|
1894
|
+
To permanently use the `ascp` of a product:
|
1738
1895
|
|
1739
1896
|
```bash
|
1740
1897
|
ascli config ascp products use 'Aspera Connect'
|
@@ -1780,7 +1937,7 @@ ascli config ascp connect version 'Aspera Connect for Mac Intel' download enclos
|
|
1780
1937
|
```
|
1781
1938
|
|
1782
1939
|
```output
|
1783
|
-
Time: 00:00:02
|
1940
|
+
Time: 00:00:02 =========================================================== 100% 27766 KB/sec Time: 00:00:02
|
1784
1941
|
Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
1785
1942
|
```
|
1786
1943
|
|
@@ -1793,21 +1950,21 @@ This [*transfer-spec*](#transferspec) will be executed by a transfer client, her
|
|
1793
1950
|
|
1794
1951
|
There are currently 3 agents:
|
1795
1952
|
|
1796
|
-
|
1797
|
-
|
1798
|
-
|
1799
|
-
|
1800
|
-
|
1953
|
+
- [`direct`](#agt_direct) : a local execution of `ascp`
|
1954
|
+
- [`connect`](#agt_connect) : use of a local Connect Client
|
1955
|
+
- [`node`](#agt_node) : use of an Aspera Transfer Node (potentially *remote*).
|
1956
|
+
- [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1957
|
+
- [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1801
1958
|
|
1802
|
-
Note
|
1803
|
-
For
|
1959
|
+
> **Note:** All transfer operations are seen from the point of view of the agent.
|
1960
|
+
For example, a node agent executing an "upload", or "package send" operation
|
1804
1961
|
will effectively push files to the related server from the agent node.
|
1805
1962
|
|
1806
|
-
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *
|
1963
|
+
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *native* `ascp` options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1807
1964
|
|
1808
1965
|
#### <a id="agt_direct"></a>Direct
|
1809
1966
|
|
1810
|
-
The `direct` agent directly executes a local ascp
|
1967
|
+
The `direct` agent directly executes a local `ascp`.
|
1811
1968
|
This is the default agent for `ascli`.
|
1812
1969
|
This is equivalent to option `--transfer=direct`.
|
1813
1970
|
`ascli` will detect locally installed Aspera products, including SDK, and use `ascp` from that component.
|
@@ -1815,18 +1972,17 @@ Refer to section [FASP](#client).
|
|
1815
1972
|
|
1816
1973
|
The `transfer_info` option accepts the following optional parameters to control multi-session, Web Socket Session and Resume policy:
|
1817
1974
|
|
1818
|
-
|
1819
|
-
|
1820
|
-
|
1821
|
-
|
1822
|
-
|
1823
|
-
|
1824
|
-
|
1825
|
-
|
1826
|
-
|
1827
|
-
|
1828
|
-
|
1829
|
-
</table>
|
1975
|
+
| Name | Type | Description |
|
1976
|
+
|----------------------|-------|-------------|
|
1977
|
+
| wss | Bool | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
1978
|
+
| spawn_timeout_sec | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
|
1979
|
+
| spawn_delay_sec | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
|
1980
|
+
| multi_incr_udp | Bool | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
|
1981
|
+
| resume | Hash | Resume<br/>parameters<br/>See below |
|
1982
|
+
| resume.iter_max | int | Resume<br/>Max number of retry on error<br/>Default: 7 |
|
1983
|
+
| resume.sleep_initial | int | Resume<br/>First Sleep before retry<br/>Default: 2 |
|
1984
|
+
| resume.sleep_factor | int | Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2 |
|
1985
|
+
| resume.sleep_max | int | Resume<br/>Default: 60 |
|
1830
1986
|
|
1831
1987
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
1832
1988
|
Sleep between iterations is:
|
@@ -1849,35 +2005,46 @@ ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":20}}'
|
|
1849
2005
|
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
1850
2006
|
```
|
1851
2007
|
|
1852
|
-
Note
|
2008
|
+
> **Note:** The `direct` agent supports additional `transfer_spec` parameters starting with `EX_` (extended).
|
1853
2009
|
In particular the field, `EX_ascp_args` which is a list of additional command line options to `ascp`.
|
1854
2010
|
|
1855
|
-
This can be useful to activate logging using option `-L` of ascp
|
2011
|
+
This can be useful to activate logging using option `-L` of `ascp`.
|
1856
2012
|
For example the option `--ts=@json:'{"EX_ascp_args":["-DDL-"]}'` will activate debug level 2 for `ascp` (`DD`), and display those logs on the terminal (`-`).
|
1857
2013
|
This is useful if the transfer fails.
|
1858
|
-
To store ascp logs in file `aspera-scp-transfer.log` in a folder, use `--ts=@json:'{"EX_ascp_args":["-L","/path/to/folder"]}'`.
|
2014
|
+
To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--ts=@json:'{"EX_ascp_args":["-L","/path/to/folder"]}'`.
|
1859
2015
|
|
1860
|
-
> Implementation note: when transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided in option `ts` in `EX_ascp_args`).
|
2016
|
+
> **Note:** Implementation note: when transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided in option `ts` in `EX_ascp_args`).
|
1861
2017
|
|
1862
2018
|
In addition to standard methods described in section [File List](#file_list), it is possible to specify the list of file using those additional methods:
|
1863
2019
|
|
1864
|
-
|
2020
|
+
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_file_list`
|
1865
2021
|
|
1866
2022
|
```javascript
|
1867
2023
|
--sources=@ts --ts=@json:'{"EX_file_list":"filelist.txt"}'
|
1868
2024
|
```
|
1869
2025
|
|
1870
|
-
|
2026
|
+
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_ascp_args`
|
1871
2027
|
|
1872
2028
|
```javascript
|
1873
2029
|
--sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","myfilelist"]}'
|
1874
2030
|
```
|
1875
2031
|
|
1876
|
-
> File lists is shown here, there are also similar options for file pair lists.
|
2032
|
+
> **Note:** File lists is shown here, there are also similar options for file pair lists.
|
2033
|
+
>
|
2034
|
+
> **Note:** Those 2 additional methods avoid the creation of a copy of the file list: if the standard options `--sources=@lines:@file:... --src-type=...` are used, then the file is list read and parsed, and a new file list is created in a temporary folder.
|
2035
|
+
>
|
2036
|
+
> **Note:** Those methods have limitations: they apply **only** to the [`direct`](#agt_direct) transfer agent (i.e. local `ascp`) and not for Aspera on Cloud.
|
1877
2037
|
|
1878
|
-
|
2038
|
+
In addition to special transfer spec parameter `EX_ascp_args`, it is possible to provide the same `ascp` options using option `ascp_opts` of `ascli`.
|
2039
|
+
This option expects an `Array`, which can be conveniently provided with extended syntax `@list:`
|
1879
2040
|
|
1880
|
-
|
2041
|
+
```bash
|
2042
|
+
ascli server download /aspera-test-dir-large/200MB --ascp-opts=@list:' -l 10m -k 3'
|
2043
|
+
```
|
2044
|
+
|
2045
|
+
> **Note:** Using `@list:`, the use of quotes and leading character "space" here is important: The `@list:` expects a single string which must not be parsed by the shell (so protected with quotes) and the leading space is the separator character.
|
2046
|
+
>
|
2047
|
+
> **Note:** The option `ascp_opts` are appended to `EX_ascp_args` if present.
|
1881
2048
|
|
1882
2049
|
#### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
1883
2050
|
|
@@ -1889,13 +2056,12 @@ By specifying option: `--transfer=node`, the CLI will start transfers in an Aspe
|
|
1889
2056
|
Transfer Server using the Node API, either on a local or remote node.
|
1890
2057
|
Parameters provided in option `transfer_info` are:
|
1891
2058
|
|
1892
|
-
|
1893
|
-
|
1894
|
-
|
1895
|
-
|
1896
|
-
|
1897
|
-
|
1898
|
-
</table>
|
2059
|
+
| Name | Type | Description |
|
2060
|
+
|------|------|-------------|
|
2061
|
+
| url | string | URL of the node API</br>Mandatory |
|
2062
|
+
| username | string | node api user or access key</br>Mandatory |
|
2063
|
+
| password | string | password, secret or bearer token</br>Mandatory |
|
2064
|
+
| root_id | string | password or secret</br>Mandatory only for bearer token |
|
1899
2065
|
|
1900
2066
|
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
1901
2067
|
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
@@ -1907,7 +2073,15 @@ If the `password` value begins with `Bearer` then the `username` is expected to
|
|
1907
2073
|
|
1908
2074
|
#### <a id="agt_httpgw"></a>HTTP Gateway
|
1909
2075
|
|
1910
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
2076
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
2077
|
+
|
2078
|
+
Parameters provided in option `transfer_info` are:
|
2079
|
+
|
2080
|
+
| Name | Type | Description |
|
2081
|
+
|------|------|-------------|
|
2082
|
+
| url | string | URL of the HTTP GW</br>Mandatory |
|
2083
|
+
| upload_bar_refresh_sec | float | Refresh rate for upload progress bar |
|
2084
|
+
| upload_chunksize | int | Size in bytes of chunks for upload |
|
1911
2085
|
|
1912
2086
|
Example:
|
1913
2087
|
|
@@ -1915,7 +2089,7 @@ Example:
|
|
1915
2089
|
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1916
2090
|
```
|
1917
2091
|
|
1918
|
-
Note
|
2092
|
+
> **Note:** The gateway only supports transfers authorized with a token.
|
1919
2093
|
|
1920
2094
|
#### <a id="agt_trsdk"></a>Transfer SDK
|
1921
2095
|
|
@@ -1923,7 +2097,8 @@ Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
|
1923
2097
|
|
1924
2098
|
By default it will listen on local port `55002` on `127.0.0.1`.
|
1925
2099
|
|
1926
|
-
The gem `grpc` was removed from dependencies, as it requires compilation of a native part.
|
2100
|
+
The gem `grpc` was removed from dependencies, as it requires compilation of a native part.
|
2101
|
+
So, to use the Transfer SDK you should install this gem:
|
1927
2102
|
|
1928
2103
|
```bash
|
1929
2104
|
gem install grpc
|
@@ -1931,9 +2106,9 @@ gem install grpc
|
|
1931
2106
|
|
1932
2107
|
On Windows the compilation may fail for various reasons (3.1.1):
|
1933
2108
|
|
1934
|
-
|
2109
|
+
- `cannot find -lx64-ucrt-ruby310`
|
1935
2110
|
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
1936
|
-
|
2111
|
+
- `conflicting types for 'gettimeofday'`
|
1937
2112
|
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)` ,i.e. change `struct timezone` to `void`
|
1938
2113
|
|
1939
2114
|
### <a id="transferspec"></a>Transfer Specification
|
@@ -1941,22 +2116,22 @@ On Windows the compilation may fail for various reasons (3.1.1):
|
|
1941
2116
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
1942
2117
|
is described in a [*transfer-spec*](#transferspec) (Transfer Specification), such as:
|
1943
2118
|
|
1944
|
-
|
1945
|
-
|
1946
|
-
|
1947
|
-
|
1948
|
-
|
2119
|
+
- server address
|
2120
|
+
- transfer user name
|
2121
|
+
- credentials
|
2122
|
+
- file list
|
2123
|
+
- etc...
|
1949
2124
|
|
1950
2125
|
`ascli` builds a default [*transfer-spec*](#transferspec) internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1951
2126
|
|
1952
2127
|
If needed, it is possible to modify or add any of the supported [*transfer-spec*](#transferspec) parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several [*transfer-spec*](#transferspec) parameters. Multiple `ts` options on command line are cumulative.
|
1953
2128
|
|
1954
|
-
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [*transfer-spec*](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
2129
|
+
It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agt_direct) using the special [*transfer-spec*](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for `ascp` command line parameters not supported yet in the transfer spec.
|
1955
2130
|
|
1956
2131
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
1957
2132
|
|
1958
|
-
|
1959
|
-
|
2133
|
+
- common to all [Transfer Agent](#agents)
|
2134
|
+
- not dependent on command line limitations (special characters...)
|
1960
2135
|
|
1961
2136
|
A [*transfer-spec*](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
1962
2137
|
|
@@ -1967,9 +2142,9 @@ All standard [*transfer-spec*](#transferspec) parameters can be specified.
|
|
1967
2142
|
|
1968
2143
|
References:
|
1969
2144
|
|
1970
|
-
|
1971
|
-
|
1972
|
-
|
2145
|
+
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22) → /opt/transfers
|
2146
|
+
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22) → Guides → API Ref → Transfer Spec V1
|
2147
|
+
- [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
1973
2148
|
|
1974
2149
|
Parameters can be displayed with commands:
|
1975
2150
|
|
@@ -1980,106 +2155,106 @@ ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
|
1980
2155
|
|
1981
2156
|
Columns:
|
1982
2157
|
|
1983
|
-
|
1984
|
-
|
1985
|
-
|
2158
|
+
- D=Direct (local `ascp` execution)
|
2159
|
+
- N=Node API
|
2160
|
+
- C=Connect Client
|
1986
2161
|
|
1987
2162
|
`ascp` argument or environment variable is provided in description.
|
1988
2163
|
|
1989
2164
|
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in `ascli`).
|
1990
2165
|
|
1991
|
-
|
1992
|
-
|
1993
|
-
|
1994
|
-
|
1995
|
-
|
1996
|
-
|
1997
|
-
|
1998
|
-
|
1999
|
-
|
2000
|
-
|
2001
|
-
|
2002
|
-
|
2003
|
-
|
2004
|
-
|
2005
|
-
|
2006
|
-
|
2007
|
-
|
2008
|
-
|
2009
|
-
|
2010
|
-
|
2011
|
-
|
2012
|
-
|
2013
|
-
|
2014
|
-
|
2015
|
-
|
2016
|
-
|
2017
|
-
|
2018
|
-
|
2019
|
-
|
2020
|
-
|
2021
|
-
<
|
2022
|
-
|
2023
|
-
|
2024
|
-
|
2025
|
-
|
2026
|
-
|
2027
|
-
|
2028
|
-
|
2029
|
-
|
2030
|
-
|
2031
|
-
|
2032
|
-
|
2033
|
-
|
2034
|
-
|
2035
|
-
|
2036
|
-
|
2037
|
-
|
2038
|
-
|
2039
|
-
<
|
2040
|
-
|
2041
|
-
|
2042
|
-
|
2043
|
-
|
2044
|
-
|
2045
|
-
|
2046
|
-
|
2047
|
-
|
2048
|
-
|
2049
|
-
|
2050
|
-
|
2051
|
-
|
2052
|
-
|
2053
|
-
|
2054
|
-
|
2055
|
-
|
2056
|
-
|
2057
|
-
|
2058
|
-
|
2059
|
-
|
2060
|
-
|
2061
|
-
|
2062
|
-
|
2063
|
-
|
2064
|
-
|
2065
|
-
|
2066
|
-
|
2067
|
-
|
2068
|
-
|
2069
|
-
|
2070
|
-
|
2071
|
-
|
2072
|
-
|
2073
|
-
|
2074
|
-
|
2075
|
-
|
2166
|
+
| Field | Type | D | N | C | Description |
|
2167
|
+
|-------|------|---|---|---|-------------|
|
2168
|
+
| EX_ascp_args | array | Y | | | Add native command line arguments to ascp |
|
2169
|
+
| EX_at_rest_password | string | Y | | | DEPRECATED: Prefer to use standard parameter: content_protection_password<br/>(env:ASPERA_SCP_FILEPASS) |
|
2170
|
+
| EX_file_list | string | Y | | | source file list |
|
2171
|
+
| EX_file_pair_list | string | Y | | | source file pair list |
|
2172
|
+
| EX_http_proxy_url | string | Y | | | Specify the proxy server address used by HTTP Fallback<br/>(-x {string}) |
|
2173
|
+
| EX_http_transfer_jpeg | int | Y | | | HTTP transfers as JPEG file<br/>(-j {int}) |
|
2174
|
+
| EX_license_text | string | Y | | | License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE) |
|
2175
|
+
| EX_no_read | bool | Y | | | no read source<br/>(--no-read) |
|
2176
|
+
| EX_no_write | bool | Y | | | no write on destination<br/>(--no-write) |
|
2177
|
+
| EX_proxy_password | string | Y | | | Password used for Aspera proxy server authentication.<br/>May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS) |
|
2178
|
+
| EX_ssh_key_paths | array | Y | | | Use public key authentication for SSH and specify the private key file paths<br/>(-i {array}) |
|
2179
|
+
| apply_local_docroot | bool | Y | | | (--apply-local-docroot) |
|
2180
|
+
| authentication | string | | | Y | value=token for SSH bypass keys, else password asked if not provided. |
|
2181
|
+
| cipher | string | Y | Y | Y | In transit encryption type.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum}) |
|
2182
|
+
| cipher_allowed | string | Y | Y | Y | returned by node API. Valid literals include "aes-128" and "none". |
|
2183
|
+
| content_protection | string | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt {enum}) |
|
2184
|
+
| content_protection_password | string | Y | Y | Y | Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS) |
|
2185
|
+
| cookie | string | Y | Y | Y | Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE) |
|
2186
|
+
| create_dir | bool | Y | Y | Y | Specifies whether to create new directories.<br/>(-d) |
|
2187
|
+
| delete_before_transfer | bool | Y | Y | Y | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects<br/>on the source are not deleted.<br/>(--delete-before-transfer) |
|
2188
|
+
| delete_source | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
2189
|
+
| destination_root | string | Y | Y | Y | Destination root directory. |
|
2190
|
+
| dgram_size | int | Y | Y | Y | UDP datagram size in bytes<br/>(-Z {int}) |
|
2191
|
+
| direction | string | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode (conversion){enum}) |
|
2192
|
+
| exclude_newer_than | int | Y | | | skip src files with mtime > arg<br/>(--exclude-newer-than {int}) |
|
2193
|
+
| exclude_older_than | int | Y | | | skip src files with mtime < arg<br/>(--exclude-older-than {int}) |
|
2194
|
+
| fasp_port | int | Y | Y | Y | Specifies fasp (UDP) port.<br/>(-O {int}) |
|
2195
|
+
| file_checksum | string | Y | Y | | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none |
|
2196
|
+
| http_fallback | bool<br/>string | Y | Y | Y | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){bool}|{string}) |
|
2197
|
+
| http_fallback_port | int | Y | | | Specifies http port when no cipher is used<br/>(-t {int}) |
|
2198
|
+
| https_fallback_port | int | Y | Y | Y | Specifies https port when cipher is used<br/>(-t {int}) |
|
2199
|
+
| lock_min_rate | bool | Y | Y | Y | |
|
2200
|
+
| lock_min_rate_kbps | bool | Y | Y | Y | |
|
2201
|
+
| lock_rate_policy | bool | Y | Y | Y | |
|
2202
|
+
| lock_target_rate | bool | Y | Y | Y | |
|
2203
|
+
| lock_target_rate_kbps | bool | Y | Y | Y | |
|
2204
|
+
| min_rate_cap_kbps | int | Y | Y | Y | |
|
2205
|
+
| min_rate_kbps | int | Y | Y | Y | Set the minimum transfer rate in kilobits per second.<br/>(-m {int}) |
|
2206
|
+
| move_after_transfer | string | Y | Y | | The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer {string}) |
|
2207
|
+
| multi_session | int | Y | Y | Y | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/> |
|
2208
|
+
| multi_session_threshold | int | Y | Y | | Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold {int}) |
|
2209
|
+
| overwrite | string | Y | Y | Y | Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite {enum}) |
|
2210
|
+
| password | string | | Y | | Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only. |
|
2211
|
+
| paths | array | Y | Y | Y | Array of path to the source (required) and a path to the destination (optional). |
|
2212
|
+
| precalculate_job_size | bool | Y | Y | Y | Specifies whether to precalculate the job size.<br/>(--precalculate-job-size) |
|
2213
|
+
| preserve_access_time | bool | Y | Y | Y | (--preserve-access-time) |
|
2214
|
+
| preserve_acls | string | Y | | | Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls {enum}) |
|
2215
|
+
| preserve_creation_time | bool | Y | Y | Y | (--preserve-creation-time) |
|
2216
|
+
| preserve_file_owner_gid | bool | Y | | | Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid) |
|
2217
|
+
| preserve_file_owner_uid | bool | Y | | | Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid) |
|
2218
|
+
| preserve_modification_time | bool | Y | Y | Y | (--preserve-modification-time) |
|
2219
|
+
| preserve_remote_acls | string | Y | | | Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls {enum}) |
|
2220
|
+
| preserve_source_access_time | bool | Y | | | Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time) |
|
2221
|
+
| preserve_times | bool | Y | Y | Y | (--preserve-times) |
|
2222
|
+
| proxy | string | Y | | | Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy {string}) |
|
2223
|
+
| rate_policy | string | Y | Y | Y | The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy {enum}) |
|
2224
|
+
| rate_policy_allowed | string | | | Y | Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed |
|
2225
|
+
| remote_host | string | Y | Y | Y | IP or fully qualified domain name of the remote server<br/>(--host {string}) |
|
2226
|
+
| remote_password | string | Y | Y | Y | SSH session password<br/>(env:ASPERA_SCP_PASS) |
|
2227
|
+
| remote_user | string | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user {string}) |
|
2228
|
+
| remove_after_transfer | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
2229
|
+
| remove_empty_directories | bool | Y | Y | | Specifies whether to remove empty directories.<br/>(--remove-empty-directories) |
|
2230
|
+
| remove_empty_source_directory | bool | Y | | | Remove empty source subdirectories and remove the source directory itself, if empty<br/>(--remove-empty-source-directory) |
|
2231
|
+
| remove_skipped | bool | Y | Y | Y | Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped) |
|
2232
|
+
| resume_policy | string | Y | Y | Y | If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum}) |
|
2233
|
+
| retry_duration | string<br/>int | | Y | Y | Specifies how long to wait before retrying transfer. (e.g. "5min") |
|
2234
|
+
| source_root | string | Y | Y | Y | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64 (conversion){string}) |
|
2235
|
+
| source_root_id | string | | Y | | The file ID of the source root directory. Required when using Bearer token auth for the source node. |
|
2236
|
+
| src_base | string | Y | Y | | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64 (conversion){string}) |
|
2237
|
+
| ssh_port | int | Y | Y | Y | Specifies SSH (TCP) port. Default: local:22, other:33001<br/>(-P {int}) |
|
2238
|
+
| ssh_private_key | string | Y | | | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----/nMII...<br/>Note the JSON encoding: /n for newlines.<br/>(env:ASPERA_SCP_KEY) |
|
2239
|
+
| ssh_private_key_passphrase | string | Y | | | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS) |
|
2240
|
+
| sshfp | string | Y | Y | Y | Check it against server SSH host key fingerprint<br/>(--check-sshfp {string}) |
|
2241
|
+
| symlink_policy | string | Y | Y | Y | Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links {enum}) |
|
2242
|
+
| tags | hash | Y | Y | Y | Metadata for transfer as JSON<br/>(--tags64 (conversion){hash}) |
|
2243
|
+
| target_rate_cap_kbps | int | | | Y | Returned by upload/download_setup node API. |
|
2244
|
+
| target_rate_kbps | int | Y | Y | Y | Specifies desired speed for the transfer.<br/>(-l {int}) |
|
2245
|
+
| target_rate_percentage | string | Y | Y | Y | |
|
2246
|
+
| title | string | | Y | Y | Title of the transfer |
|
2247
|
+
| token | string | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
2248
|
+
| use_ascp4 | bool | Y | Y | | specify version of protocol |
|
2249
|
+
| wss_enabled | bool | Y | Y | Y | |
|
2250
|
+
| wss_port | int | Y | Y | Y | TCP port used for websocket service feed. |
|
2076
2251
|
|
2077
2252
|
#### Destination folder for transfers
|
2078
2253
|
|
2079
2254
|
The destination folder is set by `ascli` by default to:
|
2080
2255
|
|
2081
|
-
|
2082
|
-
|
2256
|
+
- `.` for downloads
|
2257
|
+
- `/` for uploads
|
2083
2258
|
|
2084
2259
|
It is specified by the [*transfer-spec*](#transferspec) parameter `destination_root`.
|
2085
2260
|
As such, it can be modified with option: `--ts=@json:'{"destination_root":"<path>"}'`.
|
@@ -2101,7 +2276,7 @@ The `sources` and `src_type` options provide convenient ways to populate the tra
|
|
2101
2276
|
|
2102
2277
|
Possible values for option `sources` are:
|
2103
2278
|
|
2104
|
-
|
2279
|
+
- `@args` : (default) the list of files (or file pair) is directly provided on the command line (after commands): unused arguments (not starting with `-`) are considered as source files.
|
2105
2280
|
So, by default, the list of files to transfer will be simply specified on the command line. Example:
|
2106
2281
|
|
2107
2282
|
```bash
|
@@ -2114,13 +2289,13 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2114
2289
|
ascli server upload --sources=@args --src-type=list ~/mysample.file secondfile
|
2115
2290
|
```
|
2116
2291
|
|
2117
|
-
|
2292
|
+
- an [Extended Value](#extended) with type **Array of String**
|
2118
2293
|
|
2119
|
-
> Note
|
2294
|
+
> **Note:** extended values can be tested with the command `conf echo`
|
2120
2295
|
|
2121
2296
|
Examples:
|
2122
2297
|
|
2123
|
-
|
2298
|
+
- Using extended value
|
2124
2299
|
|
2125
2300
|
Create the file list:
|
2126
2301
|
|
@@ -2135,28 +2310,28 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2135
2310
|
--sources=@lines:@file:myfilelist.txt
|
2136
2311
|
```
|
2137
2312
|
|
2138
|
-
|
2313
|
+
- Using JSON array
|
2139
2314
|
|
2140
2315
|
```javascript
|
2141
2316
|
--sources=@json:'["file1","file2"]'
|
2142
2317
|
```
|
2143
2318
|
|
2144
|
-
|
2319
|
+
- Using STDIN, one path per line
|
2145
2320
|
|
2146
2321
|
```bash
|
2147
2322
|
--sources=@lines:@stdin:
|
2148
2323
|
```
|
2149
2324
|
|
2150
|
-
|
2325
|
+
- Using ruby code (one path per line in file)
|
2151
2326
|
|
2152
2327
|
```ruby
|
2153
2328
|
--sources=@ruby:'File.read("myfilelist.txt").split("\n")'
|
2154
2329
|
```
|
2155
2330
|
|
2156
|
-
|
2331
|
+
- `@ts` : the user provides the list of files directly in the `paths` field of transfer spec (option `ts`).
|
2157
2332
|
Examples:
|
2158
2333
|
|
2159
|
-
|
2334
|
+
- Using transfer spec
|
2160
2335
|
|
2161
2336
|
```javascript
|
2162
2337
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
@@ -2164,12 +2339,12 @@ Examples:
|
|
2164
2339
|
|
2165
2340
|
The option `src_type` allows specifying if the list specified in option `sources` is a simple file list or if it is a file pair list.
|
2166
2341
|
|
2167
|
-
> Note
|
2342
|
+
> **Note:** Option `src_type` is not used if option `sources` is set to `@ts`
|
2168
2343
|
|
2169
2344
|
Supported values for `src_type` are:
|
2170
2345
|
|
2171
|
-
|
2172
|
-
|
2346
|
+
- `list` : (default) the path of destination is the same as source and each entry is a source file path
|
2347
|
+
- `pair` : the first element is the first source, the second element is the first destination, and so on.
|
2173
2348
|
|
2174
2349
|
Example: Source file `200KB.1` is renamed `sample1` on destination:
|
2175
2350
|
|
@@ -2177,13 +2352,13 @@ Example: Source file `200KB.1` is renamed `sample1` on destination:
|
|
2177
2352
|
ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
2178
2353
|
```
|
2179
2354
|
|
2180
|
-
> Note
|
2355
|
+
> **Note:** There are some specific rules to specify a file list when using **Aspera on Cloud**, refer to the AoC plugin section.
|
2181
2356
|
|
2182
2357
|
#### <a id="multisession"></a>Support of multi-session
|
2183
2358
|
|
2184
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
2359
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one `ascp` process per session) is supported on "direct" and "node" agents, not yet on connect.
|
2185
2360
|
|
2186
|
-
|
2361
|
+
- when agent=node :
|
2187
2362
|
|
2188
2363
|
```javascript
|
2189
2364
|
--ts=@json:'{"multi_session":10,"multi_session_threshold":1}'
|
@@ -2191,16 +2366,15 @@ Multi session, i.e. starting a transfer of a file set using multiple sessions (o
|
|
2191
2366
|
|
2192
2367
|
Multi-session is directly supported by the node daemon.
|
2193
2368
|
|
2194
|
-
|
2369
|
+
- when agent=direct :
|
2195
2370
|
|
2196
2371
|
```javascript
|
2197
2372
|
--ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'
|
2198
2373
|
```
|
2199
2374
|
|
2200
|
-
Note:
|
2201
|
-
shall be preferred.
|
2375
|
+
Note: `resume_policy` set to `attr` may cause problems: `none` or `sparse_csum` shall be preferred.
|
2202
2376
|
|
2203
|
-
Multi-session
|
2377
|
+
`ascli` starts multiple `ascp` for Multi-session using `direct` agent.
|
2204
2378
|
|
2205
2379
|
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
2206
2380
|
|
@@ -2212,8 +2386,8 @@ using a passphrase, only known by users sharing files. Files stay encrypted on s
|
|
2212
2386
|
|
2213
2387
|
activating CSEAR consists in using transfer spec parameters:
|
2214
2388
|
|
2215
|
-
|
2216
|
-
|
2389
|
+
- `content_protection` : activate encryption (`encrypt` for upload) or decryption (`decrypt` for download)
|
2390
|
+
- `content_protection_password` : the passphrase to be used.
|
2217
2391
|
|
2218
2392
|
Example: parameter to download a faspex package and decrypt on the fly
|
2219
2393
|
|
@@ -2221,7 +2395,7 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
2221
2395
|
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"_pass_here_"}'
|
2222
2396
|
```
|
2223
2397
|
|
2224
|
-
Note
|
2398
|
+
> **Note:** Up to version `ascli` 4.6.0, the following parameters should be used for agent `direct`:
|
2225
2399
|
|
2226
2400
|
```javascript
|
2227
2401
|
--ts=@json:'{"EX_ascp_args":["--file-crypt=decrypt"],"EX_at_rest_password":"_secret_here_"}'
|
@@ -2229,44 +2403,48 @@ Note that up to version 4.6.0, the following parameters should be used for agent
|
|
2229
2403
|
|
2230
2404
|
#### Transfer Spec Examples
|
2231
2405
|
|
2232
|
-
|
2406
|
+
- Change target rate
|
2233
2407
|
|
2234
2408
|
```javascript
|
2235
2409
|
--ts=@json:'{"target_rate_kbps":500000}'
|
2236
2410
|
```
|
2237
2411
|
|
2238
|
-
|
2412
|
+
- Override the FASP SSH port to a specific TCP port:
|
2239
2413
|
|
2240
2414
|
```javascript
|
2241
2415
|
--ts=@json:'{"ssh_port":33002}'
|
2242
2416
|
```
|
2243
2417
|
|
2244
|
-
|
2418
|
+
- Force http fallback mode:
|
2245
2419
|
|
2246
2420
|
```javascript
|
2247
2421
|
--ts=@json:'{"http_fallback":"force"}'
|
2248
2422
|
```
|
2249
2423
|
|
2250
|
-
|
2424
|
+
- Activate progress when not activated by default on server
|
2251
2425
|
|
2252
2426
|
```javascript
|
2253
2427
|
--ts=@json:'{"precalculate_job_size":true}'
|
2254
2428
|
```
|
2255
2429
|
|
2256
|
-
### <a id="scheduling"></a>
|
2430
|
+
### <a id="scheduling"></a>Scheduling
|
2431
|
+
|
2432
|
+
It is useful to configure automated scheduled execution.
|
2433
|
+
|
2434
|
+
#### <a id="locking"></a>Locking for exclusive execution
|
2257
2435
|
|
2258
|
-
|
2436
|
+
It is also useful to ensure that `ascli` is not executed several times in parallel.
|
2259
2437
|
|
2260
2438
|
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
2261
2439
|
|
2262
|
-
|
2263
|
-
|
2264
|
-
|
2440
|
+
- Executing instances may pile-up and kill the system
|
2441
|
+
- The same file may be transferred by multiple instances at the same time.
|
2442
|
+
- `preview` may generate the same files in multiple instances.
|
2265
2443
|
|
2266
2444
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
2267
2445
|
|
2268
|
-
|
2269
|
-
|
2446
|
+
- The Windows scheduler does this by default
|
2447
|
+
- Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
2270
2448
|
|
2271
2449
|
```bash
|
2272
2450
|
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
@@ -2289,6 +2467,18 @@ The first instance will sleep 30 seconds, the second one will immediately exit l
|
|
2289
2467
|
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
2290
2468
|
```
|
2291
2469
|
|
2470
|
+
#### <a id="scheduler"></a>Scheduler
|
2471
|
+
|
2472
|
+
`ascli` does not provide an internal scheduler.
|
2473
|
+
|
2474
|
+
Instead, use the service provided by the Operating system:
|
2475
|
+
|
2476
|
+
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2477
|
+
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2478
|
+
- etc...
|
2479
|
+
|
2480
|
+
Linux also provides `anacron`, if tasks are hourly or daily.
|
2481
|
+
|
2292
2482
|
### "Provençale"
|
2293
2483
|
|
2294
2484
|
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
@@ -2343,8 +2533,8 @@ faux:///filename?filesize
|
|
2343
2533
|
|
2344
2534
|
where:
|
2345
2535
|
|
2346
|
-
|
2347
|
-
|
2536
|
+
- `filename` is the name that will be assigned to the file on the destination
|
2537
|
+
- `filesize` is the number of bytes that will be sent (in decimal).
|
2348
2538
|
|
2349
2539
|
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
2350
2540
|
|
@@ -2358,8 +2548,8 @@ faux:///dirname?<arg1>=<val1>&...
|
|
2358
2548
|
|
2359
2549
|
where:
|
2360
2550
|
|
2361
|
-
|
2362
|
-
|
2551
|
+
- `dirname` is the folder name and can contain `/` to specify a subfolder.
|
2552
|
+
- supported arguments are:
|
2363
2553
|
|
2364
2554
|
| Name | Type | Description |
|
2365
2555
|
|--------|------|-------------|
|
@@ -2372,19 +2562,19 @@ where:
|
|
2372
2562
|
|
2373
2563
|
The sequence parameter is applied as follows:
|
2374
2564
|
|
2375
|
-
|
2565
|
+
- If `seq` is `random` then each file size is:
|
2376
2566
|
|
2377
|
-
|
2378
|
-
|
2379
|
-
|
2380
|
-
|
2567
|
+
- size +/- (inc * rand())
|
2568
|
+
- Where rand is a random number between 0 and 1
|
2569
|
+
- Note that file size must not be negative, inc will be set to size if it is greater than size
|
2570
|
+
- Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
2381
2571
|
|
2382
|
-
|
2572
|
+
- If `seq` is `sequential` then each file size is:
|
2383
2573
|
|
2384
|
-
|
2385
|
-
|
2386
|
-
|
2387
|
-
|
2574
|
+
- `size + ((fileindex - 1) * inc)`
|
2575
|
+
- Where first file is index 1
|
2576
|
+
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
2577
|
+
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
2388
2578
|
|
2389
2579
|
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
2390
2580
|
|
@@ -2392,19 +2582,19 @@ To discard data at the destination, the destination argument is set to `faux://`
|
|
2392
2582
|
|
2393
2583
|
Examples:
|
2394
2584
|
|
2395
|
-
|
2585
|
+
- Upload 20 gibibytes of random data to file myfile to directory /Upload
|
2396
2586
|
|
2397
2587
|
```bash
|
2398
2588
|
ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
2399
2589
|
```
|
2400
2590
|
|
2401
|
-
|
2591
|
+
- Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
2402
2592
|
|
2403
2593
|
```bash
|
2404
2594
|
ascli server upload /tmp/sample --to-folder=faux://
|
2405
2595
|
```
|
2406
2596
|
|
2407
|
-
|
2597
|
+
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
2408
2598
|
|
2409
2599
|
```bash
|
2410
2600
|
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
@@ -2415,7 +2605,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
2415
2605
|
```text
|
2416
2606
|
ascli -h
|
2417
2607
|
NAME
|
2418
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
2608
|
+
ascli -- a command line tool for Aspera Applications (v4.11.0)
|
2419
2609
|
|
2420
2610
|
SYNOPSIS
|
2421
2611
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -2438,23 +2628,23 @@ COMMANDS
|
|
2438
2628
|
OPTIONS
|
2439
2629
|
Options begin with a '-' (minus), and value is provided on command line.
|
2440
2630
|
Special values are supported beginning with special prefix @pfx:, where pfx is one of:
|
2441
|
-
base64, json, zlib, ruby, csvt, lines, list, incps, val, file, path, env, uri, stdin, preset
|
2631
|
+
base64, json, zlib, ruby, csvt, lines, list, incps, vault, val, file, path, env, uri, stdin, preset
|
2442
2632
|
Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'
|
2443
2633
|
|
2444
2634
|
ARGS
|
2445
2635
|
Some commands require mandatory arguments, e.g. a path.
|
2446
2636
|
|
2447
2637
|
OPTIONS: global
|
2448
|
-
--interactive=ENUM use interactive input of missing params:
|
2449
|
-
--ask-options=ENUM ask even optional options:
|
2638
|
+
--interactive=ENUM use interactive input of missing params: [no], yes
|
2639
|
+
--ask-options=ENUM ask even optional options: [no], yes
|
2450
2640
|
--format=ENUM output format: text, nagios, ruby, json, jsonpp, yaml, [table], csv
|
2451
2641
|
--display=ENUM output only some information: [info], data, error
|
2452
2642
|
--fields=VALUE comma separated list of fields, or ALL, or DEF
|
2453
2643
|
--select=VALUE select only some items in lists, extended value: hash (column, value)
|
2454
2644
|
--table-style=VALUE table display style
|
2455
|
-
--flat-hash=ENUM display hash values as additional keys: [yes]
|
2456
|
-
--transpose-single=ENUM single object fields output vertically: [yes]
|
2457
|
-
--show-secrets=ENUM show secrets on command output:
|
2645
|
+
--flat-hash=ENUM display hash values as additional keys: no, [yes]
|
2646
|
+
--transpose-single=ENUM single object fields output vertically: no, [yes]
|
2647
|
+
--show-secrets=ENUM show secrets on command output: [no], yes
|
2458
2648
|
-h, --help Show this message.
|
2459
2649
|
--bash-comp generate bash completion for command
|
2460
2650
|
--show-config Display parameters used for the provided action.
|
@@ -2465,52 +2655,55 @@ OPTIONS: global
|
|
2465
2655
|
--log-level=ENUM Log level: debug, info, [warn], error, fatal, unknown
|
2466
2656
|
--logger=ENUM log method: [stderr], stdout, syslog
|
2467
2657
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
2468
|
-
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2469
2658
|
--http-options=VALUE options for http socket (extended value)
|
2470
|
-
--insecure=ENUM do not validate HTTPS certificate: [yes]
|
2471
|
-
--once-only=ENUM process only new items (some commands):
|
2472
|
-
--log-secrets=ENUM show passwords in logs:
|
2473
|
-
--cache-tokens=ENUM save and reuse Oauth tokens: [yes]
|
2659
|
+
--insecure=ENUM do not validate HTTPS certificate: no, [yes]
|
2660
|
+
--once-only=ENUM process only new items (some commands): [no], yes
|
2661
|
+
--log-secrets=ENUM show passwords in logs: [no], yes
|
2662
|
+
--cache-tokens=ENUM save and reuse Oauth tokens: no, [yes]
|
2474
2663
|
|
2475
2664
|
COMMAND: config
|
2476
|
-
SUBCOMMANDS:
|
2665
|
+
SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test export_to_cli file flush_tokens folder gem genkey id initdemo list lookup open overview plugin preset proxy_check secure smtp_settings vault wizard
|
2477
2666
|
OPTIONS:
|
2667
|
+
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2478
2668
|
--value=VALUE extended value for create, update, list filter
|
2479
2669
|
--property=VALUE name of property to set
|
2480
2670
|
--id=VALUE resource identifier (modify,delete,show)
|
2481
|
-
--bulk=ENUM Bulk operation (only some):
|
2671
|
+
--bulk=ENUM Bulk operation (only some): [no], yes
|
2672
|
+
--bfail=ENUM Bulk operation error handling: [no], yes
|
2482
2673
|
--config-file=VALUE read parameters from file in YAML format, current=/usershome/.aspera/ascli/config.yaml
|
2483
2674
|
-N, --no-default do not load default configuration for plugin
|
2484
|
-
--override=ENUM Wizard: override existing value:
|
2485
|
-
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id:
|
2486
|
-
--default=ENUM Wizard: set as default configuration for specified plugin (also: update):
|
2487
|
-
--test-mode=ENUM Wizard: skip private key check step:
|
2675
|
+
--override=ENUM Wizard: override existing value: [no], yes
|
2676
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: [no], yes
|
2677
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): [no], yes
|
2678
|
+
--test-mode=ENUM Wizard: skip private key check step: [no], yes
|
2488
2679
|
-P, --presetVALUE load the named option preset from current config file
|
2489
2680
|
--pkeypath=VALUE Wizard: path to private key for JWT
|
2490
|
-
--ascp-path=VALUE
|
2491
|
-
--use-product=VALUE
|
2492
|
-
--smtp=VALUE
|
2493
|
-
--fpac=VALUE
|
2494
|
-
--
|
2495
|
-
--
|
2681
|
+
--ascp-path=VALUE Path to ascp
|
2682
|
+
--use-product=VALUE Use ascp from specified product
|
2683
|
+
--smtp=VALUE SMTP configuration (extended value: hash)
|
2684
|
+
--fpac=VALUE Proxy auto configuration script
|
2685
|
+
--proxy-credentials=VALUE HTTP proxy credentials (Array with user and password)
|
2686
|
+
--secret=VALUE Secret for access keys
|
2687
|
+
--vault=VALUE Vault for secrets
|
2688
|
+
--vault-password=VALUE Vault password
|
2496
2689
|
--sdk-url=VALUE URL to get SDK
|
2497
2690
|
--sdk-folder=VALUE SDK folder path
|
2498
|
-
--notif-to=VALUE
|
2499
|
-
--notif-template=VALUE
|
2500
|
-
--version-check-days=VALUE
|
2501
|
-
--plugin-folder=VALUE
|
2502
|
-
--ts=VALUE
|
2503
|
-
--
|
2504
|
-
--
|
2505
|
-
--
|
2506
|
-
--src-type=ENUM
|
2507
|
-
--transfer=ENUM
|
2508
|
-
--transfer-info=VALUE
|
2509
|
-
--progress=ENUM
|
2691
|
+
--notif-to=VALUE Email recipient for notification of transfers
|
2692
|
+
--notif-template=VALUE Email ERB template for notification of transfers
|
2693
|
+
--version-check-days=VALUE Period in days to check new version (zero to disable)
|
2694
|
+
--plugin-folder=VALUE Folder where to find additional plugins
|
2695
|
+
--ts=VALUE Override transfer spec values (Hash, e.g. use @json: prefix), current={"create_dir"=>true}
|
2696
|
+
--to-folder=VALUE Destination folder for transfered files
|
2697
|
+
--sources=VALUE How list of transfered files is provided (@args,@ts,Array)
|
2698
|
+
--ascp-opts=VALUE Options for ascp in its native format
|
2699
|
+
--src-type=ENUM Type of file list: list, pair
|
2700
|
+
--transfer=ENUM Type of transfer agent: direct, node, connect, httpgw, trsdk
|
2701
|
+
--transfer-info=VALUE Parameters for transfer agent
|
2702
|
+
--progress=ENUM Type of progress bar: none, native, multi
|
2510
2703
|
|
2511
2704
|
|
2512
2705
|
COMMAND: shares
|
2513
|
-
SUBCOMMANDS: health repository
|
2706
|
+
SUBCOMMANDS: admin health repository
|
2514
2707
|
OPTIONS:
|
2515
2708
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2516
2709
|
--username=VALUE username to log in
|
@@ -2518,7 +2711,7 @@ OPTIONS:
|
|
2518
2711
|
|
2519
2712
|
|
2520
2713
|
COMMAND: node
|
2521
|
-
SUBCOMMANDS:
|
2714
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
2522
2715
|
OPTIONS:
|
2523
2716
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2524
2717
|
--username=VALUE username to log in
|
@@ -2526,18 +2719,20 @@ OPTIONS:
|
|
2526
2719
|
--validator=VALUE identifier of validator (optional for central)
|
2527
2720
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
2528
2721
|
--sync-name=VALUE sync name
|
2722
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
2529
2723
|
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
2724
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
2530
2725
|
|
2531
2726
|
|
2532
2727
|
COMMAND: orchestrator
|
2533
|
-
SUBCOMMANDS: health info
|
2728
|
+
SUBCOMMANDS: health info plugins processes workflow
|
2534
2729
|
OPTIONS:
|
2535
2730
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2536
2731
|
--username=VALUE username to log in
|
2537
2732
|
--password=VALUE user's password
|
2538
2733
|
--params=VALUE parameters hash table, use @json:{"param":"value"}
|
2539
2734
|
--result=VALUE specify result value as: 'work step:parameter'
|
2540
|
-
--synchronous=ENUM work step:parameter expected as result:
|
2735
|
+
--synchronous=ENUM work step:parameter expected as result: [no], yes
|
2541
2736
|
--ret-style=ENUM how return type is requested in api: header, arg, ext
|
2542
2737
|
--auth-style=ENUM authentication type: arg_pass, head_basic, apikey
|
2543
2738
|
|
@@ -2559,7 +2754,7 @@ OPTIONS:
|
|
2559
2754
|
|
2560
2755
|
|
2561
2756
|
COMMAND: ats
|
2562
|
-
SUBCOMMANDS:
|
2757
|
+
SUBCOMMANDS: access_key api_key aws_trust_policy cluster
|
2563
2758
|
OPTIONS:
|
2564
2759
|
--ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
|
2565
2760
|
--instance=VALUE ATS instance in ibm cloud
|
@@ -2571,7 +2766,7 @@ OPTIONS:
|
|
2571
2766
|
|
2572
2767
|
|
2573
2768
|
COMMAND: faspex5
|
2574
|
-
SUBCOMMANDS: health package
|
2769
|
+
SUBCOMMANDS: admin bearer_token health package user version
|
2575
2770
|
OPTIONS:
|
2576
2771
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2577
2772
|
--username=VALUE username to log in
|
@@ -2579,7 +2774,7 @@ OPTIONS:
|
|
2579
2774
|
--client-id=VALUE OAuth client identifier
|
2580
2775
|
--client-secret=VALUE OAuth client secret
|
2581
2776
|
--redirect-uri=VALUE OAuth redirect URI for web authentication
|
2582
|
-
--auth=ENUM OAuth type of authentication: web, jwt
|
2777
|
+
--auth=ENUM OAuth type of authentication: boot, web, jwt
|
2583
2778
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2584
2779
|
--passphrase=VALUE RSA private key passphrase
|
2585
2780
|
|
@@ -2597,7 +2792,7 @@ OPTIONS:
|
|
2597
2792
|
|
2598
2793
|
|
2599
2794
|
COMMAND: faspex
|
2600
|
-
SUBCOMMANDS: health package source
|
2795
|
+
SUBCOMMANDS: address_book dropbox health login_methods me package source v4
|
2601
2796
|
OPTIONS:
|
2602
2797
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2603
2798
|
--username=VALUE username to log in
|
@@ -2611,7 +2806,7 @@ OPTIONS:
|
|
2611
2806
|
|
2612
2807
|
|
2613
2808
|
COMMAND: preview
|
2614
|
-
SUBCOMMANDS:
|
2809
|
+
SUBCOMMANDS: check events scan test trevents
|
2615
2810
|
OPTIONS:
|
2616
2811
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2617
2812
|
--username=VALUE username to log in
|
@@ -2625,7 +2820,7 @@ OPTIONS:
|
|
2625
2820
|
--case=VALUE basename of output for for test
|
2626
2821
|
--scan-path=VALUE subpath in folder id to start scan in (default=/)
|
2627
2822
|
--scan-id=VALUE forder id in storage to start scan in, default is access key main folder id
|
2628
|
-
--mimemagic=ENUM use Mime type detection of gem mimemagic:
|
2823
|
+
--mimemagic=ENUM use Mime type detection of gem mimemagic: [no], yes
|
2629
2824
|
--overwrite=ENUM when to overwrite result file: always, never, [mtime]
|
2630
2825
|
--file-access=ENUM how to read and write files in repository: [local], remote
|
2631
2826
|
--max-size=VALUE maximum size (in bytes) of preview file
|
@@ -2646,14 +2841,14 @@ OPTIONS:
|
|
2646
2841
|
|
2647
2842
|
|
2648
2843
|
COMMAND: sync
|
2649
|
-
SUBCOMMANDS: start
|
2844
|
+
SUBCOMMANDS: admin start
|
2650
2845
|
OPTIONS:
|
2651
|
-
--
|
2652
|
-
--session
|
2846
|
+
--sync-info=VALUE Information for sync instance and sessions (Hash)
|
2847
|
+
--sync-session=VALUE Name of session to use for admin commands. default: first in parameters
|
2653
2848
|
|
2654
2849
|
|
2655
2850
|
COMMAND: aoc
|
2656
|
-
SUBCOMMANDS:
|
2851
|
+
SUBCOMMANDS: admin automation bearer_token files gateway organization packages reminder servers tier_restrictions user
|
2657
2852
|
OPTIONS:
|
2658
2853
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2659
2854
|
--username=VALUE username to log in
|
@@ -2664,20 +2859,28 @@ OPTIONS:
|
|
2664
2859
|
--client-secret=VALUE OAuth API client passcode
|
2665
2860
|
--redirect-uri=VALUE OAuth API client redirect URI
|
2666
2861
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2667
|
-
--passphrase=VALUE RSA private key passphrase
|
2668
|
-
--workspace=VALUE name of workspace
|
2669
|
-
--name=VALUE resource name
|
2670
|
-
--path=VALUE file or folder path
|
2671
|
-
--link=VALUE public link to shared resource
|
2672
|
-
--new-user-option=VALUE new user creation option for unknown package recipients
|
2673
|
-
--from-folder=VALUE share to share source folder
|
2674
2862
|
--scope=VALUE OAuth scope for AoC API calls
|
2675
|
-
--
|
2676
|
-
--
|
2863
|
+
--passphrase=VALUE RSA private key passphrase
|
2864
|
+
--workspace=VALUE Name of workspace
|
2865
|
+
--name=VALUE Resource name (prefer to use keyword name)
|
2866
|
+
--link=VALUE Public link to shared resource
|
2867
|
+
--new-user-option=VALUE New user creation option for unknown package recipients
|
2868
|
+
--from-folder=VALUE Source folder for Folder-to-Folder transfer
|
2869
|
+
--validate-metadata=ENUM Validate shared inbox metadata: [no], yes
|
2870
|
+
|
2871
|
+
COMMAND: node
|
2872
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
2873
|
+
OPTIONS:
|
2874
|
+
--validator=VALUE identifier of validator (optional for central)
|
2875
|
+
--asperabrowserurl=VALUE URL for simple aspera web ui
|
2876
|
+
--sync-name=VALUE sync name
|
2877
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
2878
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
2879
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
2677
2880
|
|
2678
2881
|
|
2679
2882
|
COMMAND: server
|
2680
|
-
SUBCOMMANDS:
|
2883
|
+
SUBCOMMANDS: browse cp delete df download du health info ls md5sum mkdir mv rename rm sync upload
|
2681
2884
|
OPTIONS:
|
2682
2885
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2683
2886
|
--username=VALUE username to log in
|
@@ -2687,7 +2890,7 @@ OPTIONS:
|
|
2687
2890
|
|
2688
2891
|
|
2689
2892
|
COMMAND: console
|
2690
|
-
SUBCOMMANDS: transfer
|
2893
|
+
SUBCOMMANDS: health transfer
|
2691
2894
|
OPTIONS:
|
2692
2895
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2693
2896
|
--username=VALUE username to log in
|
@@ -2698,7 +2901,7 @@ OPTIONS:
|
|
2698
2901
|
|
2699
2902
|
```
|
2700
2903
|
|
2701
|
-
Note
|
2904
|
+
> **Note:** commands and parameter values can be written in short form.
|
2702
2905
|
|
2703
2906
|
### Bulk creation and deletion of resources
|
2704
2907
|
|
@@ -2706,7 +2909,7 @@ Bulk creation and deletion of resources are possible using option `bulk` (yes,no
|
|
2706
2909
|
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2707
2910
|
This option is available only for some of the resources: if you need it: try and see if the entities you try to create or delete support this option.
|
2708
2911
|
|
2709
|
-
## <a id="aoc"></a>Plugin: Aspera on Cloud
|
2912
|
+
## <a id="aoc"></a>Plugin: `aoc`: IBM Aspera on Cloud
|
2710
2913
|
|
2711
2914
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
2712
2915
|
|
@@ -2749,15 +2952,15 @@ ascli config wizard --value=aoc
|
|
2749
2952
|
|
2750
2953
|
### <a id="aocmanual"></a>Configuration: using manual setup
|
2751
2954
|
|
2752
|
-
> If you used the wizard (recommended): skip this section.
|
2955
|
+
> **Note:** If you used the wizard (recommended): skip this section.
|
2753
2956
|
|
2754
2957
|
#### Configuration details
|
2755
2958
|
|
2756
2959
|
Several types of OAuth authentication are supported:
|
2757
2960
|
|
2758
|
-
|
2759
|
-
|
2760
|
-
|
2961
|
+
- JSON Web Token (JWT) : authentication is secured by a private key (recommended for CLI)
|
2962
|
+
- Web based authentication : authentication is made by user using a browser
|
2963
|
+
- URL Token : external users authentication with url tokens (public links)
|
2761
2964
|
|
2762
2965
|
The authentication method is controlled by option `auth`.
|
2763
2966
|
|
@@ -2773,19 +2976,19 @@ If you use the built-in client_id and client_secret, skip this and do not set th
|
|
2773
2976
|
|
2774
2977
|
Else you can use a specific OAuth API client_id, the first step is to declare `ascli` in Aspera on Cloud using the admin interface.
|
2775
2978
|
|
2776
|
-
(
|
2979
|
+
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client) ).
|
2777
2980
|
|
2778
2981
|
Let's start by a registration with web based authentication (auth=web):
|
2779
2982
|
|
2780
|
-
|
2781
|
-
|
2782
|
-
|
2783
|
-
|
2784
|
-
|
2785
|
-
|
2786
|
-
|
2787
|
-
|
2788
|
-
|
2983
|
+
- Open a web browser, log to your instance: e.g. `https://myorg.ibmaspera.com/`
|
2984
|
+
- Go to Apps → Admin → Organization → Integrations
|
2985
|
+
- Click "Create New"
|
2986
|
+
- Client Name: `ascli`
|
2987
|
+
- Redirect URIs: `http://localhost:12345`
|
2988
|
+
- Origins: `localhost`
|
2989
|
+
- uncheck "Prompt users to allow client to access"
|
2990
|
+
- leave the JWT part for now
|
2991
|
+
- Save
|
2789
2992
|
|
2790
2993
|
Note: for web based authentication, `ascli` listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For ``ascli`, HTTP is required, and 12345 is the default port.
|
2791
2994
|
|
@@ -2826,16 +3029,16 @@ a [private/public key pair](#private_key) must be used.
|
|
2826
3029
|
|
2827
3030
|
If you are not using the built-in client_id and secret, JWT needs to be authorized in Aspera on Cloud. This can be done in two manners:
|
2828
3031
|
|
2829
|
-
|
3032
|
+
- Graphically
|
2830
3033
|
|
2831
|
-
|
2832
|
-
|
2833
|
-
|
2834
|
-
|
2835
|
-
|
2836
|
-
|
3034
|
+
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
3035
|
+
- Go to Apps → Admin → Organization → Integrations
|
3036
|
+
- Click on the previously created application
|
3037
|
+
- select tab : "JSON Web Token Auth"
|
3038
|
+
- Modify options if necessary, for instance: activate both options in section "Settings"
|
3039
|
+
- Click "Save"
|
2837
3040
|
|
2838
|
-
|
3041
|
+
- Using command line
|
2839
3042
|
|
2840
3043
|
```bash
|
2841
3044
|
ascli aoc admin res client list
|
@@ -2865,11 +3068,11 @@ The public key must be assigned to your user. This can be done in two manners:
|
|
2865
3068
|
|
2866
3069
|
Open the previously generated public key located here: `$HOME/.aspera/ascli/my_private_key.pub`
|
2867
3070
|
|
2868
|
-
|
2869
|
-
|
2870
|
-
|
2871
|
-
|
2872
|
-
|
3071
|
+
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
3072
|
+
- Click on the user's icon (top right)
|
3073
|
+
- Select "Account Settings"
|
3074
|
+
- Paste the *Public Key* in the "Public Key" section
|
3075
|
+
- Click on "Submit"
|
2873
3076
|
|
2874
3077
|
##### Using command line
|
2875
3078
|
|
@@ -2900,9 +3103,9 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
2900
3103
|
|
2901
3104
|
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the following:
|
2902
3105
|
|
2903
|
-
|
2904
|
-
|
2905
|
-
|
3106
|
+
- change auth method to JWT
|
3107
|
+
- provide location of private key
|
3108
|
+
- provide username to login as (OAuth "subject")
|
2906
3109
|
|
2907
3110
|
Execute:
|
2908
3111
|
|
@@ -2962,13 +3165,13 @@ The option `query` can be optionally used. It expects a Hash using [Extended Val
|
|
2962
3165
|
|
2963
3166
|
The following parameters are supported:
|
2964
3167
|
|
2965
|
-
|
2966
|
-
|
2967
|
-
|
2968
|
-
|
2969
|
-
|
2970
|
-
|
2971
|
-
|
3168
|
+
- `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
3169
|
+
- `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
3170
|
+
- `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
3171
|
+
- `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
3172
|
+
- `page` : native api parameter, in general do not use (added by
|
3173
|
+
- `per_page` : native api parameter, number of items par api call, in general do not use
|
3174
|
+
- Other specific parameters depending on resource type.
|
2972
3175
|
|
2973
3176
|
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
2974
3177
|
|
@@ -2982,19 +3185,19 @@ Other parameters depend on the type of entity (refer to AoC API).
|
|
2982
3185
|
|
2983
3186
|
Examples:
|
2984
3187
|
|
2985
|
-
|
3188
|
+
- List users with `laurent` in name:
|
2986
3189
|
|
2987
3190
|
```javascript
|
2988
3191
|
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2989
3192
|
```
|
2990
3193
|
|
2991
|
-
|
3194
|
+
- List users who logged-in before a date:
|
2992
3195
|
|
2993
3196
|
```javascript
|
2994
3197
|
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2995
3198
|
```
|
2996
3199
|
|
2997
|
-
|
3200
|
+
- List external users and sort in reverse alphabetical order using name:
|
2998
3201
|
|
2999
3202
|
```javascript
|
3000
3203
|
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
@@ -3002,7 +3205,7 @@ ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"s
|
|
3002
3205
|
|
3003
3206
|
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
3004
3207
|
|
3005
|
-
Note
|
3208
|
+
> **Note:** The option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
3006
3209
|
|
3007
3210
|
#### <a id="res_select"></a>Selecting a resource
|
3008
3211
|
|
@@ -3010,10 +3213,10 @@ Resources are identified by a unique `id`, as well as a unique `name` (case inse
|
|
3010
3213
|
|
3011
3214
|
To execute an action on a specific resource, select it using one of those methods:
|
3012
3215
|
|
3013
|
-
|
3014
|
-
|
3015
|
-
|
3016
|
-
|
3216
|
+
- *recommended*: give id directly on command line *after the action*: `aoc admin res node show 123`
|
3217
|
+
- give name on command line *after the action*: `aoc admin res node show name abc`
|
3218
|
+
- provide option `id` : `aoc admin res node show --id=123`
|
3219
|
+
- provide option `name` : `aoc admin res node show --name=abc`
|
3017
3220
|
|
3018
3221
|
#### <a id="res_create"></a>Creating a resource
|
3019
3222
|
|
@@ -3054,22 +3257,23 @@ If the command returns an error, example:
|
|
3054
3257
|
| | request_id: b0f45d5b-c00a-4711-acef-72b633f8a6ea |
|
3055
3258
|
| | api.ibmaspera.com 422 Unprocessable Entity |
|
3056
3259
|
+----+-----------------------------------------------------------------------------------+```
|
3260
|
+
```
|
3057
3261
|
|
3058
3262
|
Well, remove the offending parameters and try again.
|
3059
3263
|
|
3060
|
-
Note
|
3264
|
+
> **Note:** Some properties that are shown in the web UI, such as membership, are not listed directly in the resource, but instead another resource is created to link a user and its group: `group_membership`
|
3061
3265
|
|
3062
3266
|
#### Access Key secrets
|
3063
3267
|
|
3064
|
-
In order to access some administrative actions on
|
3065
|
-
|
3268
|
+
In order to access some administrative actions on **nodes** (in fact, access keys), the associated secret is required.
|
3269
|
+
The secret is provided using the `secret` option.
|
3066
3270
|
For example in a command like:
|
3067
3271
|
|
3068
3272
|
```bash
|
3069
3273
|
ascli aoc admin res node --id=123 --secret="secret1" v3 info
|
3070
3274
|
```
|
3071
3275
|
|
3072
|
-
It is also possible to
|
3276
|
+
It is also possible to store secrets in the [secret vault](#vault) and then automatically find the related secret using the [config finder](#config_finder).
|
3073
3277
|
|
3074
3278
|
#### Activity
|
3075
3279
|
|
@@ -3097,7 +3301,7 @@ Thank you.
|
|
3097
3301
|
|
3098
3302
|
The environment provided contains the following additional variable:
|
3099
3303
|
|
3100
|
-
|
3304
|
+
- ev : all details on the transfer event
|
3101
3305
|
|
3102
3306
|
Example:
|
3103
3307
|
|
@@ -3109,11 +3313,11 @@ ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
|
3109
3313
|
|
3110
3314
|
Options:
|
3111
3315
|
|
3112
|
-
|
3113
|
-
|
3114
|
-
|
3316
|
+
- `once_only` keep track of last date it was called, so next call will get only new events
|
3317
|
+
- `query` filter (on API call)
|
3318
|
+
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
3115
3319
|
|
3116
|
-
Note
|
3320
|
+
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
3117
3321
|
|
3118
3322
|
#### Transfer: Using specific transfer ports
|
3119
3323
|
|
@@ -3321,7 +3525,7 @@ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_wor
|
|
3321
3525
|
|
3322
3526
|
#### Example: create a group, add to workspace and add user to group
|
3323
3527
|
|
3324
|
-
|
3528
|
+
- Create the group and take note of `id`
|
3325
3529
|
|
3326
3530
|
```bash
|
3327
3531
|
ascli aoc admin res group create @json:'{"name":"group 1","description":"my super group"}'
|
@@ -3329,7 +3533,7 @@ ascli aoc admin res group create @json:'{"name":"group 1","description":"my supe
|
|
3329
3533
|
|
3330
3534
|
Group: `11111`
|
3331
3535
|
|
3332
|
-
|
3536
|
+
- Get the workspace id
|
3333
3537
|
|
3334
3538
|
```bash
|
3335
3539
|
ascli aoc admin res workspace list --query=@json:'{"q":"myworkspace"}' --fields=id --format=csv --display=data
|
@@ -3337,13 +3541,13 @@ ascli aoc admin res workspace list --query=@json:'{"q":"myworkspace"}' --fields=
|
|
3337
3541
|
|
3338
3542
|
Workspace: 22222
|
3339
3543
|
|
3340
|
-
|
3544
|
+
- Add group to workspace
|
3341
3545
|
|
3342
3546
|
```bash
|
3343
3547
|
ascli aoc admin res workspace_membership create @json:'{"workspace_id":22222,"member_type":"user","member_id":11111}'
|
3344
3548
|
```
|
3345
3549
|
|
3346
|
-
|
3550
|
+
- Get a user's id
|
3347
3551
|
|
3348
3552
|
```bash
|
3349
3553
|
ascli aoc admin res user list --query=@json:'{"q":"manu.macron@example.com"}' --fields=id --format=csv --display=data
|
@@ -3351,7 +3555,7 @@ ascli aoc admin res user list --query=@json:'{"q":"manu.macron@example.com"}' --
|
|
3351
3555
|
|
3352
3556
|
User: 33333
|
3353
3557
|
|
3354
|
-
|
3558
|
+
- Add user to group
|
3355
3559
|
|
3356
3560
|
```bash
|
3357
3561
|
ascli aoc admin res group_membership create @json:'{"group_id":11111,"member_type":"user","member_id":33333}'
|
@@ -3430,16 +3634,17 @@ ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res cli
|
|
3430
3634
|
|
3431
3635
|
AoC nodes as actually composed with two related entities:
|
3432
3636
|
|
3433
|
-
|
3434
|
-
|
3637
|
+
- An access key created on the Transfer Server (HSTS/ATS)
|
3638
|
+
- a `node` resource in the AoC application.
|
3435
3639
|
|
3436
3640
|
The web UI allows creation of both entities in one shot.
|
3437
3641
|
For more flexibility, `ascli` allows this in two separate steps.
|
3438
|
-
|
3642
|
+
|
3643
|
+
> **Note:** When selecting "Use existing access key" in the web UI, this actually skips access key creation (first step).
|
3439
3644
|
|
3440
3645
|
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
3441
3646
|
|
3442
|
-
|
3647
|
+
- Create the access key on ATS
|
3443
3648
|
|
3444
3649
|
The creation options are the ones of ATS API, refer to the [section on ATS](#ats_params) for more details and examples.
|
3445
3650
|
|
@@ -3447,9 +3652,11 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3447
3652
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
3448
3653
|
```
|
3449
3654
|
|
3450
|
-
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed
|
3655
|
+
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed.
|
3656
|
+
|
3657
|
+
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking to IBM support.
|
3451
3658
|
|
3452
|
-
|
3659
|
+
- Create the AoC node entity
|
3453
3660
|
|
3454
3661
|
First, Retrieve the ATS node address
|
3455
3662
|
|
@@ -3467,22 +3674,23 @@ Creation of a node with a self-managed node is similar, but the command `aoc adm
|
|
3467
3674
|
|
3468
3675
|
### List of files to transfer
|
3469
3676
|
|
3470
|
-
Source files are provided as a list with the `sources` option.
|
3677
|
+
Source files are provided as a list with the `sources` option.
|
3678
|
+
Refer to section [File list](#file_list)
|
3471
3679
|
|
3472
|
-
Note
|
3680
|
+
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file id` API).
|
3473
3681
|
|
3474
3682
|
Source files are located on "Aspera on cloud", when :
|
3475
3683
|
|
3476
|
-
|
3477
|
-
|
3684
|
+
- the server is Aspera on Cloud, and executing a download or recv
|
3685
|
+
- the agent is Aspera on Cloud, and executing an upload or send
|
3478
3686
|
|
3479
3687
|
In this case:
|
3480
3688
|
|
3481
|
-
|
3482
|
-
|
3483
|
-
|
3484
|
-
|
3485
|
-
|
3689
|
+
- If there is a single file : specify the full path
|
3690
|
+
- Else, if there are multiple files:
|
3691
|
+
- All source files must be in the same source folder
|
3692
|
+
- Specify the source folder as first item in the list
|
3693
|
+
- followed by the list of file names.
|
3486
3694
|
|
3487
3695
|
### Packages
|
3488
3696
|
|
@@ -3498,14 +3706,14 @@ ascli aoc packages send --value=[package extended value] [other parameters such
|
|
3498
3706
|
|
3499
3707
|
Notes:
|
3500
3708
|
|
3501
|
-
|
3502
|
-
|
3503
|
-
|
3504
|
-
|
3505
|
-
|
3506
|
-
|
3507
|
-
|
3508
|
-
|
3709
|
+
- The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
3710
|
+
- List allowed shared inbox destinations with: `ascli aoc packages shared_inboxes list`
|
3711
|
+
- Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
3712
|
+
- Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
3713
|
+
- or just names: `"recipients":[{"The Dest"}]` . ascli will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
3714
|
+
- If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
3715
|
+
- if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
3716
|
+
- if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
3509
3717
|
|
3510
3718
|
#### Example: Send a package with one file to two users, using their email
|
3511
3719
|
|
@@ -3584,23 +3792,32 @@ It is possible to automatically download new packages, like using Aspera Cargo:
|
|
3584
3792
|
ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
3585
3793
|
```
|
3586
3794
|
|
3587
|
-
|
3588
|
-
|
3589
|
-
|
3795
|
+
- `--id=ALL` (case sensitive) will download all packages
|
3796
|
+
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
3797
|
+
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
3590
3798
|
|
3591
|
-
Typically, one would execute this command on a regular basis, using the method of your choice:
|
3592
|
-
|
3593
|
-
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
3594
|
-
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
3595
|
-
* etc...
|
3799
|
+
Typically, one would execute this command on a regular basis, using the method of your choice: see [Scheduler](#scheduler).
|
3596
3800
|
|
3597
3801
|
### Files
|
3598
3802
|
|
3599
|
-
|
3803
|
+
The Files application presents a **Home** folder to users in a given workspace.
|
3804
|
+
Files located here are either user's files, or shared folders.
|
3600
3805
|
|
3601
3806
|
#### Download Files
|
3602
3807
|
|
3603
|
-
|
3808
|
+
The general download command is:
|
3809
|
+
|
3810
|
+
```bash
|
3811
|
+
ascli aoc files download <source folder path> <source filename 1> ...
|
3812
|
+
```
|
3813
|
+
|
3814
|
+
I.e. the first argument is the source folder, and the following arguments are the source file names in this folder.
|
3815
|
+
|
3816
|
+
If a single file or folder is to be downloaded, then a single argument can be provided.
|
3817
|
+
|
3818
|
+
```bash
|
3819
|
+
ascli aoc files download <single file path>
|
3820
|
+
```
|
3604
3821
|
|
3605
3822
|
#### Shared folders
|
3606
3823
|
|
@@ -3610,37 +3827,37 @@ For creation, parameters are the same as for node api [permissions](https://deve
|
|
3610
3827
|
Also, the pseudo key `with` is added: it will lookup the name in the contacts and fill the proper type and id.
|
3611
3828
|
The pseudo parameter `link_name` allows changing default "shared as" name.
|
3612
3829
|
|
3613
|
-
|
3830
|
+
- List permissions on a shared folder as user
|
3614
3831
|
|
3615
3832
|
```bash
|
3616
3833
|
ascli aoc files file --path=/shared_folder_test1 perm list
|
3617
3834
|
```
|
3618
3835
|
|
3619
|
-
|
3836
|
+
- Share a personal folder with other users
|
3620
3837
|
|
3621
3838
|
```bash
|
3622
3839
|
ascli aoc files file --path=/shared_folder_test1 perm create @json:'{"with":"laurent"}'
|
3623
3840
|
```
|
3624
3841
|
|
3625
|
-
|
3842
|
+
- Revoke shared access
|
3626
3843
|
|
3627
3844
|
```bash
|
3628
3845
|
ascli aoc files file --path=/shared_folder_test1 perm delete 6161
|
3629
3846
|
```
|
3630
3847
|
|
3631
|
-
|
3848
|
+
- List shared folders in node
|
3632
3849
|
|
3633
3850
|
```bash
|
3634
3851
|
ascli aoc admin res node --id=8669 shared_folders
|
3635
3852
|
```
|
3636
3853
|
|
3637
|
-
|
3854
|
+
- List shared folders in workspace
|
3638
3855
|
|
3639
3856
|
```bash
|
3640
3857
|
ascli aoc admin res workspace --id=10818 shared_folders
|
3641
3858
|
```
|
3642
3859
|
|
3643
|
-
|
3860
|
+
- List members of shared folder
|
3644
3861
|
|
3645
3862
|
```bash
|
3646
3863
|
ascli aoc admin res node --id=8669 v4 perm 82 show
|
@@ -3654,11 +3871,11 @@ Although optional, the creation of [option preset](#lprt) is recommended to avoi
|
|
3654
3871
|
|
3655
3872
|
Procedure to send a file from org1 to org2:
|
3656
3873
|
|
3657
|
-
|
3658
|
-
|
3659
|
-
|
3660
|
-
|
3661
|
-
|
3874
|
+
- Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
3875
|
+
- Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
3876
|
+
- Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
3877
|
+
- Check that access works and locate the destination folder `mydestfolder`
|
3878
|
+
- execute the following:
|
3662
3879
|
|
3663
3880
|
```bash
|
3664
3881
|
ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
@@ -3666,15 +3883,15 @@ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | as
|
|
3666
3883
|
|
3667
3884
|
Explanation:
|
3668
3885
|
|
3669
|
-
|
3670
|
-
|
3671
|
-
|
3672
|
-
|
3673
|
-
|
3674
|
-
|
3675
|
-
|
3676
|
-
|
3677
|
-
|
3886
|
+
- `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
3887
|
+
- `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
3888
|
+
- `--format=json` format the output in JSON (instead of default text table)
|
3889
|
+
- `--display=data` display only the result, and remove other information, such as workspace name
|
3890
|
+
- `|` the standard output of the first command is fed into the second one
|
3891
|
+
- `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
3892
|
+
- `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
3893
|
+
- `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
3894
|
+
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
3678
3895
|
|
3679
3896
|
#### Find Files
|
3680
3897
|
|
@@ -3682,34 +3899,34 @@ The command `aoc files find [--value=expression]` will recursively scan storage
|
|
3682
3899
|
|
3683
3900
|
The expression can be of 3 formats:
|
3684
3901
|
|
3685
|
-
|
3686
|
-
|
3902
|
+
- empty (default) : all files, equivalent to value: `exec:true`
|
3903
|
+
- not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
3687
3904
|
|
3688
3905
|
For instance, to find files with a special extension, use `--value='\.myext$'`
|
3689
3906
|
|
3690
|
-
|
3907
|
+
- starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
3691
3908
|
|
3692
3909
|
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
3693
3910
|
|
3694
|
-
|
3911
|
+
- Find files more recent than 100 days
|
3695
3912
|
|
3696
3913
|
```bash
|
3697
3914
|
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
3698
3915
|
```
|
3699
3916
|
|
3700
|
-
|
3917
|
+
- Find files older than 1 year on a given node and store in file list
|
3701
3918
|
|
3702
3919
|
```bash
|
3703
3920
|
ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
3704
3921
|
```
|
3705
3922
|
|
3706
|
-
|
3923
|
+
- Delete the files, one by one
|
3707
3924
|
|
3708
3925
|
```bash
|
3709
3926
|
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 delete "$path" ;done
|
3710
3927
|
```
|
3711
3928
|
|
3712
|
-
|
3929
|
+
- Delete the files in bulk
|
3713
3930
|
|
3714
3931
|
```bash
|
3715
3932
|
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='_secret_here_' v3 delete @lines:@stdin:
|
@@ -3718,18 +3935,17 @@ cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='
|
|
3718
3935
|
### AoC sample commands
|
3719
3936
|
|
3720
3937
|
```bash
|
3721
|
-
aoc -N remind --username=my_aoc_user_email
|
3722
|
-
aoc -N servers
|
3723
3938
|
aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
3724
3939
|
aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
3725
|
-
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
3726
|
-
aoc admin ats access_key delete
|
3940
|
+
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
3941
|
+
aoc admin ats access_key delete ak1ibmcloud
|
3727
3942
|
aoc admin ats access_key list --fields=name,id
|
3728
|
-
aoc admin ats access_key node
|
3943
|
+
aoc admin ats access_key node ak1ibmcloud --secret=somesecret browse /
|
3729
3944
|
aoc admin ats cluster clouds
|
3730
3945
|
aoc admin ats cluster list
|
3731
3946
|
aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
3732
3947
|
aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
3948
|
+
aoc admin auth_providers list
|
3733
3949
|
aoc admin res application list
|
3734
3950
|
aoc admin res client list
|
3735
3951
|
aoc admin res client_access_key list
|
@@ -3750,14 +3966,15 @@ aoc admin res self show
|
|
3750
3966
|
aoc admin res short_link list
|
3751
3967
|
aoc admin res user list
|
3752
3968
|
aoc admin res workspace_membership list
|
3753
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3754
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3755
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3756
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3757
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3758
|
-
aoc admin resource node
|
3969
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do browse /
|
3970
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do delete /folder1
|
3971
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do mkdir /folder1
|
3972
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
|
3973
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 events
|
3974
|
+
aoc admin resource node do name my_aoc_ak_name --secret=my_aoc_ak_secret v3 access_key delete testsub1
|
3759
3975
|
aoc admin resource workspace list
|
3760
3976
|
aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
3977
|
+
aoc admin subscription
|
3761
3978
|
aoc automation workflow action my_wf_id create --value=@json:'{"name":"toto"}'
|
3762
3979
|
aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
3763
3980
|
aoc automation workflow delete my_wf_id
|
@@ -3769,10 +3986,11 @@ aoc faspex
|
|
3769
3986
|
aoc files bearer /
|
3770
3987
|
aoc files bearer_token_node / --cache-tokens=no
|
3771
3988
|
aoc files browse /
|
3772
|
-
aoc files browse /
|
3989
|
+
aoc files browse / --link=my_aoc_publink_folder
|
3773
3990
|
aoc files delete /testsrc
|
3774
3991
|
aoc files download --transfer=connect /200KB.1
|
3775
|
-
aoc files file
|
3992
|
+
aoc files file modify --path=my_aoc_test_folder
|
3993
|
+
aoc files file permission --path=my_aoc_test_folder list
|
3776
3994
|
aoc files file show --path=/200KB.1
|
3777
3995
|
aoc files file show my_file_id
|
3778
3996
|
aoc files find / --value='\.partial$'
|
@@ -3782,12 +4000,14 @@ aoc files rename /somefolder testdst
|
|
3782
4000
|
aoc files short_link create --to-folder=/testdst --value=private
|
3783
4001
|
aoc files short_link create --to-folder=/testdst --value=public
|
3784
4002
|
aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_link"}'
|
4003
|
+
aoc files sync admin status --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
4004
|
+
aoc files sync start --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
3785
4005
|
aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
|
4006
|
+
aoc files upload --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
3786
4007
|
aoc files upload --to-folder=/testsrc testfile.bin
|
3787
|
-
aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
3788
4008
|
aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
3789
4009
|
aoc files v3 info
|
3790
|
-
aoc org
|
4010
|
+
aoc org --link=my_aoc_publink_recv_from_aocuser
|
3791
4011
|
aoc organization
|
3792
4012
|
aoc packages browse "my_package_id" /contents
|
3793
4013
|
aoc packages list
|
@@ -3796,25 +4016,27 @@ aoc packages recv "my_package_id" --to-folder=.
|
|
3796
4016
|
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
3797
4017
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_external_user"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
3798
4018
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_internal_user"],"note":"my note"}' testfile.bin
|
4019
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
4020
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
3799
4021
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' testfile.bin
|
3800
4022
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' testfile.bin
|
3801
4023
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
3802
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
3803
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
3804
4024
|
aoc packages shared_inboxes list
|
4025
|
+
aoc remind --username=my_aoc_user_email
|
4026
|
+
aoc servers
|
3805
4027
|
aoc user profile modify @json:'{"name":"dummy change"}'
|
3806
4028
|
aoc user profile show
|
3807
4029
|
aoc user workspaces current
|
3808
4030
|
aoc user workspaces list
|
3809
4031
|
```
|
3810
4032
|
|
3811
|
-
## <a id="ats"></a>Plugin: Aspera Transfer Service
|
4033
|
+
## <a id="ats"></a>Plugin: `ats`: IBM Aspera Transfer Service
|
3812
4034
|
|
3813
4035
|
ATS is usable either :
|
3814
4036
|
|
3815
|
-
|
4037
|
+
- from an AoC subscription : ascli aoc admin ats : use AoC authentication
|
3816
4038
|
|
3817
|
-
|
4039
|
+
- or from an IBM Cloud subscription : ascli ats : use IBM Cloud API key authentication
|
3818
4040
|
|
3819
4041
|
### IBM Cloud ATS : creation of api key
|
3820
4042
|
|
@@ -3843,8 +4065,8 @@ UUID ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f
|
|
3843
4065
|
|
3844
4066
|
References:
|
3845
4067
|
|
3846
|
-
|
3847
|
-
|
4068
|
+
- [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
4069
|
+
- [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
3848
4070
|
|
3849
4071
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
3850
4072
|
|
@@ -3929,13 +4151,15 @@ The parameters provided to ATS for access key creation are the ones of [ATS API]
|
|
3929
4151
|
### ATS sample commands
|
3930
4152
|
|
3931
4153
|
```bash
|
3932
|
-
ats access_key cluster
|
4154
|
+
ats access_key cluster ak2ibmcloud --secret=somesecret
|
3933
4155
|
ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
3934
|
-
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
4156
|
+
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak2ibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
4157
|
+
ats access_key delete ak2ibmcloud
|
3935
4158
|
ats access_key delete ak_aws
|
3936
|
-
ats access_key
|
4159
|
+
ats access_key entitlement ak2ibmcloud
|
3937
4160
|
ats access_key list --fields=name,id
|
3938
|
-
ats access_key node
|
4161
|
+
ats access_key node ak2ibmcloud browse / --secret=somesecret
|
4162
|
+
ats access_key show ak2ibmcloud
|
3939
4163
|
ats api_key create
|
3940
4164
|
ats api_key instances
|
3941
4165
|
ats api_key list
|
@@ -3945,17 +4169,21 @@ ats cluster show --cloud=aws --region=eu-west-1
|
|
3945
4169
|
ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
3946
4170
|
```
|
3947
4171
|
|
3948
|
-
## Plugin: IBM Aspera High Speed Transfer Server (
|
4172
|
+
## <a id="server"></a>Plugin: `server`: IBM Aspera High Speed Transfer Server (SSH)
|
3949
4173
|
|
3950
|
-
|
4174
|
+
The `server` plugin is used for operations on Aspera HSTS using SSH authentication.
|
3951
4175
|
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
3952
|
-
|
4176
|
+
An SSH session is established, authenticated with either a password or an SSH private key,
|
4177
|
+
then commands `ascp` (for transfers) and `ascmd` (for file operations) are executed.
|
4178
|
+
|
4179
|
+
> **Note:** The URL to be provided is usually: `ssh://_server_address_:33001`
|
3953
4180
|
|
3954
4181
|
### Server sample commands
|
3955
4182
|
|
3956
4183
|
```bash
|
3957
|
-
server
|
4184
|
+
server br /
|
3958
4185
|
server browse /
|
4186
|
+
server browse NEW_SERVER_FOLDER/testfile.bin
|
3959
4187
|
server browse folder_1/target_hot
|
3960
4188
|
server cp NEW_SERVER_FOLDER/testfile.bin folder_1/200KB.2
|
3961
4189
|
server delete NEW_SERVER_FOLDER
|
@@ -3971,34 +4199,39 @@ server md5sum NEW_SERVER_FOLDER/testfile.bin
|
|
3971
4199
|
server mkdir NEW_SERVER_FOLDER --logger=stdout
|
3972
4200
|
server mkdir folder_1/target_hot
|
3973
4201
|
server mv folder_1/200KB.2 folder_1/to.delete
|
4202
|
+
server sync admin status --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"}}'
|
4203
|
+
server sync admin status --sync-session=mysync --sync-info=@json:'{"sessions":[{"name":"mysync","local_dir":"syncdir"}]}'
|
4204
|
+
server sync start --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"},"remote":{"path":"'"NEW_SERVER_FOLDER"'"},"reset":true,"quiet":false}'
|
4205
|
+
server sync start --sync-info=@json:'{"sessions":[{"name":"mysync","direction":"pull","remote_dir":"'"NEW_SERVER_FOLDER"'","local_dir":"syncdir","reset":true}]}'
|
3974
4206
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","'"filelist.txt"'"]}' --to-folder=NEW_SERVER_FOLDER
|
3975
4207
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-pair-list","'"filepairlist.txt"'"]}'
|
3976
4208
|
server upload --sources=@ts --ts=@json:'{"EX_file_list":"'"filelist.txt"'"}' --to-folder=NEW_SERVER_FOLDER
|
3977
4209
|
server upload --sources=@ts --ts=@json:'{"EX_file_pair_list":"'"filepairlist.txt"'"}'
|
3978
4210
|
server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
3979
4211
|
server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
3980
|
-
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
4212
|
+
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email --ascp-opts=@list:' -l 10m'
|
3981
4213
|
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"_secret_here_","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
|
3982
4214
|
server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
3983
|
-
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
4215
|
+
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}' --progress=multi
|
3984
4216
|
```
|
3985
4217
|
|
3986
4218
|
### Authentication on Server with SSH session
|
3987
4219
|
|
3988
|
-
If SSH is the session
|
4220
|
+
If SSH is the session protocol (by default i.e. not WSS), then following session authentication methods are supported:
|
3989
4221
|
|
3990
|
-
|
3991
|
-
|
4222
|
+
- `password`: SSH password
|
4223
|
+
- `ssh_keys`: SSH keys (Multiple SSH key paths can be provided.)
|
3992
4224
|
|
3993
|
-
If username is not provided
|
4225
|
+
If `username` is not provided then the default transfer user `xfer` is used.
|
3994
4226
|
|
3995
|
-
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
4227
|
+
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
4228
|
+
Example:
|
3996
4229
|
|
3997
|
-
```
|
3998
|
-
ascli server --url=ssh
|
4230
|
+
```bash
|
4231
|
+
ascli server --url=ssh://_server_address_:33001 ... --ts=@json:'{"token":"Basic abc123"}'
|
3999
4232
|
```
|
4000
4233
|
|
4001
|
-
> Note
|
4234
|
+
> **Note:** If you need to use the Aspera public keys, then specify an empty token: `--ts=@json:'{"token":""}'` : Aspera public SSH keys will be used, but the protocol will ignore the empty token.
|
4002
4235
|
|
4003
4236
|
The value of the `ssh_keys` option can be a single value or an array.
|
4004
4237
|
Each value is a **path** to a private key and is expanded (`~` is replaced with the user's home folder).
|
@@ -4011,7 +4244,7 @@ ascli server --ssh-keys=@list:,~/.ssh/id_rsa
|
|
4011
4244
|
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
4012
4245
|
```
|
4013
4246
|
|
4014
|
-
For
|
4247
|
+
For file operation command (browse, delete), the ruby SSH client library `Net::SSH` is used and provides several options settable using option `ssh_options`.
|
4015
4248
|
For a list of SSH client options, refer to the ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html).
|
4016
4249
|
|
4017
4250
|
By default the SSH library expect that a local ssh-agent is running.
|
@@ -4030,10 +4263,10 @@ ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant p
|
|
4030
4263
|
|
4031
4264
|
This means that you don't have such an SSH agent running, then:
|
4032
4265
|
|
4033
|
-
|
4034
|
-
|
4035
|
-
|
4036
|
-
|
4266
|
+
- Check env var: `SSH_AGENT_SOCK`
|
4267
|
+
- Check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option)
|
4268
|
+
- [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
4269
|
+
- To disable the use of `ssh-agent`, use the option `ssh_options` like this:
|
4037
4270
|
|
4038
4271
|
```bash
|
4039
4272
|
ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
@@ -4041,7 +4274,17 @@ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
|
4041
4274
|
|
4042
4275
|
This can also be set as default using a global preset.
|
4043
4276
|
|
4044
|
-
###
|
4277
|
+
### Other session channels for `server`
|
4278
|
+
|
4279
|
+
URL schemes `local` and `https` are also supported, mainly for testing purpose.
|
4280
|
+
(`--url=local:` , `--url=https://...`)
|
4281
|
+
|
4282
|
+
- `local` will execute `ascmd` locally, instead of using a SSH cnnection.
|
4283
|
+
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
4284
|
+
|
4285
|
+
As, most of the time, SSH is used, if an `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
4286
|
+
|
4287
|
+
### Examples: `server`
|
4045
4288
|
|
4046
4289
|
One can test the `server` application using the well known demo server:
|
4047
4290
|
|
@@ -4053,7 +4296,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
4053
4296
|
|
4054
4297
|
`initdemo` creates a [option preset](#lprt) `demoserver` and set it as default for plugin `server`.
|
4055
4298
|
|
4056
|
-
## Plugin: IBM Aspera High Speed Transfer Server
|
4299
|
+
## <a id="node"></a>Plugin: `node`: IBM Aspera High Speed Transfer Server Node
|
4057
4300
|
|
4058
4301
|
This plugin gives access to capabilities provided by HSTS node API.
|
4059
4302
|
|
@@ -4061,27 +4304,27 @@ This plugin gives access to capabilities provided by HSTS node API.
|
|
4061
4304
|
|
4062
4305
|
It is possible to:
|
4063
4306
|
|
4064
|
-
|
4065
|
-
|
4066
|
-
|
4307
|
+
- browse
|
4308
|
+
- transfer (upload / download)
|
4309
|
+
- ...
|
4067
4310
|
|
4068
4311
|
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
4069
4312
|
|
4070
|
-
|
4071
|
-
|
4072
|
-
|
4313
|
+
- `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token, used as is.
|
4314
|
+
- `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
4315
|
+
- `basic` : transfer spec is created like this:
|
4073
4316
|
|
4074
4317
|
```javascript
|
4075
4318
|
{
|
4076
|
-
"remote_host": address of node url,
|
4319
|
+
"remote_host": "<address of node url>",
|
4077
4320
|
"remote_user": "xfer",
|
4078
4321
|
"ssh_port": 33001,
|
4079
4322
|
"token": "Basic <base 64 encoded user/pass>",
|
4080
|
-
"direction": send
|
4323
|
+
"direction": "[send|receive]"
|
4081
4324
|
}
|
4082
4325
|
```
|
4083
4326
|
|
4084
|
-
Note
|
4327
|
+
> **Note:** the port is assumed to be the default Aspera SSH port `33001` and transfer user is assumed to be `xfer`.
|
4085
4328
|
|
4086
4329
|
### Central
|
4087
4330
|
|
@@ -4111,8 +4354,8 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
4111
4354
|
|
4112
4355
|
`ascli` supports remote operations through the node API. Operations are:
|
4113
4356
|
|
4114
|
-
|
4115
|
-
|
4357
|
+
- Start watchd and watchfolderd services running as a system user having access to files
|
4358
|
+
- configure a watchfolder to define automated transfers
|
4116
4359
|
|
4117
4360
|
```javascript
|
4118
4361
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -4168,9 +4411,19 @@ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"myst
|
|
4168
4411
|
### Node sample commands
|
4169
4412
|
|
4170
4413
|
```bash
|
4171
|
-
node
|
4172
|
-
node
|
4173
|
-
node access_key do my_aoc_ak_name
|
4414
|
+
node access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
4415
|
+
node access_key delete aoc_1
|
4416
|
+
node access_key do my_aoc_ak_name browse /
|
4417
|
+
node access_key do my_aoc_ak_name delete /folder2
|
4418
|
+
node access_key do my_aoc_ak_name delete testfile1
|
4419
|
+
node access_key do my_aoc_ak_name download testfile1 --to-folder=.
|
4420
|
+
node access_key do my_aoc_ak_name file show --path=/testfile1
|
4421
|
+
node access_key do my_aoc_ak_name file show 1
|
4422
|
+
node access_key do my_aoc_ak_name find /
|
4423
|
+
node access_key do my_aoc_ak_name mkdir /folder1
|
4424
|
+
node access_key do my_aoc_ak_name node_info /
|
4425
|
+
node access_key do my_aoc_ak_name rename /folder1 folder2
|
4426
|
+
node access_key do my_aoc_ak_name upload 'faux:///testfile1?1k' --default_ports=no
|
4174
4427
|
node access_key list
|
4175
4428
|
node api_details
|
4176
4429
|
node async bandwidth 1
|
@@ -4182,7 +4435,7 @@ node async show ALL
|
|
4182
4435
|
node basic_token
|
4183
4436
|
node browse / -r
|
4184
4437
|
node delete /todelete
|
4185
|
-
node delete @list
|
4438
|
+
node delete @list:,folder_1/todelete,folder_1/tdlink,folder_1/delfile
|
4186
4439
|
node delete folder_1/10MB.1
|
4187
4440
|
node delete testfile.bin
|
4188
4441
|
node download testfile.bin --to-folder=.
|
@@ -4190,15 +4443,26 @@ node download testfile.bin --to-folder=. --token-type=hybrid
|
|
4190
4443
|
node health
|
4191
4444
|
node info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
|
4192
4445
|
node license
|
4193
|
-
node mkdir /todelete
|
4194
|
-
node mkfile /delfile1 "hello world"
|
4195
|
-
node mklink /todelete /tdlink
|
4196
|
-
node rename
|
4446
|
+
node mkdir folder_1/todelete
|
4447
|
+
node mkfile folder_1/delfile1 "hello world"
|
4448
|
+
node mklink folder_1/todelete folder_1/tdlink
|
4449
|
+
node rename folder_1 delfile1 delfile
|
4197
4450
|
node search / --value=@json:'{"sort":"mtime"}'
|
4198
4451
|
node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
4199
4452
|
node service delete service1
|
4200
4453
|
node service list
|
4201
4454
|
node space /
|
4455
|
+
node sync bandwidth my_syncid
|
4456
|
+
node sync counters my_syncid
|
4457
|
+
node sync create --value=@json:'{"configuration":{"name":"sync1","local":{"path":"my_local_path"},"remote":{"host":"my_host","port":my_port,"user":"my_username","pass":"my_password","path":"my_remote_path"}}}'
|
4458
|
+
node sync delete my_syncid
|
4459
|
+
node sync files my_syncid
|
4460
|
+
node sync list
|
4461
|
+
node sync show my_syncid
|
4462
|
+
node sync start my_syncid
|
4463
|
+
node sync state my_syncid
|
4464
|
+
node sync stop my_syncid
|
4465
|
+
node sync summary my_syncid
|
4202
4466
|
node transfer list --value=@json:'{"active_only":true}'
|
4203
4467
|
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass"}'
|
4204
4468
|
node upload --username=my_aoc_ak_name --password=my_aoc_ak_secret testfile.bin --token-type=basic
|
@@ -4206,123 +4470,130 @@ node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps
|
|
4206
4470
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}' --token-type=hybrid
|
4207
4471
|
```
|
4208
4472
|
|
4209
|
-
## Plugin: IBM Aspera
|
4210
|
-
|
4211
|
-
This is currently in beta, limited operations are supported.
|
4473
|
+
## <a id="faspex5"></a>Plugin: `faspex5`: IBM Aspera Faspex v5
|
4212
4474
|
|
4213
|
-
|
4214
|
-
|
4215
|
-
The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "IBM Aspera Faspex API".
|
4216
|
-
|
4217
|
-
### Faspex 5 authentication
|
4475
|
+
IBM Aspera's newer self-managed application.
|
4218
4476
|
|
4219
4477
|
3 authentication methods are supported:
|
4220
4478
|
|
4221
|
-
|
4222
|
-
|
4223
|
-
|
4479
|
+
- jwt
|
4480
|
+
- web
|
4481
|
+
- boot
|
4224
4482
|
|
4225
|
-
|
4483
|
+
### Faspex 5 JWT authentication
|
4226
4484
|
|
4227
|
-
This is the
|
4485
|
+
This is the **recomended** method to use.
|
4228
4486
|
|
4229
4487
|
For `jwt`, create an API client in Faspex with JWT support:
|
4230
4488
|
|
4231
|
-
|
4232
|
-
|
4233
|
-
|
4234
|
-
|
4235
|
-
|
4236
|
-
|
4489
|
+
- Select a private key file: if you don't have any refer to section [Private Key](#private_key)
|
4490
|
+
- Navigate to the web UI: Admin → Configurations → API Clients → Create
|
4491
|
+
- Activate JWT
|
4492
|
+
- Paste **public** key in the appropriate section
|
4493
|
+
- Click on Create Button
|
4494
|
+
- Take note of Client Id (and Client Secret, but not used in current version)
|
4237
4495
|
|
4238
4496
|
Then use these options:
|
4239
4497
|
|
4240
4498
|
```text
|
4241
4499
|
--auth=jwt
|
4242
|
-
--client-id=
|
4243
|
-
--client-secret=
|
4244
|
-
--username=
|
4500
|
+
--client-id=_client_id_here_
|
4501
|
+
--client-secret=_secret_here_
|
4502
|
+
--username=_username_here_
|
4245
4503
|
--private-key=@file:.../path/to/key.pem
|
4246
4504
|
```
|
4247
4505
|
|
4248
|
-
|
4506
|
+
> **Note:** The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
4507
|
+
|
4508
|
+
### Faspex 5 web authentication
|
4249
4509
|
|
4250
4510
|
For `web` method, create an API client in Faspex without JWT:
|
4251
4511
|
|
4252
|
-
|
4253
|
-
|
4254
|
-
|
4255
|
-
|
4256
|
-
|
4512
|
+
- Navigate to the web UI: Admin → Configurations → API Clients → Create
|
4513
|
+
- Do not Activate JWT
|
4514
|
+
- Set **Redirect URI** to `https://127.0.0.1:8888`
|
4515
|
+
- Click on Create Button
|
4516
|
+
- Take note of Client Id (and Client Secret, but not used in current version)
|
4257
4517
|
|
4258
4518
|
Then use options:
|
4259
4519
|
|
4260
4520
|
```text
|
4261
4521
|
--auth=web
|
4262
|
-
--client-id=
|
4263
|
-
--client-secret=
|
4522
|
+
--client-id=_client_id_here_
|
4523
|
+
--client-secret=_secret_here_
|
4264
4524
|
--redirect-uri=https://127.0.0.1:8888
|
4265
4525
|
```
|
4266
4526
|
|
4267
|
-
|
4527
|
+
### Faspex 5 bootstrap authentication
|
4268
4528
|
|
4269
4529
|
For `boot` method: (will be removed in future)
|
4270
4530
|
|
4271
|
-
|
4272
|
-
|
4273
|
-
|
4274
|
-
|
4531
|
+
- Open a Web Browser
|
4532
|
+
- Start developer mode
|
4533
|
+
- Login to Faspex 5
|
4534
|
+
- Find the first API call with `Authorization` header, and copy the value of the token (series of base64 values with dots)
|
4275
4535
|
|
4276
|
-
Use
|
4536
|
+
Use this token as password and use `--auth=boot`.
|
4277
4537
|
|
4278
4538
|
```bash
|
4279
|
-
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=
|
4539
|
+
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=_token_here_
|
4280
4540
|
```
|
4281
4541
|
|
4282
4542
|
### Faspex 5 sample commands
|
4283
4543
|
|
4544
|
+
Most commands are directly REST API calls.
|
4545
|
+
Parameters to commandsa are carried through option `value`, as extended value.
|
4546
|
+
Usually using JSON format with prefix `@json:`.
|
4547
|
+
|
4548
|
+
> **Note:** The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
4549
|
+
|
4284
4550
|
```bash
|
4285
4551
|
faspex5 admin res accounts list
|
4286
4552
|
faspex5 admin res contacts list
|
4287
4553
|
faspex5 admin res jobs list
|
4554
|
+
faspex5 admin res metadata_profiles list
|
4288
4555
|
faspex5 admin res node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
4289
4556
|
faspex5 admin res oauth_clients list
|
4290
4557
|
faspex5 admin res registrations list
|
4291
4558
|
faspex5 admin res saml_configs list
|
4292
4559
|
faspex5 admin res shared_inboxes list
|
4293
4560
|
faspex5 admin res workgroups list
|
4561
|
+
faspex5 bearer_token
|
4294
4562
|
faspex5 health
|
4295
4563
|
faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
4296
4564
|
faspex5 package receive "my_package_id" --to-folder=. --ts=@json:'{"content_protection_password":"abc123_yo"}'
|
4297
4565
|
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin --ts=@json:'{"content_protection_password":"_content_prot_here_"}'
|
4566
|
+
faspex5 package show "my_package_id"
|
4567
|
+
faspex5 user profile modify @json:'{"preference":{"connect_disabled":false}}'
|
4568
|
+
faspex5 user profile show
|
4298
4569
|
```
|
4299
4570
|
|
4300
|
-
|
4571
|
+
Other examples:
|
4301
4572
|
|
4302
|
-
|
4573
|
+
- List all shared inboxes
|
4303
4574
|
|
4304
4575
|
```javascript
|
4305
4576
|
ascli faspex5 admin res shared list --value=@json:'{"all":true}' --fields=id,name
|
4306
4577
|
```
|
4307
4578
|
|
4308
|
-
|
4579
|
+
- Create Metadata profile
|
4309
4580
|
|
4310
4581
|
```javascript
|
4311
4582
|
ascli faspex5 admin res metadata_profiles create --value=@json:'{"name":"the profile","default":false,"title":{"max_length":200,"illegal_chars":[]},"note":{"max_length":400,"illegal_chars":[],"enabled":false},"fields":[{"ordering":0,"name":"field1","type":"text_area","require":true,"illegal_chars":[],"max_length":100},{"ordering":1,"name":"fff2","type":"option_list","require":false,"choices":["opt1","opt2"]}]}'
|
4312
4583
|
```
|
4313
4584
|
|
4314
|
-
|
4585
|
+
- Create a Shared inbox with specific metadata profile
|
4315
4586
|
|
4316
4587
|
```javascript
|
4317
4588
|
ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox","metadata_profile_id":1}'
|
4318
4589
|
```
|
4319
4590
|
|
4320
|
-
## Plugin: IBM Aspera Faspex
|
4591
|
+
## <a id="faspex"></a>Plugin: `faspex`: IBM Aspera Faspex v4
|
4321
4592
|
|
4322
4593
|
Notes:
|
4323
4594
|
|
4324
|
-
|
4325
|
-
|
4595
|
+
- The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
4596
|
+
- For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
4326
4597
|
|
4327
4598
|
### Listing Packages
|
4328
4599
|
|
@@ -4336,18 +4607,18 @@ By default it looks in box `inbox`, but the following boxes are also supported:
|
|
4336
4607
|
|
4337
4608
|
A user can receive a package because the recipient is:
|
4338
4609
|
|
4339
|
-
|
4340
|
-
|
4610
|
+
- the user himself (default)
|
4611
|
+
- the user is member of a dropbox/workgroup: filter using option `recipient` set with value `*<name of dropbox/workgroup>`
|
4341
4612
|
|
4342
4613
|
#### Option `query`
|
4343
4614
|
|
4344
4615
|
As inboxes may be large, it is possible to use the following query parameters:
|
4345
4616
|
|
4346
|
-
|
4347
|
-
|
4348
|
-
|
4349
|
-
|
4350
|
-
|
4617
|
+
- `count` : (native) number items in one API call (default=0, equivalent to 10)
|
4618
|
+
- `page` : (native) id of page in call (default=0)
|
4619
|
+
- `startIndex` : (native) index of item to start, default=0, oldest index=0
|
4620
|
+
- `max` : maximum number of items
|
4621
|
+
- `pmax` : maximum number of pages
|
4351
4622
|
|
4352
4623
|
(SQL query is `LIMIT <startIndex>, <count>`)
|
4353
4624
|
|
@@ -4367,9 +4638,9 @@ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in recei
|
|
4367
4638
|
|
4368
4639
|
The command is `package recv`, possible methods are:
|
4369
4640
|
|
4370
|
-
|
4371
|
-
|
4372
|
-
|
4641
|
+
- provide a package id with option `id`
|
4642
|
+
- provide a public link with option `link`
|
4643
|
+
- provide a `faspe:` URI with option `link`
|
4373
4644
|
|
4374
4645
|
```bash
|
4375
4646
|
ascli faspex package recv --id=12345
|
@@ -4401,8 +4672,8 @@ If the recipient is a dropbox or workgroup: provide the name of the dropbox or w
|
|
4401
4672
|
|
4402
4673
|
Additional optional parameters in `delivery_info`:
|
4403
4674
|
|
4404
|
-
|
4405
|
-
|
4675
|
+
- Package Note: : `"note":"note this and that"`
|
4676
|
+
- Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
4406
4677
|
|
4407
4678
|
### Email notification on transfer
|
4408
4679
|
|
@@ -4500,7 +4771,7 @@ faspex v4 wmembership list
|
|
4500
4771
|
faspex v4 workgroup list
|
4501
4772
|
```
|
4502
4773
|
|
4503
|
-
## Plugin: IBM Aspera Shares
|
4774
|
+
## <a id="shares"></a>Plugin: `shares`: IBM Aspera Shares v1
|
4504
4775
|
|
4505
4776
|
Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)
|
4506
4777
|
|
@@ -4508,33 +4779,39 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
4508
4779
|
|
4509
4780
|
```bash
|
4510
4781
|
shares admin share list
|
4511
|
-
shares admin share
|
4782
|
+
shares admin share list --fields=-status,status_message
|
4783
|
+
shares admin share user_permissions 1
|
4512
4784
|
shares admin user app_authorizations 1
|
4785
|
+
shares admin user ldap_import --value=the_name
|
4513
4786
|
shares admin user list
|
4787
|
+
shares admin user saml_import --value=@json:'{"id":"the_id","name_id":"the_name"}'
|
4514
4788
|
shares admin user share_permissions 1
|
4789
|
+
shares health
|
4515
4790
|
shares repository browse /
|
4516
4791
|
shares repository delete my_shares_upload/testfile.bin
|
4517
4792
|
shares repository download --to-folder=. my_shares_upload/testfile.bin
|
4518
|
-
shares repository download --to-folder=. my_shares_upload/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy
|
4793
|
+
shares repository download --to-folder=. my_shares_upload/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
4519
4794
|
shares repository upload --to-folder=my_shares_upload testfile.bin
|
4520
|
-
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy
|
4795
|
+
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
4521
4796
|
```
|
4522
4797
|
|
4523
|
-
## Plugin: Console
|
4798
|
+
## <a id="console"></a>Plugin: `console`: IBM Aspera Console
|
4524
4799
|
|
4525
4800
|
### Console sample commands
|
4526
4801
|
|
4527
4802
|
```bash
|
4528
|
-
console
|
4529
|
-
console transfer
|
4803
|
+
console health
|
4804
|
+
console transfer current list
|
4805
|
+
console transfer smart list
|
4530
4806
|
console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
4531
4807
|
```
|
4532
4808
|
|
4533
|
-
## Plugin: Orchestrator
|
4809
|
+
## <a id="orchestrator"></a>Plugin: `orchestrator`:IBM Aspera Orchestrator
|
4534
4810
|
|
4535
4811
|
### Orchestrator sample commands
|
4536
4812
|
|
4537
4813
|
```bash
|
4814
|
+
orchestrator health
|
4538
4815
|
orchestrator info
|
4539
4816
|
orchestrator plugins
|
4540
4817
|
orchestrator processes
|
@@ -4548,22 +4825,24 @@ orchestrator workflow status ALL
|
|
4548
4825
|
orchestrator workflow status my_orch_workflow_id
|
4549
4826
|
```
|
4550
4827
|
|
4551
|
-
## Plugin: IBM Cloud Object Storage
|
4828
|
+
## <a id="cos"></a>Plugin: `cos`: IBM Cloud Object Storage
|
4552
4829
|
|
4553
4830
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
4554
4831
|
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
4555
4832
|
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
4556
4833
|
|
4557
|
-
There are two possibilities to provide credentials.
|
4834
|
+
There are two possibilities to provide credentials.
|
4835
|
+
If you already have the endpoint, apikey and CRN, use the first method.
|
4836
|
+
If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
4558
4837
|
|
4559
4838
|
### Using endpoint, apikey and Resource Instance ID (CRN)
|
4560
4839
|
|
4561
4840
|
If you have those parameters already, then following options shall be provided:
|
4562
4841
|
|
4563
|
-
|
4564
|
-
|
4565
|
-
|
4566
|
-
|
4842
|
+
- `bucket` bucket name
|
4843
|
+
- `endpoint` storage endpoint url, e.g. `https://s3.hkg02.cloud-object-storage.appdomain.cloud`
|
4844
|
+
- `apikey` API Key
|
4845
|
+
- `crn` resource instance id
|
4567
4846
|
|
4568
4847
|
For example, let us create a default configuration:
|
4569
4848
|
|
@@ -4576,9 +4855,17 @@ Then, jump to the transfer example.
|
|
4576
4855
|
|
4577
4856
|
### Using service credential file
|
4578
4857
|
|
4579
|
-
If you are the COS administrator and don't have yet the credential:
|
4858
|
+
If you are the COS administrator and don't have yet the credential:
|
4859
|
+
Service credentials are directly created using the IBM cloud Console (web UI).
|
4860
|
+
Navigate to:
|
4580
4861
|
|
4581
|
-
|
4862
|
+
- → Navigation Menu
|
4863
|
+
- → [Resource List](https://cloud.ibm.com/resources)
|
4864
|
+
- → [Storage](https://cloud.ibm.com/objectstorage)
|
4865
|
+
- → Select your storage instance
|
4866
|
+
- → Service Credentials
|
4867
|
+
- → New credentials (Leave default role: Writer, no special options)
|
4868
|
+
- → Copy to clipboard
|
4582
4869
|
|
4583
4870
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
4584
4871
|
|
@@ -4617,9 +4904,9 @@ The field `apikey` is for option `apikey`
|
|
4617
4904
|
|
4618
4905
|
The required options for this method are:
|
4619
4906
|
|
4620
|
-
|
4621
|
-
|
4622
|
-
|
4907
|
+
- `bucket` bucket name
|
4908
|
+
- `region` bucket region, e.g. eu-de
|
4909
|
+
- `service_credentials` see below
|
4623
4910
|
|
4624
4911
|
For example, let us create a default configuration:
|
4625
4912
|
|
@@ -4639,39 +4926,51 @@ ascli cos node info
|
|
4639
4926
|
ascli cos node upload 'faux:///sample1G?1g'
|
4640
4927
|
```
|
4641
4928
|
|
4642
|
-
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
4929
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man `ascp` and section above), but you can of course send a real file by specifying a real file instead.
|
4643
4930
|
|
4644
4931
|
### COS sample commands
|
4645
4932
|
|
4646
4933
|
```bash
|
4647
|
-
cos
|
4648
|
-
cos
|
4934
|
+
cos --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
4935
|
+
cos --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
4649
4936
|
cos node access_key show self
|
4650
4937
|
cos node download testfile.bin --to-folder=.
|
4651
4938
|
cos node info
|
4652
4939
|
cos node upload testfile.bin
|
4653
4940
|
```
|
4654
4941
|
|
4655
|
-
## Plugin: IBM Aspera Sync
|
4942
|
+
## <a id="async"></a>Plugin: `async`: IBM Aspera Sync
|
4656
4943
|
|
4657
4944
|
A basic plugin to start an "async" using `ascli`.
|
4658
|
-
The main advantage is the possibility to
|
4945
|
+
The main advantage over bare `async` command line is the possibility to use a configuration file, using `ascli` standard options.
|
4946
|
+
|
4947
|
+
Also, the `sync` command is also made available through the `server sync` and `aoc files sync` commands.
|
4948
|
+
In this case, some of the `sync` parameters are fill from parameters of the related plugin.
|
4949
|
+
|
4950
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
4951
|
+
>
|
4952
|
+
> **Note:** Two JSON syntax are supported for option `sync_info`.
|
4953
|
+
> The first is same sync payload as specified on the `async` option `--conf` or in the latest node API, this is the prefered syntax and allows a single session definition.
|
4954
|
+
> The second (legacy) is specific to `ascli` and allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
4955
|
+
|
4956
|
+
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
4659
4957
|
|
4660
4958
|
### Sync sample commands
|
4661
4959
|
|
4662
4960
|
```bash
|
4663
|
-
sync
|
4961
|
+
sync admin status --sync-info=@json:'{"sessions":[{"name":"test","local_dir":"contents"}]}'
|
4962
|
+
sync start --sync-info=@json:'{"instance":{"quiet":true},"sessions":[{"name":"test","reset":true,"remote_dir":"/sync_test","local_dir":"contents","host":"my_remote_host","tcp_port":33001,"user":"my_remote_user","private_key_paths":["my_local_user_key"]}]}'
|
4664
4963
|
```
|
4665
4964
|
|
4666
|
-
## Plugin: Preview
|
4965
|
+
## <a id="preview"></a>Plugin: `preview`: Preview generator for AoC
|
4667
4966
|
|
4668
4967
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
4669
4968
|
It uses the **node API** of Aspera HSTS and requires use of Access Keys and it's **storage root**.
|
4670
4969
|
Several parameters can be used to tune several aspects:
|
4671
4970
|
|
4672
|
-
|
4673
|
-
|
4674
|
-
|
4971
|
+
- Methods for detection of new files needing generation
|
4972
|
+
- Methods for generation of video preview
|
4973
|
+
- Parameters for video handling
|
4675
4974
|
|
4676
4975
|
### Aspera Server configuration
|
4677
4976
|
|
@@ -4713,10 +5012,10 @@ Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* whi
|
|
4713
5012
|
|
4714
5013
|
The tool requires the following external tools available in the `PATH`:
|
4715
5014
|
|
4716
|
-
|
4717
|
-
|
4718
|
-
|
4719
|
-
|
5015
|
+
- ImageMagick : `convert` `composite`
|
5016
|
+
- OptiPNG : `optipng`
|
5017
|
+
- FFmpeg : `ffmpeg` `ffprobe`
|
5018
|
+
- Libreoffice : `libreoffice`
|
4720
5019
|
|
4721
5020
|
Here shown on Redhat/CentOS.
|
4722
5021
|
|
@@ -4752,13 +5051,13 @@ If you don't want to have preview for office documents or if it is too complex y
|
|
4752
5051
|
|
4753
5052
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
4754
5053
|
|
4755
|
-
|
5054
|
+
- CentOS 8
|
4756
5055
|
|
4757
5056
|
```bash
|
4758
5057
|
dnf install unoconv
|
4759
5058
|
```
|
4760
5059
|
|
4761
|
-
|
5060
|
+
- Amazon Linux
|
4762
5061
|
|
4763
5062
|
```bash
|
4764
5063
|
amazon-linux-extras enable libreoffice
|
@@ -4775,7 +5074,7 @@ The preview generator is run as a user, preferably a regular user (not root). Wh
|
|
4775
5074
|
|
4776
5075
|
Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). The configuration file must be created with the same user used to run so that it is properly used on runtime.
|
4777
5076
|
|
4778
|
-
|
5077
|
+
The `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
4779
5078
|
|
4780
5079
|
```bash
|
4781
5080
|
su -s /bin/bash - xfer
|
@@ -4798,11 +5097,11 @@ This shall list the contents of the storage root of the access key.
|
|
4798
5097
|
|
4799
5098
|
### Execution
|
4800
5099
|
|
4801
|
-
The tool intentionally supports only a
|
4802
|
-
It needs to be run on a regular basis to create or update preview files.
|
4803
|
-
|
5100
|
+
The tool intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
5101
|
+
It needs to be run on a regular basis to create or update preview files.
|
5102
|
+
For that use your best reliable scheduler, see [Scheduling](#scheduling).
|
4804
5103
|
|
4805
|
-
Typically, for
|
5104
|
+
Typically, for **Access key** access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
|
4806
5105
|
|
4807
5106
|
Lets do a one shot test, using the configuration previously created:
|
4808
5107
|
|
@@ -4818,12 +5117,12 @@ On subsequent run it reads this file and check that previews are generated for t
|
|
4818
5117
|
|
4819
5118
|
### Configuration for Execution in scheduler
|
4820
5119
|
|
4821
|
-
Here is an example of configuration for use with cron on Linux.
|
4822
|
-
Adapt the scripts to your own
|
5120
|
+
Here is an example of configuration for use with `cron` on Linux.
|
5121
|
+
Adapt the scripts to your own needs.
|
4823
5122
|
|
4824
5123
|
We assume here that a configuration preset was created as shown previously.
|
4825
5124
|
|
4826
|
-
Lets first setup a script that will be used in the scheduler and
|
5125
|
+
Lets first setup a script that will be used in the scheduler and set up the environment.
|
4827
5126
|
|
4828
5127
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
4829
5128
|
|
@@ -4845,25 +5144,25 @@ crontab<<EOF
|
|
4845
5144
|
EOF
|
4846
5145
|
```
|
4847
5146
|
|
4848
|
-
Note
|
5147
|
+
> **Note:** The logging options are kept here in the cronfile instead of conf file to allow execution on command line with output on command line.
|
4849
5148
|
|
4850
5149
|
### Candidate detection for creation or update (or deletion)
|
4851
5150
|
|
4852
5151
|
The tool generates preview files using those commands:
|
4853
5152
|
|
4854
|
-
|
4855
|
-
|
4856
|
-
|
4857
|
-
|
5153
|
+
- `trevents` : only recently uploaded files will be tested (transfer events)
|
5154
|
+
- `events` : only recently uploaded files will be tested (file events: not working)
|
5155
|
+
- `scan` : recursively scan all files under the access key's "storage root"
|
5156
|
+
- `test` : test using a local file
|
4858
5157
|
|
4859
5158
|
Once candidate are selected, once candidates are selected,
|
4860
5159
|
a preview is always generated if it does not exist already,
|
4861
5160
|
else if a preview already exist, it will be generated
|
4862
5161
|
using one of three values for the `overwrite` option:
|
4863
5162
|
|
4864
|
-
|
4865
|
-
|
4866
|
-
|
5163
|
+
- `always` : preview is always generated, even if it already exists and is newer than original
|
5164
|
+
- `never` : preview is generated only if it does not exist already
|
5165
|
+
- `mtime` : preview is generated only if the original file is newer than the existing
|
4867
5166
|
|
4868
5167
|
Deletion of preview for deleted source files: not implemented yet (TODO).
|
4869
5168
|
|
@@ -4887,8 +5186,8 @@ For instance to filter out files beginning with `._` do:
|
|
4887
5186
|
|
4888
5187
|
Two types of preview can be generated:
|
4889
5188
|
|
4890
|
-
|
4891
|
-
|
5189
|
+
- png: thumbnail
|
5190
|
+
- mp4: video preview (only for video)
|
4892
5191
|
|
4893
5192
|
Use option `skip_format` to skip generation of a format.
|
4894
5193
|
|
@@ -4896,11 +5195,11 @@ Use option `skip_format` to skip generation of a format.
|
|
4896
5195
|
|
4897
5196
|
The preview generator supports rendering of those file categories:
|
4898
5197
|
|
4899
|
-
|
4900
|
-
|
4901
|
-
|
4902
|
-
|
4903
|
-
|
5198
|
+
- image
|
5199
|
+
- pdf
|
5200
|
+
- plaintext
|
5201
|
+
- office
|
5202
|
+
- video
|
4904
5203
|
|
4905
5204
|
To avoid generation for some categories, specify a list using option `skip_types`.
|
4906
5205
|
|
@@ -4923,31 +5222,31 @@ In this case the `preview` command will first analyze the file content using mim
|
|
4923
5222
|
|
4924
5223
|
If the `mimemagic` gem complains about missing mime info file:
|
4925
5224
|
|
4926
|
-
|
5225
|
+
- any OS:
|
4927
5226
|
|
4928
|
-
|
4929
|
-
|
4930
|
-
|
5227
|
+
- Examine the error message
|
5228
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
5229
|
+
- move and rename this file to one of the locations expected by mimemagic as specified in the error message
|
4931
5230
|
|
4932
|
-
|
5231
|
+
- Windows:
|
4933
5232
|
|
4934
|
-
|
4935
|
-
|
4936
|
-
|
5233
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
5234
|
+
- Place this file in the root of Ruby (or elsewhere): `C:\RubyVV-x64\freedesktop.org.xml.in`
|
5235
|
+
- Set a global variable using `SystemPropertiesAdvanced.exe` or using `cmd` (replace `VV` with version) to the exact path of this file:
|
4937
5236
|
|
4938
5237
|
```cmd
|
4939
5238
|
SETX FREEDESKTOP_MIME_TYPES_PATH C:\RubyVV-x64\freedesktop.org.xml.in
|
4940
5239
|
```
|
4941
5240
|
|
4942
|
-
|
5241
|
+
- Close the `cmd` and restart a new one if needed to get refreshed env vars
|
4943
5242
|
|
4944
|
-
|
5243
|
+
- Linux:
|
4945
5244
|
|
4946
5245
|
```bash
|
4947
5246
|
yum install shared-mime-info
|
4948
5247
|
```
|
4949
5248
|
|
4950
|
-
|
5249
|
+
- macOS:
|
4951
5250
|
|
4952
5251
|
```bash
|
4953
5252
|
brew install shared-mime-info
|
@@ -4986,17 +5285,17 @@ preview trevents --once-only=yes --skip-types=office --log-level=info
|
|
4986
5285
|
Aspera CLI can send email, for that setup SMTP configuration. This is done with option `smtp`.
|
4987
5286
|
|
4988
5287
|
The `smtp` option is a hash table (extended value) with the following fields:
|
4989
|
-
|
4990
|
-
|
4991
|
-
|
4992
|
-
|
4993
|
-
|
4994
|
-
<
|
4995
|
-
|
4996
|
-
|
4997
|
-
|
4998
|
-
|
4999
|
-
|
5288
|
+
|
5289
|
+
| field | default | example | description |
|
5290
|
+
|--------------|---------------------|----------------------------|----------------------------------|
|
5291
|
+
| `server` | - | smtp.gmail.com | SMTP server address |
|
5292
|
+
| `tls` | true | false | use of TLS |
|
5293
|
+
| `port` | TLS: 587<br/>25 | 587 | port for service |
|
5294
|
+
| `domain` | domain of server | gmail.com | email domain of user |
|
5295
|
+
| `username` | - | john@example.com | user to authenticate on SMTP server, leave empty for open auth. |
|
5296
|
+
| `password` | - | my_password_here | password for above username |
|
5297
|
+
| `from_email` | username if defined | johnny@example.com | address used if receiver replies |
|
5298
|
+
| `from_name` | same as email | John Wayne | display name of sender |
|
5000
5299
|
|
5001
5300
|
### Example of configuration
|
5002
5301
|
|
@@ -5033,9 +5332,9 @@ The template is the full SMTP message, including headers.
|
|
5033
5332
|
|
5034
5333
|
The following variables are defined by default:
|
5035
5334
|
|
5036
|
-
|
5037
|
-
|
5038
|
-
|
5335
|
+
- `from_name`
|
5336
|
+
- `from_email`
|
5337
|
+
- `to`
|
5039
5338
|
|
5040
5339
|
Other variables are defined depending on context.
|
5041
5340
|
|
@@ -5058,10 +5357,10 @@ A default e-mail template is used, but it can be overridden with option `notif_t
|
|
5058
5357
|
|
5059
5358
|
The environment provided contains the following additional variables:
|
5060
5359
|
|
5061
|
-
|
5062
|
-
|
5063
|
-
|
5064
|
-
|
5360
|
+
- subject
|
5361
|
+
- body
|
5362
|
+
- global_transfer_status
|
5363
|
+
- ts
|
5065
5364
|
|
5066
5365
|
Example of template:
|
5067
5366
|
|
@@ -5079,9 +5378,9 @@ This gem comes with a second executable tool providing a simplified standardized
|
|
5079
5378
|
|
5080
5379
|
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [*transfer-spec*](#transferspec) is:
|
5081
5380
|
|
5082
|
-
|
5083
|
-
|
5084
|
-
|
5381
|
+
- common to Aspera Node API (HTTP POST /ops/transfer)
|
5382
|
+
- common to Aspera Connect API (browser javascript startTransfer)
|
5383
|
+
- easy to generate by using any third party language specific JSON library
|
5085
5384
|
|
5086
5385
|
Hopefully, IBM integrates this diectly in `ascp`, and this tool is made redundant.
|
5087
5386
|
|
@@ -5091,26 +5390,26 @@ The tool expect one single argument: a [*transfer-spec*](#transferspec).
|
|
5091
5390
|
|
5092
5391
|
If no argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [*transfer-spec*](#transferspec) on stdin.
|
5093
5392
|
|
5094
|
-
Note
|
5393
|
+
> **Note:** If JSON is the format, specify `@json:` to tell `ascli` to decode the hash using JSON syntax.
|
5095
5394
|
|
5096
5395
|
During execution, it generates all low level events, one per line, in JSON format on stdout.
|
5097
5396
|
|
5098
|
-
|
5397
|
+
There are special "extended" [*transfer-spec*](#transferspec) parameters supported by `asession`:
|
5099
5398
|
|
5100
|
-
|
5101
|
-
|
5399
|
+
- `EX_loglevel` to change log level of the tool
|
5400
|
+
- `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
5102
5401
|
|
5103
|
-
Note
|
5402
|
+
> **Note:** In addition, many "EX_" [*transfer-spec*](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [*transfer-spec*](#transferspec).
|
5104
5403
|
|
5105
5404
|
### Comparison of interfaces
|
5106
5405
|
|
5107
|
-
|
5108
|
-
|
5109
|
-
|
5110
|
-
|
5111
|
-
|
5112
|
-
|
5113
|
-
|
5406
|
+
| feature/tool | asession | `ascp` | FaspManager | Transfer SDK |
|
5407
|
+
|--------------|----------|--------|-------------|--------------|
|
5408
|
+
| language integration | any | any | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | many |
|
5409
|
+
| required additional components to `ascp` | Ruby<br/>Aspera | - | library<br/>(headers) | daemon |
|
5410
|
+
| startup | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) | command line arguments | API | daemon |
|
5411
|
+
| events | JSON on stdout | none by default<br/>or need to open management port<br/>and proprietary text syntax | callback | callback |
|
5412
|
+
| platforms | any with ruby and `ascp` | any with `ascp` (and SDK if compiled) | any with `ascp` | any with `ascp` and transfer daemon |
|
5114
5413
|
|
5115
5414
|
### Simple session
|
5116
5415
|
|
@@ -5122,13 +5421,13 @@ Create a file `session.json` with:
|
|
5122
5421
|
|
5123
5422
|
Then start the session:
|
5124
5423
|
|
5125
|
-
```
|
5424
|
+
```bash
|
5126
5425
|
asession < session.json
|
5127
5426
|
```
|
5128
5427
|
|
5129
5428
|
### Asynchronous commands and Persistent session
|
5130
5429
|
|
5131
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
5430
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
5132
5431
|
|
5133
5432
|
This is particularly useful for a persistent session ( with the [*transfer-spec*](#transferspec) parameter: `"keepalive":true` )
|
5134
5433
|
|
@@ -5176,13 +5475,14 @@ EXAMPLES
|
|
5176
5475
|
|
5177
5476
|
### Requirements
|
5178
5477
|
|
5179
|
-
`ascli` maybe used as a simple hot folder engine.
|
5478
|
+
`ascli` maybe used as a simple hot folder engine.
|
5479
|
+
A hot folder being defined as a tool that:
|
5180
5480
|
|
5181
|
-
|
5182
|
-
|
5183
|
-
|
5184
|
-
|
5185
|
-
|
5481
|
+
- locally (or remotely) detects new files in a top folder
|
5482
|
+
- send detected files to a remote (respectively, local) repository
|
5483
|
+
- only sends new files, do not re-send already sent files
|
5484
|
+
- optionally: sends only files that are not still "growing"
|
5485
|
+
- optionally: after transfer of files, deletes or moves to an archive
|
5186
5486
|
|
5187
5487
|
In addition: the detection should be made "continuously" or on specific time/date.
|
5188
5488
|
|
@@ -5190,42 +5490,67 @@ In addition: the detection should be made "continuously" or on specific time/dat
|
|
5190
5490
|
|
5191
5491
|
The general idea is to rely on :
|
5192
5492
|
|
5193
|
-
|
5194
|
-
|
5195
|
-
|
5493
|
+
- existing `ascp` features for detection and transfer
|
5494
|
+
- take advantage of `ascli` configuration capabilities and server side knowledge
|
5495
|
+
- the OS scheduler for reliability and continuous operation
|
5196
5496
|
|
5197
|
-
#### ascp features
|
5497
|
+
#### `ascp` features
|
5198
5498
|
|
5199
|
-
Interesting ascp features are found in its arguments: (see ascp manual):
|
5499
|
+
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
5200
5500
|
|
5201
|
-
|
5202
|
-
|
5203
|
-
|
5204
|
-
|
5501
|
+
- `ascp` already takes care of sending only "new" files: option `-k 1,2,3` (`resume_policy`)
|
5502
|
+
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
5503
|
+
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
5504
|
+
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
5205
5505
|
|
5206
5506
|
Note that:
|
5207
5507
|
|
5208
|
-
|
5209
|
-
|
5210
|
-
|
5508
|
+
- `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `--ts` parameter.
|
5509
|
+
- most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters
|
5510
|
+
- native `ascp` arguments can be provided with the [*transfer-spec*](#transferspec) parameter: `EX_ascp_args` (array), only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node)
|
5211
5511
|
|
5212
5512
|
#### server side and configuration
|
5213
5513
|
|
5214
|
-
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
5514
|
+
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
5215
5515
|
|
5216
|
-
Note
|
5516
|
+
> **Note:** file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
5517
|
+
>
|
5518
|
+
> **Note:** parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
5217
5519
|
|
5218
5520
|
#### Scheduling
|
5219
5521
|
|
5220
|
-
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
5522
|
+
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
5523
|
+
Refer to section [Scheduling](#scheduling). (on use of option `lock_port`)
|
5221
5524
|
|
5222
|
-
### Example: upload folder
|
5525
|
+
### Example: upload hot folder
|
5223
5526
|
|
5224
|
-
```
|
5225
|
-
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"
|
5527
|
+
```bash
|
5528
|
+
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
5529
|
+
```
|
5530
|
+
|
5531
|
+
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
5532
|
+
Source files are deleted after transfer.
|
5533
|
+
Growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period).
|
5534
|
+
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
5535
|
+
|
5536
|
+
### Example: unidirectional synchronization (upload) to server
|
5537
|
+
|
5538
|
+
```bash
|
5539
|
+
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
|
5540
|
+
```
|
5541
|
+
|
5542
|
+
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node:
|
5543
|
+
|
5544
|
+
### Example: unidirectional synchronization (download) from Aspera on Cloud Files
|
5545
|
+
|
5546
|
+
```bash
|
5547
|
+
ascli aoc files download . --to-folder=. --lock-port=12345 --progress=none --display=data \
|
5548
|
+
--ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
|
5226
5549
|
```
|
5227
5550
|
|
5228
|
-
|
5551
|
+
> **Note:** option `delete_before_transfer` will delete files locally, if they are not present on remote side.
|
5552
|
+
>
|
5553
|
+
> **Note:** options `progress` and `display` limit output for headless operation (e.g. cron job)
|
5229
5554
|
|
5230
5555
|
## Health check and Nagios
|
5231
5556
|
|
@@ -5245,9 +5570,9 @@ ascli console health
|
|
5245
5570
|
|
5246
5571
|
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
5247
5572
|
|
5248
|
-
|
5249
|
-
|
5250
|
-
|
5573
|
+
- issuing a transfer to the server
|
5574
|
+
- checking web app status with `asctl all:status`
|
5575
|
+
- checking daemons process status
|
5251
5576
|
|
5252
5577
|
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
5253
5578
|
|
@@ -5271,9 +5596,9 @@ OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:runnin
|
|
5271
5596
|
|
5272
5597
|
Main components:
|
5273
5598
|
|
5274
|
-
|
5275
|
-
|
5276
|
-
|
5599
|
+
- `Aspera` generic classes for REST and OAuth
|
5600
|
+
- `Aspera::Fasp`: starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.
|
5601
|
+
- `Aspera::Cli`: `ascli`.
|
5277
5602
|
|
5278
5603
|
A working example can be found in the gem, example:
|
5279
5604
|
|
@@ -5302,23 +5627,23 @@ aoc.read('self')
|
|
5302
5627
|
|
5303
5628
|
## Changes (Release notes)
|
5304
5629
|
|
5305
|
-
See
|
5630
|
+
See [CHANGELOG.md](CHANGELOG.md)
|
5306
5631
|
|
5307
5632
|
## History
|
5308
5633
|
|
5309
|
-
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
5634
|
+
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (`ascp`), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
5310
5635
|
|
5311
5636
|
There were a few pitfalls:
|
5312
5637
|
|
5313
|
-
|
5314
|
-
|
5638
|
+
- The tool was written in the aging `perl` language while most Aspera application products (but the Transfer Server) are written in `ruby`.
|
5639
|
+
- The tool was only for transfers, but not able to call other products APIs
|
5315
5640
|
|
5316
5641
|
So, it evolved into `ascli`:
|
5317
5642
|
|
5318
|
-
|
5319
|
-
|
5320
|
-
|
5321
|
-
|
5643
|
+
- portable: works on platforms supporting `ruby` (and `ascp`)
|
5644
|
+
- easy to install with the `gem` utility
|
5645
|
+
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from `ascp` command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
5646
|
+
- `ruby` is consistent with other Aspera products
|
5322
5647
|
|
5323
5648
|
## Common problems
|
5324
5649
|
|