aspera-cli 4.10.0 → 4.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +20 -0
- data/CHANGELOG.md +509 -0
- data/CONTRIBUTING.md +118 -0
- data/README.md +621 -378
- data/bin/ascli +4 -4
- data/bin/asession +11 -11
- data/docs/test_env.conf +28 -19
- data/examples/aoc.rb +4 -4
- data/examples/dascli +11 -9
- data/examples/faspex4.rb +8 -8
- data/examples/node.rb +11 -11
- data/examples/server.rb +9 -9
- data/lib/aspera/aoc.rb +273 -266
- data/lib/aspera/ascmd.rb +56 -54
- data/lib/aspera/ats_api.rb +4 -4
- data/lib/aspera/cli/basic_auth_plugin.rb +15 -12
- data/lib/aspera/cli/extended_value.rb +5 -5
- data/lib/aspera/cli/formater.rb +64 -64
- data/lib/aspera/cli/listener/line_dump.rb +1 -1
- data/lib/aspera/cli/listener/logger.rb +1 -1
- data/lib/aspera/cli/listener/progress.rb +5 -6
- data/lib/aspera/cli/listener/progress_multi.rb +14 -19
- data/lib/aspera/cli/main.rb +66 -67
- data/lib/aspera/cli/manager.rb +110 -110
- data/lib/aspera/cli/plugin.rb +54 -37
- data/lib/aspera/cli/plugins/alee.rb +4 -4
- data/lib/aspera/cli/plugins/aoc.rb +308 -669
- data/lib/aspera/cli/plugins/ats.rb +44 -46
- data/lib/aspera/cli/plugins/bss.rb +10 -10
- data/lib/aspera/cli/plugins/config.rb +447 -344
- data/lib/aspera/cli/plugins/console.rb +12 -12
- data/lib/aspera/cli/plugins/cos.rb +18 -20
- data/lib/aspera/cli/plugins/faspex.rb +110 -112
- data/lib/aspera/cli/plugins/faspex5.rb +67 -46
- data/lib/aspera/cli/plugins/node.rb +364 -288
- data/lib/aspera/cli/plugins/orchestrator.rb +46 -46
- data/lib/aspera/cli/plugins/preview.rb +122 -114
- data/lib/aspera/cli/plugins/server.rb +137 -83
- data/lib/aspera/cli/plugins/shares.rb +30 -29
- data/lib/aspera/cli/plugins/sync.rb +13 -33
- data/lib/aspera/cli/transfer_agent.rb +57 -57
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/colors.rb +3 -3
- data/lib/aspera/command_line_builder.rb +27 -27
- data/lib/aspera/cos_node.rb +22 -20
- data/lib/aspera/data_repository.rb +1 -1
- data/lib/aspera/environment.rb +30 -28
- data/lib/aspera/fasp/agent_base.rb +15 -15
- data/lib/aspera/fasp/agent_connect.rb +23 -21
- data/lib/aspera/fasp/agent_direct.rb +65 -67
- data/lib/aspera/fasp/agent_httpgw.rb +72 -68
- data/lib/aspera/fasp/agent_node.rb +23 -21
- data/lib/aspera/fasp/agent_trsdk.rb +20 -20
- data/lib/aspera/fasp/error.rb +3 -2
- data/lib/aspera/fasp/error_info.rb +11 -8
- data/lib/aspera/fasp/installation.rb +78 -78
- data/lib/aspera/fasp/listener.rb +1 -1
- data/lib/aspera/fasp/parameters.rb +75 -72
- data/lib/aspera/fasp/parameters.yaml +2 -2
- data/lib/aspera/fasp/resume_policy.rb +8 -8
- data/lib/aspera/fasp/transfer_spec.rb +35 -2
- data/lib/aspera/fasp/uri.rb +7 -7
- data/lib/aspera/faspex_gw.rb +7 -5
- data/lib/aspera/hash_ext.rb +3 -3
- data/lib/aspera/id_generator.rb +5 -5
- data/lib/aspera/keychain/encrypted_hash.rb +23 -28
- data/lib/aspera/keychain/macos_security.rb +21 -20
- data/lib/aspera/log.rb +7 -7
- data/lib/aspera/nagios.rb +19 -18
- data/lib/aspera/node.rb +209 -35
- data/lib/aspera/oauth.rb +37 -36
- data/lib/aspera/open_application.rb +19 -11
- data/lib/aspera/persistency_action_once.rb +4 -4
- data/lib/aspera/persistency_folder.rb +13 -13
- data/lib/aspera/preview/file_types.rb +8 -8
- data/lib/aspera/preview/generator.rb +67 -67
- data/lib/aspera/preview/utils.rb +27 -27
- data/lib/aspera/proxy_auto_config.js +41 -41
- data/lib/aspera/proxy_auto_config.rb +16 -16
- data/lib/aspera/rest.rb +56 -60
- data/lib/aspera/rest_call_error.rb +2 -1
- data/lib/aspera/rest_error_analyzer.rb +18 -17
- data/lib/aspera/rest_errors_aspera.rb +16 -16
- data/lib/aspera/secret_hider.rb +15 -13
- data/lib/aspera/ssh.rb +11 -10
- data/lib/aspera/sync.rb +158 -44
- data/lib/aspera/temp_file_manager.rb +2 -2
- data/lib/aspera/uri_reader.rb +4 -4
- data/lib/aspera/web_auth.rb +14 -13
- data.tar.gz.sig +0 -0
- metadata +8 -5
- metadata.gz.sig +0 -0
data/README.md
CHANGED
|
@@ -1,10 +1,13 @@
|
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
|
2
|
+
<!-- markdownlint-disable MD033 MD003 MD053 -->
|
|
2
3
|
|
|
3
4
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
|
4
5
|
|
|
5
|
-
|
|
6
|
+
##
|
|
6
7
|
|
|
7
|
-
|
|
8
|
+
Version : 4.11.0
|
|
9
|
+
|
|
10
|
+
Laurent/2016-2023
|
|
8
11
|
|
|
9
12
|
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
|
10
13
|
|
|
@@ -14,9 +17,11 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
|
14
17
|
|
|
15
18
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
|
16
19
|
|
|
17
|
-
Minimum required Ruby version: >= 2.4.
|
|
20
|
+
Minimum required Ruby version: >= 2.4.
|
|
21
|
+
|
|
22
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
|
18
23
|
|
|
19
|
-
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
|
24
|
+
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs&sort=title_asc)
|
|
20
25
|
[Link 2](https://developer.ibm.com/apis/catalog/?search=aspera)
|
|
21
26
|
|
|
22
27
|
Release notes: see [CHANGELOG.md](CHANGELOG.md)
|
|
@@ -46,7 +51,7 @@ So it is designed for:
|
|
|
46
51
|
- A configuration file (config.yaml)
|
|
47
52
|
- Advanced command line options
|
|
48
53
|
- cURL (for REST calls)
|
|
49
|
-
- Aspera transfer (ascp)
|
|
54
|
+
- Aspera transfer (`ascp`)
|
|
50
55
|
|
|
51
56
|
If the need is to perform operations programmatically in languages such as: C, Go, Python, nodejs, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
|
52
57
|
|
|
@@ -63,7 +68,6 @@ In examples, command line operations are shown using a shell such: `bash` or `zs
|
|
|
63
68
|
|
|
64
69
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
|
65
70
|
|
|
66
|
-
|
|
67
71
|
## Quick Start
|
|
68
72
|
|
|
69
73
|
This section guides you from installation, first use and advanced use.
|
|
@@ -77,7 +81,7 @@ ascli --version
|
|
|
77
81
|
```
|
|
78
82
|
|
|
79
83
|
```bash
|
|
80
|
-
4.
|
|
84
|
+
4.11.0
|
|
81
85
|
```
|
|
82
86
|
|
|
83
87
|
### First use
|
|
@@ -127,7 +131,7 @@ ascli config preset set default server myserver
|
|
|
127
131
|
```
|
|
128
132
|
|
|
129
133
|
```output
|
|
130
|
-
updated: default→server to myserver
|
|
134
|
+
updated: default → server to myserver
|
|
131
135
|
```
|
|
132
136
|
|
|
133
137
|
```bash
|
|
@@ -158,7 +162,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
|
158
162
|
```
|
|
159
163
|
|
|
160
164
|
```output
|
|
161
|
-
Time: 00:00:02
|
|
165
|
+
Time: 00:00:02 =========================================================== 100% 100 Mbps Time: 00:00:00
|
|
162
166
|
complete
|
|
163
167
|
```
|
|
164
168
|
|
|
@@ -174,9 +178,13 @@ It is possible to install *either* directly on the host operating system (Linux,
|
|
|
174
178
|
|
|
175
179
|
The direct installation is recommended and consists in installing:
|
|
176
180
|
|
|
177
|
-
- [Ruby](#ruby)
|
|
181
|
+
- [Ruby](#ruby)
|
|
178
182
|
- [aspera-cli](#the_gem)
|
|
179
|
-
- [Aspera SDK (ascp)](#fasp_prot)
|
|
183
|
+
- [Aspera SDK (`ascp`)](#fasp_prot)
|
|
184
|
+
|
|
185
|
+
Ruby version: >= 2.4.
|
|
186
|
+
|
|
187
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
|
180
188
|
|
|
181
189
|
The following sections provide information on the various installation methods.
|
|
182
190
|
|
|
@@ -184,33 +192,114 @@ An internet connection is required for the installation. If you don't have inter
|
|
|
184
192
|
|
|
185
193
|
### Docker container
|
|
186
194
|
|
|
187
|
-
|
|
195
|
+
The image is: [martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli).
|
|
196
|
+
The container contains: Ruby, `ascli` and the Aspera Transfer SDK.
|
|
197
|
+
To use the container, ensure that you have `docker` (or `podman`) installed.
|
|
188
198
|
|
|
189
|
-
|
|
190
|
-
|
|
199
|
+
```bash
|
|
200
|
+
docker --version
|
|
201
|
+
```
|
|
191
202
|
|
|
192
|
-
|
|
203
|
+
**Wanna start quickly ?** With an interactive shell ? Execute this:
|
|
193
204
|
|
|
194
205
|
```bash
|
|
195
|
-
docker --
|
|
206
|
+
docker run --tty --interactive --entrypoint bash martinlaurent/ascli:latest
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Then, execute individual `ascli` commands such as:
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
ascli conf init
|
|
213
|
+
ascli conf preset overview
|
|
214
|
+
ascli conf ascp info
|
|
215
|
+
ascli server ls /
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
That is simple, but there are limitations:
|
|
219
|
+
|
|
220
|
+
- Everything happens in the container
|
|
221
|
+
- Any generated file in the container will be lost on container (shell) exit. Including configuration files and downloaded files.
|
|
222
|
+
- No possibility to upload files located on the host system
|
|
223
|
+
|
|
224
|
+
The container image is built from this [Dockerfile](Dockerfile): the entry point is `ascli` and the default command is `help`.
|
|
225
|
+
|
|
226
|
+
The container can also be execute for individual commands like this: (add `ascli` commands and options at the end of the command line, e.g. `-v` to display the version)
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
docker run --rm --tty --interactive martinlaurent/ascli:latest
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
For more convenience, you may define a shell alias:
|
|
233
|
+
|
|
234
|
+
```bash
|
|
235
|
+
alias ascli='docker run --rm --tty --interactive martinlaurent/ascli:latest'
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
Then, you can execute the container like a local command:
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
ascli -v
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
```text
|
|
245
|
+
4.11.0
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
In order to keep persistency of configuration on the host,
|
|
249
|
+
you should mount your user's config folder to the container.
|
|
250
|
+
To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`).
|
|
251
|
+
Add options:
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
> **Note:** if you are using a `podman machine`, e.g. on Macos , make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
|
258
|
+
|
|
259
|
+
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
|
260
|
+
you can change the entry point, add option:
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
--entrypoint bash
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
You may also probably want that files downloaded in the container are in fact placed on the host.
|
|
267
|
+
In this case you need also to mount the shared transfer folder:
|
|
268
|
+
|
|
269
|
+
```bash
|
|
270
|
+
--volume $HOME/xferdir:/xferfiles
|
|
196
271
|
```
|
|
197
272
|
|
|
198
|
-
|
|
273
|
+
> **Note:** ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
|
199
274
|
|
|
200
|
-
|
|
275
|
+
And if you want all the above, simply use all the options:
|
|
201
276
|
|
|
202
|
-
|
|
277
|
+
```bash
|
|
278
|
+
alias asclish="docker run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash martinlaurent/ascli:latest"
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
export xferdir=$HOME/xferdir
|
|
283
|
+
mkdir -p $xferdir
|
|
284
|
+
chmod -R 777 $xferdir
|
|
285
|
+
mkdir -p $HOME/.aspera/ascli
|
|
286
|
+
asclish
|
|
287
|
+
```
|
|
203
288
|
|
|
204
|
-
|
|
289
|
+
A convenience sample script is also provided: download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
|
205
290
|
|
|
206
|
-
|
|
207
|
-
|-------------|------------------------------------|--------------------------|--------------------------|
|
|
208
|
-
| ASCLI__HOME | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascliconfig` |
|
|
209
|
-
| docker_args | additional options to `docker` | <empty> | `--volume /Users:/Users` |
|
|
210
|
-
| image | container image name | martinlaurent/ascli | |
|
|
211
|
-
| version | container image version | latest | `4.8.0.pre` |
|
|
291
|
+
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
|
212
292
|
|
|
213
|
-
|
|
293
|
+
Some environment variables can be set for this script to adapt its behaviour:
|
|
294
|
+
|
|
295
|
+
| env var | description | default | example |
|
|
296
|
+
|--------------|------------------------------------|--------------------------|--------------------------|
|
|
297
|
+
| ASCLI_HOME | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascliconfig` |
|
|
298
|
+
| docker_args | additional options to `docker` | <empty> | `--volume /Users:/Users` |
|
|
299
|
+
| image | container image name | martinlaurent/ascli | |
|
|
300
|
+
| version | container image version | latest | `4.8.0.pre` |
|
|
301
|
+
|
|
302
|
+
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
|
214
303
|
(value expected in the container).
|
|
215
304
|
This allows having persistent configuration on the host.
|
|
216
305
|
|
|
@@ -221,21 +310,18 @@ Example of use:
|
|
|
221
310
|
```bash
|
|
222
311
|
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
|
223
312
|
chmod a+x ascli
|
|
224
|
-
export ASCLI__HOME=$HOME/.ascliconf
|
|
225
|
-
mkdir -p $ASCLI__HOME
|
|
226
|
-
chmod -R 777 $ASCLI__HOME
|
|
227
313
|
export xferdir=$HOME/xferdir
|
|
228
314
|
mkdir -p $xferdir
|
|
229
315
|
chmod -R 777 $xferdir
|
|
230
|
-
export docker_args="--volume $xferdir:/
|
|
316
|
+
export docker_args="--volume $xferdir:/xferfiles"
|
|
231
317
|
|
|
232
318
|
./ascli conf init
|
|
233
319
|
|
|
234
|
-
|
|
235
|
-
./ascli server upload /
|
|
320
|
+
echo 'Local file to transfer' > $xferdir/samplefile.txt
|
|
321
|
+
./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload
|
|
236
322
|
```
|
|
237
323
|
|
|
238
|
-
> The local file (`samplefile`) is specified relative to storage view from container (`/
|
|
324
|
+
> **Note:** The local file (`samplefile.txt`) is specified relative to storage view from container (`/xferfiles`) mapped to the host folder `$HOME/xferdir`
|
|
239
325
|
|
|
240
326
|
### <a id="ruby"></a>Ruby
|
|
241
327
|
|
|
@@ -243,7 +329,9 @@ Use this method to install on the native host.
|
|
|
243
329
|
|
|
244
330
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
|
245
331
|
|
|
246
|
-
Required Ruby version: >= 2.4.
|
|
332
|
+
Required Ruby version: >= 2.4.
|
|
333
|
+
|
|
334
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
|
247
335
|
|
|
248
336
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
|
249
337
|
|
|
@@ -251,7 +339,7 @@ Refer to the following sections for a proposed method for specific operating sys
|
|
|
251
339
|
|
|
252
340
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
|
253
341
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
|
254
|
-
If you have a simpler better way to install Ruby : use it !
|
|
342
|
+
If you have a simpler better way to install Ruby : use it !
|
|
255
343
|
|
|
256
344
|
#### Generic: RVM: single user installation (not root)
|
|
257
345
|
|
|
@@ -318,7 +406,7 @@ Install Latest stable Ruby:
|
|
|
318
406
|
- Download the latest Ruby installer **with devkit**. (Msys2 is needed to install some native extensions, such as `grpc`)
|
|
319
407
|
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
|
320
408
|
- At the end of the installation procedure, the Msys2 installer is automatically executed, select option 3 (msys and mingw)
|
|
321
|
-
|
|
409
|
+
|
|
322
410
|
#### macOS: pre-installed or `brew`
|
|
323
411
|
|
|
324
412
|
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
|
@@ -337,15 +425,20 @@ brew install ruby
|
|
|
337
425
|
|
|
338
426
|
If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
|
|
339
427
|
|
|
340
|
-
Example:
|
|
428
|
+
Example: RHEL 8 and 9: basic installation
|
|
429
|
+
|
|
430
|
+
```bash
|
|
431
|
+
yum module install ruby:3.1
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
Example: RHEL 8, Centos 8 Stream: with extensions to compile native gems
|
|
341
435
|
|
|
342
436
|
```bash
|
|
343
437
|
yum install make automake gcc gcc-c++ kernel-devel
|
|
344
438
|
yum install redhat-rpm-config
|
|
345
439
|
dnf module reset ruby
|
|
346
|
-
dnf module enable ruby:3.
|
|
347
|
-
dnf module -y install ruby:3.
|
|
348
|
-
gem install aspera-cli
|
|
440
|
+
dnf module enable ruby:3.1
|
|
441
|
+
dnf module -y install ruby:3.1/common
|
|
349
442
|
```
|
|
350
443
|
|
|
351
444
|
Other examples:
|
|
@@ -398,7 +491,7 @@ If you already have a Java JVM on your system (`java`), it is possible to use `j
|
|
|
398
491
|
|
|
399
492
|
<https://www.jruby.org/download>
|
|
400
493
|
|
|
401
|
-
Note
|
|
494
|
+
> **Note:** Using jruby the startup time is longer than the native ruby, but the transfer speed is not impacted (executed by `ascp` binary).
|
|
402
495
|
|
|
403
496
|
### <a id="the_gem"></a>`aspera-cli` gem
|
|
404
497
|
|
|
@@ -427,7 +520,7 @@ ascli conf check_update
|
|
|
427
520
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
|
428
521
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
|
429
522
|
|
|
430
|
-
- ascp
|
|
523
|
+
- `ascp`
|
|
431
524
|
- aspera-license (in same folder, or ../etc)
|
|
432
525
|
|
|
433
526
|
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
|
@@ -467,7 +560,8 @@ Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, b
|
|
|
467
560
|
|
|
468
561
|
### <a id="offline_install"></a>Installation in air gapped environment
|
|
469
562
|
|
|
470
|
-
Note
|
|
563
|
+
> **Note:** no pre-packaged version is provided.
|
|
564
|
+
|
|
471
565
|
A method to build one is provided here:
|
|
472
566
|
|
|
473
567
|
The procedure:
|
|
@@ -528,7 +622,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
|
528
622
|
- Supports most Aspera server products (on-premise and SaaS)
|
|
529
623
|
- Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
|
530
624
|
- Supports Commands, Option values and Parameters shortcuts
|
|
531
|
-
- FASP [Transfer Agents](#agents) can be: local ascp
|
|
625
|
+
- FASP [Transfer Agents](#agents) can be: local `ascp`, or Connect Client, or any transfer node
|
|
532
626
|
- Transfer parameters can be altered by modification of [*transfer-spec*](#transferspec), this includes requiring multi-session
|
|
533
627
|
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
|
534
628
|
- Supports FaspStream creation (using Node API)
|
|
@@ -551,7 +645,7 @@ Not all `ascli` features are fully documented here, the user may explore command
|
|
|
551
645
|
|
|
552
646
|
If you want to use `ascp` directly as a command line, refer to IBM Aspera documentation of either [Desktop Client](https://www.ibm.com/docs/en/asdc), [Endpoint](https://www.ibm.com/docs/en/ahte) or [Transfer Server](https://www.ibm.com/docs/en/ahts) where [a section on `ascp` can be found](https://www.ibm.com/docs/en/ahts/4.4?topic=linux-ascp-transferring-from-command-line).
|
|
553
647
|
|
|
554
|
-
Using `ascli` with plugin `server` for command line gives advantages over ascp
|
|
648
|
+
Using `ascli` with plugin `server` for command line gives advantages over `ascp`:
|
|
555
649
|
|
|
556
650
|
- automatic resume on error
|
|
557
651
|
- configuration file
|
|
@@ -604,7 +698,8 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
|
604
698
|
```
|
|
605
699
|
|
|
606
700
|
`config echo` displays the value of the first argument using Ruby syntax: it surrounds a string with `"` and add `\` before special characters.
|
|
607
|
-
|
|
701
|
+
|
|
702
|
+
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
|
608
703
|
|
|
609
704
|
In the following examples (using a POSIX shell, such as `bash`), several sample commands are provided when equivalent.
|
|
610
705
|
For all example, most of special character handling is not specific to `ascli`: It depoends on the underlying syntax: shell , JSON, etc...
|
|
@@ -613,7 +708,7 @@ Depending on the case, a different `format` is used to display the actual value.
|
|
|
613
708
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
|
614
709
|
|
|
615
710
|
Double quotes are processed by the shell to create a single string argument.
|
|
616
|
-
For POSIX shells, single quotes can also be used in this case, or protext the special character ` ` (space) with a backslash.
|
|
711
|
+
For POSIX shells, single quotes can also be used in this case, or protext the special character ` ` (space) with a backslash. <!-- markdownlint-disable-line -->
|
|
617
712
|
|
|
618
713
|
```bash
|
|
619
714
|
ascli conf echo "Hello World" --format=text
|
|
@@ -628,8 +723,9 @@ Hello World
|
|
|
628
723
|
#### Using a shell variable, parsed by shell, in an extended value
|
|
629
724
|
|
|
630
725
|
To be evaluated by shell, the shell variable must not be in single quotes.
|
|
631
|
-
|
|
632
|
-
|
|
726
|
+
Even if the variable contains spaces it makes only one argument to `ascli` because word parsing is made before variable expansion by shell.
|
|
727
|
+
|
|
728
|
+
> **Note:** we use a simple variable here: the variable is not necessarily an environment variable.
|
|
633
729
|
|
|
634
730
|
```bash
|
|
635
731
|
MYVAR="Hello World"
|
|
@@ -787,7 +883,7 @@ ascli config echo -- --sample
|
|
|
787
883
|
"--sample"
|
|
788
884
|
```
|
|
789
885
|
|
|
790
|
-
Note
|
|
886
|
+
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
|
791
887
|
|
|
792
888
|
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
|
793
889
|
|
|
@@ -873,7 +969,7 @@ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sor
|
|
|
873
969
|
:...............................:..................................:...........:
|
|
874
970
|
```
|
|
875
971
|
|
|
876
|
-
Note
|
|
972
|
+
> **Note:** `select` filters selected elements from the result of API calls, while the `query` parameters gives filtering parameters to the API when listing elements.
|
|
877
973
|
|
|
878
974
|
#### Verbosity of output
|
|
879
975
|
|
|
@@ -923,7 +1019,7 @@ The following "readers" are supported (returns value in []):
|
|
|
923
1019
|
- @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
|
|
924
1020
|
- @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
|
925
1021
|
- @stdin: : [String] read from stdin (no value on right)
|
|
926
|
-
- @preset:NAME : [Hash] get whole option preset value by name. Subvalues can also be used using `.` as separator. e.g. foo.bar is conf[foo][bar]
|
|
1022
|
+
- @preset:NAME : [Hash] get whole option preset value by name. Subvalues can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]`
|
|
927
1023
|
|
|
928
1024
|
In addition it is possible to decode a value, using one or multiple decoders :
|
|
929
1025
|
|
|
@@ -985,7 +1081,7 @@ ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
|
|
985
1081
|
{"version"=>"0.9", "hello"=>true}
|
|
986
1082
|
```
|
|
987
1083
|
|
|
988
|
-
Note
|
|
1084
|
+
> **Note:** `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` are equivalent to: `@preset:config`
|
|
989
1085
|
|
|
990
1086
|
### <a id="native"></a>Structured Value
|
|
991
1087
|
|
|
@@ -1097,6 +1193,8 @@ If necessary, the configuration file can opened in a text editor with:
|
|
|
1097
1193
|
ascli config open
|
|
1098
1194
|
```
|
|
1099
1195
|
|
|
1196
|
+
> **Note:** this starts the editor specified by env var `EDITOR` if defined.
|
|
1197
|
+
|
|
1100
1198
|
Older format for commands are still supported:
|
|
1101
1199
|
|
|
1102
1200
|
```bash
|
|
@@ -1115,7 +1213,7 @@ This preset name is reserved and contains an array of key-value , where the key
|
|
|
1115
1213
|
|
|
1116
1214
|
When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
|
|
1117
1215
|
|
|
1118
|
-
Note
|
|
1216
|
+
> **Note:** Special plugin name: `config` can be associated with a preset that is loaded initially, typically used for default values.
|
|
1119
1217
|
|
|
1120
1218
|
Operations on this preset are done using regular `config` operations:
|
|
1121
1219
|
|
|
@@ -1131,9 +1229,9 @@ ascli config preset get default _plugin_name_
|
|
|
1131
1229
|
"_default_preset_for_plugin_"
|
|
1132
1230
|
```
|
|
1133
1231
|
|
|
1134
|
-
#### <a id="
|
|
1232
|
+
#### <a id="config"></a>Plugin: `config`: CLI Configuration
|
|
1135
1233
|
|
|
1136
|
-
Plugin `config`
|
|
1234
|
+
Plugin `config` is used to configure `ascli` and also contains global options.
|
|
1137
1235
|
|
|
1138
1236
|
When `ascli` starts, it looks for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
|
1139
1237
|
|
|
@@ -1347,7 +1445,7 @@ The vault is used with options `vault` and `vault_password`.
|
|
|
1347
1445
|
|
|
1348
1446
|
`vault_password` specifies the password for the vault.
|
|
1349
1447
|
Although it can be specified on command line, for security reason you can hide the value.
|
|
1350
|
-
For example it can be specified on command line like this:
|
|
1448
|
+
For example it can be securely specified on command line like this:
|
|
1351
1449
|
|
|
1352
1450
|
```bash
|
|
1353
1451
|
export ASCLI_VAULT_PASSWORD
|
|
@@ -1356,7 +1454,7 @@ read -s ASCLI_VAULT_PASSWORD
|
|
|
1356
1454
|
|
|
1357
1455
|
#### Vault: System keychain
|
|
1358
1456
|
|
|
1359
|
-
> **macOS only**
|
|
1457
|
+
> **Note:** **macOS only**
|
|
1360
1458
|
|
|
1361
1459
|
It is possible to manage secrets in macOS keychain (only read supported currently).
|
|
1362
1460
|
|
|
@@ -1372,7 +1470,7 @@ It is possible to store and use secrets encrypted in a file.
|
|
|
1372
1470
|
--vault=@json:'{"type":"file","name":"vault.bin"}'
|
|
1373
1471
|
```
|
|
1374
1472
|
|
|
1375
|
-
`name` is the file path, absolute or relative to the config folder `
|
|
1473
|
+
`name` is the file path, absolute or relative to the config folder `ASCLI_HOME`.
|
|
1376
1474
|
|
|
1377
1475
|
#### Vault: Operations
|
|
1378
1476
|
|
|
@@ -1395,6 +1493,24 @@ When a secret is needed by a sub command, the command can search for existing co
|
|
|
1395
1493
|
|
|
1396
1494
|
The lookup is done by comparing the service URL and username (or access key).
|
|
1397
1495
|
|
|
1496
|
+
#### Securing passwords and secrets
|
|
1497
|
+
|
|
1498
|
+
A passwords can be saved in clear in a [option preset](#lprt) together with other account information (URL, username, etc...).
|
|
1499
|
+
Example:
|
|
1500
|
+
|
|
1501
|
+
```bash
|
|
1502
|
+
`ascli` conf preset update myconf --url=... --username=... --password=...
|
|
1503
|
+
```
|
|
1504
|
+
|
|
1505
|
+
For a more secure storage one can do:
|
|
1506
|
+
|
|
1507
|
+
```bash
|
|
1508
|
+
`ascli` conf preset update myconf --url=... --username=... --password=@val:@vault:myconf.password
|
|
1509
|
+
`ascli` conf vault create myconf @json:'{"password":"__value_here__"}'
|
|
1510
|
+
```
|
|
1511
|
+
|
|
1512
|
+
> **Note:** use `@val:` in front of `@vault:` so that the extended value is not evaluated.
|
|
1513
|
+
|
|
1398
1514
|
### <a id="private_key"></a>Private Key
|
|
1399
1515
|
|
|
1400
1516
|
Some applications allow the user to be authenticated using a private key (Server, AoC, Faspex5, ...).
|
|
@@ -1545,7 +1661,7 @@ Available loggers: `stdout`, `stderr`, `syslog`.
|
|
|
1545
1661
|
|
|
1546
1662
|
Available levels: `debug`, `info`, `warn`, `error`.
|
|
1547
1663
|
|
|
1548
|
-
Note
|
|
1664
|
+
> **Note:** When using the `direct` agent (`ascp`), additional transfer logs can be activated using `ascp` option `EX_ascp_args`, see [`direct`](#agt_direct).
|
|
1549
1665
|
|
|
1550
1666
|
Examples:
|
|
1551
1667
|
|
|
@@ -1625,7 +1741,7 @@ There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is
|
|
|
1625
1741
|
The `http_proxy` environment variable (**lower case**, preferred) can be set to the URL of the proxy, e.g. `http://myproxy.org.net:3128`.
|
|
1626
1742
|
Refer to [Ruby findproxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
|
1627
1743
|
|
|
1628
|
-
> Note
|
|
1744
|
+
> **Note:** Ruby expects a URL and `myproxy.org.net:3128` alone is **not** accepted.
|
|
1629
1745
|
|
|
1630
1746
|
```bash
|
|
1631
1747
|
export http_proxy=http://proxy.example.com:3128
|
|
@@ -1689,7 +1805,7 @@ To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) para
|
|
|
1689
1805
|
|
|
1690
1806
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
|
1691
1807
|
|
|
1692
|
-
- `show` : shows the path of ascp used
|
|
1808
|
+
- `show` : shows the path of `ascp` used
|
|
1693
1809
|
- `use` : list,download connect client versions available on internet
|
|
1694
1810
|
- `products` : list Aspera transfer products available locally
|
|
1695
1811
|
- `connect` : list,download connect client versions available on internet
|
|
@@ -1718,9 +1834,9 @@ ascli config ascp info
|
|
|
1718
1834
|
|
|
1719
1835
|
#### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
|
1720
1836
|
|
|
1721
|
-
By default, `ascli` uses any found local product with ascp
|
|
1837
|
+
By default, `ascli` uses any found local product with `ascp`, including SDK.
|
|
1722
1838
|
|
|
1723
|
-
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
|
1839
|
+
To temporarily use an alternate `ascp` path use option `ascp_path` (`--ascp-path=`)
|
|
1724
1840
|
|
|
1725
1841
|
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
|
1726
1842
|
|
|
@@ -1771,11 +1887,11 @@ ascli config ascp products list
|
|
|
1771
1887
|
|
|
1772
1888
|
#### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
|
1773
1889
|
|
|
1774
|
-
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
|
1890
|
+
If no `ascp` is selected, this is equivalent to using option: `--use-product=FIRST`.
|
|
1775
1891
|
|
|
1776
|
-
Using the option use_product finds the ascp binary of the selected product.
|
|
1892
|
+
Using the option use_product finds the `ascp` binary of the selected product.
|
|
1777
1893
|
|
|
1778
|
-
To permanently use the ascp of a product:
|
|
1894
|
+
To permanently use the `ascp` of a product:
|
|
1779
1895
|
|
|
1780
1896
|
```bash
|
|
1781
1897
|
ascli config ascp products use 'Aspera Connect'
|
|
@@ -1821,7 +1937,7 @@ ascli config ascp connect version 'Aspera Connect for Mac Intel' download enclos
|
|
|
1821
1937
|
```
|
|
1822
1938
|
|
|
1823
1939
|
```output
|
|
1824
|
-
Time: 00:00:02
|
|
1940
|
+
Time: 00:00:02 =========================================================== 100% 27766 KB/sec Time: 00:00:02
|
|
1825
1941
|
Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
|
1826
1942
|
```
|
|
1827
1943
|
|
|
@@ -1840,15 +1956,15 @@ There are currently 3 agents:
|
|
|
1840
1956
|
- [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
|
1841
1957
|
- [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
|
1842
1958
|
|
|
1843
|
-
Note
|
|
1844
|
-
For
|
|
1959
|
+
> **Note:** All transfer operations are seen from the point of view of the agent.
|
|
1960
|
+
For example, a node agent executing an "upload", or "package send" operation
|
|
1845
1961
|
will effectively push files to the related server from the agent node.
|
|
1846
1962
|
|
|
1847
|
-
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *
|
|
1963
|
+
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *native* `ascp` options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
|
1848
1964
|
|
|
1849
1965
|
#### <a id="agt_direct"></a>Direct
|
|
1850
1966
|
|
|
1851
|
-
The `direct` agent directly executes a local ascp
|
|
1967
|
+
The `direct` agent directly executes a local `ascp`.
|
|
1852
1968
|
This is the default agent for `ascli`.
|
|
1853
1969
|
This is equivalent to option `--transfer=direct`.
|
|
1854
1970
|
`ascli` will detect locally installed Aspera products, including SDK, and use `ascp` from that component.
|
|
@@ -1856,18 +1972,17 @@ Refer to section [FASP](#client).
|
|
|
1856
1972
|
|
|
1857
1973
|
The `transfer_info` option accepts the following optional parameters to control multi-session, Web Socket Session and Resume policy:
|
|
1858
1974
|
|
|
1859
|
-
|
|
1860
|
-
|
|
1861
|
-
|
|
1862
|
-
|
|
1863
|
-
|
|
1864
|
-
|
|
1865
|
-
|
|
1866
|
-
|
|
1867
|
-
|
|
1868
|
-
|
|
1869
|
-
|
|
1870
|
-
</table>
|
|
1975
|
+
| Name | Type | Description |
|
|
1976
|
+
|----------------------|-------|-------------|
|
|
1977
|
+
| wss | Bool | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
|
1978
|
+
| spawn_timeout_sec | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
|
|
1979
|
+
| spawn_delay_sec | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
|
|
1980
|
+
| multi_incr_udp | Bool | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
|
|
1981
|
+
| resume | Hash | Resume<br/>parameters<br/>See below |
|
|
1982
|
+
| resume.iter_max | int | Resume<br/>Max number of retry on error<br/>Default: 7 |
|
|
1983
|
+
| resume.sleep_initial | int | Resume<br/>First Sleep before retry<br/>Default: 2 |
|
|
1984
|
+
| resume.sleep_factor | int | Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2 |
|
|
1985
|
+
| resume.sleep_max | int | Resume<br/>Default: 60 |
|
|
1871
1986
|
|
|
1872
1987
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
|
1873
1988
|
Sleep between iterations is:
|
|
@@ -1890,15 +2005,15 @@ ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":20}}'
|
|
|
1890
2005
|
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
|
1891
2006
|
```
|
|
1892
2007
|
|
|
1893
|
-
Note
|
|
2008
|
+
> **Note:** The `direct` agent supports additional `transfer_spec` parameters starting with `EX_` (extended).
|
|
1894
2009
|
In particular the field, `EX_ascp_args` which is a list of additional command line options to `ascp`.
|
|
1895
2010
|
|
|
1896
|
-
This can be useful to activate logging using option `-L` of ascp
|
|
2011
|
+
This can be useful to activate logging using option `-L` of `ascp`.
|
|
1897
2012
|
For example the option `--ts=@json:'{"EX_ascp_args":["-DDL-"]}'` will activate debug level 2 for `ascp` (`DD`), and display those logs on the terminal (`-`).
|
|
1898
2013
|
This is useful if the transfer fails.
|
|
1899
|
-
To store ascp logs in file `aspera-scp-transfer.log` in a folder, use `--ts=@json:'{"EX_ascp_args":["-L","/path/to/folder"]}'`.
|
|
2014
|
+
To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--ts=@json:'{"EX_ascp_args":["-L","/path/to/folder"]}'`.
|
|
1900
2015
|
|
|
1901
|
-
> Implementation note: when transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided in option `ts` in `EX_ascp_args`).
|
|
2016
|
+
> **Note:** Implementation note: when transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided in option `ts` in `EX_ascp_args`).
|
|
1902
2017
|
|
|
1903
2018
|
In addition to standard methods described in section [File List](#file_list), it is possible to specify the list of file using those additional methods:
|
|
1904
2019
|
|
|
@@ -1914,11 +2029,22 @@ In addition to standard methods described in section [File List](#file_list), it
|
|
|
1914
2029
|
--sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","myfilelist"]}'
|
|
1915
2030
|
```
|
|
1916
2031
|
|
|
1917
|
-
> File lists is shown here, there are also similar options for file pair lists.
|
|
2032
|
+
> **Note:** File lists is shown here, there are also similar options for file pair lists.
|
|
2033
|
+
>
|
|
2034
|
+
> **Note:** Those 2 additional methods avoid the creation of a copy of the file list: if the standard options `--sources=@lines:@file:... --src-type=...` are used, then the file is list read and parsed, and a new file list is created in a temporary folder.
|
|
2035
|
+
>
|
|
2036
|
+
> **Note:** Those methods have limitations: they apply **only** to the [`direct`](#agt_direct) transfer agent (i.e. local `ascp`) and not for Aspera on Cloud.
|
|
1918
2037
|
|
|
1919
|
-
|
|
2038
|
+
In addition to special transfer spec parameter `EX_ascp_args`, it is possible to provide the same `ascp` options using option `ascp_opts` of `ascli`.
|
|
2039
|
+
This option expects an `Array`, which can be conveniently provided with extended syntax `@list:`
|
|
2040
|
+
|
|
2041
|
+
```bash
|
|
2042
|
+
ascli server download /aspera-test-dir-large/200MB --ascp-opts=@list:' -l 10m -k 3'
|
|
2043
|
+
```
|
|
1920
2044
|
|
|
1921
|
-
>
|
|
2045
|
+
> **Note:** Using `@list:`, the use of quotes and leading character "space" here is important: The `@list:` expects a single string which must not be parsed by the shell (so protected with quotes) and the leading space is the separator character.
|
|
2046
|
+
>
|
|
2047
|
+
> **Note:** The option `ascp_opts` are appended to `EX_ascp_args` if present.
|
|
1922
2048
|
|
|
1923
2049
|
#### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
|
1924
2050
|
|
|
@@ -1930,13 +2056,12 @@ By specifying option: `--transfer=node`, the CLI will start transfers in an Aspe
|
|
|
1930
2056
|
Transfer Server using the Node API, either on a local or remote node.
|
|
1931
2057
|
Parameters provided in option `transfer_info` are:
|
|
1932
2058
|
|
|
1933
|
-
|
|
1934
|
-
|
|
1935
|
-
|
|
1936
|
-
|
|
1937
|
-
|
|
1938
|
-
|
|
1939
|
-
</table>
|
|
2059
|
+
| Name | Type | Description |
|
|
2060
|
+
|------|------|-------------|
|
|
2061
|
+
| url | string | URL of the node API</br>Mandatory |
|
|
2062
|
+
| username | string | node api user or access key</br>Mandatory |
|
|
2063
|
+
| password | string | password, secret or bearer token</br>Mandatory |
|
|
2064
|
+
| root_id | string | password or secret</br>Mandatory only for bearer token |
|
|
1940
2065
|
|
|
1941
2066
|
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
|
1942
2067
|
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
|
@@ -1952,12 +2077,11 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
|
1952
2077
|
|
|
1953
2078
|
Parameters provided in option `transfer_info` are:
|
|
1954
2079
|
|
|
1955
|
-
|
|
1956
|
-
|
|
1957
|
-
|
|
1958
|
-
|
|
1959
|
-
|
|
1960
|
-
</table>
|
|
2080
|
+
| Name | Type | Description |
|
|
2081
|
+
|------|------|-------------|
|
|
2082
|
+
| url | string | URL of the HTTP GW</br>Mandatory |
|
|
2083
|
+
| upload_bar_refresh_sec | float | Refresh rate for upload progress bar |
|
|
2084
|
+
| upload_chunksize | int | Size in bytes of chunks for upload |
|
|
1961
2085
|
|
|
1962
2086
|
Example:
|
|
1963
2087
|
|
|
@@ -1965,7 +2089,7 @@ Example:
|
|
|
1965
2089
|
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
|
1966
2090
|
```
|
|
1967
2091
|
|
|
1968
|
-
> The gateway only supports transfers authorized with a token.
|
|
2092
|
+
> **Note:** The gateway only supports transfers authorized with a token.
|
|
1969
2093
|
|
|
1970
2094
|
#### <a id="agt_trsdk"></a>Transfer SDK
|
|
1971
2095
|
|
|
@@ -2002,7 +2126,7 @@ is described in a [*transfer-spec*](#transferspec) (Transfer Specification), suc
|
|
|
2002
2126
|
|
|
2003
2127
|
If needed, it is possible to modify or add any of the supported [*transfer-spec*](#transferspec) parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several [*transfer-spec*](#transferspec) parameters. Multiple `ts` options on command line are cumulative.
|
|
2004
2128
|
|
|
2005
|
-
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [*transfer-spec*](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
|
2129
|
+
It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agt_direct) using the special [*transfer-spec*](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for `ascp` command line parameters not supported yet in the transfer spec.
|
|
2006
2130
|
|
|
2007
2131
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
|
2008
2132
|
|
|
@@ -2018,8 +2142,8 @@ All standard [*transfer-spec*](#transferspec) parameters can be specified.
|
|
|
2018
2142
|
|
|
2019
2143
|
References:
|
|
2020
2144
|
|
|
2021
|
-
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr
|
|
2022
|
-
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
|
2145
|
+
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22) → /opt/transfers
|
|
2146
|
+
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22) → Guides → API Ref → Transfer Spec V1
|
|
2023
2147
|
- [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
|
2024
2148
|
|
|
2025
2149
|
Parameters can be displayed with commands:
|
|
@@ -2039,91 +2163,91 @@ Columns:
|
|
|
2039
2163
|
|
|
2040
2164
|
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in `ascli`).
|
|
2041
2165
|
|
|
2042
|
-
|
|
2043
|
-
|
|
2044
|
-
|
|
2045
|
-
|
|
2046
|
-
|
|
2047
|
-
|
|
2048
|
-
|
|
2049
|
-
|
|
2050
|
-
|
|
2051
|
-
|
|
2052
|
-
|
|
2053
|
-
|
|
2054
|
-
|
|
2055
|
-
|
|
2056
|
-
|
|
2057
|
-
|
|
2058
|
-
|
|
2059
|
-
|
|
2060
|
-
|
|
2061
|
-
|
|
2062
|
-
|
|
2063
|
-
|
|
2064
|
-
|
|
2065
|
-
|
|
2066
|
-
|
|
2067
|
-
|
|
2068
|
-
|
|
2069
|
-
|
|
2070
|
-
|
|
2071
|
-
|
|
2072
|
-
<
|
|
2073
|
-
|
|
2074
|
-
|
|
2075
|
-
|
|
2076
|
-
|
|
2077
|
-
|
|
2078
|
-
|
|
2079
|
-
|
|
2080
|
-
|
|
2081
|
-
|
|
2082
|
-
|
|
2083
|
-
|
|
2084
|
-
|
|
2085
|
-
|
|
2086
|
-
|
|
2087
|
-
|
|
2088
|
-
|
|
2089
|
-
|
|
2090
|
-
<
|
|
2091
|
-
|
|
2092
|
-
|
|
2093
|
-
|
|
2094
|
-
|
|
2095
|
-
|
|
2096
|
-
|
|
2097
|
-
|
|
2098
|
-
|
|
2099
|
-
|
|
2100
|
-
|
|
2101
|
-
|
|
2102
|
-
|
|
2103
|
-
|
|
2104
|
-
|
|
2105
|
-
|
|
2106
|
-
|
|
2107
|
-
|
|
2108
|
-
|
|
2109
|
-
|
|
2110
|
-
|
|
2111
|
-
|
|
2112
|
-
|
|
2113
|
-
|
|
2114
|
-
|
|
2115
|
-
|
|
2116
|
-
|
|
2117
|
-
|
|
2118
|
-
|
|
2119
|
-
|
|
2120
|
-
|
|
2121
|
-
|
|
2122
|
-
|
|
2123
|
-
|
|
2124
|
-
|
|
2125
|
-
|
|
2126
|
-
|
|
2166
|
+
| Field | Type | D | N | C | Description |
|
|
2167
|
+
|-------|------|---|---|---|-------------|
|
|
2168
|
+
| EX_ascp_args | array | Y | | | Add native command line arguments to ascp |
|
|
2169
|
+
| EX_at_rest_password | string | Y | | | DEPRECATED: Prefer to use standard parameter: content_protection_password<br/>(env:ASPERA_SCP_FILEPASS) |
|
|
2170
|
+
| EX_file_list | string | Y | | | source file list |
|
|
2171
|
+
| EX_file_pair_list | string | Y | | | source file pair list |
|
|
2172
|
+
| EX_http_proxy_url | string | Y | | | Specify the proxy server address used by HTTP Fallback<br/>(-x {string}) |
|
|
2173
|
+
| EX_http_transfer_jpeg | int | Y | | | HTTP transfers as JPEG file<br/>(-j {int}) |
|
|
2174
|
+
| EX_license_text | string | Y | | | License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE) |
|
|
2175
|
+
| EX_no_read | bool | Y | | | no read source<br/>(--no-read) |
|
|
2176
|
+
| EX_no_write | bool | Y | | | no write on destination<br/>(--no-write) |
|
|
2177
|
+
| EX_proxy_password | string | Y | | | Password used for Aspera proxy server authentication.<br/>May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS) |
|
|
2178
|
+
| EX_ssh_key_paths | array | Y | | | Use public key authentication for SSH and specify the private key file paths<br/>(-i {array}) |
|
|
2179
|
+
| apply_local_docroot | bool | Y | | | (--apply-local-docroot) |
|
|
2180
|
+
| authentication | string | | | Y | value=token for SSH bypass keys, else password asked if not provided. |
|
|
2181
|
+
| cipher | string | Y | Y | Y | In transit encryption type.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum}) |
|
|
2182
|
+
| cipher_allowed | string | Y | Y | Y | returned by node API. Valid literals include "aes-128" and "none". |
|
|
2183
|
+
| content_protection | string | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt {enum}) |
|
|
2184
|
+
| content_protection_password | string | Y | Y | Y | Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS) |
|
|
2185
|
+
| cookie | string | Y | Y | Y | Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE) |
|
|
2186
|
+
| create_dir | bool | Y | Y | Y | Specifies whether to create new directories.<br/>(-d) |
|
|
2187
|
+
| delete_before_transfer | bool | Y | Y | Y | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects<br/>on the source are not deleted.<br/>(--delete-before-transfer) |
|
|
2188
|
+
| delete_source | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
|
2189
|
+
| destination_root | string | Y | Y | Y | Destination root directory. |
|
|
2190
|
+
| dgram_size | int | Y | Y | Y | UDP datagram size in bytes<br/>(-Z {int}) |
|
|
2191
|
+
| direction | string | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode (conversion){enum}) |
|
|
2192
|
+
| exclude_newer_than | int | Y | | | skip src files with mtime > arg<br/>(--exclude-newer-than {int}) |
|
|
2193
|
+
| exclude_older_than | int | Y | | | skip src files with mtime < arg<br/>(--exclude-older-than {int}) |
|
|
2194
|
+
| fasp_port | int | Y | Y | Y | Specifies fasp (UDP) port.<br/>(-O {int}) |
|
|
2195
|
+
| file_checksum | string | Y | Y | | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none |
|
|
2196
|
+
| http_fallback | bool<br/>string | Y | Y | Y | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){bool}|{string}) |
|
|
2197
|
+
| http_fallback_port | int | Y | | | Specifies http port when no cipher is used<br/>(-t {int}) |
|
|
2198
|
+
| https_fallback_port | int | Y | Y | Y | Specifies https port when cipher is used<br/>(-t {int}) |
|
|
2199
|
+
| lock_min_rate | bool | Y | Y | Y | |
|
|
2200
|
+
| lock_min_rate_kbps | bool | Y | Y | Y | |
|
|
2201
|
+
| lock_rate_policy | bool | Y | Y | Y | |
|
|
2202
|
+
| lock_target_rate | bool | Y | Y | Y | |
|
|
2203
|
+
| lock_target_rate_kbps | bool | Y | Y | Y | |
|
|
2204
|
+
| min_rate_cap_kbps | int | Y | Y | Y | |
|
|
2205
|
+
| min_rate_kbps | int | Y | Y | Y | Set the minimum transfer rate in kilobits per second.<br/>(-m {int}) |
|
|
2206
|
+
| move_after_transfer | string | Y | Y | | The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer {string}) |
|
|
2207
|
+
| multi_session | int | Y | Y | Y | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/> |
|
|
2208
|
+
| multi_session_threshold | int | Y | Y | | Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold {int}) |
|
|
2209
|
+
| overwrite | string | Y | Y | Y | Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite {enum}) |
|
|
2210
|
+
| password | string | | Y | | Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only. |
|
|
2211
|
+
| paths | array | Y | Y | Y | Array of path to the source (required) and a path to the destination (optional). |
|
|
2212
|
+
| precalculate_job_size | bool | Y | Y | Y | Specifies whether to precalculate the job size.<br/>(--precalculate-job-size) |
|
|
2213
|
+
| preserve_access_time | bool | Y | Y | Y | (--preserve-access-time) |
|
|
2214
|
+
| preserve_acls | string | Y | | | Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls {enum}) |
|
|
2215
|
+
| preserve_creation_time | bool | Y | Y | Y | (--preserve-creation-time) |
|
|
2216
|
+
| preserve_file_owner_gid | bool | Y | | | Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid) |
|
|
2217
|
+
| preserve_file_owner_uid | bool | Y | | | Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid) |
|
|
2218
|
+
| preserve_modification_time | bool | Y | Y | Y | (--preserve-modification-time) |
|
|
2219
|
+
| preserve_remote_acls | string | Y | | | Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls {enum}) |
|
|
2220
|
+
| preserve_source_access_time | bool | Y | | | Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time) |
|
|
2221
|
+
| preserve_times | bool | Y | Y | Y | (--preserve-times) |
|
|
2222
|
+
| proxy | string | Y | | | Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy {string}) |
|
|
2223
|
+
| rate_policy | string | Y | Y | Y | The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy {enum}) |
|
|
2224
|
+
| rate_policy_allowed | string | | | Y | Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed |
|
|
2225
|
+
| remote_host | string | Y | Y | Y | IP or fully qualified domain name of the remote server<br/>(--host {string}) |
|
|
2226
|
+
| remote_password | string | Y | Y | Y | SSH session password<br/>(env:ASPERA_SCP_PASS) |
|
|
2227
|
+
| remote_user | string | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user {string}) |
|
|
2228
|
+
| remove_after_transfer | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
|
2229
|
+
| remove_empty_directories | bool | Y | Y | | Specifies whether to remove empty directories.<br/>(--remove-empty-directories) |
|
|
2230
|
+
| remove_empty_source_directory | bool | Y | | | Remove empty source subdirectories and remove the source directory itself, if empty<br/>(--remove-empty-source-directory) |
|
|
2231
|
+
| remove_skipped | bool | Y | Y | Y | Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped) |
|
|
2232
|
+
| resume_policy | string | Y | Y | Y | If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum}) |
|
|
2233
|
+
| retry_duration | string<br/>int | | Y | Y | Specifies how long to wait before retrying transfer. (e.g. "5min") |
|
|
2234
|
+
| source_root | string | Y | Y | Y | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64 (conversion){string}) |
|
|
2235
|
+
| source_root_id | string | | Y | | The file ID of the source root directory. Required when using Bearer token auth for the source node. |
|
|
2236
|
+
| src_base | string | Y | Y | | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64 (conversion){string}) |
|
|
2237
|
+
| ssh_port | int | Y | Y | Y | Specifies SSH (TCP) port. Default: local:22, other:33001<br/>(-P {int}) |
|
|
2238
|
+
| ssh_private_key | string | Y | | | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----/nMII...<br/>Note the JSON encoding: /n for newlines.<br/>(env:ASPERA_SCP_KEY) |
|
|
2239
|
+
| ssh_private_key_passphrase | string | Y | | | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS) |
|
|
2240
|
+
| sshfp | string | Y | Y | Y | Check it against server SSH host key fingerprint<br/>(--check-sshfp {string}) |
|
|
2241
|
+
| symlink_policy | string | Y | Y | Y | Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links {enum}) |
|
|
2242
|
+
| tags | hash | Y | Y | Y | Metadata for transfer as JSON<br/>(--tags64 (conversion){hash}) |
|
|
2243
|
+
| target_rate_cap_kbps | int | | | Y | Returned by upload/download_setup node API. |
|
|
2244
|
+
| target_rate_kbps | int | Y | Y | Y | Specifies desired speed for the transfer.<br/>(-l {int}) |
|
|
2245
|
+
| target_rate_percentage | string | Y | Y | Y | |
|
|
2246
|
+
| title | string | | Y | Y | Title of the transfer |
|
|
2247
|
+
| token | string | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
|
2248
|
+
| use_ascp4 | bool | Y | Y | | specify version of protocol |
|
|
2249
|
+
| wss_enabled | bool | Y | Y | Y | |
|
|
2250
|
+
| wss_port | int | Y | Y | Y | TCP port used for websocket service feed. |
|
|
2127
2251
|
|
|
2128
2252
|
#### Destination folder for transfers
|
|
2129
2253
|
|
|
@@ -2167,7 +2291,7 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
|
2167
2291
|
|
|
2168
2292
|
- an [Extended Value](#extended) with type **Array of String**
|
|
2169
2293
|
|
|
2170
|
-
> Note
|
|
2294
|
+
> **Note:** extended values can be tested with the command `conf echo`
|
|
2171
2295
|
|
|
2172
2296
|
Examples:
|
|
2173
2297
|
|
|
@@ -2215,7 +2339,7 @@ Examples:
|
|
|
2215
2339
|
|
|
2216
2340
|
The option `src_type` allows specifying if the list specified in option `sources` is a simple file list or if it is a file pair list.
|
|
2217
2341
|
|
|
2218
|
-
> Note
|
|
2342
|
+
> **Note:** Option `src_type` is not used if option `sources` is set to `@ts`
|
|
2219
2343
|
|
|
2220
2344
|
Supported values for `src_type` are:
|
|
2221
2345
|
|
|
@@ -2228,11 +2352,11 @@ Example: Source file `200KB.1` is renamed `sample1` on destination:
|
|
|
2228
2352
|
ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
|
2229
2353
|
```
|
|
2230
2354
|
|
|
2231
|
-
> Note
|
|
2355
|
+
> **Note:** There are some specific rules to specify a file list when using **Aspera on Cloud**, refer to the AoC plugin section.
|
|
2232
2356
|
|
|
2233
2357
|
#### <a id="multisession"></a>Support of multi-session
|
|
2234
2358
|
|
|
2235
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
|
2359
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one `ascp` process per session) is supported on "direct" and "node" agents, not yet on connect.
|
|
2236
2360
|
|
|
2237
2361
|
- when agent=node :
|
|
2238
2362
|
|
|
@@ -2271,7 +2395,7 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
|
2271
2395
|
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"_pass_here_"}'
|
|
2272
2396
|
```
|
|
2273
2397
|
|
|
2274
|
-
Note
|
|
2398
|
+
> **Note:** Up to version `ascli` 4.6.0, the following parameters should be used for agent `direct`:
|
|
2275
2399
|
|
|
2276
2400
|
```javascript
|
|
2277
2401
|
--ts=@json:'{"EX_ascp_args":["--file-crypt=decrypt"],"EX_at_rest_password":"_secret_here_"}'
|
|
@@ -2303,9 +2427,13 @@ Note that up to version 4.6.0, the following parameters should be used for agent
|
|
|
2303
2427
|
--ts=@json:'{"precalculate_job_size":true}'
|
|
2304
2428
|
```
|
|
2305
2429
|
|
|
2306
|
-
### <a id="scheduling"></a>
|
|
2430
|
+
### <a id="scheduling"></a>Scheduling
|
|
2431
|
+
|
|
2432
|
+
It is useful to configure automated scheduled execution.
|
|
2433
|
+
|
|
2434
|
+
#### <a id="locking"></a>Locking for exclusive execution
|
|
2307
2435
|
|
|
2308
|
-
|
|
2436
|
+
It is also useful to ensure that `ascli` is not executed several times in parallel.
|
|
2309
2437
|
|
|
2310
2438
|
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
|
2311
2439
|
|
|
@@ -2339,6 +2467,18 @@ The first instance will sleep 30 seconds, the second one will immediately exit l
|
|
|
2339
2467
|
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
|
2340
2468
|
```
|
|
2341
2469
|
|
|
2470
|
+
#### <a id="scheduler"></a>Scheduler
|
|
2471
|
+
|
|
2472
|
+
`ascli` does not provide an internal scheduler.
|
|
2473
|
+
|
|
2474
|
+
Instead, use the service provided by the Operating system:
|
|
2475
|
+
|
|
2476
|
+
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
|
2477
|
+
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
|
2478
|
+
- etc...
|
|
2479
|
+
|
|
2480
|
+
Linux also provides `anacron`, if tasks are hourly or daily.
|
|
2481
|
+
|
|
2342
2482
|
### "Provençale"
|
|
2343
2483
|
|
|
2344
2484
|
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
|
@@ -2465,7 +2605,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
|
2465
2605
|
```text
|
|
2466
2606
|
ascli -h
|
|
2467
2607
|
NAME
|
|
2468
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
|
2608
|
+
ascli -- a command line tool for Aspera Applications (v4.11.0)
|
|
2469
2609
|
|
|
2470
2610
|
SYNOPSIS
|
|
2471
2611
|
ascli COMMANDS [OPTIONS] [ARGS]
|
|
@@ -2515,7 +2655,6 @@ OPTIONS: global
|
|
|
2515
2655
|
--log-level=ENUM Log level: debug, info, [warn], error, fatal, unknown
|
|
2516
2656
|
--logger=ENUM log method: [stderr], stdout, syslog
|
|
2517
2657
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
|
2518
|
-
--query=VALUE additional filter for API calls (extended value) (some commands)
|
|
2519
2658
|
--http-options=VALUE options for http socket (extended value)
|
|
2520
2659
|
--insecure=ENUM do not validate HTTPS certificate: no, [yes]
|
|
2521
2660
|
--once-only=ENUM process only new items (some commands): [no], yes
|
|
@@ -2523,8 +2662,9 @@ OPTIONS: global
|
|
|
2523
2662
|
--cache-tokens=ENUM save and reuse Oauth tokens: no, [yes]
|
|
2524
2663
|
|
|
2525
2664
|
COMMAND: config
|
|
2526
|
-
SUBCOMMANDS:
|
|
2665
|
+
SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test export_to_cli file flush_tokens folder gem genkey id initdemo list lookup open overview plugin preset proxy_check secure smtp_settings vault wizard
|
|
2527
2666
|
OPTIONS:
|
|
2667
|
+
--query=VALUE additional filter for API calls (extended value) (some commands)
|
|
2528
2668
|
--value=VALUE extended value for create, update, list filter
|
|
2529
2669
|
--property=VALUE name of property to set
|
|
2530
2670
|
--id=VALUE resource identifier (modify,delete,show)
|
|
@@ -2552,18 +2692,18 @@ OPTIONS:
|
|
|
2552
2692
|
--notif-template=VALUE Email ERB template for notification of transfers
|
|
2553
2693
|
--version-check-days=VALUE Period in days to check new version (zero to disable)
|
|
2554
2694
|
--plugin-folder=VALUE Folder where to find additional plugins
|
|
2555
|
-
--ts=VALUE
|
|
2556
|
-
--
|
|
2557
|
-
--
|
|
2558
|
-
--
|
|
2559
|
-
--src-type=ENUM
|
|
2560
|
-
--transfer=ENUM
|
|
2561
|
-
--transfer-info=VALUE
|
|
2562
|
-
--progress=ENUM
|
|
2695
|
+
--ts=VALUE Override transfer spec values (Hash, e.g. use @json: prefix), current={"create_dir"=>true}
|
|
2696
|
+
--to-folder=VALUE Destination folder for transfered files
|
|
2697
|
+
--sources=VALUE How list of transfered files is provided (@args,@ts,Array)
|
|
2698
|
+
--ascp-opts=VALUE Options for ascp in its native format
|
|
2699
|
+
--src-type=ENUM Type of file list: list, pair
|
|
2700
|
+
--transfer=ENUM Type of transfer agent: direct, node, connect, httpgw, trsdk
|
|
2701
|
+
--transfer-info=VALUE Parameters for transfer agent
|
|
2702
|
+
--progress=ENUM Type of progress bar: none, native, multi
|
|
2563
2703
|
|
|
2564
2704
|
|
|
2565
2705
|
COMMAND: shares
|
|
2566
|
-
SUBCOMMANDS: health repository
|
|
2706
|
+
SUBCOMMANDS: admin health repository
|
|
2567
2707
|
OPTIONS:
|
|
2568
2708
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2569
2709
|
--username=VALUE username to log in
|
|
@@ -2571,7 +2711,7 @@ OPTIONS:
|
|
|
2571
2711
|
|
|
2572
2712
|
|
|
2573
2713
|
COMMAND: node
|
|
2574
|
-
SUBCOMMANDS:
|
|
2714
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
|
2575
2715
|
OPTIONS:
|
|
2576
2716
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2577
2717
|
--username=VALUE username to log in
|
|
@@ -2579,11 +2719,13 @@ OPTIONS:
|
|
|
2579
2719
|
--validator=VALUE identifier of validator (optional for central)
|
|
2580
2720
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
|
2581
2721
|
--sync-name=VALUE sync name
|
|
2722
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
|
2582
2723
|
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
|
2724
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
|
2583
2725
|
|
|
2584
2726
|
|
|
2585
2727
|
COMMAND: orchestrator
|
|
2586
|
-
SUBCOMMANDS: health info
|
|
2728
|
+
SUBCOMMANDS: health info plugins processes workflow
|
|
2587
2729
|
OPTIONS:
|
|
2588
2730
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2589
2731
|
--username=VALUE username to log in
|
|
@@ -2612,7 +2754,7 @@ OPTIONS:
|
|
|
2612
2754
|
|
|
2613
2755
|
|
|
2614
2756
|
COMMAND: ats
|
|
2615
|
-
SUBCOMMANDS:
|
|
2757
|
+
SUBCOMMANDS: access_key api_key aws_trust_policy cluster
|
|
2616
2758
|
OPTIONS:
|
|
2617
2759
|
--ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
|
|
2618
2760
|
--instance=VALUE ATS instance in ibm cloud
|
|
@@ -2624,7 +2766,7 @@ OPTIONS:
|
|
|
2624
2766
|
|
|
2625
2767
|
|
|
2626
2768
|
COMMAND: faspex5
|
|
2627
|
-
SUBCOMMANDS: health
|
|
2769
|
+
SUBCOMMANDS: admin bearer_token health package user version
|
|
2628
2770
|
OPTIONS:
|
|
2629
2771
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2630
2772
|
--username=VALUE username to log in
|
|
@@ -2632,7 +2774,7 @@ OPTIONS:
|
|
|
2632
2774
|
--client-id=VALUE OAuth client identifier
|
|
2633
2775
|
--client-secret=VALUE OAuth client secret
|
|
2634
2776
|
--redirect-uri=VALUE OAuth redirect URI for web authentication
|
|
2635
|
-
--auth=ENUM OAuth type of authentication: web, jwt
|
|
2777
|
+
--auth=ENUM OAuth type of authentication: boot, web, jwt
|
|
2636
2778
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
|
2637
2779
|
--passphrase=VALUE RSA private key passphrase
|
|
2638
2780
|
|
|
@@ -2650,7 +2792,7 @@ OPTIONS:
|
|
|
2650
2792
|
|
|
2651
2793
|
|
|
2652
2794
|
COMMAND: faspex
|
|
2653
|
-
SUBCOMMANDS: health package source
|
|
2795
|
+
SUBCOMMANDS: address_book dropbox health login_methods me package source v4
|
|
2654
2796
|
OPTIONS:
|
|
2655
2797
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2656
2798
|
--username=VALUE username to log in
|
|
@@ -2664,7 +2806,7 @@ OPTIONS:
|
|
|
2664
2806
|
|
|
2665
2807
|
|
|
2666
2808
|
COMMAND: preview
|
|
2667
|
-
SUBCOMMANDS:
|
|
2809
|
+
SUBCOMMANDS: check events scan test trevents
|
|
2668
2810
|
OPTIONS:
|
|
2669
2811
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2670
2812
|
--username=VALUE username to log in
|
|
@@ -2699,14 +2841,14 @@ OPTIONS:
|
|
|
2699
2841
|
|
|
2700
2842
|
|
|
2701
2843
|
COMMAND: sync
|
|
2702
|
-
SUBCOMMANDS: start
|
|
2844
|
+
SUBCOMMANDS: admin start
|
|
2703
2845
|
OPTIONS:
|
|
2704
|
-
--
|
|
2705
|
-
--session
|
|
2846
|
+
--sync-info=VALUE Information for sync instance and sessions (Hash)
|
|
2847
|
+
--sync-session=VALUE Name of session to use for admin commands. default: first in parameters
|
|
2706
2848
|
|
|
2707
2849
|
|
|
2708
2850
|
COMMAND: aoc
|
|
2709
|
-
SUBCOMMANDS:
|
|
2851
|
+
SUBCOMMANDS: admin automation bearer_token files gateway organization packages reminder servers tier_restrictions user
|
|
2710
2852
|
OPTIONS:
|
|
2711
2853
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2712
2854
|
--username=VALUE username to log in
|
|
@@ -2717,20 +2859,28 @@ OPTIONS:
|
|
|
2717
2859
|
--client-secret=VALUE OAuth API client passcode
|
|
2718
2860
|
--redirect-uri=VALUE OAuth API client redirect URI
|
|
2719
2861
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
|
2720
|
-
--passphrase=VALUE RSA private key passphrase
|
|
2721
|
-
--workspace=VALUE name of workspace
|
|
2722
|
-
--name=VALUE resource name
|
|
2723
|
-
--path=VALUE file or folder path
|
|
2724
|
-
--link=VALUE public link to shared resource
|
|
2725
|
-
--new-user-option=VALUE new user creation option for unknown package recipients
|
|
2726
|
-
--from-folder=VALUE share to share source folder
|
|
2727
2862
|
--scope=VALUE OAuth scope for AoC API calls
|
|
2728
|
-
--
|
|
2729
|
-
--
|
|
2863
|
+
--passphrase=VALUE RSA private key passphrase
|
|
2864
|
+
--workspace=VALUE Name of workspace
|
|
2865
|
+
--name=VALUE Resource name (prefer to use keyword name)
|
|
2866
|
+
--link=VALUE Public link to shared resource
|
|
2867
|
+
--new-user-option=VALUE New user creation option for unknown package recipients
|
|
2868
|
+
--from-folder=VALUE Source folder for Folder-to-Folder transfer
|
|
2869
|
+
--validate-metadata=ENUM Validate shared inbox metadata: [no], yes
|
|
2870
|
+
|
|
2871
|
+
COMMAND: node
|
|
2872
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
|
2873
|
+
OPTIONS:
|
|
2874
|
+
--validator=VALUE identifier of validator (optional for central)
|
|
2875
|
+
--asperabrowserurl=VALUE URL for simple aspera web ui
|
|
2876
|
+
--sync-name=VALUE sync name
|
|
2877
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
|
2878
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
|
2879
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
|
2730
2880
|
|
|
2731
2881
|
|
|
2732
2882
|
COMMAND: server
|
|
2733
|
-
SUBCOMMANDS:
|
|
2883
|
+
SUBCOMMANDS: browse cp delete df download du health info ls md5sum mkdir mv rename rm sync upload
|
|
2734
2884
|
OPTIONS:
|
|
2735
2885
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2736
2886
|
--username=VALUE username to log in
|
|
@@ -2740,7 +2890,7 @@ OPTIONS:
|
|
|
2740
2890
|
|
|
2741
2891
|
|
|
2742
2892
|
COMMAND: console
|
|
2743
|
-
SUBCOMMANDS: transfer
|
|
2893
|
+
SUBCOMMANDS: health transfer
|
|
2744
2894
|
OPTIONS:
|
|
2745
2895
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
2746
2896
|
--username=VALUE username to log in
|
|
@@ -2751,7 +2901,7 @@ OPTIONS:
|
|
|
2751
2901
|
|
|
2752
2902
|
```
|
|
2753
2903
|
|
|
2754
|
-
Note
|
|
2904
|
+
> **Note:** commands and parameter values can be written in short form.
|
|
2755
2905
|
|
|
2756
2906
|
### Bulk creation and deletion of resources
|
|
2757
2907
|
|
|
@@ -2759,7 +2909,7 @@ Bulk creation and deletion of resources are possible using option `bulk` (yes,no
|
|
|
2759
2909
|
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
|
2760
2910
|
This option is available only for some of the resources: if you need it: try and see if the entities you try to create or delete support this option.
|
|
2761
2911
|
|
|
2762
|
-
## <a id="aoc"></a>Plugin: Aspera on Cloud
|
|
2912
|
+
## <a id="aoc"></a>Plugin: `aoc`: IBM Aspera on Cloud
|
|
2763
2913
|
|
|
2764
2914
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
|
2765
2915
|
|
|
@@ -2802,7 +2952,7 @@ ascli config wizard --value=aoc
|
|
|
2802
2952
|
|
|
2803
2953
|
### <a id="aocmanual"></a>Configuration: using manual setup
|
|
2804
2954
|
|
|
2805
|
-
> If you used the wizard (recommended): skip this section.
|
|
2955
|
+
> **Note:** If you used the wizard (recommended): skip this section.
|
|
2806
2956
|
|
|
2807
2957
|
#### Configuration details
|
|
2808
2958
|
|
|
@@ -2826,12 +2976,12 @@ If you use the built-in client_id and client_secret, skip this and do not set th
|
|
|
2826
2976
|
|
|
2827
2977
|
Else you can use a specific OAuth API client_id, the first step is to declare `ascli` in Aspera on Cloud using the admin interface.
|
|
2828
2978
|
|
|
2829
|
-
(
|
|
2979
|
+
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client) ).
|
|
2830
2980
|
|
|
2831
2981
|
Let's start by a registration with web based authentication (auth=web):
|
|
2832
2982
|
|
|
2833
2983
|
- Open a web browser, log to your instance: e.g. `https://myorg.ibmaspera.com/`
|
|
2834
|
-
- Go to Apps→Admin→Organization→Integrations
|
|
2984
|
+
- Go to Apps → Admin → Organization → Integrations
|
|
2835
2985
|
- Click "Create New"
|
|
2836
2986
|
- Client Name: `ascli`
|
|
2837
2987
|
- Redirect URIs: `http://localhost:12345`
|
|
@@ -2882,7 +3032,7 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
|
2882
3032
|
- Graphically
|
|
2883
3033
|
|
|
2884
3034
|
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
|
2885
|
-
- Go to Apps→Admin→Organization→Integrations
|
|
3035
|
+
- Go to Apps → Admin → Organization → Integrations
|
|
2886
3036
|
- Click on the previously created application
|
|
2887
3037
|
- select tab : "JSON Web Token Auth"
|
|
2888
3038
|
- Modify options if necessary, for instance: activate both options in section "Settings"
|
|
@@ -3055,7 +3205,7 @@ ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"s
|
|
|
3055
3205
|
|
|
3056
3206
|
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
|
3057
3207
|
|
|
3058
|
-
Note
|
|
3208
|
+
> **Note:** The option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
|
3059
3209
|
|
|
3060
3210
|
#### <a id="res_select"></a>Selecting a resource
|
|
3061
3211
|
|
|
@@ -3111,7 +3261,7 @@ If the command returns an error, example:
|
|
|
3111
3261
|
|
|
3112
3262
|
Well, remove the offending parameters and try again.
|
|
3113
3263
|
|
|
3114
|
-
Note
|
|
3264
|
+
> **Note:** Some properties that are shown in the web UI, such as membership, are not listed directly in the resource, but instead another resource is created to link a user and its group: `group_membership`
|
|
3115
3265
|
|
|
3116
3266
|
#### Access Key secrets
|
|
3117
3267
|
|
|
@@ -3167,7 +3317,7 @@ Options:
|
|
|
3167
3317
|
- `query` filter (on API call)
|
|
3168
3318
|
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
|
3169
3319
|
|
|
3170
|
-
Note
|
|
3320
|
+
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
|
3171
3321
|
|
|
3172
3322
|
#### Transfer: Using specific transfer ports
|
|
3173
3323
|
|
|
@@ -3489,7 +3639,8 @@ AoC nodes as actually composed with two related entities:
|
|
|
3489
3639
|
|
|
3490
3640
|
The web UI allows creation of both entities in one shot.
|
|
3491
3641
|
For more flexibility, `ascli` allows this in two separate steps.
|
|
3492
|
-
|
|
3642
|
+
|
|
3643
|
+
> **Note:** When selecting "Use existing access key" in the web UI, this actually skips access key creation (first step).
|
|
3493
3644
|
|
|
3494
3645
|
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
|
3495
3646
|
|
|
@@ -3503,7 +3654,7 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
|
3503
3654
|
|
|
3504
3655
|
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed.
|
|
3505
3656
|
|
|
3506
|
-
> Note
|
|
3657
|
+
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking to IBM support.
|
|
3507
3658
|
|
|
3508
3659
|
- Create the AoC node entity
|
|
3509
3660
|
|
|
@@ -3523,9 +3674,10 @@ Creation of a node with a self-managed node is similar, but the command `aoc adm
|
|
|
3523
3674
|
|
|
3524
3675
|
### List of files to transfer
|
|
3525
3676
|
|
|
3526
|
-
Source files are provided as a list with the `sources` option.
|
|
3677
|
+
Source files are provided as a list with the `sources` option.
|
|
3678
|
+
Refer to section [File list](#file_list)
|
|
3527
3679
|
|
|
3528
|
-
Note
|
|
3680
|
+
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file id` API).
|
|
3529
3681
|
|
|
3530
3682
|
Source files are located on "Aspera on cloud", when :
|
|
3531
3683
|
|
|
@@ -3644,19 +3796,28 @@ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
|
3644
3796
|
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
|
3645
3797
|
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
|
3646
3798
|
|
|
3647
|
-
Typically, one would execute this command on a regular basis, using the method of your choice:
|
|
3648
|
-
|
|
3649
|
-
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
|
3650
|
-
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
|
3651
|
-
- etc...
|
|
3799
|
+
Typically, one would execute this command on a regular basis, using the method of your choice: see [Scheduler](#scheduler).
|
|
3652
3800
|
|
|
3653
3801
|
### Files
|
|
3654
3802
|
|
|
3655
|
-
|
|
3803
|
+
The Files application presents a **Home** folder to users in a given workspace.
|
|
3804
|
+
Files located here are either user's files, or shared folders.
|
|
3656
3805
|
|
|
3657
3806
|
#### Download Files
|
|
3658
3807
|
|
|
3659
|
-
|
|
3808
|
+
The general download command is:
|
|
3809
|
+
|
|
3810
|
+
```bash
|
|
3811
|
+
ascli aoc files download <source folder path> <source filename 1> ...
|
|
3812
|
+
```
|
|
3813
|
+
|
|
3814
|
+
I.e. the first argument is the source folder, and the following arguments are the source file names in this folder.
|
|
3815
|
+
|
|
3816
|
+
If a single file or folder is to be downloaded, then a single argument can be provided.
|
|
3817
|
+
|
|
3818
|
+
```bash
|
|
3819
|
+
ascli aoc files download <single file path>
|
|
3820
|
+
```
|
|
3660
3821
|
|
|
3661
3822
|
#### Shared folders
|
|
3662
3823
|
|
|
@@ -3774,18 +3935,17 @@ cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='
|
|
|
3774
3935
|
### AoC sample commands
|
|
3775
3936
|
|
|
3776
3937
|
```bash
|
|
3777
|
-
aoc -N remind --username=my_aoc_user_email
|
|
3778
|
-
aoc -N servers
|
|
3779
3938
|
aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
|
3780
3939
|
aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
|
3781
|
-
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
|
3782
|
-
aoc admin ats access_key delete
|
|
3940
|
+
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
|
3941
|
+
aoc admin ats access_key delete ak1ibmcloud
|
|
3783
3942
|
aoc admin ats access_key list --fields=name,id
|
|
3784
|
-
aoc admin ats access_key node
|
|
3943
|
+
aoc admin ats access_key node ak1ibmcloud --secret=somesecret browse /
|
|
3785
3944
|
aoc admin ats cluster clouds
|
|
3786
3945
|
aoc admin ats cluster list
|
|
3787
3946
|
aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
|
3788
3947
|
aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
|
3948
|
+
aoc admin auth_providers list
|
|
3789
3949
|
aoc admin res application list
|
|
3790
3950
|
aoc admin res client list
|
|
3791
3951
|
aoc admin res client_access_key list
|
|
@@ -3806,14 +3966,15 @@ aoc admin res self show
|
|
|
3806
3966
|
aoc admin res short_link list
|
|
3807
3967
|
aoc admin res user list
|
|
3808
3968
|
aoc admin res workspace_membership list
|
|
3809
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
|
3810
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
|
3811
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
|
3812
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
|
3813
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
|
3814
|
-
aoc admin resource node
|
|
3969
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do browse /
|
|
3970
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do delete /folder1
|
|
3971
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do mkdir /folder1
|
|
3972
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
|
|
3973
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 events
|
|
3974
|
+
aoc admin resource node do name my_aoc_ak_name --secret=my_aoc_ak_secret v3 access_key delete testsub1
|
|
3815
3975
|
aoc admin resource workspace list
|
|
3816
3976
|
aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
|
3977
|
+
aoc admin subscription
|
|
3817
3978
|
aoc automation workflow action my_wf_id create --value=@json:'{"name":"toto"}'
|
|
3818
3979
|
aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
|
3819
3980
|
aoc automation workflow delete my_wf_id
|
|
@@ -3825,10 +3986,11 @@ aoc faspex
|
|
|
3825
3986
|
aoc files bearer /
|
|
3826
3987
|
aoc files bearer_token_node / --cache-tokens=no
|
|
3827
3988
|
aoc files browse /
|
|
3828
|
-
aoc files browse /
|
|
3989
|
+
aoc files browse / --link=my_aoc_publink_folder
|
|
3829
3990
|
aoc files delete /testsrc
|
|
3830
3991
|
aoc files download --transfer=connect /200KB.1
|
|
3831
|
-
aoc files file
|
|
3992
|
+
aoc files file modify --path=my_aoc_test_folder
|
|
3993
|
+
aoc files file permission --path=my_aoc_test_folder list
|
|
3832
3994
|
aoc files file show --path=/200KB.1
|
|
3833
3995
|
aoc files file show my_file_id
|
|
3834
3996
|
aoc files find / --value='\.partial$'
|
|
@@ -3838,12 +4000,14 @@ aoc files rename /somefolder testdst
|
|
|
3838
4000
|
aoc files short_link create --to-folder=/testdst --value=private
|
|
3839
4001
|
aoc files short_link create --to-folder=/testdst --value=public
|
|
3840
4002
|
aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_link"}'
|
|
4003
|
+
aoc files sync admin status --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
|
4004
|
+
aoc files sync start --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
|
3841
4005
|
aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
|
|
4006
|
+
aoc files upload --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
|
3842
4007
|
aoc files upload --to-folder=/testsrc testfile.bin
|
|
3843
|
-
aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
|
3844
4008
|
aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
|
3845
4009
|
aoc files v3 info
|
|
3846
|
-
aoc org
|
|
4010
|
+
aoc org --link=my_aoc_publink_recv_from_aocuser
|
|
3847
4011
|
aoc organization
|
|
3848
4012
|
aoc packages browse "my_package_id" /contents
|
|
3849
4013
|
aoc packages list
|
|
@@ -3852,19 +4016,21 @@ aoc packages recv "my_package_id" --to-folder=.
|
|
|
3852
4016
|
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
|
3853
4017
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_external_user"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
|
3854
4018
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_internal_user"],"note":"my note"}' testfile.bin
|
|
4019
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
|
4020
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
|
3855
4021
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' testfile.bin
|
|
3856
4022
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' testfile.bin
|
|
3857
4023
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
|
3858
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
|
3859
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
|
3860
4024
|
aoc packages shared_inboxes list
|
|
4025
|
+
aoc remind --username=my_aoc_user_email
|
|
4026
|
+
aoc servers
|
|
3861
4027
|
aoc user profile modify @json:'{"name":"dummy change"}'
|
|
3862
4028
|
aoc user profile show
|
|
3863
4029
|
aoc user workspaces current
|
|
3864
4030
|
aoc user workspaces list
|
|
3865
4031
|
```
|
|
3866
4032
|
|
|
3867
|
-
## <a id="ats"></a>Plugin: Aspera Transfer Service
|
|
4033
|
+
## <a id="ats"></a>Plugin: `ats`: IBM Aspera Transfer Service
|
|
3868
4034
|
|
|
3869
4035
|
ATS is usable either :
|
|
3870
4036
|
|
|
@@ -3985,13 +4151,15 @@ The parameters provided to ATS for access key creation are the ones of [ATS API]
|
|
|
3985
4151
|
### ATS sample commands
|
|
3986
4152
|
|
|
3987
4153
|
```bash
|
|
3988
|
-
ats access_key cluster
|
|
4154
|
+
ats access_key cluster ak2ibmcloud --secret=somesecret
|
|
3989
4155
|
ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
|
3990
|
-
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
|
4156
|
+
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak2ibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
|
4157
|
+
ats access_key delete ak2ibmcloud
|
|
3991
4158
|
ats access_key delete ak_aws
|
|
3992
|
-
ats access_key
|
|
4159
|
+
ats access_key entitlement ak2ibmcloud
|
|
3993
4160
|
ats access_key list --fields=name,id
|
|
3994
|
-
ats access_key node
|
|
4161
|
+
ats access_key node ak2ibmcloud browse / --secret=somesecret
|
|
4162
|
+
ats access_key show ak2ibmcloud
|
|
3995
4163
|
ats api_key create
|
|
3996
4164
|
ats api_key instances
|
|
3997
4165
|
ats api_key list
|
|
@@ -4001,17 +4169,21 @@ ats cluster show --cloud=aws --region=eu-west-1
|
|
|
4001
4169
|
ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
|
4002
4170
|
```
|
|
4003
4171
|
|
|
4004
|
-
## Plugin: IBM Aspera High Speed Transfer Server (
|
|
4172
|
+
## <a id="server"></a>Plugin: `server`: IBM Aspera High Speed Transfer Server (SSH)
|
|
4005
4173
|
|
|
4006
|
-
|
|
4174
|
+
The `server` plugin is used for operations on Aspera HSTS using SSH authentication.
|
|
4007
4175
|
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
|
4008
|
-
|
|
4176
|
+
An SSH session is established, authenticated with either a password or an SSH private key,
|
|
4177
|
+
then commands `ascp` (for transfers) and `ascmd` (for file operations) are executed.
|
|
4178
|
+
|
|
4179
|
+
> **Note:** The URL to be provided is usually: `ssh://_server_address_:33001`
|
|
4009
4180
|
|
|
4010
4181
|
### Server sample commands
|
|
4011
4182
|
|
|
4012
4183
|
```bash
|
|
4013
|
-
server
|
|
4184
|
+
server br /
|
|
4014
4185
|
server browse /
|
|
4186
|
+
server browse NEW_SERVER_FOLDER/testfile.bin
|
|
4015
4187
|
server browse folder_1/target_hot
|
|
4016
4188
|
server cp NEW_SERVER_FOLDER/testfile.bin folder_1/200KB.2
|
|
4017
4189
|
server delete NEW_SERVER_FOLDER
|
|
@@ -4027,34 +4199,39 @@ server md5sum NEW_SERVER_FOLDER/testfile.bin
|
|
|
4027
4199
|
server mkdir NEW_SERVER_FOLDER --logger=stdout
|
|
4028
4200
|
server mkdir folder_1/target_hot
|
|
4029
4201
|
server mv folder_1/200KB.2 folder_1/to.delete
|
|
4202
|
+
server sync admin status --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"}}'
|
|
4203
|
+
server sync admin status --sync-session=mysync --sync-info=@json:'{"sessions":[{"name":"mysync","local_dir":"syncdir"}]}'
|
|
4204
|
+
server sync start --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"},"remote":{"path":"'"NEW_SERVER_FOLDER"'"},"reset":true,"quiet":false}'
|
|
4205
|
+
server sync start --sync-info=@json:'{"sessions":[{"name":"mysync","direction":"pull","remote_dir":"'"NEW_SERVER_FOLDER"'","local_dir":"syncdir","reset":true}]}'
|
|
4030
4206
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","'"filelist.txt"'"]}' --to-folder=NEW_SERVER_FOLDER
|
|
4031
4207
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-pair-list","'"filepairlist.txt"'"]}'
|
|
4032
4208
|
server upload --sources=@ts --ts=@json:'{"EX_file_list":"'"filelist.txt"'"}' --to-folder=NEW_SERVER_FOLDER
|
|
4033
4209
|
server upload --sources=@ts --ts=@json:'{"EX_file_pair_list":"'"filepairlist.txt"'"}'
|
|
4034
4210
|
server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
|
4035
4211
|
server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
|
4036
|
-
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
|
4212
|
+
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email --ascp-opts=@list:' -l 10m'
|
|
4037
4213
|
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"_secret_here_","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
|
|
4038
4214
|
server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
|
4039
|
-
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
|
4215
|
+
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}' --progress=multi
|
|
4040
4216
|
```
|
|
4041
4217
|
|
|
4042
4218
|
### Authentication on Server with SSH session
|
|
4043
4219
|
|
|
4044
|
-
If SSH is the session
|
|
4220
|
+
If SSH is the session protocol (by default i.e. not WSS), then following session authentication methods are supported:
|
|
4045
4221
|
|
|
4046
|
-
- SSH password
|
|
4047
|
-
- SSH keys (Multiple SSH key paths can be provided.)
|
|
4222
|
+
- `password`: SSH password
|
|
4223
|
+
- `ssh_keys`: SSH keys (Multiple SSH key paths can be provided.)
|
|
4048
4224
|
|
|
4049
|
-
If username is not provided
|
|
4225
|
+
If `username` is not provided then the default transfer user `xfer` is used.
|
|
4050
4226
|
|
|
4051
|
-
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
|
4227
|
+
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
|
4228
|
+
Example:
|
|
4052
4229
|
|
|
4053
|
-
```
|
|
4054
|
-
ascli server --url=ssh
|
|
4230
|
+
```bash
|
|
4231
|
+
ascli server --url=ssh://_server_address_:33001 ... --ts=@json:'{"token":"Basic abc123"}'
|
|
4055
4232
|
```
|
|
4056
4233
|
|
|
4057
|
-
> Note
|
|
4234
|
+
> **Note:** If you need to use the Aspera public keys, then specify an empty token: `--ts=@json:'{"token":""}'` : Aspera public SSH keys will be used, but the protocol will ignore the empty token.
|
|
4058
4235
|
|
|
4059
4236
|
The value of the `ssh_keys` option can be a single value or an array.
|
|
4060
4237
|
Each value is a **path** to a private key and is expanded (`~` is replaced with the user's home folder).
|
|
@@ -4067,7 +4244,7 @@ ascli server --ssh-keys=@list:,~/.ssh/id_rsa
|
|
|
4067
4244
|
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
|
4068
4245
|
```
|
|
4069
4246
|
|
|
4070
|
-
For
|
|
4247
|
+
For file operation command (browse, delete), the ruby SSH client library `Net::SSH` is used and provides several options settable using option `ssh_options`.
|
|
4071
4248
|
For a list of SSH client options, refer to the ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html).
|
|
4072
4249
|
|
|
4073
4250
|
By default the SSH library expect that a local ssh-agent is running.
|
|
@@ -4086,8 +4263,8 @@ ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant p
|
|
|
4086
4263
|
|
|
4087
4264
|
This means that you don't have such an SSH agent running, then:
|
|
4088
4265
|
|
|
4089
|
-
-
|
|
4090
|
-
-
|
|
4266
|
+
- Check env var: `SSH_AGENT_SOCK`
|
|
4267
|
+
- Check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option)
|
|
4091
4268
|
- [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
|
4092
4269
|
- To disable the use of `ssh-agent`, use the option `ssh_options` like this:
|
|
4093
4270
|
|
|
@@ -4097,7 +4274,17 @@ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
|
|
4097
4274
|
|
|
4098
4275
|
This can also be set as default using a global preset.
|
|
4099
4276
|
|
|
4100
|
-
###
|
|
4277
|
+
### Other session channels for `server`
|
|
4278
|
+
|
|
4279
|
+
URL schemes `local` and `https` are also supported, mainly for testing purpose.
|
|
4280
|
+
(`--url=local:` , `--url=https://...`)
|
|
4281
|
+
|
|
4282
|
+
- `local` will execute `ascmd` locally, instead of using a SSH cnnection.
|
|
4283
|
+
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
|
4284
|
+
|
|
4285
|
+
As, most of the time, SSH is used, if an `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
|
4286
|
+
|
|
4287
|
+
### Examples: `server`
|
|
4101
4288
|
|
|
4102
4289
|
One can test the `server` application using the well known demo server:
|
|
4103
4290
|
|
|
@@ -4109,7 +4296,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
|
4109
4296
|
|
|
4110
4297
|
`initdemo` creates a [option preset](#lprt) `demoserver` and set it as default for plugin `server`.
|
|
4111
4298
|
|
|
4112
|
-
## Plugin: IBM Aspera High Speed Transfer Server
|
|
4299
|
+
## <a id="node"></a>Plugin: `node`: IBM Aspera High Speed Transfer Server Node
|
|
4113
4300
|
|
|
4114
4301
|
This plugin gives access to capabilities provided by HSTS node API.
|
|
4115
4302
|
|
|
@@ -4129,15 +4316,15 @@ For transfers, it is possible to control how transfer is authorized using option
|
|
|
4129
4316
|
|
|
4130
4317
|
```javascript
|
|
4131
4318
|
{
|
|
4132
|
-
"remote_host": address of node url,
|
|
4319
|
+
"remote_host": "<address of node url>",
|
|
4133
4320
|
"remote_user": "xfer",
|
|
4134
4321
|
"ssh_port": 33001,
|
|
4135
4322
|
"token": "Basic <base 64 encoded user/pass>",
|
|
4136
|
-
"direction": send
|
|
4323
|
+
"direction": "[send|receive]"
|
|
4137
4324
|
}
|
|
4138
4325
|
```
|
|
4139
4326
|
|
|
4140
|
-
Note
|
|
4327
|
+
> **Note:** the port is assumed to be the default Aspera SSH port `33001` and transfer user is assumed to be `xfer`.
|
|
4141
4328
|
|
|
4142
4329
|
### Central
|
|
4143
4330
|
|
|
@@ -4224,13 +4411,20 @@ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"myst
|
|
|
4224
4411
|
### Node sample commands
|
|
4225
4412
|
|
|
4226
4413
|
```bash
|
|
4227
|
-
node
|
|
4228
|
-
node
|
|
4229
|
-
node
|
|
4230
|
-
node
|
|
4231
|
-
node
|
|
4232
|
-
node
|
|
4233
|
-
node
|
|
4414
|
+
node access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
|
4415
|
+
node access_key delete aoc_1
|
|
4416
|
+
node access_key do my_aoc_ak_name browse /
|
|
4417
|
+
node access_key do my_aoc_ak_name delete /folder2
|
|
4418
|
+
node access_key do my_aoc_ak_name delete testfile1
|
|
4419
|
+
node access_key do my_aoc_ak_name download testfile1 --to-folder=.
|
|
4420
|
+
node access_key do my_aoc_ak_name file show --path=/testfile1
|
|
4421
|
+
node access_key do my_aoc_ak_name file show 1
|
|
4422
|
+
node access_key do my_aoc_ak_name find /
|
|
4423
|
+
node access_key do my_aoc_ak_name mkdir /folder1
|
|
4424
|
+
node access_key do my_aoc_ak_name node_info /
|
|
4425
|
+
node access_key do my_aoc_ak_name rename /folder1 folder2
|
|
4426
|
+
node access_key do my_aoc_ak_name upload 'faux:///testfile1?1k' --default_ports=no
|
|
4427
|
+
node access_key list
|
|
4234
4428
|
node api_details
|
|
4235
4429
|
node async bandwidth 1
|
|
4236
4430
|
node async counters 1
|
|
@@ -4241,7 +4435,7 @@ node async show ALL
|
|
|
4241
4435
|
node basic_token
|
|
4242
4436
|
node browse / -r
|
|
4243
4437
|
node delete /todelete
|
|
4244
|
-
node delete @list
|
|
4438
|
+
node delete @list:,folder_1/todelete,folder_1/tdlink,folder_1/delfile
|
|
4245
4439
|
node delete folder_1/10MB.1
|
|
4246
4440
|
node delete testfile.bin
|
|
4247
4441
|
node download testfile.bin --to-folder=.
|
|
@@ -4249,11 +4443,14 @@ node download testfile.bin --to-folder=. --token-type=hybrid
|
|
|
4249
4443
|
node health
|
|
4250
4444
|
node info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
|
|
4251
4445
|
node license
|
|
4252
|
-
node mkdir /todelete
|
|
4253
|
-
node mkfile /delfile1 "hello world"
|
|
4254
|
-
node mklink /todelete /tdlink
|
|
4255
|
-
node rename
|
|
4446
|
+
node mkdir folder_1/todelete
|
|
4447
|
+
node mkfile folder_1/delfile1 "hello world"
|
|
4448
|
+
node mklink folder_1/todelete folder_1/tdlink
|
|
4449
|
+
node rename folder_1 delfile1 delfile
|
|
4256
4450
|
node search / --value=@json:'{"sort":"mtime"}'
|
|
4451
|
+
node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
|
4452
|
+
node service delete service1
|
|
4453
|
+
node service list
|
|
4257
4454
|
node space /
|
|
4258
4455
|
node sync bandwidth my_syncid
|
|
4259
4456
|
node sync counters my_syncid
|
|
@@ -4273,7 +4470,7 @@ node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps
|
|
|
4273
4470
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}' --token-type=hybrid
|
|
4274
4471
|
```
|
|
4275
4472
|
|
|
4276
|
-
## Plugin: IBM Aspera
|
|
4473
|
+
## <a id="faspex5"></a>Plugin: `faspex5`: IBM Aspera Faspex v5
|
|
4277
4474
|
|
|
4278
4475
|
IBM Aspera's newer self-managed application.
|
|
4279
4476
|
|
|
@@ -4306,7 +4503,7 @@ Then use these options:
|
|
|
4306
4503
|
--private-key=@file:.../path/to/key.pem
|
|
4307
4504
|
```
|
|
4308
4505
|
|
|
4309
|
-
> The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
|
4506
|
+
> **Note:** The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
|
4310
4507
|
|
|
4311
4508
|
### Faspex 5 web authentication
|
|
4312
4509
|
|
|
@@ -4348,12 +4545,13 @@ Most commands are directly REST API calls.
|
|
|
4348
4545
|
Parameters to commandsa are carried through option `value`, as extended value.
|
|
4349
4546
|
Usually using JSON format with prefix `@json:`.
|
|
4350
4547
|
|
|
4351
|
-
> The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
|
4548
|
+
> **Note:** The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
|
4352
4549
|
|
|
4353
4550
|
```bash
|
|
4354
4551
|
faspex5 admin res accounts list
|
|
4355
4552
|
faspex5 admin res contacts list
|
|
4356
4553
|
faspex5 admin res jobs list
|
|
4554
|
+
faspex5 admin res metadata_profiles list
|
|
4357
4555
|
faspex5 admin res node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
|
4358
4556
|
faspex5 admin res oauth_clients list
|
|
4359
4557
|
faspex5 admin res registrations list
|
|
@@ -4365,6 +4563,9 @@ faspex5 health
|
|
|
4365
4563
|
faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
|
4366
4564
|
faspex5 package receive "my_package_id" --to-folder=. --ts=@json:'{"content_protection_password":"abc123_yo"}'
|
|
4367
4565
|
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin --ts=@json:'{"content_protection_password":"_content_prot_here_"}'
|
|
4566
|
+
faspex5 package show "my_package_id"
|
|
4567
|
+
faspex5 user profile modify @json:'{"preference":{"connect_disabled":false}}'
|
|
4568
|
+
faspex5 user profile show
|
|
4368
4569
|
```
|
|
4369
4570
|
|
|
4370
4571
|
Other examples:
|
|
@@ -4387,7 +4588,7 @@ ascli faspex5 admin res metadata_profiles create --value=@json:'{"name":"the pro
|
|
|
4387
4588
|
ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox","metadata_profile_id":1}'
|
|
4388
4589
|
```
|
|
4389
4590
|
|
|
4390
|
-
## Plugin: IBM Aspera Faspex
|
|
4591
|
+
## <a id="faspex"></a>Plugin: `faspex`: IBM Aspera Faspex v4
|
|
4391
4592
|
|
|
4392
4593
|
Notes:
|
|
4393
4594
|
|
|
@@ -4570,7 +4771,7 @@ faspex v4 wmembership list
|
|
|
4570
4771
|
faspex v4 workgroup list
|
|
4571
4772
|
```
|
|
4572
4773
|
|
|
4573
|
-
## Plugin: IBM Aspera Shares
|
|
4774
|
+
## <a id="shares"></a>Plugin: `shares`: IBM Aspera Shares v1
|
|
4574
4775
|
|
|
4575
4776
|
Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)
|
|
4576
4777
|
|
|
@@ -4578,10 +4779,14 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
|
4578
4779
|
|
|
4579
4780
|
```bash
|
|
4580
4781
|
shares admin share list
|
|
4782
|
+
shares admin share list --fields=-status,status_message
|
|
4581
4783
|
shares admin share user_permissions 1
|
|
4582
4784
|
shares admin user app_authorizations 1
|
|
4785
|
+
shares admin user ldap_import --value=the_name
|
|
4583
4786
|
shares admin user list
|
|
4787
|
+
shares admin user saml_import --value=@json:'{"id":"the_id","name_id":"the_name"}'
|
|
4584
4788
|
shares admin user share_permissions 1
|
|
4789
|
+
shares health
|
|
4585
4790
|
shares repository browse /
|
|
4586
4791
|
shares repository delete my_shares_upload/testfile.bin
|
|
4587
4792
|
shares repository download --to-folder=. my_shares_upload/testfile.bin
|
|
@@ -4590,7 +4795,7 @@ shares repository upload --to-folder=my_shares_upload testfile.bin
|
|
|
4590
4795
|
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
|
4591
4796
|
```
|
|
4592
4797
|
|
|
4593
|
-
## Plugin: Console
|
|
4798
|
+
## <a id="console"></a>Plugin: `console`: IBM Aspera Console
|
|
4594
4799
|
|
|
4595
4800
|
### Console sample commands
|
|
4596
4801
|
|
|
@@ -4601,11 +4806,12 @@ console transfer smart list
|
|
|
4601
4806
|
console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
|
4602
4807
|
```
|
|
4603
4808
|
|
|
4604
|
-
## Plugin: Orchestrator
|
|
4809
|
+
## <a id="orchestrator"></a>Plugin: `orchestrator`:IBM Aspera Orchestrator
|
|
4605
4810
|
|
|
4606
4811
|
### Orchestrator sample commands
|
|
4607
4812
|
|
|
4608
4813
|
```bash
|
|
4814
|
+
orchestrator health
|
|
4609
4815
|
orchestrator info
|
|
4610
4816
|
orchestrator plugins
|
|
4611
4817
|
orchestrator processes
|
|
@@ -4619,13 +4825,15 @@ orchestrator workflow status ALL
|
|
|
4619
4825
|
orchestrator workflow status my_orch_workflow_id
|
|
4620
4826
|
```
|
|
4621
4827
|
|
|
4622
|
-
## Plugin: IBM Cloud Object Storage
|
|
4828
|
+
## <a id="cos"></a>Plugin: `cos`: IBM Cloud Object Storage
|
|
4623
4829
|
|
|
4624
4830
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
|
4625
4831
|
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
|
4626
4832
|
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
|
4627
4833
|
|
|
4628
|
-
There are two possibilities to provide credentials.
|
|
4834
|
+
There are two possibilities to provide credentials.
|
|
4835
|
+
If you already have the endpoint, apikey and CRN, use the first method.
|
|
4836
|
+
If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
|
4629
4837
|
|
|
4630
4838
|
### Using endpoint, apikey and Resource Instance ID (CRN)
|
|
4631
4839
|
|
|
@@ -4647,9 +4855,17 @@ Then, jump to the transfer example.
|
|
|
4647
4855
|
|
|
4648
4856
|
### Using service credential file
|
|
4649
4857
|
|
|
4650
|
-
If you are the COS administrator and don't have yet the credential:
|
|
4858
|
+
If you are the COS administrator and don't have yet the credential:
|
|
4859
|
+
Service credentials are directly created using the IBM cloud Console (web UI).
|
|
4860
|
+
Navigate to:
|
|
4651
4861
|
|
|
4652
|
-
|
|
4862
|
+
- → Navigation Menu
|
|
4863
|
+
- → [Resource List](https://cloud.ibm.com/resources)
|
|
4864
|
+
- → [Storage](https://cloud.ibm.com/objectstorage)
|
|
4865
|
+
- → Select your storage instance
|
|
4866
|
+
- → Service Credentials
|
|
4867
|
+
- → New credentials (Leave default role: Writer, no special options)
|
|
4868
|
+
- → Copy to clipboard
|
|
4653
4869
|
|
|
4654
4870
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
4655
4871
|
|
|
@@ -4710,31 +4926,43 @@ ascli cos node info
|
|
|
4710
4926
|
ascli cos node upload 'faux:///sample1G?1g'
|
|
4711
4927
|
```
|
|
4712
4928
|
|
|
4713
|
-
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
|
4929
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man `ascp` and section above), but you can of course send a real file by specifying a real file instead.
|
|
4714
4930
|
|
|
4715
4931
|
### COS sample commands
|
|
4716
4932
|
|
|
4717
4933
|
```bash
|
|
4718
|
-
cos
|
|
4719
|
-
cos
|
|
4934
|
+
cos --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
|
4935
|
+
cos --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
|
4720
4936
|
cos node access_key show self
|
|
4721
4937
|
cos node download testfile.bin --to-folder=.
|
|
4722
4938
|
cos node info
|
|
4723
4939
|
cos node upload testfile.bin
|
|
4724
4940
|
```
|
|
4725
4941
|
|
|
4726
|
-
## Plugin: IBM Aspera Sync
|
|
4942
|
+
## <a id="async"></a>Plugin: `async`: IBM Aspera Sync
|
|
4727
4943
|
|
|
4728
4944
|
A basic plugin to start an "async" using `ascli`.
|
|
4729
|
-
The main advantage is the possibility to
|
|
4945
|
+
The main advantage over bare `async` command line is the possibility to use a configuration file, using `ascli` standard options.
|
|
4946
|
+
|
|
4947
|
+
Also, the `sync` command is also made available through the `server sync` and `aoc files sync` commands.
|
|
4948
|
+
In this case, some of the `sync` parameters are fill from parameters of the related plugin.
|
|
4949
|
+
|
|
4950
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
|
4951
|
+
>
|
|
4952
|
+
> **Note:** Two JSON syntax are supported for option `sync_info`.
|
|
4953
|
+
> The first is same sync payload as specified on the `async` option `--conf` or in the latest node API, this is the prefered syntax and allows a single session definition.
|
|
4954
|
+
> The second (legacy) is specific to `ascli` and allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
|
4955
|
+
|
|
4956
|
+
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
|
4730
4957
|
|
|
4731
4958
|
### Sync sample commands
|
|
4732
4959
|
|
|
4733
4960
|
```bash
|
|
4734
|
-
sync
|
|
4961
|
+
sync admin status --sync-info=@json:'{"sessions":[{"name":"test","local_dir":"contents"}]}'
|
|
4962
|
+
sync start --sync-info=@json:'{"instance":{"quiet":true},"sessions":[{"name":"test","reset":true,"remote_dir":"/sync_test","local_dir":"contents","host":"my_remote_host","tcp_port":33001,"user":"my_remote_user","private_key_paths":["my_local_user_key"]}]}'
|
|
4735
4963
|
```
|
|
4736
4964
|
|
|
4737
|
-
## Plugin: Preview
|
|
4965
|
+
## <a id="preview"></a>Plugin: `preview`: Preview generator for AoC
|
|
4738
4966
|
|
|
4739
4967
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
|
4740
4968
|
It uses the **node API** of Aspera HSTS and requires use of Access Keys and it's **storage root**.
|
|
@@ -4846,7 +5074,7 @@ The preview generator is run as a user, preferably a regular user (not root). Wh
|
|
|
4846
5074
|
|
|
4847
5075
|
Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). The configuration file must be created with the same user used to run so that it is properly used on runtime.
|
|
4848
5076
|
|
|
4849
|
-
|
|
5077
|
+
The `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
|
4850
5078
|
|
|
4851
5079
|
```bash
|
|
4852
5080
|
su -s /bin/bash - xfer
|
|
@@ -4869,11 +5097,11 @@ This shall list the contents of the storage root of the access key.
|
|
|
4869
5097
|
|
|
4870
5098
|
### Execution
|
|
4871
5099
|
|
|
4872
|
-
The tool intentionally supports only a
|
|
4873
|
-
It needs to be run on a regular basis to create or update preview files.
|
|
4874
|
-
|
|
5100
|
+
The tool intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
|
5101
|
+
It needs to be run on a regular basis to create or update preview files.
|
|
5102
|
+
For that use your best reliable scheduler, see [Scheduling](#scheduling).
|
|
4875
5103
|
|
|
4876
|
-
Typically, for
|
|
5104
|
+
Typically, for **Access key** access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
|
|
4877
5105
|
|
|
4878
5106
|
Lets do a one shot test, using the configuration previously created:
|
|
4879
5107
|
|
|
@@ -4889,12 +5117,12 @@ On subsequent run it reads this file and check that previews are generated for t
|
|
|
4889
5117
|
|
|
4890
5118
|
### Configuration for Execution in scheduler
|
|
4891
5119
|
|
|
4892
|
-
Here is an example of configuration for use with cron on Linux.
|
|
4893
|
-
Adapt the scripts to your own
|
|
5120
|
+
Here is an example of configuration for use with `cron` on Linux.
|
|
5121
|
+
Adapt the scripts to your own needs.
|
|
4894
5122
|
|
|
4895
5123
|
We assume here that a configuration preset was created as shown previously.
|
|
4896
5124
|
|
|
4897
|
-
Lets first setup a script that will be used in the scheduler and
|
|
5125
|
+
Lets first setup a script that will be used in the scheduler and set up the environment.
|
|
4898
5126
|
|
|
4899
5127
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
|
4900
5128
|
|
|
@@ -4916,7 +5144,7 @@ crontab<<EOF
|
|
|
4916
5144
|
EOF
|
|
4917
5145
|
```
|
|
4918
5146
|
|
|
4919
|
-
Note
|
|
5147
|
+
> **Note:** The logging options are kept here in the cronfile instead of conf file to allow execution on command line with output on command line.
|
|
4920
5148
|
|
|
4921
5149
|
### Candidate detection for creation or update (or deletion)
|
|
4922
5150
|
|
|
@@ -4997,12 +5225,12 @@ If the `mimemagic` gem complains about missing mime info file:
|
|
|
4997
5225
|
- any OS:
|
|
4998
5226
|
|
|
4999
5227
|
- Examine the error message
|
|
5000
|
-
- Download the file:
|
|
5228
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
|
5001
5229
|
- move and rename this file to one of the locations expected by mimemagic as specified in the error message
|
|
5002
5230
|
|
|
5003
5231
|
- Windows:
|
|
5004
5232
|
|
|
5005
|
-
- Download the file:
|
|
5233
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
|
5006
5234
|
- Place this file in the root of Ruby (or elsewhere): `C:\RubyVV-x64\freedesktop.org.xml.in`
|
|
5007
5235
|
- Set a global variable using `SystemPropertiesAdvanced.exe` or using `cmd` (replace `VV` with version) to the exact path of this file:
|
|
5008
5236
|
|
|
@@ -5057,17 +5285,17 @@ preview trevents --once-only=yes --skip-types=office --log-level=info
|
|
|
5057
5285
|
Aspera CLI can send email, for that setup SMTP configuration. This is done with option `smtp`.
|
|
5058
5286
|
|
|
5059
5287
|
The `smtp` option is a hash table (extended value) with the following fields:
|
|
5060
|
-
|
|
5061
|
-
|
|
5062
|
-
|
|
5063
|
-
|
|
5064
|
-
|
|
5065
|
-
<
|
|
5066
|
-
|
|
5067
|
-
|
|
5068
|
-
|
|
5069
|
-
|
|
5070
|
-
|
|
5288
|
+
|
|
5289
|
+
| field | default | example | description |
|
|
5290
|
+
|--------------|---------------------|----------------------------|----------------------------------|
|
|
5291
|
+
| `server` | - | smtp.gmail.com | SMTP server address |
|
|
5292
|
+
| `tls` | true | false | use of TLS |
|
|
5293
|
+
| `port` | TLS: 587<br/>25 | 587 | port for service |
|
|
5294
|
+
| `domain` | domain of server | gmail.com | email domain of user |
|
|
5295
|
+
| `username` | - | john@example.com | user to authenticate on SMTP server, leave empty for open auth. |
|
|
5296
|
+
| `password` | - | my_password_here | password for above username |
|
|
5297
|
+
| `from_email` | username if defined | johnny@example.com | address used if receiver replies |
|
|
5298
|
+
| `from_name` | same as email | John Wayne | display name of sender |
|
|
5071
5299
|
|
|
5072
5300
|
### Example of configuration
|
|
5073
5301
|
|
|
@@ -5104,9 +5332,9 @@ The template is the full SMTP message, including headers.
|
|
|
5104
5332
|
|
|
5105
5333
|
The following variables are defined by default:
|
|
5106
5334
|
|
|
5107
|
-
- from_name
|
|
5108
|
-
- from_email
|
|
5109
|
-
- to
|
|
5335
|
+
- `from_name`
|
|
5336
|
+
- `from_email`
|
|
5337
|
+
- `to`
|
|
5110
5338
|
|
|
5111
5339
|
Other variables are defined depending on context.
|
|
5112
5340
|
|
|
@@ -5162,26 +5390,26 @@ The tool expect one single argument: a [*transfer-spec*](#transferspec).
|
|
|
5162
5390
|
|
|
5163
5391
|
If no argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [*transfer-spec*](#transferspec) on stdin.
|
|
5164
5392
|
|
|
5165
|
-
Note
|
|
5393
|
+
> **Note:** If JSON is the format, specify `@json:` to tell `ascli` to decode the hash using JSON syntax.
|
|
5166
5394
|
|
|
5167
5395
|
During execution, it generates all low level events, one per line, in JSON format on stdout.
|
|
5168
5396
|
|
|
5169
|
-
|
|
5397
|
+
There are special "extended" [*transfer-spec*](#transferspec) parameters supported by `asession`:
|
|
5170
5398
|
|
|
5171
5399
|
- `EX_loglevel` to change log level of the tool
|
|
5172
5400
|
- `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
|
5173
5401
|
|
|
5174
|
-
Note
|
|
5402
|
+
> **Note:** In addition, many "EX_" [*transfer-spec*](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [*transfer-spec*](#transferspec).
|
|
5175
5403
|
|
|
5176
5404
|
### Comparison of interfaces
|
|
5177
5405
|
|
|
5178
|
-
|
|
5179
|
-
|
|
5180
|
-
|
|
5181
|
-
|
|
5182
|
-
|
|
5183
|
-
|
|
5184
|
-
|
|
5406
|
+
| feature/tool | asession | `ascp` | FaspManager | Transfer SDK |
|
|
5407
|
+
|--------------|----------|--------|-------------|--------------|
|
|
5408
|
+
| language integration | any | any | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | many |
|
|
5409
|
+
| required additional components to `ascp` | Ruby<br/>Aspera | - | library<br/>(headers) | daemon |
|
|
5410
|
+
| startup | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) | command line arguments | API | daemon |
|
|
5411
|
+
| events | JSON on stdout | none by default<br/>or need to open management port<br/>and proprietary text syntax | callback | callback |
|
|
5412
|
+
| platforms | any with ruby and `ascp` | any with `ascp` (and SDK if compiled) | any with `ascp` | any with `ascp` and transfer daemon |
|
|
5185
5413
|
|
|
5186
5414
|
### Simple session
|
|
5187
5415
|
|
|
@@ -5193,13 +5421,13 @@ Create a file `session.json` with:
|
|
|
5193
5421
|
|
|
5194
5422
|
Then start the session:
|
|
5195
5423
|
|
|
5196
|
-
```
|
|
5424
|
+
```bash
|
|
5197
5425
|
asession < session.json
|
|
5198
5426
|
```
|
|
5199
5427
|
|
|
5200
5428
|
### Asynchronous commands and Persistent session
|
|
5201
5429
|
|
|
5202
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
|
5430
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
|
5203
5431
|
|
|
5204
5432
|
This is particularly useful for a persistent session ( with the [*transfer-spec*](#transferspec) parameter: `"keepalive":true` )
|
|
5205
5433
|
|
|
@@ -5266,9 +5494,9 @@ The general idea is to rely on :
|
|
|
5266
5494
|
- take advantage of `ascli` configuration capabilities and server side knowledge
|
|
5267
5495
|
- the OS scheduler for reliability and continuous operation
|
|
5268
5496
|
|
|
5269
|
-
#### ascp features
|
|
5497
|
+
#### `ascp` features
|
|
5270
5498
|
|
|
5271
|
-
Interesting ascp features are found in its arguments: (see ascp manual):
|
|
5499
|
+
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
|
5272
5500
|
|
|
5273
5501
|
- `ascp` already takes care of sending only "new" files: option `-k 1,2,3` (`resume_policy`)
|
|
5274
5502
|
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
|
@@ -5279,14 +5507,15 @@ Note that:
|
|
|
5279
5507
|
|
|
5280
5508
|
- `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `--ts` parameter.
|
|
5281
5509
|
- most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters
|
|
5282
|
-
- native ascp arguments can be provided with the [*transfer-spec*](#transferspec) parameter: `EX_ascp_args` (array), only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node)
|
|
5510
|
+
- native `ascp` arguments can be provided with the [*transfer-spec*](#transferspec) parameter: `EX_ascp_args` (array), only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node)
|
|
5283
5511
|
|
|
5284
5512
|
#### server side and configuration
|
|
5285
5513
|
|
|
5286
5514
|
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
|
5287
|
-
> file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
|
5288
5515
|
|
|
5289
|
-
>
|
|
5516
|
+
> **Note:** file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
|
5517
|
+
>
|
|
5518
|
+
> **Note:** parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
|
5290
5519
|
|
|
5291
5520
|
#### Scheduling
|
|
5292
5521
|
|
|
@@ -5295,7 +5524,7 @@ Refer to section [Scheduling](#scheduling). (on use of option `lock_port`)
|
|
|
5295
5524
|
|
|
5296
5525
|
### Example: upload hot folder
|
|
5297
5526
|
|
|
5298
|
-
```
|
|
5527
|
+
```bash
|
|
5299
5528
|
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
|
5300
5529
|
```
|
|
5301
5530
|
|
|
@@ -5304,11 +5533,25 @@ Source files are deleted after transfer.
|
|
|
5304
5533
|
Growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period).
|
|
5305
5534
|
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
|
5306
5535
|
|
|
5307
|
-
### Example: unidirectional synchronization
|
|
5536
|
+
### Example: unidirectional synchronization (upload) to server
|
|
5308
5537
|
|
|
5309
|
-
```
|
|
5310
|
-
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than:-8,"src_base":"source_sync"}'
|
|
5538
|
+
```bash
|
|
5539
|
+
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
|
|
5540
|
+
```
|
|
5541
|
+
|
|
5542
|
+
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node:
|
|
5543
|
+
|
|
5544
|
+
### Example: unidirectional synchronization (download) from Aspera on Cloud Files
|
|
5545
|
+
|
|
5546
|
+
```bash
|
|
5547
|
+
ascli aoc files download . --to-folder=. --lock-port=12345 --progress=none --display=data \
|
|
5548
|
+
--ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
|
|
5311
5549
|
```
|
|
5550
|
+
|
|
5551
|
+
> **Note:** option `delete_before_transfer` will delete files locally, if they are not present on remote side.
|
|
5552
|
+
>
|
|
5553
|
+
> **Note:** options `progress` and `display` limit output for headless operation (e.g. cron job)
|
|
5554
|
+
|
|
5312
5555
|
## Health check and Nagios
|
|
5313
5556
|
|
|
5314
5557
|
Most plugin provide a `health` command that will check the health status of the application. Example:
|
|
@@ -5384,11 +5627,11 @@ aoc.read('self')
|
|
|
5384
5627
|
|
|
5385
5628
|
## Changes (Release notes)
|
|
5386
5629
|
|
|
5387
|
-
See
|
|
5630
|
+
See [CHANGELOG.md](CHANGELOG.md)
|
|
5388
5631
|
|
|
5389
5632
|
## History
|
|
5390
5633
|
|
|
5391
|
-
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
|
5634
|
+
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (`ascp`), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
|
5392
5635
|
|
|
5393
5636
|
There were a few pitfalls:
|
|
5394
5637
|
|
|
@@ -5399,7 +5642,7 @@ So, it evolved into `ascli`:
|
|
|
5399
5642
|
|
|
5400
5643
|
- portable: works on platforms supporting `ruby` (and `ascp`)
|
|
5401
5644
|
- easy to install with the `gem` utility
|
|
5402
|
-
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from ascp command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
|
5645
|
+
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from `ascp` command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
|
5403
5646
|
- `ruby` is consistent with other Aspera products
|
|
5404
5647
|
|
|
5405
5648
|
## Common problems
|