aspera-cli 4.15.0 → 4.16.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +29 -3
- data/CHANGELOG.md +292 -228
- data/CONTRIBUTING.md +69 -18
- data/README.md +1102 -952
- data/bin/ascli +13 -31
- data/bin/asession +3 -1
- data/examples/dascli +2 -2
- data/lib/aspera/aoc.rb +28 -33
- data/lib/aspera/ascmd.rb +3 -6
- data/lib/aspera/assert.rb +45 -0
- data/lib/aspera/cli/extended_value.rb +5 -5
- data/lib/aspera/cli/formatter.rb +26 -13
- data/lib/aspera/cli/hints.rb +4 -3
- data/lib/aspera/cli/main.rb +16 -3
- data/lib/aspera/cli/manager.rb +45 -36
- data/lib/aspera/cli/plugin.rb +20 -13
- data/lib/aspera/cli/plugins/aoc.rb +103 -73
- data/lib/aspera/cli/plugins/ats.rb +4 -3
- data/lib/aspera/cli/plugins/config.rb +114 -119
- data/lib/aspera/cli/plugins/cos.rb +2 -2
- data/lib/aspera/cli/plugins/faspex.rb +23 -19
- data/lib/aspera/cli/plugins/faspex5.rb +75 -43
- data/lib/aspera/cli/plugins/node.rb +28 -15
- data/lib/aspera/cli/plugins/orchestrator.rb +4 -2
- data/lib/aspera/cli/plugins/preview.rb +9 -7
- data/lib/aspera/cli/plugins/server.rb +6 -3
- data/lib/aspera/cli/plugins/shares.rb +30 -26
- data/lib/aspera/cli/sync_actions.rb +9 -9
- data/lib/aspera/cli/transfer_agent.rb +21 -14
- data/lib/aspera/cli/transfer_progress.rb +2 -3
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +13 -11
- data/lib/aspera/cos_node.rb +3 -2
- data/lib/aspera/coverage.rb +22 -0
- data/lib/aspera/data_repository.rb +33 -2
- data/lib/aspera/environment.rb +4 -2
- data/lib/aspera/fasp/{agent_aspera.rb → agent_alpha.rb} +29 -39
- data/lib/aspera/fasp/agent_base.rb +17 -7
- data/lib/aspera/fasp/agent_direct.rb +88 -84
- data/lib/aspera/fasp/agent_httpgw.rb +4 -3
- data/lib/aspera/fasp/agent_node.rb +3 -2
- data/lib/aspera/fasp/agent_trsdk.rb +79 -37
- data/lib/aspera/fasp/installation.rb +51 -12
- data/lib/aspera/fasp/management.rb +11 -6
- data/lib/aspera/fasp/parameters.rb +53 -47
- data/lib/aspera/fasp/resume_policy.rb +7 -5
- data/lib/aspera/fasp/sync.rb +273 -0
- data/lib/aspera/fasp/transfer_spec.rb +10 -8
- data/lib/aspera/fasp/uri.rb +2 -2
- data/lib/aspera/faspex_gw.rb +11 -8
- data/lib/aspera/faspex_postproc.rb +6 -5
- data/lib/aspera/id_generator.rb +3 -1
- data/lib/aspera/json_rpc.rb +10 -8
- data/lib/aspera/keychain/encrypted_hash.rb +46 -11
- data/lib/aspera/keychain/macos_security.rb +15 -13
- data/lib/aspera/log.rb +4 -3
- data/lib/aspera/nagios.rb +7 -2
- data/lib/aspera/node.rb +17 -16
- data/lib/aspera/node_simulator.rb +214 -0
- data/lib/aspera/oauth.rb +22 -19
- data/lib/aspera/persistency_action_once.rb +13 -14
- data/lib/aspera/persistency_folder.rb +3 -2
- data/lib/aspera/preview/file_types.rb +53 -267
- data/lib/aspera/preview/generator.rb +7 -5
- data/lib/aspera/preview/terminal.rb +14 -5
- data/lib/aspera/preview/utils.rb +8 -7
- data/lib/aspera/proxy_auto_config.rb +6 -3
- data/lib/aspera/rest.rb +29 -13
- data/lib/aspera/rest_error_analyzer.rb +1 -0
- data/lib/aspera/rest_errors_aspera.rb +2 -0
- data/lib/aspera/secret_hider.rb +5 -2
- data/lib/aspera/ssh.rb +10 -8
- data/lib/aspera/temp_file_manager.rb +1 -1
- data/lib/aspera/web_server_simple.rb +2 -1
- data.tar.gz.sig +0 -0
- metadata +96 -45
- metadata.gz.sig +0 -0
- data/lib/aspera/sync.rb +0 -219
data/README.md
CHANGED
@@ -10,9 +10,9 @@
|
|
10
10
|
|
11
11
|
## Introduction
|
12
12
|
|
13
|
-
Version : 4.
|
13
|
+
Version : 4.16.0
|
14
14
|
|
15
|
-
Laurent/2016-
|
15
|
+
Laurent/2016-2024
|
16
16
|
|
17
17
|
This gem provides the `ascli` CLI (Command Line Interface) to IBM Aspera software.
|
18
18
|
|
@@ -44,16 +44,16 @@ Refer to [BUGS.md](BUGS.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
44
44
|
- Execute commands remotely on Aspera products
|
45
45
|
- Transfer to/from Aspera products
|
46
46
|
|
47
|
-
|
47
|
+
It is designed for:
|
48
48
|
|
49
49
|
- Interactive operations on a text terminal (typically, VT100 compatible), e.g. for maintenance
|
50
50
|
- Scripting, e.g. batch operations in (shell) scripts (e.g. cron job)
|
51
51
|
|
52
52
|
`ascli` can be seen as a command line tool integrating:
|
53
53
|
|
54
|
-
- A configuration file (config.yaml)
|
55
|
-
- Advanced command line options
|
56
|
-
-
|
54
|
+
- A configuration file (`config.yaml`)
|
55
|
+
- Advanced command line options ([Extended Value](#extended))
|
56
|
+
- `curl` (for REST calls)
|
57
57
|
- Aspera transfer (`ascp`)
|
58
58
|
|
59
59
|
If the need is to perform operations programmatically in languages such as: C, Go, Python, nodejs, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
@@ -62,22 +62,23 @@ If the need is to perform operations programmatically in languages such as: C, G
|
|
62
62
|
- Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, Ruby, etc...)
|
63
63
|
|
64
64
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
65
|
+
Code examples here: <https://github.com/laurent-martin/aspera-api-examples>
|
65
66
|
|
66
67
|
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
67
68
|
|
68
69
|
### Notations, Shell, Examples
|
69
70
|
|
70
|
-
Command line operations examples are shown using a shell such: `bash` or `zsh`.
|
71
|
+
Command line operations examples are shown using a shell such as: `bash` or `zsh`.
|
71
72
|
|
72
|
-
Command line parameters in examples beginning with `my_`,
|
73
|
+
Command line parameters in examples beginning with `my_`, e.g. `my_param_value`, are user-provided value and not fixed value commands.
|
73
74
|
|
74
75
|
`ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, etc...)
|
75
76
|
|
76
77
|
Some commands will start an Aspera-based transfer (e.g. `upload`).
|
77
|
-
The transfer is not directly implemented in `ascli`, rather `ascli` uses
|
78
|
+
The transfer is not directly implemented in `ascli`, rather `ascli` uses one of the external Aspera Transfer Clients called **[Transfer Agents](#agents)**.
|
78
79
|
|
79
|
-
> **Note:**
|
80
|
-
The
|
80
|
+
> **Note:** A **[Transfer Agents](#agents)** is a client for the remote Transfer Server (HSTS).
|
81
|
+
The **[Transfer Agents](#agents)** may be local or remote...
|
81
82
|
For example a remote Aspera Server may be used as a transfer agent (using node API).
|
82
83
|
i.e. using option `--transfer=node`
|
83
84
|
|
@@ -85,23 +86,20 @@ i.e. using option `--transfer=node`
|
|
85
86
|
|
86
87
|
This section guides you from installation, first use and advanced use.
|
87
88
|
|
88
|
-
First, follow
|
89
|
+
First, follow section: [Installation](#installation) (Ruby, Gem, FASP) to start using `ascli`.
|
89
90
|
|
90
91
|
Once the gem is installed, `ascli` shall be accessible:
|
91
92
|
|
92
|
-
```
|
93
|
-
ascli --version
|
94
|
-
|
95
|
-
|
96
|
-
```bash
|
97
|
-
4.15.0
|
93
|
+
```console
|
94
|
+
$ ascli --version
|
95
|
+
4.16.0
|
98
96
|
```
|
99
97
|
|
100
98
|
### First use
|
101
99
|
|
102
100
|
Once installation is completed, you can proceed to the first use with a demo server:
|
103
101
|
|
104
|
-
If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
102
|
+
If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard).
|
105
103
|
|
106
104
|
To test with Aspera demo transfer server, setup the environment and then test:
|
107
105
|
|
@@ -124,19 +122,21 @@ ascli server browse /
|
|
124
122
|
+------------+--------+-----------+-------+---------------------------+-----------------------+
|
125
123
|
```
|
126
124
|
|
127
|
-
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options.
|
125
|
+
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options.
|
126
|
+
The following example will:
|
128
127
|
|
129
|
-
-
|
130
|
-
-
|
131
|
-
-
|
132
|
-
-
|
128
|
+
- Create a [option preset](#lprt)
|
129
|
+
- Define it as default for the `server` plugin
|
130
|
+
- List files in a folder
|
131
|
+
- Download a file
|
133
132
|
|
134
133
|
```bash
|
135
134
|
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here
|
136
135
|
```
|
137
136
|
|
138
137
|
```output
|
139
|
-
|
138
|
+
Updated: myserver
|
139
|
+
Saving config file.
|
140
140
|
```
|
141
141
|
|
142
142
|
```bash
|
@@ -144,7 +144,8 @@ ascli config preset set default server myserver
|
|
144
144
|
```
|
145
145
|
|
146
146
|
```output
|
147
|
-
|
147
|
+
Updated: default: server <- myserver
|
148
|
+
Saving config file.
|
148
149
|
```
|
149
150
|
|
150
151
|
```bash
|
@@ -182,7 +183,7 @@ Then, follow the section relative to the product you want to interact with ( Asp
|
|
182
183
|
|
183
184
|
## <a id="installation"></a>Installation
|
184
185
|
|
185
|
-
It is possible to install **either** directly on the host operating system (Linux, macOS, Windows) or as a container (`docker`).
|
186
|
+
It is possible to install **either** directly on the host operating system (Linux, macOS, Windows) or as a [container](#container) (`docker`, `podman`, `singularity`).
|
186
187
|
|
187
188
|
The direct installation is recommended and consists in installing:
|
188
189
|
|
@@ -196,215 +197,16 @@ Ruby version: >= 2.6.
|
|
196
197
|
|
197
198
|
The following sections provide information on the various installation methods.
|
198
199
|
|
199
|
-
An internet connection is required for the installation.
|
200
|
-
|
201
|
-
### Container
|
202
|
-
|
203
|
-
The container image is: [`martinlaurent/ascli`](https://hub.docker.com/r/martinlaurent/ascli).
|
204
|
-
The container contains: Ruby, `ascli` and the Aspera Transfer SDK.
|
205
|
-
To use the container, ensure that you have `podman` (or `docker`) installed.
|
206
|
-
|
207
|
-
```bash
|
208
|
-
podman --version
|
209
|
-
```
|
210
|
-
|
211
|
-
#### Container: Quick start
|
212
|
-
|
213
|
-
**Wanna start quickly ?** With an interactive shell ? Execute this:
|
214
|
-
|
215
|
-
```bash
|
216
|
-
podman run --tty --interactive --entrypoint bash martinlaurent/ascli:latest
|
217
|
-
```
|
218
|
-
|
219
|
-
Then, execute individual `ascli` commands such as:
|
220
|
-
|
221
|
-
```bash
|
222
|
-
ascli conf init
|
223
|
-
ascli conf preset overview
|
224
|
-
ascli conf ascp info
|
225
|
-
ascli server ls /
|
226
|
-
```
|
227
|
-
|
228
|
-
That is simple, but there are limitations:
|
229
|
-
|
230
|
-
- Everything happens in the container
|
231
|
-
- Any generated file in the container will be lost on container (shell) exit. Including configuration files and downloaded files.
|
232
|
-
- No possibility to upload files located on the host system
|
233
|
-
|
234
|
-
#### Container: Details
|
235
|
-
|
236
|
-
The container image is built from this [Dockerfile](Dockerfile.tmpl.erb): the entry point is `ascli` and the default command is `help`.
|
237
|
-
|
238
|
-
If you want to run the image with a shell, execute with option: `--entrypoint bash`, and give argument `-l` (`bash` login option to override the `help` default argument)
|
239
|
-
|
240
|
-
The container can also be executed for individual commands like this: (add `ascli` commands and options at the end of the command line, e.g. `-v` to display the version)
|
241
|
-
|
242
|
-
```bash
|
243
|
-
podman run --rm --tty --interactive martinlaurent/ascli:latest
|
244
|
-
```
|
245
|
-
|
246
|
-
For more convenience, you may define a shell alias:
|
247
|
-
|
248
|
-
```bash
|
249
|
-
alias ascli='podman run --rm --tty --interactive martinlaurent/ascli:latest'
|
250
|
-
```
|
251
|
-
|
252
|
-
Then, you can execute the container like a local command:
|
253
|
-
|
254
|
-
```bash
|
255
|
-
ascli -v
|
256
|
-
```
|
257
|
-
|
258
|
-
```text
|
259
|
-
4.15.0
|
260
|
-
```
|
261
|
-
|
262
|
-
In order to keep persistency of configuration on the host,
|
263
|
-
you should specify your user's config folder as a volume for the container.
|
264
|
-
To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`).
|
265
|
-
Add options:
|
266
|
-
|
267
|
-
```bash
|
268
|
-
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
269
|
-
```
|
270
|
-
|
271
|
-
> **Note:** if you are using a `podman machine`, e.g. on macOS , make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
272
|
-
|
273
|
-
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
274
|
-
you can change the entry point, add option:
|
275
|
-
|
276
|
-
```bash
|
277
|
-
--entrypoint bash
|
278
|
-
```
|
279
|
-
|
280
|
-
You may also probably want that files downloaded in the container are directed to the host.
|
281
|
-
In this case you need also to specify the shared transfer folder as a volume:
|
282
|
-
|
283
|
-
```bash
|
284
|
-
--volume $HOME/xferdir:/xferfiles
|
285
|
-
```
|
286
|
-
|
287
|
-
> **Note:** ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
288
|
-
|
289
|
-
And if you want all the above, simply use all the options:
|
290
|
-
|
291
|
-
```bash
|
292
|
-
alias asclish="podman run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash martinlaurent/ascli:latest"
|
293
|
-
```
|
294
|
-
|
295
|
-
```bash
|
296
|
-
export xferdir=$HOME/xferdir
|
297
|
-
mkdir -p $xferdir
|
298
|
-
chmod -R 777 $xferdir
|
299
|
-
mkdir -p $HOME/.aspera/ascli
|
300
|
-
asclish
|
301
|
-
```
|
302
|
-
|
303
|
-
#### Container: Sample start script
|
304
|
-
|
305
|
-
A convenience sample script is also provided: download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
306
|
-
|
307
|
-
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
308
|
-
|
309
|
-
Some environment variables can be set for this script to adapt its behavior:
|
310
|
-
|
311
|
-
| env var | description | default | example |
|
312
|
-
|----------------|------------------------------------|--------------------------|--------------------------|
|
313
|
-
| `ASCLI_HOME` | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascli_config` |
|
314
|
-
| `docker_args` | additional options to `podman` | <empty> | `--volume /Users:/Users` |
|
315
|
-
| `image` | container image name | `martinlaurent/ascli` | |
|
316
|
-
| `version` | container image version | latest | `4.8.0.pre` |
|
317
|
-
|
318
|
-
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
319
|
-
(value expected in the container).
|
320
|
-
This allows having persistent configuration on the host.
|
321
|
-
|
322
|
-
To add local storage as a volume, you can use the env var `docker_args`:
|
323
|
-
|
324
|
-
Example of use:
|
325
|
-
|
326
|
-
```bash
|
327
|
-
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
328
|
-
chmod a+x ascli
|
329
|
-
export xferdir=$HOME/xferdir
|
330
|
-
mkdir -p $xferdir
|
331
|
-
chmod -R 777 $xferdir
|
332
|
-
export docker_args="--volume $xferdir:/xferfiles"
|
333
|
-
|
334
|
-
./ascli conf init
|
335
|
-
|
336
|
-
echo 'Local file to transfer' > $xferdir/samplefile.txt
|
337
|
-
./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload
|
338
|
-
```
|
339
|
-
|
340
|
-
> **Note:** The local file (`samplefile.txt`) is specified relative to storage view from container (`/xferfiles`) mapped to the host folder `$HOME/xferdir`
|
341
|
-
>
|
342
|
-
> **Note:** Do not use too many volumes, as the legacy `aufs` limits the number. (anyway, prefer to use `overlay2`)
|
343
|
-
|
344
|
-
#### Container: Offline installation
|
345
|
-
|
346
|
-
- First create the image archive:
|
347
|
-
|
348
|
-
```bash
|
349
|
-
podman pull martinlaurent/ascli
|
350
|
-
podman save martinlaurent/ascli|gzip>ascli_image_latest.tar.gz
|
351
|
-
```
|
352
|
-
|
353
|
-
- Then, on air-gapped system:
|
354
|
-
|
355
|
-
```bash
|
356
|
-
podman load -i ascli_image_latest.tar.gz
|
357
|
-
```
|
358
|
-
|
359
|
-
#### Container: `aspera.conf`
|
360
|
-
|
361
|
-
`ascp`'s configuration file `aspera.conf` is located in the container at: `/aspera_sdk/aspera.conf` (see Dockerfile).
|
362
|
-
As the container is immutable, it is not recommended to modify this file.
|
363
|
-
If one wants to change the content, it is possible to tell `ascp` to use another file using `ascp` option `-f`, e.g. by locating it on the host folder `$HOME/.aspera/ascli` mapped to the container folder `/home/cliuser/.aspera/ascli`:
|
364
|
-
|
365
|
-
```bash
|
366
|
-
echo '<CONF/>' > $HOME/.aspera/ascli/aspera.conf
|
367
|
-
```
|
200
|
+
An internet connection is required for the installation.
|
201
|
+
If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
368
202
|
|
369
|
-
|
370
|
-
|
371
|
-
```bash
|
372
|
-
--transfer-info=@json:'{"ascp_args":["-f","/home/cliuser/.aspera/ascli/aspera.conf"]}'
|
373
|
-
```
|
374
|
-
|
375
|
-
#### Container: Singularity
|
376
|
-
|
377
|
-
Singularity is another type of use of container.
|
378
|
-
|
379
|
-
On Linux install:
|
380
|
-
|
381
|
-
```bash
|
382
|
-
dnf install singularity-ce
|
383
|
-
```
|
384
|
-
|
385
|
-
Build an image like this:
|
386
|
-
|
387
|
-
```bash
|
388
|
-
singularity build ascli.sif docker://martinlaurent/ascli
|
389
|
-
```
|
390
|
-
|
391
|
-
Then, start `ascli` like this:
|
392
|
-
|
393
|
-
```bash
|
394
|
-
singularity run ascli.sif
|
395
|
-
```
|
396
|
-
|
397
|
-
Or get a shell with access to `ascli` like this:
|
398
|
-
|
399
|
-
```bash
|
400
|
-
singularity shell ascli.sif
|
401
|
-
```
|
203
|
+
A package with pre-installed Ruby, gem and ascp may also be provided.
|
402
204
|
|
403
205
|
### <a id="ruby"></a>Ruby
|
404
206
|
|
405
|
-
Use this method to install on the native host.
|
207
|
+
Use this method to install on the native host (e.g. your Windows, macOS or Linux system).
|
406
208
|
|
407
|
-
A Ruby interpreter is required to run `ascli
|
209
|
+
A Ruby interpreter is required to run `ascli`.
|
408
210
|
|
409
211
|
Required Ruby version: >= 2.6.
|
410
212
|
|
@@ -412,24 +214,20 @@ Required Ruby version: >= 2.6.
|
|
412
214
|
|
413
215
|
**Ruby can be installed using any method** : rpm, yum, dnf, rvm, brew, Windows installer, ... .
|
414
216
|
|
415
|
-
In priority
|
217
|
+
**In priority**, refer to the official Ruby documentation:
|
416
218
|
|
417
219
|
- [Download Ruby](https://www.ruby-lang.org/en/downloads/)
|
418
220
|
- [Installation Guide](https://www.ruby-lang.org/en/documentation/installation/)
|
419
221
|
|
420
|
-
|
421
|
-
|
422
|
-
The recommended installation method is `rvm` for Unix-like systems (Linux, AIX, macOS, Windows with cygwin, etc...).
|
423
|
-
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
424
|
-
If you have a simpler better way to install Ruby : use it !
|
222
|
+
For convenience, you may refer to the following sections for a proposed method for specific operating systems.
|
425
223
|
|
426
|
-
####
|
224
|
+
#### Unix-like: RVM: Single user installation (not root)
|
427
225
|
|
428
|
-
|
226
|
+
Install `rvm`.
|
227
|
+
Follow [https://rvm.io/](https://rvm.io/).
|
429
228
|
|
430
|
-
|
431
|
-
|
432
|
-
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
229
|
+
Execute the shell/curl command.
|
230
|
+
As regular user, it installs in the user's home: `~/.rvm` .
|
433
231
|
|
434
232
|
```bash
|
435
233
|
\curl -sSL https://get.rvm.io | bash -s stable
|
@@ -437,7 +235,7 @@ Execute the shell/curl command. As regular user, it install in the user's home:
|
|
437
235
|
|
438
236
|
Follow on-screen instructions to install keys, and then re-execute the command.
|
439
237
|
|
440
|
-
|
238
|
+
Upon RVM installation, open a new terminal or initialize with:
|
441
239
|
|
442
240
|
```bash
|
443
241
|
source ~/.rvm/scripts/rvm
|
@@ -457,19 +255,16 @@ rvm install 3.2.2
|
|
457
255
|
|
458
256
|
Ruby is now installed for the user, go to [Gem installation](#the_gem).
|
459
257
|
|
460
|
-
|
461
|
-
|
462
|
-
|
463
|
-
|
464
|
-
As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
|
465
|
-
One can install in another location with :
|
258
|
+
Alternatively RVM can be installed system-wide, for this execute as `root`.
|
259
|
+
It then installs by default in `/usr/local/rvm` for all users and creates `/etc/profile.d/rvm.sh`.
|
260
|
+
One can install in another location with:
|
466
261
|
|
467
262
|
```bash
|
468
263
|
curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
469
264
|
```
|
470
265
|
|
471
266
|
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
472
|
-
If so, one can rename the
|
267
|
+
If so, one can rename the environment script so that it is not loaded by default:
|
473
268
|
|
474
269
|
```bash
|
475
270
|
mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok
|
@@ -493,13 +288,13 @@ Manual installation:
|
|
493
288
|
- Download the latest Ruby installer **"with devkit"**. (`Msys2` is needed to install some native extensions, such as `grpc`)
|
494
289
|
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
495
290
|
- At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and mingw)
|
496
|
-
-
|
291
|
+
- Then install the aspera-cli gem and Aspera SDK (see next sections)
|
497
292
|
|
498
293
|
Automated installation, with internet access:
|
499
294
|
|
500
|
-
The
|
295
|
+
The Ruby installer supports silent installation, to see the options, execute it with `/help`, or refer to the [Ruby Installer FAQ](https://github.com/oneclick/rubyinstaller2/wiki/FAQ)
|
501
296
|
|
502
|
-
Download the
|
297
|
+
Download the Ruby installer executable from <https://rubyinstaller.org/downloads/> and then install:
|
503
298
|
|
504
299
|
```bat
|
505
300
|
rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\aspera-cli
|
@@ -507,9 +302,15 @@ rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\asper
|
|
507
302
|
|
508
303
|
Installation without network:
|
509
304
|
|
510
|
-
It is essentially the same procedure, but instead of retrieving files from internet,
|
305
|
+
It is essentially the same procedure, but instead of retrieving files from internet, copy the files from a machine with internet access, and then install from those archives:
|
306
|
+
|
307
|
+
- Download the `exe` Ruby installer from <https://rubyinstaller.org/downloads/>
|
308
|
+
|
309
|
+
```bash
|
310
|
+
v=$(curl -s https://rubyinstaller.org/downloads/|sed -nEe 's|.*(https://.*/releases/download/.*exe).*|\1|p'|head -n 1)
|
311
|
+
curl -o ${v##*/} $v
|
312
|
+
```
|
511
313
|
|
512
|
-
- Download the `exe` ruby installer from <https://rubyinstaller.org/downloads/>
|
513
314
|
- Create an archive with necessary gems: <https://help.rubygems.org/kb/rubygems/installing-gems-with-no-network>
|
514
315
|
|
515
316
|
```bat
|
@@ -537,25 +338,20 @@ rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\asper
|
|
537
338
|
gem install --force --local *.gem
|
538
339
|
```
|
539
340
|
|
540
|
-
-
|
341
|
+
- Install the SDK
|
541
342
|
|
542
343
|
```bash
|
543
|
-
ascli
|
344
|
+
ascli config ascp install --sdk-url=file:///sdk.zip
|
544
345
|
```
|
545
346
|
|
546
347
|
> **Note:** An example of installation script is provided: [docs/install.bat](docs/install.bat)
|
547
348
|
|
548
|
-
#### macOS:
|
549
|
-
|
550
|
-
macOS 10.13+ (High Sierra) comes with a recent Ruby.
|
551
|
-
So you can use it directly.
|
552
|
-
You will need to install aspera-cli using `sudo` :
|
349
|
+
#### macOS: `brew`
|
553
350
|
|
554
|
-
|
555
|
-
|
556
|
-
```
|
351
|
+
**macOS** come with Ruby.
|
352
|
+
Nevertheless, [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes), and it will be removed in the future, so better not to rely on it.
|
557
353
|
|
558
|
-
|
354
|
+
The recommended way is to either user `RVM` or use [Homebrew](https://brew.sh/).
|
559
355
|
|
560
356
|
```bash
|
561
357
|
brew install ruby
|
@@ -567,13 +363,13 @@ If your Linux distribution provides a standard Ruby package, you can use it prov
|
|
567
363
|
|
568
364
|
**Example:** RHEL 8+, Rocky Linux 8+, Centos 8 Stream: with extensions to compile native gems
|
569
365
|
|
570
|
-
- Check available
|
366
|
+
- Check available Ruby versions:
|
571
367
|
|
572
368
|
```bash
|
573
369
|
dnf module list ruby
|
574
370
|
```
|
575
371
|
|
576
|
-
- If
|
372
|
+
- If Ruby was already installed with an older version, remove it:
|
577
373
|
|
578
374
|
```bash
|
579
375
|
dnf module -y reset ruby
|
@@ -602,7 +398,7 @@ yum install -y ruby ruby-devel rubygems ruby-json
|
|
602
398
|
apt install -y ruby ruby-dev rubygems ruby-json
|
603
399
|
```
|
604
400
|
|
605
|
-
One can
|
401
|
+
One can remove all installed gems, for example to start fresh:
|
606
402
|
|
607
403
|
```bash
|
608
404
|
gem uninstall -axI $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
@@ -610,7 +406,7 @@ gem uninstall -axI $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
|
610
406
|
|
611
407
|
#### Other Unixes (AIX)
|
612
408
|
|
613
|
-
Ruby is sometimes made available as installable package through third party providers.
|
409
|
+
Ruby is sometimes made available as an installable package through third party providers.
|
614
410
|
For example for AIX, one can look at:
|
615
411
|
|
616
412
|
<https://www.ibm.com/support/pages/aix-toolbox-open-source-software-downloads-alpha#R>
|
@@ -618,7 +414,7 @@ For example for AIX, one can look at:
|
|
618
414
|
If your Unix does not provide a pre-built Ruby, you can get it using one of those
|
619
415
|
[methods](https://www.ruby-lang.org/en/documentation/installation/).
|
620
416
|
|
621
|
-
For instance to build from source
|
417
|
+
For instance to build from source and install in `/opt/ruby` :
|
622
418
|
|
623
419
|
```bash
|
624
420
|
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
|
@@ -644,9 +440,25 @@ If you already have a Java JVM on your system (`java`), it is possible to use `j
|
|
644
440
|
|
645
441
|
> **Note:** Using `jruby`, the startup time is longer than the native Ruby, but transfer speed is not impacted (executed by `ascp` binary).
|
646
442
|
|
443
|
+
#### Optional gems
|
444
|
+
|
445
|
+
Some additional gems can be installed to provide additional features:
|
446
|
+
|
447
|
+
- `rmagick` : to generate thumbnails of images
|
448
|
+
- `grpc` : to use the transfer SDK (gRPC)
|
449
|
+
- `mimemagic` : to detect MIME type of files for `preview` command
|
450
|
+
|
451
|
+
Install like this:
|
452
|
+
|
453
|
+
```bash
|
454
|
+
gem install rmagick grpc mimemagic
|
455
|
+
```
|
456
|
+
|
457
|
+
> **Note:** Those are not installed as part of dependencies because they involve compilation of native code.
|
458
|
+
|
647
459
|
### <a id="the_gem"></a>`aspera-cli` gem
|
648
460
|
|
649
|
-
Once you have Ruby and rights to install gems
|
461
|
+
Once you have Ruby and rights to install gems, install the `aspera-cli` gem and its dependencies:
|
650
462
|
|
651
463
|
```bash
|
652
464
|
gem install aspera-cli
|
@@ -658,27 +470,27 @@ To upgrade to the latest version:
|
|
658
470
|
gem update aspera-cli
|
659
471
|
```
|
660
472
|
|
661
|
-
`ascli` checks every week if a new version is available and
|
473
|
+
During its execution, `ascli` checks every week if a new version is available and notifies the user in a WARN log.
|
662
474
|
To de-activate this feature, globally set the option `version_check_days` to `0`, or specify a different period in days.
|
663
475
|
|
664
476
|
To check if a new version is available (independently of `version_check_days`):
|
665
477
|
|
666
478
|
```bash
|
667
|
-
ascli
|
479
|
+
ascli config check_update
|
668
480
|
```
|
669
481
|
|
670
482
|
### <a id="fasp_prot"></a>FASP Protocol
|
671
483
|
|
672
|
-
Most file transfers will be
|
484
|
+
Most file transfers will be executed using the **FASP** protocol, using `ascp`.
|
673
485
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
674
486
|
|
675
487
|
- `ascp`
|
676
|
-
- aspera-license (in same folder, or ../etc)
|
488
|
+
- `aspera-license` (in same folder, or ../etc)
|
677
489
|
|
678
490
|
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
679
491
|
|
680
492
|
```bash
|
681
|
-
ascli
|
493
|
+
ascli config ascp install
|
682
494
|
```
|
683
495
|
|
684
496
|
If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
|
@@ -688,7 +500,7 @@ curl -Lso sdk.zip https://ibm.biz/aspera_sdk
|
|
688
500
|
```
|
689
501
|
|
690
502
|
```bash
|
691
|
-
ascli
|
503
|
+
ascli config ascp install --sdk-url=file:///sdk.zip
|
692
504
|
```
|
693
505
|
|
694
506
|
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
@@ -700,37 +512,39 @@ If the embedded method is not used, the following packages are also suitable:
|
|
700
512
|
- IBM Aspera High Speed Transfer Server (Licensed)
|
701
513
|
- IBM Aspera High Speed Transfer EndPoint (Licensed)
|
702
514
|
|
703
|
-
For instance, Aspera Connect Client can be installed
|
704
|
-
|
515
|
+
For instance, Aspera Connect Client can be installed by visiting the page:
|
516
|
+
[https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
705
517
|
|
706
|
-
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
518
|
+
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found by default.
|
707
519
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
708
520
|
|
709
521
|
Several methods are provided to start a transfer.
|
710
|
-
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available.
|
522
|
+
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available.
|
523
|
+
Refer to section: [Transfer Agents](#agents)
|
711
524
|
|
712
525
|
### <a id="offline_install"></a>Installation in air gapped environment
|
713
526
|
|
714
|
-
> **Note:**
|
527
|
+
> **Note:** No pre-packaged version is provided yet.
|
715
528
|
|
716
529
|
A method to build one is provided here:
|
717
530
|
|
718
531
|
The procedure:
|
719
532
|
|
720
533
|
- Follow the non-root installation procedure with RVM, including gem
|
534
|
+
|
721
535
|
- Archive (zip, tar) the main RVM folder (includes ascli):
|
722
536
|
|
723
537
|
```bash
|
724
538
|
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
725
539
|
```
|
726
540
|
|
727
|
-
-
|
541
|
+
- Show the Aspera SDK URL
|
728
542
|
|
729
543
|
```bash
|
730
|
-
ascli
|
544
|
+
ascli --show-config --fields=sdk_url
|
731
545
|
```
|
732
546
|
|
733
|
-
- Download the SDK archive from that URL
|
547
|
+
- Download the SDK archive from that URL
|
734
548
|
|
735
549
|
```bash
|
736
550
|
curl -Lso sdk.zip https://ibm.biz/aspera_sdk
|
@@ -747,7 +561,7 @@ tar zxvf rvm-ascli.tgz
|
|
747
561
|
|
748
562
|
source ~/.rvm/scripts/rvm
|
749
563
|
|
750
|
-
ascli
|
564
|
+
ascli config ascp install --sdk-url=file:///sdk.zip
|
751
565
|
```
|
752
566
|
|
753
567
|
- Add those lines to shell init (`.profile`)
|
@@ -756,9 +570,218 @@ ascli conf ascp install --sdk-url=file:///sdk.zip
|
|
756
570
|
source ~/.rvm/scripts/rvm
|
757
571
|
```
|
758
572
|
|
573
|
+
> **Note:** Alternatively, to download all necessary gems in folder `my_gems`, execute:
|
574
|
+
|
575
|
+
```bash
|
576
|
+
gem install aspera-cli -N -i my_gems
|
577
|
+
```
|
578
|
+
|
579
|
+
### Container
|
580
|
+
|
581
|
+
The container image is: [`martinlaurent/ascli`](https://hub.docker.com/r/martinlaurent/ascli).
|
582
|
+
The container contains: Ruby, `ascli` and the Aspera Transfer SDK.
|
583
|
+
To use the container, ensure that you have `podman` (or `docker`) installed.
|
584
|
+
|
585
|
+
```bash
|
586
|
+
podman --version
|
587
|
+
```
|
588
|
+
|
589
|
+
#### Container: Quick start
|
590
|
+
|
591
|
+
**Wanna start quickly ?** With an interactive shell ?
|
592
|
+
Execute this:
|
593
|
+
|
594
|
+
```bash
|
595
|
+
podman run --rm --tty --interactive --entrypoint bash martinlaurent/ascli:latest
|
596
|
+
```
|
597
|
+
|
598
|
+
> **Note:** This command changes the entrypoint to an interactive shell instead of direct execution of `ascli`.
|
599
|
+
|
600
|
+
Then, execute individual `ascli` commands such as:
|
601
|
+
|
602
|
+
```bash
|
603
|
+
ascli config init
|
604
|
+
ascli config preset overview
|
605
|
+
ascli config ascp info
|
606
|
+
ascli server ls /
|
607
|
+
```
|
608
|
+
|
609
|
+
That is simple, but there are limitations:
|
610
|
+
|
611
|
+
- Everything happens in the container
|
612
|
+
- Any generated file in the container will be lost on container (shell) exit. Including configuration files and downloaded files.
|
613
|
+
- No possibility to upload files located on the host system
|
614
|
+
|
615
|
+
#### Container: Details
|
616
|
+
|
617
|
+
The container image is built from this [Dockerfile](Dockerfile.tmpl.erb).
|
618
|
+
The entry point is `ascli` and the default command is `help`.
|
619
|
+
|
620
|
+
The container can be executed for individual commands like this: (add `ascli` commands and options at the end of the command line, e.g. `-v` to display the version)
|
621
|
+
|
622
|
+
```bash
|
623
|
+
podman run --rm --tty --interactive martinlaurent/ascli:latest
|
624
|
+
```
|
625
|
+
|
626
|
+
For more convenience, you may define a shell alias:
|
627
|
+
|
628
|
+
```bash
|
629
|
+
alias ascli='podman run --rm --tty --interactive martinlaurent/ascli:latest'
|
630
|
+
```
|
631
|
+
|
632
|
+
Then, you can execute the container like a local command:
|
633
|
+
|
634
|
+
```bash
|
635
|
+
ascli -v
|
636
|
+
```
|
637
|
+
|
638
|
+
```text
|
639
|
+
4.16.0
|
640
|
+
```
|
641
|
+
|
642
|
+
In order to keep persistency of configuration on the host, you should specify your user's configuration folder as a volume for the container.
|
643
|
+
To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`).
|
644
|
+
Add options:
|
645
|
+
|
646
|
+
```bash
|
647
|
+
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
648
|
+
```
|
649
|
+
|
650
|
+
> **Note:** If you are using a `podman machine`, e.g. on macOS , make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
651
|
+
|
652
|
+
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
653
|
+
you can change the entry point, add option:
|
654
|
+
|
655
|
+
```bash
|
656
|
+
--entrypoint bash
|
657
|
+
```
|
658
|
+
|
659
|
+
You may also probably want that files downloaded in the container are directed to the host.
|
660
|
+
In this case you need also to specify the shared transfer folder as a volume:
|
661
|
+
|
662
|
+
```bash
|
663
|
+
--volume $HOME/xferdir:/xferfiles
|
664
|
+
```
|
665
|
+
|
666
|
+
> **Note:** ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
667
|
+
|
668
|
+
And if you want all the above, simply use all the options:
|
669
|
+
|
670
|
+
```bash
|
671
|
+
alias asclish="podman run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash martinlaurent/ascli:latest"
|
672
|
+
```
|
673
|
+
|
674
|
+
```bash
|
675
|
+
export xferdir=$HOME/xferdir
|
676
|
+
mkdir -p $xferdir
|
677
|
+
chmod -R 777 $xferdir
|
678
|
+
mkdir -p $HOME/.aspera/ascli
|
679
|
+
asclish
|
680
|
+
```
|
681
|
+
|
682
|
+
#### Container: Sample start script
|
683
|
+
|
684
|
+
A convenience sample script is also provided: download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
685
|
+
|
686
|
+
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli config gem path)/../examples/dascli ascli`
|
687
|
+
|
688
|
+
Some environment variables can be set for this script to adapt its behavior:
|
689
|
+
|
690
|
+
| env var | Description | Default | Example |
|
691
|
+
|----------------|------------------------------------|--------------------------|--------------------------|
|
692
|
+
| `ASCLI_HOME` | Configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascli_config` |
|
693
|
+
| `docker_args` | Additional options to `podman` | <empty> | `--volume /Users:/Users` |
|
694
|
+
| `image` | Container image name | `martinlaurent/ascli` | |
|
695
|
+
| `version` | Container image version | Latest | `4.8.0.pre` |
|
696
|
+
|
697
|
+
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
698
|
+
(value expected in the container).
|
699
|
+
This allows having persistent configuration on the host.
|
700
|
+
|
701
|
+
To add local storage as a volume, you can use the env var `docker_args`:
|
702
|
+
|
703
|
+
Example of use:
|
704
|
+
|
705
|
+
```bash
|
706
|
+
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
707
|
+
chmod a+x ascli
|
708
|
+
export xferdir=$HOME/xferdir
|
709
|
+
mkdir -p $xferdir
|
710
|
+
chmod -R 777 $xferdir
|
711
|
+
export docker_args="--volume $xferdir:/xferfiles"
|
712
|
+
|
713
|
+
./ascli config init
|
714
|
+
|
715
|
+
echo 'Local file to transfer' > $xferdir/samplefile.txt
|
716
|
+
./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload
|
717
|
+
```
|
718
|
+
|
719
|
+
> **Note:** The local file (`samplefile.txt`) is specified relative to storage view from container (`/xferfiles`) mapped to the host folder `$HOME/xferdir`
|
720
|
+
>
|
721
|
+
> **Note:** Do not use too many volumes, as the legacy `aufs` limits the number. (anyway, prefer to use `overlay2`)
|
722
|
+
|
723
|
+
#### Container: Offline installation
|
724
|
+
|
725
|
+
- First create the image archive:
|
726
|
+
|
727
|
+
```bash
|
728
|
+
podman pull martinlaurent/ascli
|
729
|
+
podman save martinlaurent/ascli|gzip>ascli_image_latest.tar.gz
|
730
|
+
```
|
731
|
+
|
732
|
+
- Then, on air-gapped system:
|
733
|
+
|
734
|
+
```bash
|
735
|
+
podman load -i ascli_image_latest.tar.gz
|
736
|
+
```
|
737
|
+
|
738
|
+
#### Container: `aspera.conf`
|
739
|
+
|
740
|
+
`ascp`'s configuration file `aspera.conf` is located in the container at: `/aspera_sdk/aspera.conf` (see Dockerfile).
|
741
|
+
As the container is immutable, it is not recommended to modify this file.
|
742
|
+
If one wants to change the content, it is possible to tell `ascp` to use another file using `ascp` option `-f`, e.g. by locating it on the host folder `$HOME/.aspera/ascli` mapped to the container folder `/home/cliuser/.aspera/ascli`:
|
743
|
+
|
744
|
+
```bash
|
745
|
+
echo '<CONF/>' > $HOME/.aspera/ascli/aspera.conf
|
746
|
+
```
|
747
|
+
|
748
|
+
Then, tell `ascp` to use that other configuration file:
|
749
|
+
|
750
|
+
```bash
|
751
|
+
--transfer-info=@json:'{"ascp_args":["-f","/home/cliuser/.aspera/ascli/aspera.conf"]}'
|
752
|
+
```
|
753
|
+
|
754
|
+
#### Container: Singularity
|
755
|
+
|
756
|
+
Singularity is another type of use of container.
|
757
|
+
|
758
|
+
On Linux install:
|
759
|
+
|
760
|
+
```bash
|
761
|
+
dnf install singularity-ce
|
762
|
+
```
|
763
|
+
|
764
|
+
Build an image like this:
|
765
|
+
|
766
|
+
```bash
|
767
|
+
singularity build ascli.sif docker://martinlaurent/ascli
|
768
|
+
```
|
769
|
+
|
770
|
+
Then, start `ascli` like this:
|
771
|
+
|
772
|
+
```bash
|
773
|
+
singularity run ascli.sif
|
774
|
+
```
|
775
|
+
|
776
|
+
Or get a shell with access to `ascli` like this:
|
777
|
+
|
778
|
+
```bash
|
779
|
+
singularity shell ascli.sif
|
780
|
+
```
|
781
|
+
|
759
782
|
## <a id="cli"></a>Command Line Interface: `ascli`
|
760
783
|
|
761
|
-
The `aspera-cli`
|
784
|
+
The `aspera-cli` gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
762
785
|
|
763
786
|
- IBM Aspera High Speed Transfer Server (FASP and Node)
|
764
787
|
- IBM Aspera on Cloud (including ATS)
|
@@ -766,21 +789,21 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
766
789
|
- IBM Aspera Shares
|
767
790
|
- IBM Aspera Console
|
768
791
|
- IBM Aspera Orchestrator
|
769
|
-
-
|
792
|
+
- And more...
|
770
793
|
|
771
794
|
`ascli` provides the following features:
|
772
795
|
|
773
|
-
- Supports
|
774
|
-
- Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
796
|
+
- Supports commands to Aspera server products (on-premise and SaaS)
|
797
|
+
- Any command line **options** (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files, ...
|
775
798
|
- Supports Commands, Option values and Parameters shortcuts
|
776
799
|
- FASP [Transfer Agents](#agents) can be: local `ascp`, or Connect Client, or any transfer node
|
777
800
|
- Transfer parameters can be altered by modification of [*transfer-spec*](#transferspec), this includes requiring multi-session
|
778
801
|
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
779
802
|
- Supports FaspStream creation (using Node API)
|
780
|
-
- Supports
|
803
|
+
- Supports **Watchfolder** creation (using Node API)
|
781
804
|
- Additional command plugins can be written by the user
|
782
805
|
- Supports download of faspex and Aspera on Cloud "external" links
|
783
|
-
- Supports
|
806
|
+
- Supports **Legacy** SSH based FASP transfers and remote commands (`ascmd`)
|
784
807
|
|
785
808
|
Basic usage is displayed by executing:
|
786
809
|
|
@@ -790,7 +813,7 @@ ascli -h
|
|
790
813
|
|
791
814
|
Refer to sections: [Usage](#usage).
|
792
815
|
|
793
|
-
|
816
|
+
> **Note:** `ascli` features are not fully documented here, the user may explore commands on the command line.
|
794
817
|
|
795
818
|
### `ascp` command line
|
796
819
|
|
@@ -798,17 +821,17 @@ If you want to use `ascp` directly as a command line, refer to IBM Aspera docume
|
|
798
821
|
|
799
822
|
Using `ascli` with plugin `server` for command line gives advantages over `ascp`:
|
800
823
|
|
801
|
-
-
|
802
|
-
-
|
803
|
-
-
|
804
|
-
-
|
824
|
+
- Automatic resume on error
|
825
|
+
- Configuration file
|
826
|
+
- Choice of transfer agents
|
827
|
+
- Integrated support of multi-session
|
805
828
|
|
806
|
-
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args`).
|
829
|
+
Moreover all `ascp` options are supported either through transfer spec parameters (listed with `conf ascp spec`) and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args` in `transfer_info`).
|
807
830
|
|
808
831
|
### <a id="parsing"></a>Command line parsing, Special Characters
|
809
832
|
|
810
833
|
`ascli` is typically executed in a shell, either interactively or in a script.
|
811
|
-
`ascli` receives its arguments from this shell (through Operating System).
|
834
|
+
`ascli` receives its arguments from this shell (through the Operating System).
|
812
835
|
|
813
836
|
#### Shell parsing for Unix-like systems: Linux, macOS, AIX
|
814
837
|
|
@@ -819,7 +842,7 @@ In this environment the shell parses the command line, possibly replacing variab
|
|
819
842
|
See [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation).
|
820
843
|
The shell builds the list of arguments and then `fork`/`exec` Ruby with that list.
|
821
844
|
Ruby receives a list parameters from shell and gives it to `ascli`.
|
822
|
-
|
845
|
+
Special character handling (quotes, spaces, env vars, ...) is handled by the shell for any command executed.
|
823
846
|
|
824
847
|
#### Shell parsing for Windows
|
825
848
|
|
@@ -834,22 +857,22 @@ One can also run `ascli` with option `--log-level=debug` to display the command
|
|
834
857
|
|
835
858
|
The following examples give the same result on Windows:
|
836
859
|
|
837
|
-
-
|
860
|
+
- Single quote protects the double quote
|
838
861
|
|
839
862
|
```cmd
|
840
|
-
|
863
|
+
ascli config echo @json:'{"url":"https://..."}'
|
841
864
|
```
|
842
865
|
|
843
|
-
-
|
866
|
+
- Triple double quotes are replaced with a single double quote
|
844
867
|
|
845
868
|
```cmd
|
846
|
-
|
869
|
+
ascli config echo @json:{"""url""":"""https://..."""}
|
847
870
|
```
|
848
871
|
|
849
|
-
-
|
872
|
+
- Double quote is escaped with backslash within double quotes
|
850
873
|
|
851
874
|
```cmd
|
852
|
-
|
875
|
+
ascli config echo @json:"{\"url\":\"https://...\"}"
|
853
876
|
```
|
854
877
|
|
855
878
|
More details: on Windows, `cmd.exe` is typically used to start .
|
@@ -858,7 +881,8 @@ Basically it handles I/O redirection (`<>|`), shell variables (`%`), multiple co
|
|
858
881
|
Eventually, all those special characters are removed from the command line unless escaped with `^` or `"`.
|
859
882
|
`"` are kept and given to the program.
|
860
883
|
|
861
|
-
Then, Windows `CreateProcess`
|
884
|
+
Then, the shell calls Windows' `CreateProcess` with just the whole command line as a single string.
|
885
|
+
Unlike Unix-like systems where the command line is split into arguments by the shell.
|
862
886
|
|
863
887
|
It's up to the program to split arguments:
|
864
888
|
|
@@ -869,26 +893,29 @@ It's up to the program to split arguments:
|
|
869
893
|
Ruby vaguely follows the Microsoft C/C++ parameter parsing rules.
|
870
894
|
(See `w32_cmdvector` in Ruby source [`win32.c`](https://github.com/ruby/ruby/blob/master/win32/win32.c#L1766)) : <!--cspell:disable-line-->
|
871
895
|
|
872
|
-
-
|
873
|
-
-
|
874
|
-
-
|
875
|
-
-
|
876
|
-
-
|
896
|
+
- Space characters: split arguments (space, tab, newline)
|
897
|
+
- Backslash: `\` escape single special character
|
898
|
+
- Globing characters: `*?[]{}` for file globing
|
899
|
+
- Double quotes: `"`
|
900
|
+
- Single quotes: `'`
|
877
901
|
|
878
902
|
#### Extended Values (JSON, Ruby, ...)
|
879
903
|
|
880
|
-
Some of the `ascli` parameters are expected to be [Extended Values](#extended), i.e. not a simple `String`, but a
|
904
|
+
Some of the `ascli` parameters are expected to be [Extended Values](#extended), i.e. not a simple `String`, but a composite structure (`Hash`, `Array`).
|
881
905
|
Typically, the `@json:` modifier is used, it expects a [JSON](https://www.json.org/) string.
|
882
906
|
JSON itself has some special syntax: for example `"` is used to enclose a `String`.
|
883
907
|
|
884
908
|
#### Testing Extended Values
|
885
909
|
|
886
|
-
In case of doubt of argument values after parsing, one can test using command `config echo`.
|
910
|
+
In case of doubt of argument values after parsing, one can test using command `config echo`.
|
911
|
+
`config echo` takes exactly **one** argument which can use the [Extended Value](#extended) syntax.
|
912
|
+
Unprocessed command line arguments are shown in the error message.
|
887
913
|
|
888
|
-
Example:
|
914
|
+
Example:
|
915
|
+
The shell parses three arguments (as `String`: `1`, `2` and `3`), so the additional two arguments are not processed by the `echo` command.
|
889
916
|
|
890
917
|
```bash
|
891
|
-
ascli
|
918
|
+
ascli config echo 1 2 3
|
892
919
|
```
|
893
920
|
|
894
921
|
```ruby
|
@@ -900,9 +927,10 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
900
927
|
|
901
928
|
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
902
929
|
|
903
|
-
In the following examples (using a POSIX shell, such as `bash`), several sample commands are provided
|
904
|
-
For all example, most of special character handling is not specific to `ascli`:
|
905
|
-
|
930
|
+
In the following examples (using a POSIX shell, such as `bash`), several equivalent sample commands are provided.
|
931
|
+
For all example, most of special character handling is not specific to `ascli`:
|
932
|
+
It depends on the underlying syntax: shell , JSON, etc...
|
933
|
+
Depending on the case, a different `format` option is used to display the actual value.
|
906
934
|
|
907
935
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
908
936
|
|
@@ -910,9 +938,9 @@ Double quotes are processed by the shell to create a single string argument.
|
|
910
938
|
For **POSIX shells**, single quotes can also be used in this case, or protect the special character ` ` (space) with a backslash. <!-- markdownlint-disable-line -->
|
911
939
|
|
912
940
|
```bash
|
913
|
-
ascli
|
914
|
-
ascli
|
915
|
-
ascli
|
941
|
+
ascli config echo "Hello World" --format=text
|
942
|
+
ascli config echo 'Hello World' --format=text
|
943
|
+
ascli config echo Hello\ World --format=text
|
916
944
|
```
|
917
945
|
|
918
946
|
```output
|
@@ -922,14 +950,14 @@ Hello World
|
|
922
950
|
#### Using a shell variable, parsed by shell, in an extended value
|
923
951
|
|
924
952
|
To be evaluated by shell, the shell variable must not be in single quotes.
|
925
|
-
Even if the variable contains spaces it
|
953
|
+
Even if the variable contains spaces it results only in one argument for `ascli` because word parsing is made before variable expansion by shell.
|
926
954
|
|
927
|
-
> **Note:**
|
955
|
+
> **Note:** We use a shell variable here: the variable is not necessarily an environment variable (`export`).
|
928
956
|
|
929
957
|
```bash
|
930
958
|
MYVAR="Hello World"
|
931
|
-
ascli
|
932
|
-
ascli
|
959
|
+
ascli config echo @json:'{"title":"'$MYVAR'"}' --format=json
|
960
|
+
ascli config echo @json:{\"title\":\"$MYVAR\"} --format=json
|
933
961
|
```
|
934
962
|
|
935
963
|
```json
|
@@ -942,8 +970,8 @@ Double quote is a shell special character.
|
|
942
970
|
Like any shell special character, it can be protected either by preceding with a backslash or by enclosing in a single quote.
|
943
971
|
|
944
972
|
```bash
|
945
|
-
ascli
|
946
|
-
ascli
|
973
|
+
ascli config echo \"
|
974
|
+
ascli config echo '"'
|
947
975
|
```
|
948
976
|
|
949
977
|
```output
|
@@ -953,9 +981,9 @@ ascli conf echo '"'
|
|
953
981
|
Double quote in JSON is a little tricky because `"` is special both for the shell and JSON. Both shell and JSON syntax allow to protect `"`, but only the shell allows protection using single quote.
|
954
982
|
|
955
983
|
```bash
|
956
|
-
ascli
|
957
|
-
ascli
|
958
|
-
ascli
|
984
|
+
ascli config echo @json:'"\""' --format=text
|
985
|
+
ascli config echo @json:\"\\\"\" --format=text
|
986
|
+
ascli config echo @ruby:\'\"\' --format=text
|
959
987
|
```
|
960
988
|
|
961
989
|
```output
|
@@ -988,9 +1016,9 @@ Both `"` and `\` are special characters for JSON and Ruby and can be protected w
|
|
988
1016
|
- Then, since the value will be evaluated by shell, any shell special characters must be protected, either using preceding `\` for each character to protect, or by enclosing in single quote:
|
989
1017
|
|
990
1018
|
```bash
|
991
|
-
ascli
|
992
|
-
ascli
|
993
|
-
ascli
|
1019
|
+
ascli config echo @json:{\"title\":\"Test\ \\\"\ \'\ \&\ \\\\\"} --format=json
|
1020
|
+
ascli config echo @json:'{"title":"Test \" '\'' & \\"}' --format=json
|
1021
|
+
ascli config echo @ruby:"{'title'=>%q{Test \" ' & \\\\}}" --format=json
|
994
1022
|
```
|
995
1023
|
|
996
1024
|
```json
|
@@ -1002,7 +1030,7 @@ ascli conf echo @ruby:"{'title'=>%q{Test \" ' & \\\\}}" --format=json
|
|
1002
1030
|
If `ascli` is used interactively (a user typing on terminal), it is easy to require the user to type values:
|
1003
1031
|
|
1004
1032
|
```bash
|
1005
|
-
ascli
|
1033
|
+
ascli config echo @ruby:"{'title'=>gets.chomp}" --format=json
|
1006
1034
|
```
|
1007
1035
|
|
1008
1036
|
`gets` is Ruby's method of terminal input (terminated by `\n`), and `chomp` removes the trailing `\n`.
|
@@ -1012,13 +1040,13 @@ ascli conf echo @ruby:"{'title'=>gets.chomp}" --format=json
|
|
1012
1040
|
If you need to provide a list of command line argument from lines that are in a file, on Linux you can use the `xargs` command:
|
1013
1041
|
|
1014
1042
|
```bash
|
1015
|
-
xargs -a lines.txt -d \\n ascli
|
1043
|
+
xargs -a lines.txt -d \\n ascli config echo
|
1016
1044
|
```
|
1017
1045
|
|
1018
1046
|
This is equivalent to execution of:
|
1019
1047
|
|
1020
1048
|
```bash
|
1021
|
-
ascli
|
1049
|
+
ascli config echo [line1] [line2] [line3] ...
|
1022
1050
|
```
|
1023
1051
|
|
1024
1052
|
If there are spaces in the lines, those are not taken as separator, as we provide option `-d \\n` to `xargs`.
|
@@ -1037,8 +1065,8 @@ Using those values will not require any escaping of characters since values do n
|
|
1037
1065
|
If the value is to be assigned directly to an option of ascli, then you can directly use the content of the file or env var using the `@file:` or `@env:` readers:
|
1038
1066
|
|
1039
1067
|
```bash
|
1040
|
-
ascli
|
1041
|
-
ascli
|
1068
|
+
ascli config echo @file:title.txt --format=text
|
1069
|
+
ascli config echo @env:MYTITLE --format=text
|
1042
1070
|
```
|
1043
1071
|
|
1044
1072
|
```output
|
@@ -1048,8 +1076,8 @@ Test " ' & \
|
|
1048
1076
|
If the value to be used is in a more complex structure, then the `@ruby:` modifier can be used: it allows any Ruby code in expression, including reading from file or env var. In those cases, there is no character to protect because values are not parsed by the shell, or JSON or even Ruby.
|
1049
1077
|
|
1050
1078
|
```bash
|
1051
|
-
ascli
|
1052
|
-
ascli
|
1079
|
+
ascli config echo @ruby:"{'title'=>File.read('title.txt')}" --format=json
|
1080
|
+
ascli config echo @ruby:"{'title'=>ENV['MYTITLE']}" --format=json
|
1053
1081
|
```
|
1054
1082
|
|
1055
1083
|
```json
|
@@ -1061,11 +1089,11 @@ ascli conf echo @ruby:"{'title'=>ENV['MYTITLE']}" --format=json
|
|
1061
1089
|
Command line arguments are the units of command line typically separated by spaces (the `argv` of C).
|
1062
1090
|
The tokenization of the command line is typically done by the shell, refer to the previous section [Command Line Parsing](#parsing).
|
1063
1091
|
|
1064
|
-
`ascli`
|
1092
|
+
`ascli` handles three types of command line arguments:
|
1065
1093
|
|
1066
1094
|
- Commands
|
1067
|
-
- Options
|
1068
1095
|
- Positional Arguments
|
1096
|
+
- Options
|
1069
1097
|
|
1070
1098
|
For example:
|
1071
1099
|
|
@@ -1073,29 +1101,30 @@ For example:
|
|
1073
1101
|
ascli command subcommand --option-name=VAL1 VAL2
|
1074
1102
|
```
|
1075
1103
|
|
1076
|
-
-
|
1077
|
-
-
|
1078
|
-
-
|
1104
|
+
- Executes **command**: `command subcommand`
|
1105
|
+
- With one **option**: `option_name` and its **value**: `VAL1`
|
1106
|
+
- The command has one additional **positional argument**: `VAL2`
|
1079
1107
|
|
1080
1108
|
If the value of a command, option or argument is constrained by a fixed list of values, then it is possible to use a few of the first letters of the value, provided that it uniquely identifies the value.
|
1081
|
-
For example `ascli
|
1109
|
+
For example `ascli config pre ov` is the same as `ascli config preset overview`.
|
1082
1110
|
|
1083
1111
|
The value of options and arguments is evaluated with the [Extended Value Syntax](#extended).
|
1084
1112
|
|
1085
1113
|
#### Commands
|
1086
1114
|
|
1087
1115
|
Commands are typically entity types or verbs to act on those entities.
|
1116
|
+
Its value is a `String` that must belong to a fixed list of values in a given context.
|
1088
1117
|
|
1089
1118
|
Example:
|
1090
1119
|
|
1091
1120
|
```bash
|
1092
|
-
ascli
|
1121
|
+
ascli config ascp info
|
1093
1122
|
```
|
1094
1123
|
|
1095
1124
|
- `ascli` is the executable executed by the shell
|
1096
|
-
- `conf` is the first level command
|
1097
|
-
- `ascp` is the second level command
|
1098
|
-
- `info` is the third level command
|
1125
|
+
- `conf` is the first level command: name of the plugin to be used
|
1126
|
+
- `ascp` is the second level command: name of the component (singleton)
|
1127
|
+
- `info` is the third level command: action to be performed
|
1099
1128
|
|
1100
1129
|
Typically, commands are located at the **beginning** of the command line.
|
1101
1130
|
Order is significant.
|
@@ -1104,20 +1133,37 @@ If wrong, or no command is provided when expected, an error message is displayed
|
|
1104
1133
|
|
1105
1134
|
Some sub-commands appear after entity selection, e.g. `ascli aoc admin res node do 8669 browse /`: `browse` is a sub-command of `node`.
|
1106
1135
|
|
1136
|
+
#### Positional Arguments
|
1137
|
+
|
1138
|
+
Positional Arguments are typically mandatory values for a command, such as entity creation data.
|
1139
|
+
|
1140
|
+
It could also be designed as an option, but since it is mandatory and typically creation parameters need not be set in a configuration file, then it is better to use a positional argument, and not define specific options.
|
1141
|
+
|
1142
|
+
The advantages of using a positional argument instead of an option for the same are that the command line is shorter(no option name, just the position) and the value is clearly mandatory.
|
1143
|
+
|
1144
|
+
The disadvantage is that it is not possible to define a default value in a configuration file or environment variable like for options.
|
1145
|
+
Nevertheless, [Extended Values](#extended) syntax is supported, so it is possible to retrieve a value from the configuration file or environment variable (using `@preset:`).
|
1146
|
+
|
1147
|
+
If a Positional Arguments begins with `-`, then either use the `@val:` syntax (see [Extended Values](#extended)), or use the `--` separator (see below).
|
1148
|
+
|
1149
|
+
A few positional arguments are optional, they are located at the end of the command line.
|
1150
|
+
|
1107
1151
|
#### Options
|
1108
1152
|
|
1109
1153
|
All options, e.g. `--log-level=debug`, are command line arguments that:
|
1110
1154
|
|
1111
|
-
-
|
1112
|
-
-
|
1113
|
-
-
|
1114
|
-
-
|
1155
|
+
- Start with `--`
|
1156
|
+
- Have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
|
1157
|
+
- Have a value, separated from name with a `=`
|
1158
|
+
- Can be used by prefix (avoid), provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
|
1115
1159
|
|
1116
1160
|
Exceptions:
|
1117
1161
|
|
1118
|
-
-
|
1119
|
-
-
|
1120
|
-
-
|
1162
|
+
- Some options accept a short form, e.g. `-Ptoto` is equivalent to `--preset=toto`, refer to the manual or `-h`.
|
1163
|
+
- Some options (flags) don't take a value, e.g. `-N`
|
1164
|
+
- The special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`.
|
1165
|
+
|
1166
|
+
Example:
|
1121
1167
|
|
1122
1168
|
```bash
|
1123
1169
|
ascli config echo -- --sample
|
@@ -1129,14 +1175,12 @@ ascli config echo -- --sample
|
|
1129
1175
|
|
1130
1176
|
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
1131
1177
|
|
1132
|
-
Options may have
|
1133
|
-
|
1134
|
-
Options can be placed anywhere on command line and evaluated in order.
|
1178
|
+
Options may have a (hardcoded) default value.
|
1135
1179
|
|
1136
|
-
Options are
|
1180
|
+
Options can be placed anywhere on command line and are evaluated in order.
|
1137
1181
|
|
1138
|
-
|
1139
|
-
|
1182
|
+
Options are typically optional: to change the default behavior.
|
1183
|
+
But some are mandatory, so they can be placed in a configuration file, for example: connection information.
|
1140
1184
|
|
1141
1185
|
The value for **any** options can come from the following locations (in this order, last value evaluated overrides previous value):
|
1142
1186
|
|
@@ -1146,27 +1190,14 @@ The value for **any** options can come from the following locations (in this ord
|
|
1146
1190
|
|
1147
1191
|
Environment variable starting with prefix: ASCLI_ are taken as option values, e.g. `ASCLI_OPTION_NAME` is for `--option-name`.
|
1148
1192
|
|
1149
|
-
|
1150
|
-
|
1151
|
-
|
1152
|
-
|
1153
|
-
- it is a mandatory parameters that would benefit from being set in a config file or environment variable
|
1154
|
-
- it is optional
|
1193
|
+
Option `show_config` dry runs the configuration, and then returns currently set values for options.
|
1194
|
+
`ascli --show-config` outputs global options only, and `ascli [plugin] --show-config` outputs global and plugin default options.
|
1195
|
+
In addition, option `--show-config` can be added at the end of any full command line, this displays the options that would be used for the command.
|
1155
1196
|
|
1156
|
-
|
1157
|
-
|
1158
|
-
Positional Arguments are typically mandatory values for a command, such as entity creation data.
|
1197
|
+
A parameter is typically designed as option if:
|
1159
1198
|
|
1160
|
-
It
|
1161
|
-
|
1162
|
-
The advantages of using a positional argument instead of an option for the same are that the command line is shorter(no option name, just the position) and the value is clearly mandatory.
|
1163
|
-
|
1164
|
-
The disadvantage is that it is not possible to define a default value in a config file or environment variable like for options.
|
1165
|
-
Nevertheless, [Extended Values](#extended) syntax is supported, so it is possible to retrieve a value from the config file or environment variable.
|
1166
|
-
|
1167
|
-
If a Positional Arguments begins with `-`, then either use the `@val:` syntax (see [Extended Values](#extended)), or use the `--` separator (see above).
|
1168
|
-
|
1169
|
-
Very few positional arguments are optional, they are located at the end of the command line.
|
1199
|
+
- It is optional, or
|
1200
|
+
- It is a mandatory parameter that would benefit from being set persistently (i.e. in a configuration file or environment variable, e.g. URL and credentials).
|
1170
1201
|
|
1171
1202
|
### Interactive Input
|
1172
1203
|
|
@@ -1186,6 +1217,8 @@ The behavior can be controlled with:
|
|
1186
1217
|
Command execution will result in output (terminal, stdout/stderr).
|
1187
1218
|
The information displayed depends on the action.
|
1188
1219
|
|
1220
|
+
To redirect results to a file, use option `output`.
|
1221
|
+
|
1189
1222
|
#### Types of output data
|
1190
1223
|
|
1191
1224
|
Depending on action, the output will contain:
|
@@ -1202,10 +1235,10 @@ Depending on action, the output will contain:
|
|
1202
1235
|
By default, result of type single_object and object_list are displayed using format `table`.
|
1203
1236
|
The table style can be customized with parameter: `table_style` (horizontal, vertical and intersection characters) and is `:.:` by default.
|
1204
1237
|
|
1205
|
-
In a table format, when displaying
|
1238
|
+
In a table format, when displaying **Objects** (single, or list), by default, sub object are
|
1206
1239
|
flattened (option `flat_hash`). So, object `{"user":{"id":1,"name":"toto"}}` will have attributes: `user.id` and `user.name`.
|
1207
1240
|
Setting `flat_hash` to `false` will only display one field: `user` and value is the sub `Hash`.
|
1208
|
-
When in flatten mode, it is possible to filter fields by
|
1241
|
+
When in flatten mode, it is possible to filter fields by compound field name using dot as separator.
|
1209
1242
|
|
1210
1243
|
Object lists are displayed one per line, with attributes as columns. Single objects are transposed: one attribute per line.
|
1211
1244
|
If transposition of single object is not desired, use option: `transpose_single` set to `no`.
|
@@ -1220,17 +1253,6 @@ The style of output can be set using the `format` parameter, supporting:
|
|
1220
1253
|
- `yaml` : YAML
|
1221
1254
|
- `csv` : Comma Separated Values
|
1222
1255
|
|
1223
|
-
#### Entity identifier
|
1224
|
-
|
1225
|
-
When a command is executed on a single entity, the entity is identified by a unique identifier that follows the command: e.g. `ascli aoc admin res user show 1234` where `1234` is the user's identifier.
|
1226
|
-
|
1227
|
-
> **Note:** The legacy option `id` is deprecated: `--id=1234` as it does not provide the possibility to have sub-entities.
|
1228
|
-
|
1229
|
-
Only some commands provide the following capability:
|
1230
|
-
If the entity can also be uniquely identified by a name, then the name can be used instead of the identifier, using the **percent selector**: `ascli aoc admin res user show %name:john` where `john` is the user name.
|
1231
|
-
|
1232
|
-
Syntax: `%<field>:<value>`
|
1233
|
-
|
1234
1256
|
#### Verbosity of output
|
1235
1257
|
|
1236
1258
|
Output messages are categorized in 3 types:
|
@@ -1260,9 +1282,9 @@ Elements of the list can be:
|
|
1260
1282
|
|
1261
1283
|
- `DEF` : default display of columns (that's the default, when not set)
|
1262
1284
|
- `ALL` : all columns available
|
1263
|
-
-
|
1264
|
-
- property : add property to the current list
|
1265
|
-
-
|
1285
|
+
- `-`**property** : remove property from the current list
|
1286
|
+
- **property** : add property to the current list
|
1287
|
+
- A Ruby `RegEx` : using `@ruby:'/.../'`
|
1266
1288
|
|
1267
1289
|
Examples:
|
1268
1290
|
|
@@ -1300,6 +1322,20 @@ In above example, the same result is obtained with option:
|
|
1300
1322
|
--select=@ruby:'->(i){i["ats_admin"]}'
|
1301
1323
|
```
|
1302
1324
|
|
1325
|
+
#### Percent selector
|
1326
|
+
|
1327
|
+
The percent selector allows identification of an entity by another unique identifier other than the native identifier.
|
1328
|
+
|
1329
|
+
When a command is executed on a single entity, the entity is identified by a unique identifier that follows the command:
|
1330
|
+
e.g. `ascli aoc admin res user show 1234` where `1234` is the user's identifier.
|
1331
|
+
|
1332
|
+
Some commands provide the following capability:
|
1333
|
+
If the entity can also be uniquely identified by a name, then the name can be used instead of the identifier, using the **percent selector**: `ascli aoc admin res user show %name:john` where `john` is the user name.
|
1334
|
+
|
1335
|
+
Syntax: `%<field>:<value>`
|
1336
|
+
|
1337
|
+
> **Note:** The legacy option `id` is deprecated: `--id=1234` (options have a single value and thus do not provide the possibility to identify sub-entities)
|
1338
|
+
|
1303
1339
|
### <a id="extended"></a>Extended Value Syntax
|
1304
1340
|
|
1305
1341
|
Most options and arguments are specified by a simple string (e.g. username or url).
|
@@ -1318,27 +1354,30 @@ Decoders act like a function with its parameter on right hand side and are recog
|
|
1318
1354
|
|
1319
1355
|
The following decoders are supported:
|
1320
1356
|
|
1321
|
-
|
|
1322
|
-
|
1323
|
-
| base64
|
1324
|
-
| `csvt`
|
1325
|
-
| `env`
|
1326
|
-
| `file`
|
1327
|
-
| `json`
|
1328
|
-
| `lines`
|
1329
|
-
| `list`
|
1330
|
-
| `none`
|
1331
|
-
| `path`
|
1332
|
-
| preset
|
1333
|
-
| extend
|
1334
|
-
| `re`
|
1335
|
-
| `ruby`
|
1336
|
-
| secret
|
1337
|
-
| `stdin`
|
1338
|
-
| `uri`
|
1339
|
-
| `val`
|
1340
|
-
| `yaml`
|
1341
|
-
| `zlib`
|
1357
|
+
| Decoder | Parameter| Returns | Description |
|
1358
|
+
|----------|----------|---------|-------------|
|
1359
|
+
| `base64` | String | String | Decode a base64 encoded string |
|
1360
|
+
| `csvt` | String | Array | Decode a titled CSV value |
|
1361
|
+
| `env` | String | String | Read from a named env var name, e.g. `--password=@env:MYPASSVAR` |
|
1362
|
+
| `file` | String | String | Read value from specified file (prefix `~/` is replaced with the users home folder), e.g. `--key=@file:~/.ssh/mykey` |
|
1363
|
+
| `json` | String | Any | Decode JSON values (convenient to provide complex structures) |
|
1364
|
+
| `lines` | String | Array | Split a string in multiple lines and return an array |
|
1365
|
+
| `list` | String | Array | Split a string in multiple items taking first character as separator and return an array |
|
1366
|
+
| `none` | None | Nil | A null value |
|
1367
|
+
| `path` | String | String | Performs path expansion on specified path (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml` |
|
1368
|
+
| `preset` | String | Hash | Get whole option preset value by name. Sub-values can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]` |
|
1369
|
+
| `extend` | String | String | Evaluates embedded extended value syntax in string |
|
1370
|
+
| `re` | String | Regexp | Ruby Regular Expression (short for `@ruby:/.../`) |
|
1371
|
+
| `ruby` | String | Any | Execute specified Ruby code |
|
1372
|
+
| `secret` | None | String | Ask password interactively (hides input) |
|
1373
|
+
| `stdin` | None | String | Read from stdin (no value on right) |
|
1374
|
+
| `uri` | String | String | Read value from specified URL, e.g. `--fpac=@uri:http://serv/f.pac` |
|
1375
|
+
| `val` | String | String | Prevent decoders on the right to be decoded. e.g. `--key=@val:@file:foo` sets the option `key` to value `@file:foo`. |
|
1376
|
+
| `yaml` | String | Any | Decode YAML |
|
1377
|
+
| `zlib` | String | String | Un-compress zlib data |
|
1378
|
+
|
1379
|
+
> **Note:** A few commands support a value of type `Proc` (lambda expression).
|
1380
|
+
For example, the **Extended Value** `@ruby:'->(i){i["attr"]}'` is a lambda expression that returns the value of attribute `attr` of the `Hash` `i`.
|
1342
1381
|
|
1343
1382
|
To display the result of an extended value, use the `config echo` command.
|
1344
1383
|
|
@@ -1406,7 +1445,7 @@ ascli config echo @json:@extend:'{"hello":true,"version":"@preset:config.version
|
|
1406
1445
|
Example: Create a `Hash` from YAML provided as **heredoc**:
|
1407
1446
|
|
1408
1447
|
```bash
|
1409
|
-
ascli
|
1448
|
+
ascli config echo @yaml:@stdin: --format=json<<EOF
|
1410
1449
|
key1: value1
|
1411
1450
|
key2:
|
1412
1451
|
- item1
|
@@ -1528,7 +1567,7 @@ If necessary, the configuration file can opened in a text editor with:
|
|
1528
1567
|
ascli config open
|
1529
1568
|
```
|
1530
1569
|
|
1531
|
-
> **Note:**
|
1570
|
+
> **Note:** This starts the editor specified by env var `EDITOR` if defined.
|
1532
1571
|
|
1533
1572
|
Older format for commands are still supported:
|
1534
1573
|
|
@@ -1568,35 +1607,35 @@ ascli config preset get default _plugin_name_
|
|
1568
1607
|
|
1569
1608
|
Plugin `config` provides general commands for `ascli`:
|
1570
1609
|
|
1571
|
-
- Option preset,
|
1572
|
-
- wizard
|
1573
|
-
- vault
|
1610
|
+
- Option preset, configuration file operations
|
1611
|
+
- `wizard`
|
1612
|
+
- `vault`
|
1574
1613
|
- `ascp`
|
1575
1614
|
|
1576
1615
|
The default preset for `config` is read for any plugin invocation, this allows setting global options, such as `--log-level` or `--interactive`.
|
1577
1616
|
When `ascli` starts, it looks for the `default` Option preset and checks the value for `config`.
|
1578
1617
|
If set, it loads the options independently of the plugin used.
|
1579
1618
|
|
1580
|
-
> **Note:** If no global default is set by the user, `ascli` will use `global_common_defaults` when setting global parameters (e.g. `
|
1619
|
+
> **Note:** If no global default is set by the user, `ascli` will use `global_common_defaults` when setting global parameters (e.g. `config ascp use`)
|
1581
1620
|
>
|
1582
1621
|
> **Note:** If you don't know the name of the global preset, you can use `GLOBAL` to refer to it.
|
1583
1622
|
|
1584
1623
|
Show current default (global) Option preset (`config` plugin):
|
1585
1624
|
|
1586
1625
|
```console
|
1587
|
-
$ ascli
|
1626
|
+
$ ascli config preset get default config
|
1588
1627
|
global_common_defaults
|
1589
1628
|
```
|
1590
1629
|
|
1591
1630
|
```bash
|
1592
|
-
ascli
|
1631
|
+
ascli config preset set GLOBAL version_check_days 0
|
1593
1632
|
```
|
1594
1633
|
|
1595
1634
|
If the default global Option preset is not set, and you want to use a different name:
|
1596
1635
|
|
1597
1636
|
```bash
|
1598
|
-
ascli
|
1599
|
-
ascli
|
1637
|
+
ascli config preset set GLOBAL version_check_days 0
|
1638
|
+
ascli config preset set default config my_common_defaults
|
1600
1639
|
```
|
1601
1640
|
|
1602
1641
|
#### Config sample commands
|
@@ -1619,6 +1658,7 @@ config ascp use /usr/bin/ascp
|
|
1619
1658
|
config check_update
|
1620
1659
|
config coffee
|
1621
1660
|
config coffee --ui=text
|
1661
|
+
config coffee --ui=text --query=@json:'{"text":"true"}'
|
1622
1662
|
config detect https://faspex4.example.com/path
|
1623
1663
|
config detect https://faspex5.example.com/path
|
1624
1664
|
config detect https://node.example.com/path
|
@@ -1626,6 +1666,7 @@ config detect https://shares.example.com/path shares
|
|
1626
1666
|
config detect my_org aoc
|
1627
1667
|
config doc
|
1628
1668
|
config doc transfer-parameters
|
1669
|
+
config echo -- --special-string
|
1629
1670
|
config echo @base64:SGVsbG8gV29ybGQK
|
1630
1671
|
config echo @csvt:@stdin:
|
1631
1672
|
config echo @env:USER
|
@@ -1661,6 +1702,7 @@ config preset show conf_name
|
|
1661
1702
|
config preset unset conf_name param
|
1662
1703
|
config preset update conf_name --p1=v1 --p2=v2
|
1663
1704
|
config proxy_check --fpac=@file:examples/proxy.pac https://eudemo.asperademo.com --proxy-credentials=@list:,user,pass
|
1705
|
+
config pubkey @file:my_key
|
1664
1706
|
config vault create my_label @json:'{"password":"my_password_here","description":"my secret"}'
|
1665
1707
|
config vault delete my_label
|
1666
1708
|
config vault list
|
@@ -1671,8 +1713,8 @@ config wizard https://faspex5.example.com/path faspex5 --key-path=my_private_key
|
|
1671
1713
|
config wizard https://node.example.com/path node --username=test --password=test
|
1672
1714
|
config wizard https://orch.example.com/path orchestrator --username=test --password=test
|
1673
1715
|
config wizard https://shares.example.com/path shares --username=test --password=test
|
1674
|
-
config wizard my_org aoc --key-path= --username=my_user_email
|
1675
|
-
config wizard my_org aoc --key-path= --username=my_user_email --use-generic-client=yes
|
1716
|
+
config wizard my_org aoc --key-path=my_private_key --username=my_user_email
|
1717
|
+
config wizard my_org aoc --key-path=my_private_key --username=my_user_email --use-generic-client=yes
|
1676
1718
|
```
|
1677
1719
|
|
1678
1720
|
#### Format of file
|
@@ -1696,9 +1738,9 @@ demo_server:
|
|
1696
1738
|
We can see here:
|
1697
1739
|
|
1698
1740
|
- The configuration was created with `ascli` version 0.3.7
|
1699
|
-
-
|
1700
|
-
-
|
1701
|
-
-
|
1741
|
+
- The default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
1742
|
+
- The [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
1743
|
+
- The default [option preset](#lprt) to load in any case is : `cli_default`
|
1702
1744
|
|
1703
1745
|
Two [option presets](#lprt) are reserved:
|
1704
1746
|
|
@@ -1710,7 +1752,7 @@ The user may create as many [option presets](#lprt) as needed. For instance, a p
|
|
1710
1752
|
|
1711
1753
|
Values in the configuration also follow the [Extended Value Syntax](#extended).
|
1712
1754
|
|
1713
|
-
> **Note:**
|
1755
|
+
> **Note:** If the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
|
1714
1756
|
|
1715
1757
|
```bash
|
1716
1758
|
ascli config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/ascli/my_private_key"
|
@@ -1719,10 +1761,8 @@ ascli config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/ascli/m
|
|
1719
1761
|
This creates the [option preset](#lprt):
|
1720
1762
|
|
1721
1763
|
```yaml
|
1722
|
-
...
|
1723
1764
|
my_aoc_org:
|
1724
|
-
private_key: @file
|
1725
|
-
...
|
1765
|
+
private_key: "@file:/Users/laurent/.aspera/ascli/my_private_key"
|
1726
1766
|
```
|
1727
1767
|
|
1728
1768
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
@@ -1734,7 +1774,7 @@ Some options are global, some options are available only for some plugins. (the
|
|
1734
1774
|
Options are loaded using this algorithm:
|
1735
1775
|
|
1736
1776
|
- If option `--no-default` (or `-N`) is specified, then no default value is loaded for the plugin
|
1737
|
-
-
|
1777
|
+
- Else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
1738
1778
|
- If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or if the value is a `Hash`, it uses it as options values.
|
1739
1779
|
- Environment variables are evaluated
|
1740
1780
|
- Command line options are evaluated
|
@@ -1743,8 +1783,9 @@ Parameters are evaluated in the order of command line.
|
|
1743
1783
|
|
1744
1784
|
To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
|
1745
1785
|
|
1746
|
-
On command line, words in parameter names are separated by a dash
|
1747
|
-
|
1786
|
+
On command line, words in parameter names are separated by a dash (`-`).
|
1787
|
+
In configuration file, separator is an underscore.
|
1788
|
+
E.g. `--xxx-yyy` on command line gives `xxx_yyy` in configuration file.
|
1748
1789
|
|
1749
1790
|
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
1750
1791
|
|
@@ -1789,14 +1830,14 @@ ascli config wizard
|
|
1789
1830
|
#### Example of configuration for a plugin
|
1790
1831
|
|
1791
1832
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
1792
|
-
only username/password and url are required (either on command line, or from
|
1833
|
+
only username/password and url are required (either on command line, or from configuration file).
|
1793
1834
|
Those can usually be provided on the command line:
|
1794
1835
|
|
1795
1836
|
```bash
|
1796
1837
|
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=my_password_here
|
1797
1838
|
```
|
1798
1839
|
|
1799
|
-
This can also be provisioned in a
|
1840
|
+
This can also be provisioned in a configuration file:
|
1800
1841
|
|
1801
1842
|
- Build [option preset](#lprt)
|
1802
1843
|
|
@@ -1838,25 +1879,46 @@ ascli shares repo browse /
|
|
1838
1879
|
|
1839
1880
|
### <a id="vault"></a>Secret Vault
|
1840
1881
|
|
1841
|
-
|
1882
|
+
Secrets (e.g. passwords) are usually command options.
|
1842
1883
|
They can be provided on command line, env vars, files etc.
|
1843
|
-
|
1884
|
+
|
1885
|
+
For security reasons, those secrets shall not be exposed in clear, either:
|
1886
|
+
|
1887
|
+
- On terminal during input
|
1888
|
+
- In logs
|
1889
|
+
- In command output
|
1890
|
+
|
1891
|
+
Instead, they shall be hidden or encrypted.
|
1892
|
+
|
1893
|
+
Terminal output secret removal is controlled by option `show_secrets` (default: `no`).
|
1894
|
+
Log secret removal is controlled by option `log_secrets` (default: `no`).
|
1895
|
+
Mandatory command line options can be requested interactively (e.g. password) using option `interactive`.
|
1896
|
+
Or it is possible to use extended value `@secret:[name]` to ask for a secret interactively.
|
1897
|
+
It is also possible to enter an option as an environment variable, e.g. `ASCLI_PASSWORD` for option `password` and read the env var like this:
|
1898
|
+
|
1899
|
+
```bash
|
1900
|
+
read -s ASCLI_PASSWORD
|
1901
|
+
export ASCLI_PASSWORD
|
1902
|
+
```
|
1903
|
+
|
1904
|
+
Another possibility is to retrieve values from a secret vault.
|
1844
1905
|
|
1845
1906
|
The vault is used with options `vault` and `vault_password`.
|
1846
1907
|
|
1847
|
-
`vault`
|
1908
|
+
`vault` shall be a `Hash` describing the vault:
|
1848
1909
|
|
1849
1910
|
```json
|
1850
1911
|
{"type":"system","name":"ascli"}
|
1851
1912
|
```
|
1852
1913
|
|
1853
1914
|
`vault_password` specifies the password for the vault.
|
1854
|
-
|
1915
|
+
|
1916
|
+
Although it can be specified on command line, for security reason you should avoid exposing the secret.
|
1855
1917
|
For example it can be securely specified on command line like this:
|
1856
1918
|
|
1857
1919
|
```bash
|
1858
|
-
export ASCLI_VAULT_PASSWORD
|
1859
1920
|
read -s ASCLI_VAULT_PASSWORD
|
1921
|
+
export ASCLI_VAULT_PASSWORD
|
1860
1922
|
```
|
1861
1923
|
|
1862
1924
|
#### Vault: System key chain
|
@@ -1871,13 +1933,13 @@ It is possible to manage secrets in macOS key chain (only read supported current
|
|
1871
1933
|
|
1872
1934
|
#### Vault: Encrypted file
|
1873
1935
|
|
1874
|
-
It is possible to store and use secrets encrypted in a file
|
1936
|
+
It is possible to store and use secrets encrypted in a file using option `vault` set to:
|
1875
1937
|
|
1876
|
-
```
|
1877
|
-
|
1938
|
+
```json
|
1939
|
+
{"type":"file","name":"vault.bin"}
|
1878
1940
|
```
|
1879
1941
|
|
1880
|
-
`name` is the file path, absolute or relative to the
|
1942
|
+
`name` is the file path, absolute or relative to the configuration folder `ASCLI_HOME`.
|
1881
1943
|
|
1882
1944
|
#### Vault: Operations
|
1883
1945
|
|
@@ -1891,12 +1953,12 @@ Then secrets can be manipulated using commands:
|
|
1891
1953
|
- `delete`
|
1892
1954
|
|
1893
1955
|
```bash
|
1894
|
-
ascli
|
1956
|
+
ascli config vault create mylabel @json:'{"password":"my_password_here","description":"for this account"}'
|
1895
1957
|
```
|
1896
1958
|
|
1897
1959
|
#### <a id="config_finder"></a>Configuration Finder
|
1898
1960
|
|
1899
|
-
When a secret is needed by a sub command, the command can search for existing configurations in the
|
1961
|
+
When a secret is needed by a sub command, the command can search for existing configurations in the configuration file.
|
1900
1962
|
|
1901
1963
|
The lookup is done by comparing the service URL and username (or access key).
|
1902
1964
|
|
@@ -1906,20 +1968,20 @@ A passwords can be saved in clear in a [option preset](#lprt) together with othe
|
|
1906
1968
|
Example:
|
1907
1969
|
|
1908
1970
|
```bash
|
1909
|
-
ascli
|
1971
|
+
ascli config preset update myconf --url=... --username=... --password=...
|
1910
1972
|
```
|
1911
1973
|
|
1912
1974
|
For a more secure storage one can do:
|
1913
1975
|
|
1914
1976
|
```bash
|
1915
|
-
ascli
|
1977
|
+
ascli config preset update myconf --url=... --username=... --password=@val:@vault:myconf.password
|
1916
1978
|
```
|
1917
1979
|
|
1918
1980
|
```bash
|
1919
|
-
ascli
|
1981
|
+
ascli config vault create myconf @json:'{"password":"my_password_here"}'
|
1920
1982
|
```
|
1921
1983
|
|
1922
|
-
> **Note:**
|
1984
|
+
> **Note:** Use `@val:` in front of `@vault:` so that the extended value is not evaluated.
|
1923
1985
|
|
1924
1986
|
### <a id="private_key"></a>Private Key
|
1925
1987
|
|
@@ -1955,10 +2017,16 @@ ascli config genkey ${PRIVKEYFILE} 4096
|
|
1955
2017
|
|
1956
2018
|
> **Note:** `ascli` uses the `openssl` library.
|
1957
2019
|
|
2020
|
+
To display the public key of a private key:
|
2021
|
+
|
2022
|
+
```bash
|
2023
|
+
ascli config pubkey @file:${PRIVKEYFILE}
|
2024
|
+
```
|
2025
|
+
|
1958
2026
|
To display the version of **openssl** used in `ascli`:
|
1959
2027
|
|
1960
2028
|
```bash
|
1961
|
-
ascli config echo @ruby:OpenSSL::OPENSSL_VERSION
|
2029
|
+
ascli config echo @ruby:OpenSSL::OPENSSL_VERSION --format=text
|
1962
2030
|
```
|
1963
2031
|
|
1964
2032
|
#### `ssh-keygen`
|
@@ -2014,8 +2082,8 @@ Ruby's default values can be overridden using env vars: `SSL_CERT_FILE` and `SSL
|
|
2014
2082
|
> **Note:** One can display those values like this:
|
2015
2083
|
|
2016
2084
|
```bash
|
2017
|
-
ascli
|
2018
|
-
ascli
|
2085
|
+
ascli config echo @ruby:OpenSSL::X509::DEFAULT_CERT_DIR --format=text
|
2086
|
+
ascli config echo @ruby:OpenSSL::X509::DEFAULT_CERT_FILE --format=text
|
2019
2087
|
```
|
2020
2088
|
|
2021
2089
|
`ascp` also needs to validate certificates when using **WSS**.
|
@@ -2023,13 +2091,13 @@ ascli conf echo @ruby:OpenSSL::X509::DEFAULT_CERT_FILE --format=text
|
|
2023
2091
|
> **Note:** `ascli` overrides the default hardcoded location used by `ascp` for WSS (e.g. on macOS: `/Library/Aspera/ssl`) and uses the same locations as specified in `cert_stores` (using `-i` switch of `ascp`). Hardcoded locations can be found using:
|
2024
2092
|
|
2025
2093
|
```bash
|
2026
|
-
|
2094
|
+
ascli config ascp info --fields=openssldir
|
2027
2095
|
```
|
2028
2096
|
|
2029
2097
|
or
|
2030
2098
|
|
2031
2099
|
```bash
|
2032
|
-
ascli
|
2100
|
+
strings $(ascli config ascp info --fields=ascp)|grep -w OPENSSLDIR
|
2033
2101
|
```
|
2034
2102
|
|
2035
2103
|
To update trusted root certificates for `ascli`:
|
@@ -2039,29 +2107,29 @@ Typically done by updating the system's root certificate store.
|
|
2039
2107
|
An up-to-date version of the certificate bundle can be retrieved with:
|
2040
2108
|
|
2041
2109
|
```bash
|
2042
|
-
ascli
|
2110
|
+
ascli config echo @uri:https://curl.haxx.se/ca/cacert.pem --format=text
|
2043
2111
|
```
|
2044
2112
|
|
2045
2113
|
To download that certificate store:
|
2046
2114
|
|
2047
2115
|
```bash
|
2048
|
-
ascli
|
2116
|
+
ascli config echo @uri:https://curl.haxx.se/ca/cacert.pem --format=text > /tmp/cacert.pem
|
2049
2117
|
```
|
2050
2118
|
|
2051
|
-
Then, use this store by setting the option
|
2119
|
+
Then, use this store by setting the option `cert_stores` or env var `SSL_CERT_FILE`.
|
2052
2120
|
|
2053
|
-
To trust a certificate (e.g. self-signed), provided that the `CN` is correct
|
2121
|
+
To trust a specific certificate (e.g. self-signed), **provided that the `CN` is correct**, save the certificate to a file:
|
2054
2122
|
|
2055
2123
|
```bash
|
2056
|
-
ascli
|
2124
|
+
ascli config remote_certificate https://localhost:9092 > myserver.pem
|
2057
2125
|
```
|
2058
2126
|
|
2059
|
-
> **Note:**
|
2127
|
+
> **Note:** The saved certificate shows the CN as first line.
|
2060
2128
|
|
2061
2129
|
Then, use this file as certificate store (e.g. here, Node API):
|
2062
2130
|
|
2063
2131
|
```bash
|
2064
|
-
ascli
|
2132
|
+
ascli config echo @uri:https://localhost:9092/ping --cert-stores=myserver.pem
|
2065
2133
|
```
|
2066
2134
|
|
2067
2135
|
### Image and video thumbnails
|
@@ -2072,18 +2140,18 @@ It's also available when using the `show` command of `preview` plugin.
|
|
2072
2140
|
|
2073
2141
|
The following options can be specified in the option `query`:
|
2074
2142
|
|
2075
|
-
|
|
2143
|
+
| Option | Description |
|
2076
2144
|
|------------|-------------|
|
2077
|
-
| text |
|
2078
|
-
| double |
|
2145
|
+
| text | Display text instead of image (Bool) |
|
2146
|
+
| double | Display double text resolution (half characters) (Bool) |
|
2079
2147
|
| font_ratio | Font height/width ratio in terminal (Float) |
|
2080
2148
|
|
2081
2149
|
### <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
|
2082
2150
|
|
2083
2151
|
Some actions may require the use of a graphical tool:
|
2084
2152
|
|
2085
|
-
-
|
2086
|
-
-
|
2153
|
+
- A browser for Aspera on Cloud authentication (web auth method)
|
2154
|
+
- A text editor for configuration file edition
|
2087
2155
|
|
2088
2156
|
By default `ascli` assumes that a graphical environment is available on Windows, and on other systems, rely on the presence of the `DISPLAY` environment variable.
|
2089
2157
|
It is also possible to force the graphical mode with option --ui :
|
@@ -2110,16 +2178,16 @@ Available levels: `debug`, `info`, `warn`, `error`.
|
|
2110
2178
|
|
2111
2179
|
Examples:
|
2112
2180
|
|
2113
|
-
-
|
2181
|
+
- Display debugging log on `stdout`:
|
2114
2182
|
|
2115
2183
|
```bash
|
2116
|
-
ascli
|
2184
|
+
ascli config pre over --log-level=debug --logger=stdout
|
2117
2185
|
```
|
2118
2186
|
|
2119
|
-
-
|
2187
|
+
- Log errors to `syslog`:
|
2120
2188
|
|
2121
2189
|
```bash
|
2122
|
-
ascli
|
2190
|
+
ascli config pre over --log-level=error --logger=syslog
|
2123
2191
|
```
|
2124
2192
|
|
2125
2193
|
When `ascli` is used interactively in a shell, the shell itself will usually log executed commands in the history file.
|
@@ -2135,14 +2203,18 @@ It will display the exact content of HTTP requests and responses.
|
|
2135
2203
|
|
2136
2204
|
### <a id="http_options"></a>HTTP socket parameters
|
2137
2205
|
|
2138
|
-
To ignore SSL certificate for any address/port, use option: `insecure`, i.e. `--insecure=yes`.
|
2139
|
-
To ignore SSL certificate for specific address/port, use option `ignore_certificate`, set to an `Array` of URL for which certificate will be ignored (only the address and port are matched), e.g. `--ignore-certificate=@list:,https://127.0.0.1:9092`
|
2206
|
+
To ignore SSL certificate for **any** address/port, use option: `insecure`, i.e. `--insecure=yes`.
|
2207
|
+
To ignore SSL certificate for a list of specific address/port, use option `ignore_certificate`, set to an `Array` of URL for which certificate will be ignored (only the address and port are matched), e.g. `--ignore-certificate=@list:,https://127.0.0.1:9092`
|
2140
2208
|
|
2141
2209
|
> **Note:** Ignoring certificate also applies to `ascp`'s wss.
|
2142
2210
|
|
2211
|
+
Ignoring a certificate is not recommended, it is better to add the certificate to the trusted store.
|
2212
|
+
So, a warning is displayed when a certificate is ignored.
|
2213
|
+
To disable the warning, use option `silent_insecure` set to `no`.
|
2214
|
+
|
2143
2215
|
HTTP connection parameters (not `ascp` wss) can be adjusted using option `http_options`:
|
2144
2216
|
|
2145
|
-
|
|
2217
|
+
| Parameter | Default |
|
2146
2218
|
|----------------------|---------|
|
2147
2219
|
| `read_timeout` | 60 |
|
2148
2220
|
| `write_timeout` | 60 |
|
@@ -2153,7 +2225,7 @@ Values are in set **seconds** and can be of type either integer or float.
|
|
2153
2225
|
Default values are the ones of Ruby:
|
2154
2226
|
For a full list, refer to the Ruby library: [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html).
|
2155
2227
|
|
2156
|
-
Like any other option, those can be set either on command line, or in
|
2228
|
+
Like any other option, those can be set either on command line, or in configuration file, either in a global preset or server-specific one.
|
2157
2229
|
|
2158
2230
|
Example:
|
2159
2231
|
|
@@ -2165,13 +2237,15 @@ ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
|
2165
2237
|
|
2166
2238
|
There are several types of network connections, each of them use a different mechanism to define a (forward) **proxy**:
|
2167
2239
|
|
2168
|
-
-
|
2169
|
-
- Legacy Aspera HTTP/S Fallback
|
2170
|
-
- Aspera FASP
|
2240
|
+
- REST calls (APIs) and HTTP Gateway
|
2241
|
+
- `ascp` WSS and Legacy Aspera HTTP/S Fallback
|
2242
|
+
- `ascp` SSH and UDP (Aspera FASP)
|
2171
2243
|
|
2172
2244
|
Refer to the following sections.
|
2173
2245
|
|
2174
|
-
|
2246
|
+
#### Proxy for REST and HTTP Gateway
|
2247
|
+
|
2248
|
+
REST API calls and transfers based on HTTP Gateway both use Ruby `Net::HTTP` gem.
|
2175
2249
|
|
2176
2250
|
There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is used.
|
2177
2251
|
|
@@ -2185,7 +2259,7 @@ Refer to [Ruby find proxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_p
|
|
2185
2259
|
export http_proxy=http://proxy.example.com:3128
|
2186
2260
|
```
|
2187
2261
|
|
2188
|
-
|
2262
|
+
Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) javascript value.
|
2189
2263
|
To read the script from a URL (`http:`, `https:` and `file:`), use prefix: `@uri:`.
|
2190
2264
|
A minimal script can be specified to define the use of a local proxy:
|
2191
2265
|
|
@@ -2197,7 +2271,7 @@ The result of a PAC file can be tested with command: `config proxy_check`.
|
|
2197
2271
|
Example, using command line option:
|
2198
2272
|
|
2199
2273
|
```bash
|
2200
|
-
ascli
|
2274
|
+
ascli config proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:3128;DIRECT";}' http://example.com
|
2201
2275
|
```
|
2202
2276
|
|
2203
2277
|
```text
|
@@ -2230,11 +2304,11 @@ ascli --proxy-credentials=@json:'["__username_here__","__password_here__"]' ...
|
|
2230
2304
|
ascli --proxy-credentials=@list::__username_here__:__password_here__ ...
|
2231
2305
|
```
|
2232
2306
|
|
2233
|
-
|
2307
|
+
#### Proxy for Legacy Aspera HTTP/S Fallback
|
2234
2308
|
|
2235
2309
|
Only supported with the `direct` agent: To specify a proxy for legacy HTTP fallback, use `ascp` native option `-x` and `ascp_args`: `--transfer-info=@json:'{"ascp_args":["-x","url_here"]}'`. Alternatively, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url`.
|
2236
2310
|
|
2237
|
-
|
2311
|
+
#### FASP proxy (forward) for transfers
|
2238
2312
|
|
2239
2313
|
To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) parameter: `proxy` (only supported with the `direct` agent).
|
2240
2314
|
|
@@ -2244,9 +2318,9 @@ The `config` plugin also allows specification for the use of a local FASP **clie
|
|
2244
2318
|
It provides the following commands for `ascp` subcommand:
|
2245
2319
|
|
2246
2320
|
- `show` : shows the path of `ascp` used
|
2247
|
-
- `use` :
|
2321
|
+
- `use` : specify the ascp path to use
|
2248
2322
|
- `products` : list Aspera transfer products available locally
|
2249
|
-
- `connect` : list
|
2323
|
+
- `connect` : list and download connect client versions available on internet
|
2250
2324
|
|
2251
2325
|
#### Show path of currently used `ascp`
|
2252
2326
|
|
@@ -2272,7 +2346,7 @@ ascli config ascp info
|
|
2272
2346
|
|
2273
2347
|
#### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
2274
2348
|
|
2275
|
-
By default, `ascli` uses any found local product with `ascp`, including SDK.
|
2349
|
+
By default, `ascli` uses any found local product with `ascp`, including Transfer SDK.
|
2276
2350
|
|
2277
2351
|
To temporarily use an alternate `ascp` path use option `ascp_path` (`--ascp-path=`)
|
2278
2352
|
|
@@ -2327,7 +2401,7 @@ ascli config ascp products list
|
|
2327
2401
|
|
2328
2402
|
If no `ascp` is selected, this is equivalent to using option: `--use-product=FIRST`.
|
2329
2403
|
|
2330
|
-
Using the option use_product finds the `ascp` binary of the selected product.
|
2404
|
+
Using the option `use_product` finds the `ascp` binary of the selected product.
|
2331
2405
|
|
2332
2406
|
To permanently use the `ascp` of a product:
|
2333
2407
|
|
@@ -2395,7 +2469,7 @@ There are currently 3 agents, set with option `transfer`:
|
|
2395
2469
|
- [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
2396
2470
|
|
2397
2471
|
> **Note:** All transfer operations are seen from the point of view of the agent.
|
2398
|
-
For example, a node agent executing an
|
2472
|
+
For example, a node agent executing an **upload**, or **package send** operation
|
2399
2473
|
will effectively push files to the related server from the agent node.
|
2400
2474
|
|
2401
2475
|
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of **native** `ascp` options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
@@ -2405,25 +2479,24 @@ Specific options for agents are provided with option `transfer_info`, cumulative
|
|
2405
2479
|
#### <a id="agt_direct"></a>Direct
|
2406
2480
|
|
2407
2481
|
The `direct` agent directly executes a local `ascp`.
|
2408
|
-
This is the default agent for `ascli
|
2409
|
-
This is equivalent to option `--transfer=direct`.
|
2482
|
+
This is the default agent for `ascli` (option `--transfer=direct`).
|
2410
2483
|
`ascli` will search locally installed Aspera products, including SDK, and use `ascp` from that component.
|
2411
2484
|
Refer to section [FASP](#client).
|
2412
2485
|
|
2413
2486
|
The `transfer_info` option accepts the following optional parameters to control multi-session, Web Socket Session and Resume policy:
|
2414
2487
|
|
2415
|
-
| Name
|
2416
|
-
|
2417
|
-
| `wss`
|
2418
|
-
| `ascp_args`
|
2419
|
-
| `spawn_timeout_sec`
|
2420
|
-
| `spawn_delay_sec`
|
2421
|
-
| `multi_incr_udp`
|
2422
|
-
| `resume`
|
2423
|
-
| `resume.iter_max`
|
2488
|
+
| Name | Type | Description |
|
2489
|
+
|------------------------|-------|-------------|
|
2490
|
+
| `wss` | Bool | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
2491
|
+
| `ascp_args` | Array | Array of strings with native `ascp` arguments<br/>Use this instead of deprecated `EX_ascp_args`.<br/>Default: [] |
|
2492
|
+
| `spawn_timeout_sec` | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
|
2493
|
+
| `spawn_delay_sec` | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
|
2494
|
+
| `multi_incr_udp` | Bool | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
|
2495
|
+
| `resume` | Hash | Resume<br/>parameters<br/>See below |
|
2496
|
+
| `resume.iter_max` | int | Resume<br/>Max number of retry on error<br/>Default: 7 |
|
2424
2497
|
| `resume.sleep_initial` | int | Resume<br/>First Sleep before retry<br/>Default: 2 |
|
2425
2498
|
| `resume.sleep_factor` | int | Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2 |
|
2426
|
-
| `resume.sleep_max`
|
2499
|
+
| `resume.sleep_max` | int | Resume<br/>Default: 60 |
|
2427
2500
|
|
2428
2501
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
2429
2502
|
Sleep between iterations is:
|
@@ -2493,7 +2566,7 @@ asconfigurator -x 'set_node_data;transfer_in_bandwidth_aggregate_trunk_id,1'
|
|
2493
2566
|
asconfigurator -x 'set_node_data;transfer_out_bandwidth_aggregate_trunk_id,2'
|
2494
2567
|
```
|
2495
2568
|
|
2496
|
-
But this command is not available on clients, so edit the file `aspera.conf`, you can find the location with: `ascli
|
2569
|
+
But this command is not available on clients, so edit the file `aspera.conf`, you can find the location with: `ascli config ascp info --fields=aspera_conf` and modify the sections `default` and `trunks` like this for a global 100 Mbps virtual link:
|
2497
2570
|
|
2498
2571
|
```xml
|
2499
2572
|
<?xml version='1.0' encoding='UTF-8'?>
|
@@ -2555,9 +2628,9 @@ Parameters provided in option `transfer_info` are:
|
|
2555
2628
|
| Name | Type | Description |
|
2556
2629
|
|----------|--------|-------------|
|
2557
2630
|
| url | string | URL of the node API</br>Mandatory |
|
2558
|
-
| username | string |
|
2559
|
-
| password | string |
|
2560
|
-
| root_id | string |
|
2631
|
+
| username | string | Node api user or access key</br>Mandatory |
|
2632
|
+
| password | string | Password, secret or bearer token</br>Mandatory |
|
2633
|
+
| root_id | string | Root file id</br>Mandatory only for bearer token |
|
2561
2634
|
|
2562
2635
|
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
2563
2636
|
|
@@ -2585,9 +2658,9 @@ Parameters provided in option `transfer_info` are:
|
|
2585
2658
|
| Name | Type | Description |
|
2586
2659
|
|------------------------|--------|---------------------------------------|
|
2587
2660
|
| url | string | URL of the HTTP GW</br>Mandatory |
|
2588
|
-
| upload_chunk_size | int | Size in bytes of chunks for upload<br/>Default: 64000 |
|
2589
|
-
| api_version | string |
|
2590
|
-
| synchronous | bool |
|
2661
|
+
| upload_chunk_size | int | Size in bytes of chunks for upload<br/>Default: `64000` |
|
2662
|
+
| api_version | string | Force use of version (`v1`, `v2`)<br/>Default: `v2` |
|
2663
|
+
| synchronous | bool | Wait for each message acknowledgment<br/>Default: `false` |
|
2591
2664
|
|
2592
2665
|
Example:
|
2593
2666
|
|
@@ -2606,11 +2679,11 @@ Options for `transfer_info` are:
|
|
2606
2679
|
|
2607
2680
|
| Name | Type | Description |
|
2608
2681
|
|----------|--------|-------------|
|
2609
|
-
| url | string | IP address and port listened by the daemon</br>Mandatory<br/>Default:
|
2610
|
-
| external | bool | Use external daemon, do not start<br/>Default: false |
|
2611
|
-
| keep | bool | Keep the daemon running after exiting `ascli`<br/>Default: false |
|
2682
|
+
| `url` | string | IP address and port listened by the daemon</br>Mandatory<br/>Default: `:0` |
|
2683
|
+
| `external` | bool | Use external daemon, do not start<br/>Default: `false` |
|
2684
|
+
| `keep` | bool | Keep the daemon running after exiting `ascli`<br/>Default: `false` |
|
2612
2685
|
|
2613
|
-
> **Note:**
|
2686
|
+
> **Note:** If port zero is specified in the URL, then the daemon will listen on a random available port. If no address is specified, then `127.0.0.1` is used.
|
2614
2687
|
|
2615
2688
|
The gem `grpc` was removed from dependencies, as it requires compilation of a native part.
|
2616
2689
|
So, to use the Transfer SDK you should install this gem:
|
@@ -2638,11 +2711,11 @@ On Windows the compilation may fail for various reasons (3.1.1):
|
|
2638
2711
|
Some commands lead to file transfer (upload/download).
|
2639
2712
|
All parameters necessary for this transfer are described in a [*transfer-spec*](#transferspec) (Transfer Specification), such as:
|
2640
2713
|
|
2641
|
-
-
|
2642
|
-
-
|
2643
|
-
-
|
2644
|
-
-
|
2645
|
-
-
|
2714
|
+
- Server address
|
2715
|
+
- Transfer user name
|
2716
|
+
- Credentials
|
2717
|
+
- File list
|
2718
|
+
- Etc...
|
2646
2719
|
|
2647
2720
|
`ascli` builds the [*transfer-spec*](#transferspec) internally as a `Hash`.
|
2648
2721
|
It is not necessary to provide additional parameters on the command line for this transfer.
|
@@ -2660,13 +2733,13 @@ This is especially useful for `ascp` command line parameters not supported in th
|
|
2660
2733
|
|
2661
2734
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
2662
2735
|
|
2663
|
-
-
|
2664
|
-
-
|
2736
|
+
- Common to all [Transfer Agent](#agents)
|
2737
|
+
- Not dependent on command line limitations (special characters...)
|
2665
2738
|
|
2666
2739
|
### <a id="transferparams"></a>Transfer Parameters
|
2667
2740
|
|
2668
2741
|
All standard [*transfer-spec*](#transferspec) parameters can be specified.
|
2669
|
-
[*transfer-spec*](#transferspec) can also be saved/overridden in the
|
2742
|
+
[*transfer-spec*](#transferspec) can also be saved/overridden in the configuration file.
|
2670
2743
|
|
2671
2744
|
References:
|
2672
2745
|
|
@@ -2830,9 +2903,9 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2830
2903
|
ascli server upload --sources=@args --src-type=list ~/mysample.file secondfile
|
2831
2904
|
```
|
2832
2905
|
|
2833
|
-
-
|
2906
|
+
- An [Extended Value](#extended) with type **Array of String**
|
2834
2907
|
|
2835
|
-
> **Note:** extended values can be tested with the command `
|
2908
|
+
> **Note:** extended values can be tested with the command `config echo`
|
2836
2909
|
|
2837
2910
|
Examples:
|
2838
2911
|
|
@@ -2903,10 +2976,10 @@ The transfer destination is normally expected to designate a destination folder.
|
|
2903
2976
|
|
2904
2977
|
But there is one exception: The destination specifies the new item name when the following are met:
|
2905
2978
|
|
2906
|
-
-
|
2907
|
-
-
|
2908
|
-
-
|
2909
|
-
-
|
2979
|
+
- There is a single source item (file or folder)
|
2980
|
+
- Transfer spec `create_dir` is not set to `true` (`ascp` option `-d` not provided)
|
2981
|
+
- Destination is not an existing folder
|
2982
|
+
- The `dirname` of destination is an existing folder
|
2910
2983
|
|
2911
2984
|
For this reason it is recommended to set `create_dir` to `true` for consistent behavior between single and multiple items transfer, this is the default in `ascli`.
|
2912
2985
|
|
@@ -3054,7 +3127,7 @@ For example, on Linux it is convenient to create a wrapping script, e.g. `cron_a
|
|
3054
3127
|
|
3055
3128
|
```bash
|
3056
3129
|
#!/bin/bash
|
3057
|
-
# load the
|
3130
|
+
# load the Ruby environment
|
3058
3131
|
. /etc/profile.d/rvm.sh
|
3059
3132
|
rvm use 2.6 --quiet
|
3060
3133
|
# set a timeout protection, just in case ascli is frozen
|
@@ -3072,7 +3145,7 @@ crontab<<EOF
|
|
3072
3145
|
EOF
|
3073
3146
|
```
|
3074
3147
|
|
3075
|
-
> **Note:**
|
3148
|
+
> **Note:** Logging options are kept here in the cron file instead of configuration file to allow execution on command line with output on command line.
|
3076
3149
|
|
3077
3150
|
### <a id="locking"></a>Locking for exclusive execution
|
3078
3151
|
|
@@ -3119,7 +3192,7 @@ Several **PVCL** adapters are available, one is embedded in `ascp`, the others a
|
|
3119
3192
|
The list of supported **PVCL** adapters can be retrieved with command:
|
3120
3193
|
|
3121
3194
|
```bash
|
3122
|
-
ascli
|
3195
|
+
ascli config ascp info --fields=@re:'^pvcl'
|
3123
3196
|
```
|
3124
3197
|
|
3125
3198
|
```output
|
@@ -3135,7 +3208,7 @@ stdio-tar v1
|
|
3135
3208
|
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
3136
3209
|
|
3137
3210
|
Those adapters can be used wherever a file path is used in `ascp` including configuration.
|
3138
|
-
They act as a pseudo
|
3211
|
+
They act as a **pseudo drive**.
|
3139
3212
|
|
3140
3213
|
The simplified format is:
|
3141
3214
|
|
@@ -3166,7 +3239,7 @@ where:
|
|
3166
3239
|
- `filename` is the name that will be assigned to the file on the destination
|
3167
3240
|
- `filesize` is the number of bytes that will be sent (in decimal).
|
3168
3241
|
|
3169
|
-
> **Note:**
|
3242
|
+
> **Note:** Characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
3170
3243
|
|
3171
3244
|
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
3172
3245
|
|
@@ -3179,12 +3252,12 @@ faux:///dirname?<arg1>=<val1>&...
|
|
3179
3252
|
where:
|
3180
3253
|
|
3181
3254
|
- `dirname` is the folder name and can contain `/` to specify a subfolder.
|
3182
|
-
-
|
3255
|
+
- Supported arguments are:
|
3183
3256
|
|
3184
3257
|
| Name | Type | Description |
|
3185
3258
|
|--------|------|-------------|
|
3186
|
-
|count |int |
|
3187
|
-
|file |string|Basename for files<br>Default:
|
3259
|
+
|count |int |Number of files<br/>Mandatory|
|
3260
|
+
|file |string|Basename for files<br>Default: `file`|
|
3188
3261
|
|size |int |Size of first file.<br>Default: 0|
|
3189
3262
|
|inc |int |Increment applied to determine next file size<br>Default: 0|
|
3190
3263
|
|seq |enum |Sequence in determining next file size<br/>Values: random, sequential<br/>Default: sequential|
|
@@ -3233,7 +3306,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
3233
3306
|
```text
|
3234
3307
|
ascli -h
|
3235
3308
|
NAME
|
3236
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
3309
|
+
ascli -- a command line tool for Aspera Applications (v4.16.0)
|
3237
3310
|
|
3238
3311
|
SYNOPSIS
|
3239
3312
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -3265,6 +3338,7 @@ OPTIONS: global
|
|
3265
3338
|
--interactive=ENUM Use interactive input of missing params: [no], yes
|
3266
3339
|
--ask-options=ENUM Ask even optional options: [no], yes
|
3267
3340
|
--format=ENUM Output format: text, nagios, ruby, json, jsonpp, yaml, [table], csv
|
3341
|
+
--output=VALUE Destination for results (String)
|
3268
3342
|
--display=ENUM Output only some information: [info], data, error
|
3269
3343
|
--fields=VALUE Comma separated list of: fields, or ALL, or DEF (String, Array, Regexp, Proc)
|
3270
3344
|
--select=VALUE Select only some items in lists: column, value (Hash, Proc)
|
@@ -3287,7 +3361,7 @@ OPTIONS: global
|
|
3287
3361
|
--pid-file=VALUE Write process identifier to file, delete on exit (String)
|
3288
3362
|
|
3289
3363
|
COMMAND: config
|
3290
|
-
SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test file flush_tokens folder gem genkey initdemo open plugins preset proxy_check remote_certificate smtp_settings throw vault wizard
|
3364
|
+
SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test file flush_tokens folder gem genkey initdemo open plugins preset proxy_check pubkey remote_certificate smtp_settings throw vault wizard
|
3291
3365
|
OPTIONS:
|
3292
3366
|
--home=VALUE Home folder for tool (String)
|
3293
3367
|
--config-file=VALUE Path to YAML file with preset configuration
|
@@ -3317,7 +3391,8 @@ OPTIONS:
|
|
3317
3391
|
--notify-to=VALUE Email recipient for notification of transfers
|
3318
3392
|
--notify-template=VALUE Email ERB template for notification of transfers
|
3319
3393
|
--insecure=ENUM Do not validate any HTTPS certificate: [no], yes
|
3320
|
-
--ignore-certificate=VALUE
|
3394
|
+
--ignore-certificate=VALUE Do not validate HTTPS certificate for these URLs (Array)
|
3395
|
+
--silent-insecure=ENUM Issue a warning if certificate is ignored: no, [yes]
|
3321
3396
|
--cert-stores=VALUE List of folder with trusted certificates (Array, String)
|
3322
3397
|
--http-options=VALUE Options for HTTP/S socket (Hash)
|
3323
3398
|
--cache-tokens=ENUM Save and reuse Oauth tokens: no, [yes]
|
@@ -3327,7 +3402,7 @@ OPTIONS:
|
|
3327
3402
|
--to-folder=VALUE Destination folder for transferred files
|
3328
3403
|
--sources=VALUE How list of transferred files is provided (@args,@ts,Array)
|
3329
3404
|
--src-type=ENUM Type of file list: [list], pair
|
3330
|
-
--transfer=ENUM Type of transfer agent: [direct],
|
3405
|
+
--transfer=ENUM Type of transfer agent: trsdk, [direct], httpgw, connect, node, alpha
|
3331
3406
|
--transfer-info=VALUE Parameters for transfer agent (Hash)
|
3332
3407
|
|
3333
3408
|
|
@@ -3337,11 +3412,10 @@ OPTIONS:
|
|
3337
3412
|
--url=VALUE URL of application, e.g. https://faspex.example.com/aspera/faspex
|
3338
3413
|
--username=VALUE Username to log in
|
3339
3414
|
--password=VALUE User's password
|
3340
|
-
--type=ENUM Type of user/group for operations: [any], local, ldap, saml
|
3341
3415
|
|
3342
3416
|
|
3343
3417
|
COMMAND: node
|
3344
|
-
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse central delete download events health http_node_download info license mkdir mkfile mklink rename search service slash space ssync stream sync transfer upload watch_folder
|
3418
|
+
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse central delete download events health http_node_download info license mkdir mkfile mklink rename search service simulator slash space ssync stream sync transfer upload watch_folder
|
3345
3419
|
OPTIONS:
|
3346
3420
|
--url=VALUE URL of application, e.g. https://faspex.example.com/aspera/faspex
|
3347
3421
|
--username=VALUE Username to log in
|
@@ -3406,7 +3480,7 @@ OPTIONS:
|
|
3406
3480
|
--auth=ENUM OAuth type of authentication: boot, web, [jwt]
|
3407
3481
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
3408
3482
|
--passphrase=VALUE OAuth JWT RSA private key passphrase
|
3409
|
-
--box=VALUE Package inbox, either shared inbox name or one of
|
3483
|
+
--box=VALUE Package inbox, either shared inbox name or one of: inbox, inbox_history, inbox_all, inbox_all_history, outbox, outbox_history, pending, pending_history, all or ALL
|
3410
3484
|
--shared-folder=VALUE Send package with files from shared folder
|
3411
3485
|
--group-type=ENUM Type of shared box: [shared_inboxes], workgroups
|
3412
3486
|
|
@@ -3515,7 +3589,7 @@ OPTIONS:
|
|
3515
3589
|
|
3516
3590
|
```
|
3517
3591
|
|
3518
|
-
> **Note:**
|
3592
|
+
> **Note:** Commands and parameter values can be written in short form.
|
3519
3593
|
|
3520
3594
|
### Bulk creation and deletion of resources
|
3521
3595
|
|
@@ -3528,12 +3602,12 @@ This option is available only for some of the resources: if you need it: try and
|
|
3528
3602
|
`ascli` uses a plugin mechanism.
|
3529
3603
|
The first level command (just after `ascli` on the command line) is the name of the concerned plugin which will execute the command.
|
3530
3604
|
Each plugin usually represents commands sent to a specific application.
|
3531
|
-
For instance, the plugin `faspex` allows operations on
|
3605
|
+
For instance, the plugin `faspex` allows operations on **Aspera Faspex**.
|
3532
3606
|
|
3533
3607
|
Available plugins can be found using command:
|
3534
3608
|
|
3535
3609
|
```bash
|
3536
|
-
ascli
|
3610
|
+
ascli config plugin list
|
3537
3611
|
```
|
3538
3612
|
|
3539
3613
|
```output
|
@@ -3561,7 +3635,7 @@ ascli --show-config --select=@json:'{"key":"plugin_folder"}'
|
|
3561
3635
|
You can create the skeleton of a new plugin like this:
|
3562
3636
|
|
3563
3637
|
```bash
|
3564
|
-
ascli
|
3638
|
+
ascli config plugin create foo .
|
3565
3639
|
```
|
3566
3640
|
|
3567
3641
|
```output
|
@@ -3590,7 +3664,7 @@ Here is a sample invocation :
|
|
3590
3664
|
|
3591
3665
|
```text
|
3592
3666
|
ascli config wizard
|
3593
|
-
option: url> https://
|
3667
|
+
option: url> https://_your_instance_.ibmaspera.com
|
3594
3668
|
Detected: Aspera on Cloud
|
3595
3669
|
Preparing preset: aoc_myorg
|
3596
3670
|
Please provide path to your private RSA key, or empty to generate one:
|
@@ -3602,21 +3676,25 @@ option: username> john@example.com
|
|
3602
3676
|
Updating profile with new key
|
3603
3677
|
creating new config preset: aoc_myorg
|
3604
3678
|
Setting config preset as default for aspera
|
3605
|
-
saving
|
3679
|
+
saving configuration file
|
3606
3680
|
Done.
|
3607
3681
|
You can test with:
|
3608
3682
|
ascli aoc user profile show
|
3609
3683
|
```
|
3610
3684
|
|
3611
|
-
|
3685
|
+
> **Note:** In above example, replace `https://_your_instance_.ibmaspera.com` with your actual AoC URL.
|
3686
|
+
|
3687
|
+
Optionally, it is possible to create a new organization-specific integration, i.e. client application identification.
|
3612
3688
|
For this, specify the option: `--use-generic-client=no`.
|
3613
3689
|
|
3614
3690
|
If you already know the application, and want to limit the detection to it, provide url and plugin name:
|
3615
3691
|
|
3616
3692
|
```bash
|
3617
|
-
ascli config wizard
|
3693
|
+
ascli config wizard _your_instance_ aoc
|
3618
3694
|
```
|
3619
3695
|
|
3696
|
+
> **Note:** In above example, replace `_your_instance_` with the first part of your actual AoC URL: `https://_your_instance_.ibmaspera.com`.
|
3697
|
+
|
3620
3698
|
### <a id="aocmanual"></a>Configuration: Using manual setup
|
3621
3699
|
|
3622
3700
|
> **Note:** If you used the wizard (recommended): skip this section.
|
@@ -3635,7 +3713,7 @@ For a **quick start**, follow the mandatory and sufficient section: [API Client
|
|
3635
3713
|
|
3636
3714
|
For a more convenient, browser-less, experience follow the [JWT](#jwt) section (auth=jwt) in addition to Client Registration.
|
3637
3715
|
|
3638
|
-
In Oauth, a
|
3716
|
+
In Oauth, a **Bearer** token is generated to authenticate REST calls.
|
3639
3717
|
Bearer tokens are valid for a period of time defined (by the AoC app, configurable by admin) at its creation.
|
3640
3718
|
`ascli` saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
3641
3719
|
|
@@ -3649,34 +3727,37 @@ Else you can use a specific OAuth API client_id, the first step is to declare `a
|
|
3649
3727
|
|
3650
3728
|
Let's start by a registration with web based authentication (auth=web):
|
3651
3729
|
|
3652
|
-
- Open a web browser, log to your instance: e.g. `https://
|
3730
|
+
- Open a web browser, log to your instance: e.g. `https://_your_instance_.ibmaspera.com/`
|
3731
|
+
(use your actual AoC instance URL)
|
3653
3732
|
- Go to Apps → Admin → Organization → Integrations
|
3654
|
-
- Click
|
3655
|
-
- Client Name
|
3656
|
-
- Redirect URIs
|
3657
|
-
- Origins
|
3658
|
-
- uncheck
|
3733
|
+
- Click **Create New**
|
3734
|
+
- **Client Name**: `ascli`
|
3735
|
+
- **Redirect URIs**: `http://localhost:12345`
|
3736
|
+
- **Origins**: `localhost`
|
3737
|
+
- uncheck **Prompt users to allow client to access**
|
3659
3738
|
- leave the JWT part for now
|
3660
|
-
- Save
|
3739
|
+
- **Save**
|
3661
3740
|
|
3662
|
-
> **Note:**
|
3741
|
+
> **Note:** For web based authentication, `ascli` listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For ``ascli`, HTTP is required, and 12345 is the default port.
|
3663
3742
|
|
3664
|
-
Once the client is registered, a
|
3743
|
+
Once the client is registered, a **Client ID** and **Secret** are created, these values will be used in the next step.
|
3665
3744
|
|
3666
3745
|
#### <a id="aocpreset"></a>[option preset](#lprt) for Aspera on Cloud
|
3667
3746
|
|
3668
3747
|
If you did not use the wizard, you can also manually create a [option preset](#lprt) for `ascli` in its configuration file.
|
3669
3748
|
|
3670
|
-
Lets create
|
3749
|
+
Lets create a [option preset](#lprt) called: `my_aoc_org` using `ask` for interactive input (client info from previous step):
|
3671
3750
|
|
3672
3751
|
```bash
|
3673
3752
|
ascli config preset ask my_aoc_org url client_id client_secret
|
3674
|
-
option: url> https://
|
3753
|
+
option: url> https://_your_instance_.ibmaspera.com/
|
3675
3754
|
option: client_id> my_client_id_here
|
3676
3755
|
option: client_secret> my_client_secret_here
|
3677
3756
|
updated: my_aoc_org
|
3678
3757
|
```
|
3679
3758
|
|
3759
|
+
> **Note:** In above example, replace `https://_your_instance_.ibmaspera.com` with your actual AoC URL.
|
3760
|
+
|
3680
3761
|
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
3681
3762
|
|
3682
3763
|
Define this [option preset](#lprt) as default configuration for the `aspera` plugin:
|
@@ -3700,12 +3781,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
3700
3781
|
|
3701
3782
|
- Graphically
|
3702
3783
|
|
3703
|
-
- Open a web browser, log to your instance: `https://
|
3784
|
+
- Open a web browser, log to your instance: `https://_your_instance_.ibmaspera.com/`
|
3785
|
+
(Use your actual AoC instance URL)
|
3704
3786
|
- Go to Apps → Admin → Organization → Integrations
|
3705
3787
|
- Click on the previously created application
|
3706
|
-
- select tab :
|
3707
|
-
- Modify options if necessary, for instance: activate both options in section
|
3708
|
-
-
|
3788
|
+
- select tab : **JSON Web Token Auth**
|
3789
|
+
- Modify options if necessary, for instance: activate both options in section **Settings**
|
3790
|
+
- **Save**
|
3709
3791
|
|
3710
3792
|
- Using command line
|
3711
3793
|
|
@@ -3737,11 +3819,12 @@ The public key must be assigned to your user. This can be done in two manners:
|
|
3737
3819
|
|
3738
3820
|
Open the previously generated public key located here: `$HOME/.aspera/ascli/my_private_key.pub`
|
3739
3821
|
|
3740
|
-
- Open a web browser, log to your instance: `https://
|
3822
|
+
- Open a web browser, log to your instance: `https://_your_instance_.ibmaspera.com/`
|
3823
|
+
(Use your actual AoC instance URL)
|
3741
3824
|
- Click on the user's icon (top right)
|
3742
|
-
- Select
|
3743
|
-
- Paste the
|
3744
|
-
- Click on
|
3825
|
+
- Select **Account Settings**
|
3826
|
+
- Paste the Public Key PEM value in the **Public Key** section
|
3827
|
+
- Click on **Submit**
|
3745
3828
|
|
3746
3829
|
##### Using command line
|
3747
3830
|
|
@@ -3766,15 +3849,15 @@ ascli aoc user profile modify @ruby:'{"public_key"=>File.read(File.expand_path("
|
|
3766
3849
|
modified
|
3767
3850
|
```
|
3768
3851
|
|
3769
|
-
> **Note:**
|
3852
|
+
> **Note:** The `aspera user info show` command can be used to verify modifications.
|
3770
3853
|
|
3771
3854
|
#### [option preset](#lprt) modification for JWT
|
3772
3855
|
|
3773
3856
|
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the following:
|
3774
3857
|
|
3775
|
-
-
|
3776
|
-
-
|
3777
|
-
-
|
3858
|
+
- Change auth method to JWT
|
3859
|
+
- Provide location of private key
|
3860
|
+
- Provide username to login as (OAuth **subject**)
|
3778
3861
|
|
3779
3862
|
Execute:
|
3780
3863
|
|
@@ -3782,7 +3865,10 @@ Execute:
|
|
3782
3865
|
ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/ascli/my_private_key --username=someuser@example.com
|
3783
3866
|
```
|
3784
3867
|
|
3785
|
-
> **Note:**
|
3868
|
+
> **Note:** The private key argument represents the actual PEM string.
|
3869
|
+
In order to read the content from a file, use the `@file:` prefix.
|
3870
|
+
But if the @file: argument is used as is, it will read the file and set in the configuration file.
|
3871
|
+
So, to keep the `@file:` tag in the configuration file, the `@val:` prefix is added.
|
3786
3872
|
|
3787
3873
|
After this last step, commands do not require web login anymore.
|
3788
3874
|
|
@@ -3837,7 +3923,7 @@ ascli aoc admin res node v4 1234 --secret=_ak_secret_here_ bearer_token_node /
|
|
3837
3923
|
|
3838
3924
|
The `admin` command allows several administrative tasks (and require admin privilege).
|
3839
3925
|
|
3840
|
-
It allows actions (create, update, delete) on
|
3926
|
+
It allows actions (create, update, delete) on **resources**: users, group, nodes, workspace, etc... with the `admin resource` command.
|
3841
3927
|
|
3842
3928
|
#### Listing resources
|
3843
3929
|
|
@@ -3898,9 +3984,9 @@ Resources are identified by a unique `id`, as well as a unique `name` (case inse
|
|
3898
3984
|
To execute an action on a specific resource, select it using one of those methods:
|
3899
3985
|
|
3900
3986
|
- **recommended**: give id directly on command line **after the action**: `aoc admin res node show 123`
|
3901
|
-
-
|
3902
|
-
-
|
3903
|
-
-
|
3987
|
+
- Give name on command line **after the action**: `aoc admin res node show name abc`
|
3988
|
+
- Provide option `id` : `aoc admin res node show 123`
|
3989
|
+
- Provide option `name` : `aoc admin res node show --name=abc`
|
3904
3990
|
|
3905
3991
|
#### <a id="res_create"></a>Creating a resource
|
3906
3992
|
|
@@ -3971,7 +4057,7 @@ It can also support filters and send notification using option `notify_to`. a te
|
|
3971
4057
|
|
3972
4058
|
`mytemplate.erb`:
|
3973
4059
|
|
3974
|
-
```
|
4060
|
+
```yaml
|
3975
4061
|
From: <%=from_name%> <<%=from_email%>>
|
3976
4062
|
To: <<%=ev['user_email']%>>
|
3977
4063
|
Subject: <%=ev['files_completed']%> files received
|
@@ -3985,7 +4071,7 @@ Thank you.
|
|
3985
4071
|
|
3986
4072
|
The environment provided contains the following additional variable:
|
3987
4073
|
|
3988
|
-
- ev : all details on the transfer event
|
4074
|
+
- `ev` : all details on the transfer event
|
3989
4075
|
|
3990
4076
|
Example:
|
3991
4077
|
|
@@ -4008,7 +4094,7 @@ The option `default_ports` ([yes]/no) allows ascli to retrieve the server ports
|
|
4008
4094
|
|
4009
4095
|
#### Using ATS
|
4010
4096
|
|
4011
|
-
Refer to section
|
4097
|
+
Refer to section **Examples** of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
|
4012
4098
|
|
4013
4099
|
#### Files with type `link`
|
4014
4100
|
|
@@ -4107,7 +4193,7 @@ ascli aoc user workspaces list
|
|
4107
4193
|
+------+----------------------------+
|
4108
4194
|
```
|
4109
4195
|
|
4110
|
-
#### Example: Create a sub access key in a
|
4196
|
+
#### Example: Create a sub access key in a `node`
|
4111
4197
|
|
4112
4198
|
Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
4113
4199
|
|
@@ -4217,7 +4303,7 @@ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:
|
|
4217
4303
|
+-------------------------------+
|
4218
4304
|
```
|
4219
4305
|
|
4220
|
-
#### Example: List
|
4306
|
+
#### Example: List **Limited** users
|
4221
4307
|
|
4222
4308
|
```bash
|
4223
4309
|
ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
@@ -4268,7 +4354,7 @@ In this example, a user has access to a workspace where two shared folders are l
|
|
4268
4354
|
First, setup the environment (skip if already done)
|
4269
4355
|
|
4270
4356
|
```bash
|
4271
|
-
ascli
|
4357
|
+
ascli config wizard --url=https://sedemo.ibmaspera.com --username=someuser@example.com
|
4272
4358
|
```
|
4273
4359
|
|
4274
4360
|
```output
|
@@ -4287,13 +4373,13 @@ Once updated or validated, press enter.
|
|
4287
4373
|
|
4288
4374
|
creating new config preset: aoc_sedemo
|
4289
4375
|
Setting config preset as default for aspera
|
4290
|
-
saving
|
4376
|
+
saving configuration file
|
4291
4377
|
Done.
|
4292
4378
|
You can test with:
|
4293
4379
|
ascli aoc user profile show
|
4294
4380
|
```
|
4295
4381
|
|
4296
|
-
This creates the option preset
|
4382
|
+
This creates the option preset `aoc_[org name]` to allow seamless command line access and sets it as default for aspera on cloud.
|
4297
4383
|
|
4298
4384
|
Then, create two shared folders located in two regions, in your files home, in a workspace.
|
4299
4385
|
|
@@ -4335,12 +4421,12 @@ ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res cli
|
|
4335
4421
|
AoC nodes as actually composed with two related entities:
|
4336
4422
|
|
4337
4423
|
- An access key created on the Transfer Server (HSTS/ATS)
|
4338
|
-
-
|
4424
|
+
- A `node` resource in the AoC application.
|
4339
4425
|
|
4340
4426
|
The web UI allows creation of both entities in one shot.
|
4341
4427
|
For more flexibility, `ascli` allows this in two separate steps.
|
4342
4428
|
|
4343
|
-
> **Note:** When selecting
|
4429
|
+
> **Note:** When selecting **Use existing access key** in the web UI, this actually skips access key creation (first step).
|
4344
4430
|
|
4345
4431
|
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
4346
4432
|
|
@@ -4379,10 +4465,10 @@ Refer to section [File list](#file_list)
|
|
4379
4465
|
|
4380
4466
|
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file id` API).
|
4381
4467
|
|
4382
|
-
Source files are located on
|
4468
|
+
Source files are located on **Aspera on cloud**, when :
|
4383
4469
|
|
4384
|
-
-
|
4385
|
-
-
|
4470
|
+
- The server is Aspera on Cloud, and executing a download or recv
|
4471
|
+
- The agent is Aspera on Cloud, and executing an upload or send
|
4386
4472
|
|
4387
4473
|
In this case:
|
4388
4474
|
|
@@ -4538,7 +4624,7 @@ For creation, parameters are the same as for node API [permissions](https://deve
|
|
4538
4624
|
`ascli` expects the same payload for creation, but it will automatically populate required tags if needed.
|
4539
4625
|
|
4540
4626
|
Also, the pseudo key `with` is available: it will lookup the name in the contacts and fill the proper type and id.
|
4541
|
-
The pseudo parameter `link_name` allows changing default
|
4627
|
+
The pseudo parameter `link_name` allows changing default **shared as** name.
|
4542
4628
|
|
4543
4629
|
- List permissions on a shared folder as user
|
4544
4630
|
|
@@ -4579,7 +4665,7 @@ Procedure to send a file from org1 to org2:
|
|
4579
4665
|
- Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
4580
4666
|
- Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
4581
4667
|
- Check that access works and locate the destination folder `mydestfolder`
|
4582
|
-
-
|
4668
|
+
- Execute the following:
|
4583
4669
|
|
4584
4670
|
```bash
|
4585
4671
|
ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
@@ -4612,7 +4698,9 @@ For instructions, refer to section `find` for plugin `node`.
|
|
4612
4698
|
### AoC sample commands
|
4613
4699
|
|
4614
4700
|
```bash
|
4701
|
+
aoc admin analytics transfers nodes
|
4615
4702
|
aoc admin analytics transfers organization --query=@json:'{"status":"completed","direction":"receive"}' --notify-to=my_email_external --notify-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
4703
|
+
aoc admin analytics transfers users --once_only=yes
|
4616
4704
|
aoc admin ats access_key create --cloud=aws --region=my_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_bucket_name","credentials":{"access_key_id":"my_bucket_key","secret_access_key":"my_bucket_secret"},"path":"/"}}'
|
4617
4705
|
aoc admin ats access_key create --cloud=softlayer --region=my_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"my_secret_here","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_bucket_name","credentials":{"access_key_id":"my_bucket_key","secret_access_key":"my_bucket_secret"},"path":"/"}}'
|
4618
4706
|
aoc admin ats access_key delete ak1ibmcloud
|
@@ -4691,25 +4779,29 @@ aoc files upload --to-folder=/ test_file.bin --url=my_public_link_folder_no_pass
|
|
4691
4779
|
aoc files upload --to-folder=/testsrc test_file.bin
|
4692
4780
|
aoc files upload --workspace=my_other_workspace --to-folder=my_other_folder test_file.bin --transfer=node --transfer-info=@json:@stdin:
|
4693
4781
|
aoc files v3 info
|
4694
|
-
aoc gateway --pid-file=
|
4782
|
+
aoc gateway --pid-file=pid_aocfxgw https://localhost:12345/aspera/faspex &
|
4695
4783
|
aoc org --url=my_public_link_recv_from_aoc_user
|
4696
4784
|
aoc organization
|
4697
4785
|
aoc packages browse package_id3 /contents
|
4698
4786
|
aoc packages list
|
4699
4787
|
aoc packages list --query=@json:'{"dropbox_name":"my_shared_inbox_name","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
4700
|
-
aoc packages
|
4701
|
-
aoc packages
|
4702
|
-
aoc packages
|
4703
|
-
aoc packages
|
4704
|
-
aoc packages send --workspace=my_workspace_shared_inbox @json:'{"name":"
|
4705
|
-
aoc packages send --workspace=my_workspace_shared_inbox @json:'{"name":"
|
4706
|
-
aoc packages send @json:'{"name":"
|
4707
|
-
aoc packages send @json:'{"name":"
|
4708
|
-
aoc packages send @json:'{"name":"
|
4709
|
-
aoc packages send @json:'{"name":"
|
4788
|
+
aoc packages receive ALL --once-only=yes --to-folder=. --lock-port=12345
|
4789
|
+
aoc packages receive ALL --once-only=yes --to-folder=. --lock-port=12345 --query=@json:'{"dropbox_name":"my_shared_inbox_name","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
4790
|
+
aoc packages receive INIT --once-only=yes --query=@json:'{"dropbox_name":"my_shared_inbox_name"}'
|
4791
|
+
aoc packages receive package_id3 --to-folder=.
|
4792
|
+
aoc packages send --workspace=my_workspace_shared_inbox --validate-metadata=yes @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_shared_inbox_meta"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' test_file.bin
|
4793
|
+
aoc packages send --workspace=my_workspace_shared_inbox --validate-metadata=yes @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_shared_inbox_meta"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' test_file.bin
|
4794
|
+
aoc packages send --workspace=my_workspace_shared_inbox --validate-metadata=yes @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_shared_inbox_meta"],"metadata":{"Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' test_file.bin
|
4795
|
+
aoc packages send --workspace=my_workspace_shared_inbox @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_shared_inbox_name"]}' test_file.bin
|
4796
|
+
aoc packages send @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_email_external"]}' --new-user-option=@json:'{"package_contact":true}' test_file.bin
|
4797
|
+
aoc packages send @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_email_internal"],"note":"my note"}' test_file.bin
|
4798
|
+
aoc packages send @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE"}' test_file.bin --url=my_public_link_send_aoc_user --password=my_public_link_send_use_pass
|
4799
|
+
aoc packages send @json:'{"name":"$(notdir test) PACKAGE_TITLE_BASE"}' test_file.bin --url=my_public_link_send_shared_inbox
|
4710
4800
|
aoc packages shared_inboxes list
|
4711
4801
|
aoc remind --username=my_user_email
|
4712
4802
|
aoc servers
|
4803
|
+
aoc user pref modify @json:'{"default_language":"en-us"}'
|
4804
|
+
aoc user pref show
|
4713
4805
|
aoc user profile modify @json:'{"name":"dummy change"}'
|
4714
4806
|
aoc user profile show
|
4715
4807
|
aoc user workspaces current
|
@@ -4720,9 +4812,9 @@ aoc user workspaces list
|
|
4720
4812
|
|
4721
4813
|
ATS is usable either :
|
4722
4814
|
|
4723
|
-
-
|
4815
|
+
- From an AoC subscription : ascli aoc admin ats : use AoC authentication
|
4724
4816
|
|
4725
|
-
-
|
4817
|
+
- Or from an IBM Cloud subscription : ascli ats : use IBM Cloud API key authentication
|
4726
4818
|
|
4727
4819
|
### IBM Cloud ATS : Creation of api key
|
4728
4820
|
|
@@ -4872,6 +4964,7 @@ then commands `ascp` (for transfers) and `ascmd` (for file operations) are execu
|
|
4872
4964
|
|
4873
4965
|
```bash
|
4874
4966
|
server browse /
|
4967
|
+
server browse / --password=@none: --ssh-options=@json:'{"number_of_password_prompts":0}' --ssh-keys=$aspera_key_path
|
4875
4968
|
server browse my_inside_folder/test_file.bin
|
4876
4969
|
server browse my_upload_folder/target_hot
|
4877
4970
|
server cp my_inside_folder/test_file.bin my_upload_folder/200KB.2
|
@@ -4899,7 +4992,7 @@ server upload --sources=@ts --transfer-info=@json:'{"ascp_args":["--file-list","
|
|
4899
4992
|
server upload --sources=@ts --transfer-info=@json:'{"ascp_args":["--file-pair-list","file_pair_list.txt"]}'
|
4900
4993
|
server upload --sources=@ts --ts=@json:'{"paths":[{"source":"test_file.bin","destination":"my_inside_folder/other_name_4"}]}' --transfer=trsdk
|
4901
4994
|
server upload --src-type=pair 'test_file.bin' my_inside_folder/other_name_2 --notify-to=my_email_external --transfer-info=@json:'{"ascp_args":["-l","100m"]}'
|
4902
|
-
server upload --src-type=pair --sources=@json:'["test_file.bin","my_inside_folder/other_name_3"]' --transfer-info=@json:'{"quiet":false}' --progress=no
|
4995
|
+
server upload --src-type=pair --sources=@json:'["test_file.bin","my_inside_folder/other_name_3"]' --transfer-info=@json:'{"quiet":false}' --ts=@json:'{"use_ascp4":true}' --progress=no
|
4903
4996
|
server upload --src-type=pair test_file.bin my_upload_folder/other_name_5 --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"my_secret_here","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
|
4904
4997
|
server upload --to-folder=my_upload_folder/target_hot --lock-port=12345 --transfer-info=@json:'{"ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
4905
4998
|
```
|
@@ -5014,9 +5107,9 @@ The authentication is `username` and `password` or `access_key` and `secret` thr
|
|
5014
5107
|
|
5015
5108
|
It is possible to do **gen3/node user** operations:
|
5016
5109
|
|
5017
|
-
- browse
|
5018
|
-
-
|
5019
|
-
- delete
|
5110
|
+
- `browse`
|
5111
|
+
- Transfer (`upload` / `download` / `sync`)
|
5112
|
+
- `delete`
|
5020
5113
|
- ...
|
5021
5114
|
|
5022
5115
|
When using an access key, so called **gen4/access key** API is also supported through sub commands using `access_keys do self`.
|
@@ -5038,7 +5131,7 @@ It recursively scans storage to find files/folders matching a criteria and then
|
|
5038
5131
|
`[filter_expr]` is either:
|
5039
5132
|
|
5040
5133
|
- Optional (default) : all files and folder are selected
|
5041
|
-
-
|
5134
|
+
- Type `String` : the expression is similar to shell globing, refer to **Ruby** function: [`File.fnmatch`](https://ruby-doc.org/3.2.2/File.html#method-c-fnmatch)
|
5042
5135
|
- Type `Proc` : the expression is a Ruby lambda that takes one argument: a `Hash` that contains the current folder entry to test. Refer to the following examples.
|
5043
5136
|
|
5044
5137
|
Examples of expressions:
|
@@ -5093,13 +5186,13 @@ The following are examples of `ruby_lambda` to be provided in the following temp
|
|
5093
5186
|
->(f){f["name"].match?(/\.gif$/)}
|
5094
5187
|
```
|
5095
5188
|
|
5096
|
-
`ascli` commands can be piped in order to combine operations, such as
|
5189
|
+
`ascli` commands can be piped in order to combine operations, such as **find and delete**:
|
5097
5190
|
|
5098
5191
|
```bash
|
5099
5192
|
ascli node access_keys do self find / @ruby:'->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}' --fields=path --format=csv | ascli node --bulk=yes delete @lines:@stdin:
|
5100
5193
|
```
|
5101
5194
|
|
5102
|
-
> **Note:**
|
5195
|
+
> **Note:** The pipe `|` character on the last line.
|
5103
5196
|
|
5104
5197
|
### Central
|
5105
5198
|
|
@@ -5114,7 +5207,7 @@ ascli node central file list
|
|
5114
5207
|
|
5115
5208
|
By providing the `validator` option, offline transfer validation can be done.
|
5116
5209
|
|
5117
|
-
> **Note:**
|
5210
|
+
> **Note:** See later in this doc, refer to HSTS doc.
|
5118
5211
|
|
5119
5212
|
### Sync
|
5120
5213
|
|
@@ -5134,14 +5227,14 @@ Use the command `ascli node stream create --ts=@json:<value>`, with [*transfer-s
|
|
5134
5227
|
{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"my_pass_here"}
|
5135
5228
|
```
|
5136
5229
|
|
5137
|
-
###
|
5230
|
+
### Watchfolder
|
5138
5231
|
|
5139
5232
|
Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/entsrv/3.7.4/es_admin_linux/webhelp/index.html#watchfolder_external/dita/json_conf.html) for watch folder creation.
|
5140
5233
|
|
5141
5234
|
`ascli` supports remote operations through the node API. Operations are:
|
5142
5235
|
|
5143
5236
|
- Start `watchd` and `watchfolderd` services running as a system user having access to files
|
5144
|
-
-
|
5237
|
+
- Configure a **watchfolder** to define automated transfers
|
5145
5238
|
|
5146
5239
|
```bash
|
5147
5240
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -5153,6 +5246,8 @@ ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","
|
|
5153
5246
|
|
5154
5247
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
5155
5248
|
|
5249
|
+
The following command lists one file that requires validation, and assign it to the unique validator identifier provided:
|
5250
|
+
|
5156
5251
|
```bash
|
5157
5252
|
ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
5158
5253
|
```
|
@@ -5165,6 +5260,8 @@ ascli node central file list --validator=ascli --data=@json:'{"file_transfer_fil
|
|
5165
5260
|
+--------------+--------------+------------+--------------------------------------+
|
5166
5261
|
```
|
5167
5262
|
|
5263
|
+
To update the status of the file, use the following command:
|
5264
|
+
|
5168
5265
|
```bash
|
5169
5266
|
ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
5170
5267
|
```
|
@@ -5178,8 +5275,8 @@ updated
|
|
5178
5275
|
Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner.
|
5179
5276
|
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
|
5180
5277
|
Simply create an **Aspera Transfer Service** instance, which provides access to the node API.
|
5181
|
-
Then create a configuration for the **SHOD** instance in the configuration file: in section
|
5182
|
-
Create another configuration for the Azure ATS instance: in section
|
5278
|
+
Then create a configuration for the **SHOD** instance in the configuration file: in section **shares**, a configuration named: `aws_shod`.
|
5279
|
+
Create another configuration for the Azure ATS instance: in section **node**, named `azure_ats`.
|
5183
5280
|
Then execute the following command:
|
5184
5281
|
|
5185
5282
|
```bash
|
@@ -5188,7 +5285,7 @@ ascli node download /share/sourcefile --to-folder=/destination_folder --preset=a
|
|
5188
5285
|
|
5189
5286
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.
|
5190
5287
|
|
5191
|
-
###
|
5288
|
+
### Node file information
|
5192
5289
|
|
5193
5290
|
When node api is used with an **Access key**, extra information can be retrieved, such as preview.
|
5194
5291
|
|
@@ -5213,7 +5310,7 @@ ascli aoc files thumbnail /preview_samples/Aspera.mpg
|
|
5213
5310
|
|
5214
5311
|
> **Note:** To specify the file by its file id, use the selector syntax: `%id:_file_id_here_`
|
5215
5312
|
>
|
5216
|
-
> **Note:** To force textual display of the preview on **iTerm**, prefix command with: `env -u TERM_PROGRAM -u LC_TERMINAL`
|
5313
|
+
> **Note:** To force textual display of the preview on **iTerm**, prefix command with: `env -u TERM_PROGRAM -u LC_TERMINAL` or use option: ``
|
5217
5314
|
|
5218
5315
|
### Create access key
|
5219
5316
|
|
@@ -5249,8 +5346,8 @@ A bearer token is authorized on the node by creating `permissions` on a **folder
|
|
5249
5346
|
|
5250
5347
|
Bearer tokens can be generated using command `bearer_token`: it takes two arguments:
|
5251
5348
|
|
5252
|
-
-
|
5253
|
-
-
|
5349
|
+
- The private key used to sign the token
|
5350
|
+
- The token information, which is a `Hash` containing the following elements:
|
5254
5351
|
|
5255
5352
|
| parameter | Default | type | description |
|
5256
5353
|
| ------------------------ |-----------------------------|-----------|----------------------------------------------------------|
|
@@ -5270,7 +5367,7 @@ Bearer tokens can be generated using command `bearer_token`: it takes two argume
|
|
5270
5367
|
|
5271
5368
|
#### Bearer token: Environment
|
5272
5369
|
|
5273
|
-
- If a self-managed Aspera node is used, then a
|
5370
|
+
- If a self-managed Aspera node is used, then a **node user admin** must be created:
|
5274
5371
|
It has no `docroot` but has at least one file restriction (for testing, one can use `*` to accept creation of an access key with any storage root path).
|
5275
5372
|
Refer to the Aspera HSTS documentation.
|
5276
5373
|
|
@@ -5289,7 +5386,7 @@ Let's assume that the access key was created, and a default configuration is set
|
|
5289
5386
|
|
5290
5387
|
```bash
|
5291
5388
|
my_private_pem=./myorgkey.pem
|
5292
|
-
ascli
|
5389
|
+
ascli config genkey $my_private_pem
|
5293
5390
|
```
|
5294
5391
|
|
5295
5392
|
> **Note:** This key is not used for authentication, it is used to sign bearer tokens.
|
@@ -5345,9 +5442,9 @@ Let's assume that the access key was created, and a default configuration is set
|
|
5345
5442
|
|
5346
5443
|
Now, let's assume we are the user, the only information received are:
|
5347
5444
|
|
5348
|
-
-
|
5349
|
-
-
|
5350
|
-
-
|
5445
|
+
- The url of the node API
|
5446
|
+
- A Bearer token
|
5447
|
+
- A file `id` for which we have access
|
5351
5448
|
|
5352
5449
|
Let's use it:
|
5353
5450
|
|
@@ -5481,7 +5578,7 @@ Setting config preset as default for faspex5
|
|
5481
5578
|
Done.
|
5482
5579
|
You can test with:
|
5483
5580
|
ascli faspex5 user profile show
|
5484
|
-
Saving
|
5581
|
+
Saving configuration file.
|
5485
5582
|
```
|
5486
5583
|
|
5487
5584
|
> **Note:** Include the public key `BEGIN` and `END` lines when pasting in the user profile.
|
@@ -5532,9 +5629,9 @@ As usual, typically a user will create preset to avoid having to type these opti
|
|
5532
5629
|
Example:
|
5533
5630
|
|
5534
5631
|
```bash
|
5535
|
-
ascli
|
5632
|
+
ascli config preset update myf5 --auth=jwt --client-id=_client_id_here_ --client-secret=my_secret_here --username=_username_here_ --private-key=@file:.../path/to/key.pem
|
5536
5633
|
|
5537
|
-
ascli
|
5634
|
+
ascli config preset set default faspx5 myf5
|
5538
5635
|
|
5539
5636
|
ascli faspex5 user profile show
|
5540
5637
|
```
|
@@ -5570,7 +5667,7 @@ For `boot` method: (will be removed in future)
|
|
5570
5667
|
Use this token as password and use `--auth=boot`.
|
5571
5668
|
|
5572
5669
|
```bash
|
5573
|
-
ascli
|
5670
|
+
ascli config preset update f5boot --url=https://localhost/aspera/faspex --auth=boot --password=_token_here_
|
5574
5671
|
```
|
5575
5672
|
|
5576
5673
|
### Faspex 5 sample commands
|
@@ -5590,35 +5687,36 @@ faspex5 admin res node list
|
|
5590
5687
|
faspex5 admin res oauth_clients list
|
5591
5688
|
faspex5 admin res registrations list
|
5592
5689
|
faspex5 admin res saml_configs list
|
5593
|
-
faspex5 admin res shared_inboxes invite %name:
|
5690
|
+
faspex5 admin res shared_inboxes invite %name:my_shared_box_name johnny@example.com
|
5594
5691
|
faspex5 admin res shared_inboxes list
|
5595
|
-
faspex5 admin res shared_inboxes
|
5596
|
-
faspex5 admin res shared_inboxes members %name:
|
5597
|
-
faspex5 admin res shared_inboxes members %name:
|
5598
|
-
faspex5 admin res shared_inboxes members %name:
|
5692
|
+
faspex5 admin res shared_inboxes list --query=@json:'{"all":true}'
|
5693
|
+
faspex5 admin res shared_inboxes members %name:my_shared_box_name create %name:john@example.com
|
5694
|
+
faspex5 admin res shared_inboxes members %name:my_shared_box_name delete %name:john@example.com
|
5695
|
+
faspex5 admin res shared_inboxes members %name:my_shared_box_name delete %name:johnny@example.com
|
5696
|
+
faspex5 admin res shared_inboxes members %name:my_shared_box_name list
|
5599
5697
|
faspex5 admin res workgroups list
|
5600
5698
|
faspex5 admin smtp show
|
5601
5699
|
faspex5 admin smtp test my_email_external
|
5602
5700
|
faspex5 bearer_token
|
5603
|
-
faspex5 gateway --pid-file=
|
5701
|
+
faspex5 gateway --pid-file=pid_f5_fxgw https://localhost:12346/aspera/faspex &
|
5604
5702
|
faspex5 health
|
5605
|
-
faspex5 packages list --box=
|
5703
|
+
faspex5 packages list --box=my_shared_box_name
|
5606
5704
|
faspex5 packages list --box=my_workgroup --group-type=workgroups
|
5607
5705
|
faspex5 packages list --query=@json:'{"mailbox":"inbox","state":["released"]}'
|
5608
|
-
faspex5 packages receive --box=
|
5706
|
+
faspex5 packages receive --box=my_shared_box_name package_box_id1 --to-folder=.
|
5609
5707
|
faspex5 packages receive --box=my_workgroup --group-type=workgroups workgroup_package_id1 --to-folder=.
|
5610
5708
|
faspex5 packages receive ALL --once-only=yes --to-folder=.
|
5611
5709
|
faspex5 packages receive INIT --once-only=yes
|
5612
5710
|
faspex5 packages receive f5_p31 --to-folder=. --ts=@json:'{"content_protection_password":"my_secret_here"}'
|
5613
5711
|
faspex5 packages send --shared-folder=%name:my_shared_folder_name @json:'{"title":"test title","recipients":["my_email_internal"]}' my_shared_folder_file
|
5614
|
-
faspex5 packages send @json:'{"title":"test title","recipients":["
|
5712
|
+
faspex5 packages send @json:'{"title":"test title","recipients":["my_shared_box_name"],"metadata":{"Options":"Opt1","TextInput":"example text"}}' test_file.bin
|
5615
5713
|
faspex5 packages send @json:'{"title":"test title","recipients":["my_workgroup"]}' test_file.bin
|
5616
5714
|
faspex5 packages send @json:'{"title":"test title","recipients":[{"name":"my_username"}]my_meta}' test_file.bin --ts=@json:'{"content_protection_password":"my_passphrase_here"}'
|
5617
|
-
faspex5 packages show --box=
|
5715
|
+
faspex5 packages show --box=my_shared_box_name package_box_id1
|
5618
5716
|
faspex5 packages show --box=my_workgroup --group-type=workgroups workgroup_package_id1
|
5619
5717
|
faspex5 packages show f5_p31
|
5620
5718
|
faspex5 packages status f5_p31
|
5621
|
-
faspex5 postprocessing --pid-file=
|
5719
|
+
faspex5 postprocessing --pid-file=pid_f5_postproc @json:'{"url":"https://localhost:8443/domain","processing":{"script_folder":""}}' &
|
5622
5720
|
faspex5 shared browse %name:my_src
|
5623
5721
|
faspex5 shared list
|
5624
5722
|
faspex5 shared_folders browse %name:my_shared_folder_name
|
@@ -5633,25 +5731,28 @@ By default, package operations (send, receive, list) are done on the user's inbo
|
|
5633
5731
|
|
5634
5732
|
To select another inbox, use option `box` with one of the following values:
|
5635
5733
|
|
5636
|
-
- inbox
|
5637
|
-
- inbox_history
|
5638
|
-
- inbox_all
|
5639
|
-
- inbox_all_history
|
5640
|
-
- outbox
|
5641
|
-
- outbox_history
|
5642
|
-
- pending
|
5643
|
-
- pending_history
|
5644
|
-
- all
|
5645
|
-
- ALL (only
|
5646
|
-
- name of a shared inbox or workgroup
|
5734
|
+
- `inbox`
|
5735
|
+
- `inbox_history`
|
5736
|
+
- `inbox_all`
|
5737
|
+
- `inbox_all_history`
|
5738
|
+
- `outbox`
|
5739
|
+
- `outbox_history`
|
5740
|
+
- `pending`
|
5741
|
+
- `pending_history`
|
5742
|
+
- `all`
|
5743
|
+
- `ALL` (**admin only**, all inboxes of all users)
|
5744
|
+
- **name of a shared inbox or workgroup**
|
5647
5745
|
|
5648
|
-
> **Note:**
|
5746
|
+
> **Note:** In case the name of the `box` is specific, use option `group_type` with either `shared_inboxes` or `workgroups` to be more specific.
|
5649
5747
|
|
5650
5748
|
### Faspex 5: Send a package
|
5651
5749
|
|
5652
|
-
The `Hash` creation parameter provided to command `faspex5 packages send` corresponds to the Faspex 5 API: `POST /packages`.
|
5750
|
+
The `Hash` creation parameter provided to command `faspex5 packages send [extended value: Hash with package info ] [files...]` corresponds to the Faspex 5 API: `POST /packages`.
|
5751
|
+
|
5752
|
+
The interface is the one of the API (Refer to Faspex5 API documentation, or look at request in browser).
|
5653
5753
|
|
5654
5754
|
Required fields are `title` and `recipients`.
|
5755
|
+
|
5655
5756
|
Example using `@json:` format:
|
5656
5757
|
|
5657
5758
|
```json
|
@@ -5660,15 +5761,16 @@ Example using `@json:` format:
|
|
5660
5761
|
|
5661
5762
|
`recipient_type` is one of (Refer to API):
|
5662
5763
|
|
5663
|
-
- user
|
5664
|
-
- workgroup
|
5665
|
-
- external_user
|
5666
|
-
- distribution_list
|
5667
|
-
- shared_inbox
|
5764
|
+
- `user`
|
5765
|
+
- `workgroup`
|
5766
|
+
- `external_user`
|
5767
|
+
- `distribution_list`
|
5768
|
+
- `shared_inbox`
|
5668
5769
|
|
5669
|
-
`ascli` adds some convenience:
|
5670
|
-
|
5671
|
-
|
5770
|
+
`ascli` adds some convenience:
|
5771
|
+
The API expects the field `recipients` to be an `Array` of `Hash`, each with field `name` and optionally `recipient_type`.
|
5772
|
+
`ascli` also accepts an `Array` of `String`, with simply a recipient name.
|
5773
|
+
Then, `ascli` will lookup existing contacts among all possible types, use it if a single match is found, and set the `name` and `recipient_type` accordingly.
|
5672
5774
|
Else an exception is sent.
|
5673
5775
|
|
5674
5776
|
> **Note:** The lookup is case insensitive and on partial matches.
|
@@ -5677,39 +5779,74 @@ Else an exception is sent.
|
|
5677
5779
|
{"title":"some title","recipients":["user@example.com"]}
|
5678
5780
|
```
|
5679
5781
|
|
5680
|
-
If the lookup needs to be only on certain types, you can specify the field: `recipient_types` with either a single value or an Array of values (from the list above). e.g. :
|
5782
|
+
If the lookup needs to be only on certain types, you can specify the field: `recipient_types` with either a single value or an `Array` of values (from the list above). e.g. :
|
5681
5783
|
|
5682
5784
|
```json
|
5683
5785
|
{"title":"test title","recipient_types":"user","recipients":["user1@example.com","user2@example.com"]}
|
5684
5786
|
```
|
5685
5787
|
|
5686
|
-
### Faspex 5:
|
5788
|
+
### Faspex 5: Send a package with metadata
|
5687
5789
|
|
5688
|
-
|
5790
|
+
It's the same as sending a package, but with an extra field `metadata` in the package info.
|
5689
5791
|
|
5690
|
-
```
|
5691
|
-
|
5792
|
+
```json
|
5793
|
+
{"title":"test title","recipients":["my shared inbox"],"metadata":{"Confidential":"Yes","Drop menu":"Option 1"}}
|
5692
5794
|
```
|
5693
5795
|
|
5694
5796
|
Basically, add the field `metadata`, with one key per metadata and the value is directly the metadata value.
|
5797
|
+
(Refer to API documentation for more details).
|
5798
|
+
|
5799
|
+
### Faspex 5: List packages
|
5800
|
+
|
5801
|
+
Option `box` can be used to list packages from a specific box (see **Inbox selection** above).
|
5802
|
+
|
5803
|
+
Option `query` can be used to filter the list of packages, based on native API parameters, directly sent to [Faspex 5 API `GET /packages`](https://developer.ibm.com/apis/catalog/aspera--ibm-aspera-faspex-5-0-api/api/API--aspera--ibm-aspera-faspex-api#getAllPackages).
|
5804
|
+
|
5805
|
+
| parameter | Type | description |
|
5806
|
+
|-----------|---------|-------------|
|
5807
|
+
| `offset` | Native | Managed by `ascli`: Offset of first package. Default: 0 |
|
5808
|
+
| `limit` | Native | Managed by `ascli`: # of packages per API call. Default: 100 |
|
5809
|
+
| `q` | Native | General search string (case insensitive, matches if value is contained in several fields) |
|
5810
|
+
| ... | Native | Other native parameters are supported (Refer to API documentation) |
|
5811
|
+
| `max` | Special | Maximum number of items to retrieve (stop pages when the maximum is passed) |
|
5812
|
+
| `pmax` | Special | Maximum number of pages to request (stop pages when the maximum is passed) |
|
5813
|
+
|
5814
|
+
A positional parameter in last position, of type `Proc`, can be used to filter the list of packages.
|
5815
|
+
This advantage of this method is that the expression can be any test, even complex, as it is Ruby code.
|
5816
|
+
But the disadvantage is that the filtering is done in `ascli` and not in Faspex 5, so it is less efficient.
|
5817
|
+
|
5818
|
+
Examples:
|
5819
|
+
|
5820
|
+
- List only available packages: (filtering is done in Faspex)
|
5821
|
+
|
5822
|
+
```bash
|
5823
|
+
ascli faspex5 packages list --query=@json:'{"status":"completed"}'
|
5824
|
+
```
|
5825
|
+
|
5826
|
+
- Similar, using filtering in `ascli`:
|
5827
|
+
|
5828
|
+
```bash
|
5829
|
+
ascli faspex5 packages list @ruby:'->(p){p["state"].eql?("released")}'
|
5830
|
+
```
|
5695
5831
|
|
5696
5832
|
### Faspex 5: Receive a package
|
5697
5833
|
|
5698
|
-
|
5834
|
+
To receive one, or several packages at once, use command `faspex5 packages receive`.
|
5835
|
+
Provide either a single package id, or an extended value `Array` of package ids, e.g. `@list:,1,2,3` as argument.
|
5699
5836
|
|
5700
|
-
|
5837
|
+
The same options as for `faspex5 packages list` can be used to select the box and filter the packages to download.
|
5838
|
+
I.e. options `box` and `query`, as well as last positional parameter `Proc` (filter).
|
5701
5839
|
|
5702
|
-
|
5840
|
+
Option `--once-only=yes` can be used, for "cargo-like" behavior.
|
5841
|
+
Special package id `INIT` initializes the persistency of already received packages when option `--once-only=yes` is used.
|
5703
5842
|
|
5704
|
-
|
5843
|
+
Special package id `ALL` selects all packages (of the selected box).
|
5844
|
+
In this case, typically, only `completed` packages should be downloaded, so use option `--query=@json:'{"status":"completed"}'`.
|
5705
5845
|
|
5706
|
-
|
5707
|
-
|
5708
|
-
- `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
5709
|
-
- `offset` : native api parameter, in general do not use (added by `ascli`)
|
5710
|
-
- `limit` : native api parameter, number of items par api call, in general do not use (added by `ascli`)
|
5846
|
+
If a package is password protected, then the content protection password is asked interactively.
|
5847
|
+
To keep the content encrypted, use option: `--ts=@json:'{"content_protection":null}'`, or provide the password instead of `null`.
|
5711
5848
|
|
5712
|
-
|
5849
|
+
> **Tip:** If you use option `query` and/or positional `filter`, you can use the `list` command for a dry run.
|
5713
5850
|
|
5714
5851
|
### Faspex 5: List all shared inboxes and work groups
|
5715
5852
|
|
@@ -5784,7 +5921,7 @@ ascli faspex5 packages send @json:'{"title":"hello","recipients":[{"name":"_reci
|
|
5784
5921
|
To receive all packages, only once, through persistency of already received packages:
|
5785
5922
|
|
5786
5923
|
```bash
|
5787
|
-
ascli faspex5 packages receive ALL --once-only=yes
|
5924
|
+
ascli faspex5 packages receive ALL --once-only=yes --query=@json:'{"status":"completed"}'
|
5788
5925
|
```
|
5789
5926
|
|
5790
5927
|
To initialize, and skip all current package so that next time `ALL` is used, only newer packages are downloaded:
|
@@ -5808,33 +5945,34 @@ The following parameters are supported:
|
|
5808
5945
|
|
5809
5946
|
| parameter | type | default | description |
|
5810
5947
|
|----------------------------|---------|------------------------|-----------------------------------------------------|
|
5811
|
-
|
5812
|
-
|
|
5813
|
-
| certificate
|
5814
|
-
| certificate.
|
5815
|
-
| certificate.
|
5816
|
-
|
|
5817
|
-
| processing
|
5818
|
-
| processing.
|
5819
|
-
| processing.
|
5948
|
+
<!-- markdownlint-disable-next-line -->
|
5949
|
+
| url | string | http://localhost:8080 | Base url on which requests are listened |
|
5950
|
+
| certificate | hash | nil | Certificate information (if HTTPS) |
|
5951
|
+
| certificate.key | string | nil | Path to private key file |
|
5952
|
+
| certificate.cert | string | nil | Path to certificate |
|
5953
|
+
| certificate.chain | string | nil | Path to intermediary certificates |
|
5954
|
+
| processing | hash | nil | Behavior of post processing |
|
5955
|
+
| processing.script_folder | string | . | Prefix added to script path |
|
5956
|
+
| processing.fail_on_error | bool | false | Fail if true and process exit with non zero |
|
5957
|
+
| processing.timeout_seconds | integer | 60 | Max. execution time before script is killed |
|
5820
5958
|
|
5821
5959
|
Parameter `url` defines:
|
5822
5960
|
|
5823
|
-
-
|
5824
|
-
-
|
5825
|
-
-
|
5961
|
+
- If `http` or `https` is used
|
5962
|
+
- The local port number
|
5963
|
+
- The **base path**, i.e. the path under which requests are received.
|
5826
5964
|
|
5827
5965
|
When a request is received the following happens:
|
5828
5966
|
|
5829
|
-
-
|
5830
|
-
-
|
5831
|
-
-
|
5832
|
-
-
|
5833
|
-
-
|
5967
|
+
- The processor get the path of the url called
|
5968
|
+
- It removes the **base path**
|
5969
|
+
- It prepends it with the value of `script_folder`
|
5970
|
+
- It executes the script
|
5971
|
+
- Upon success, a success code is returned
|
5834
5972
|
|
5835
5973
|
In Faspex 5, configure like this:
|
5836
5974
|
|
5837
|
-
|
5975
|
+
**Webhook endpoint URI** : `http://localhost:8080/processing/script1.sh`
|
5838
5976
|
|
5839
5977
|
Then, the postprocessing script executed will be `script1.sh`.
|
5840
5978
|
|
@@ -5859,8 +5997,8 @@ By default it looks in box `inbox`, but the following boxes are also supported:
|
|
5859
5997
|
|
5860
5998
|
A user can receive a package because the recipient is:
|
5861
5999
|
|
5862
|
-
-
|
5863
|
-
-
|
6000
|
+
- The user himself (default)
|
6001
|
+
- The user is member of a dropbox/workgroup: filter using option `recipient` set with value `*<name of dropbox/workgroup>`
|
5864
6002
|
|
5865
6003
|
#### Option `query`
|
5866
6004
|
|
@@ -5874,7 +6012,7 @@ As inboxes may be large, it is possible to use the following query parameters:
|
|
5874
6012
|
|
5875
6013
|
(SQL query is `LIMIT <startIndex>, <count>`)
|
5876
6014
|
|
5877
|
-
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under
|
6015
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under **Services (API v.3)**.
|
5878
6016
|
|
5879
6017
|
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`).
|
5880
6018
|
By default `count` is `0` (`10`), it can be increased to issue less HTTP calls.
|
@@ -5891,9 +6029,9 @@ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in recei
|
|
5891
6029
|
|
5892
6030
|
The command is `package recv`, possible methods are:
|
5893
6031
|
|
5894
|
-
-
|
5895
|
-
-
|
5896
|
-
-
|
6032
|
+
- Provide a package id with option `id`
|
6033
|
+
- Provide a public link with option `link`
|
6034
|
+
- Provide a `faspe:` URI with option `link`
|
5897
6035
|
|
5898
6036
|
```bash
|
5899
6037
|
ascli faspex package recv 12345
|
@@ -5951,7 +6089,7 @@ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipien
|
|
5951
6089
|
|
5952
6090
|
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
5953
6091
|
|
5954
|
-
### Operation on
|
6092
|
+
### Operation on dropboxes
|
5955
6093
|
|
5956
6094
|
Example:
|
5957
6095
|
|
@@ -5963,9 +6101,8 @@ ascli faspex v4 dropbox delete 36
|
|
5963
6101
|
|
5964
6102
|
### Remote sources
|
5965
6103
|
|
5966
|
-
Faspex lacks an API to list the contents of a remote source (available in web UI).
|
5967
|
-
the node API is used, for this it is required to
|
5968
|
-
a storage name to a node config and sub path.
|
6104
|
+
Faspex lacks an API to list the contents of a remote source (available in web UI).
|
6105
|
+
To workaround this, the node API is used, for this it is required to set option: `storage` that links a storage name to a node configuration and sub path.
|
5969
6106
|
|
5970
6107
|
Example:
|
5971
6108
|
|
@@ -5988,7 +6125,7 @@ In this example, a faspex storage named `my_storage` exists in Faspex, and is lo
|
|
5988
6125
|
under the docroot in `/mydir` (this must be the same as configured in Faspex).
|
5989
6126
|
The node configuration name is `my_faspex_node` here.
|
5990
6127
|
|
5991
|
-
> **Note:**
|
6128
|
+
> **Note:** The v4 API provides an API for nodes and shares.
|
5992
6129
|
|
5993
6130
|
### Automated package download (cargo)
|
5994
6131
|
|
@@ -6007,23 +6144,23 @@ faspex dropbox list --recipient="*my_dbx"
|
|
6007
6144
|
faspex health
|
6008
6145
|
faspex login_methods
|
6009
6146
|
faspex me
|
6010
|
-
faspex package list --box=sent --
|
6011
|
-
faspex package list --
|
6147
|
+
faspex package list --box=sent --query=@json:'{"max":1}' --fields=package_id --display=data --format=csv --output=f4_prs2
|
6148
|
+
faspex package list --query=@json:'{"max":1}' --fields=package_id --display=data --format=csv --output=f4_prs1
|
6012
6149
|
faspex package list --query=@json:'{"max":5}'
|
6013
|
-
faspex package list --recipient="*my_dbx" --format=csv --fields=package_id --query=@json:'{"max":1}'
|
6014
|
-
faspex package list --recipient="*my_wkg" --format=csv --fields=package_id --query=@json:'{"max":1}'
|
6015
|
-
faspex package
|
6016
|
-
faspex package
|
6017
|
-
faspex package
|
6018
|
-
faspex package
|
6019
|
-
faspex package
|
6020
|
-
faspex package
|
6021
|
-
faspex package send --delivery-info=@json:'{"title":"
|
6022
|
-
faspex package send --delivery-info=@json:'{"title":"
|
6023
|
-
faspex package send --delivery-info=@json:'{"title":"
|
6024
|
-
faspex package send --delivery-info=@json:'{"title":"
|
6025
|
-
faspex package send --link=https://app.example.com/send_to_dropbox_path --delivery-info=@json:'{"title":"
|
6026
|
-
faspex package send --link=https://app.example.com/send_to_user_path --delivery-info=@json:'{"title":"
|
6150
|
+
faspex package list --recipient="*my_dbx" --format=csv --fields=package_id --query=@json:'{"max":1}' --output=f4_db_id1
|
6151
|
+
faspex package list --recipient="*my_wkg" --format=csv --fields=package_id --query=@json:'{"max":1}' --output=f4_db_id2
|
6152
|
+
faspex package receive --to-folder=. --link=https://app.example.com/recv_from_user_path
|
6153
|
+
faspex package receive ALL --once-only=yes --to-folder=. --query=@json:'{"max":10}'
|
6154
|
+
faspex package receive f4_db_id1 --recipient="*my_dbx" --to-folder=.
|
6155
|
+
faspex package receive f4_db_id2 --recipient="*my_wkg" --to-folder=.
|
6156
|
+
faspex package receive f4_pri1 --to-folder=.
|
6157
|
+
faspex package receive f4_prs2 --to-folder=. --box=sent
|
6158
|
+
faspex package send --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["*my_dbx"]}' test_file.bin
|
6159
|
+
faspex package send --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["*my_wkg"]}' test_file.bin
|
6160
|
+
faspex package send --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_email_internal","my_username"]}' test_file.bin
|
6161
|
+
faspex package send --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE","recipients":["my_email_internal"]}' --remote_source=%name:my_src sample_source.txt
|
6162
|
+
faspex package send --link=https://app.example.com/send_to_dropbox_path --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE"}' test_file.bin
|
6163
|
+
faspex package send --link=https://app.example.com/send_to_user_path --delivery-info=@json:'{"title":"$(notdir test) PACKAGE_TITLE_BASE"}' test_file.bin
|
6027
6164
|
faspex source info %name:my_src --storage=@preset:faspex4_storage
|
6028
6165
|
faspex source list
|
6029
6166
|
faspex source node %name:my_src br / --storage=@preset:faspex4_storage
|
@@ -6035,25 +6172,41 @@ faspex v4 wmembership list
|
|
6035
6172
|
faspex v4 workgroup list
|
6036
6173
|
```
|
6037
6174
|
|
6038
|
-
## <a id="shares"></a>Plugin: `shares`: IBM Aspera Shares v1
|
6175
|
+
## <a id="shares"></a>Plugin: `shares`: IBM Aspera Shares v1
|
6176
|
+
|
6177
|
+
Aspera Shares supports the **node API** for the file transfer part.
|
6178
|
+
|
6179
|
+
Supported commands are listed in Share's API documentation:
|
6180
|
+
|
6181
|
+
<https://developer.ibm.com/apis/catalog/aspera--aspera-shares-api/Introduction>
|
6182
|
+
|
6183
|
+
Example:
|
6184
|
+
|
6185
|
+
```bash
|
6186
|
+
ascli shares admin share create @json:'{"node_id":1,"name":"test1","directory":"test1","create_directory":true}'
|
6187
|
+
|
6188
|
+
share_id=$(ascli shares admin share list --select=@json:'{"name":"test1"}' --fields=id)
|
6189
|
+
|
6190
|
+
user_id=$(ascli shares admin user list --select=@json:'{"username":"entity1"}' --fields=id)
|
6039
6191
|
|
6040
|
-
|
6192
|
+
ascli shares admin share user_permissions $share_id create @json:'{"user_id":'$user_id',"browse_permission":true, "download_permission":true, "mkdir_permission":true,"delete_permission":true,"rename_permission":true,"content_availability_permission":true,"manage_permission":true}'
|
6193
|
+
```
|
6041
6194
|
|
6042
6195
|
### Shares 1 sample commands
|
6043
6196
|
|
6044
6197
|
```bash
|
6045
|
-
shares admin group list
|
6198
|
+
shares admin group all list
|
6046
6199
|
shares admin node list
|
6047
6200
|
shares admin share list --fields=DEF,-status,status_message
|
6048
6201
|
shares admin share user_permissions 1 list
|
6049
|
-
shares admin user
|
6050
|
-
shares admin user app_authorizations 1
|
6051
|
-
shares admin user
|
6052
|
-
shares admin user
|
6053
|
-
shares admin user
|
6054
|
-
shares admin user
|
6055
|
-
shares admin user
|
6056
|
-
shares admin user
|
6202
|
+
shares admin user all app_authorizations 1 modify @json:'{"app_login":true}'
|
6203
|
+
shares admin user all app_authorizations 1 show
|
6204
|
+
shares admin user all list
|
6205
|
+
shares admin user all share_permissions 1 list
|
6206
|
+
shares admin user all share_permissions 1 show 1
|
6207
|
+
shares admin user ldap add the_name
|
6208
|
+
shares admin user local list
|
6209
|
+
shares admin user saml import @json:'{"id":"the_id","name_id":"the_name"}'
|
6057
6210
|
shares files browse /
|
6058
6211
|
shares files delete my_share1/test_file.bin
|
6059
6212
|
shares files download --to-folder=. my_share1/test_file.bin
|
@@ -6116,8 +6269,8 @@ If you have those parameters already, then following options shall be provided:
|
|
6116
6269
|
For example, let us create a default configuration:
|
6117
6270
|
|
6118
6271
|
```bash
|
6119
|
-
ascli
|
6120
|
-
ascli
|
6272
|
+
ascli config preset update mycos --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
6273
|
+
ascli config preset set default cos mycos
|
6121
6274
|
```
|
6122
6275
|
|
6123
6276
|
Then, jump to the transfer example.
|
@@ -6180,8 +6333,8 @@ The required options for this method are:
|
|
6180
6333
|
For example, let us create a default configuration:
|
6181
6334
|
|
6182
6335
|
```bash
|
6183
|
-
ascli
|
6184
|
-
ascli
|
6336
|
+
ascli config preset update mycos --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
6337
|
+
ascli config preset set default cos mycos
|
6185
6338
|
```
|
6186
6339
|
|
6187
6340
|
### Operations, transfers
|
@@ -6195,50 +6348,19 @@ ascli cos node info
|
|
6195
6348
|
ascli cos node upload 'faux:///sample1G?1g'
|
6196
6349
|
```
|
6197
6350
|
|
6198
|
-
> **Note:**
|
6351
|
+
> **Note:** We generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man `ascp` and section above), but you can of course send a real file by specifying a real file instead.
|
6199
6352
|
|
6200
6353
|
### COS sample commands
|
6201
6354
|
|
6202
6355
|
```bash
|
6203
6356
|
cos --bucket=my_bucket_name --endpoint=my_bucket_endpoint --apikey=my_bucket_apikey --crn=my_resource_instance_id node info
|
6204
|
-
cos --bucket=my_bucket_name --region=my_bucket_region --service-credentials=@json:@file:
|
6357
|
+
cos --bucket=my_bucket_name --region=my_bucket_region --service-credentials=@json:@file:my_cos_svc_cred node info
|
6205
6358
|
cos node access_key show self
|
6206
6359
|
cos node download test_file.bin --to-folder=.
|
6207
6360
|
cos node info --log-level=trace2
|
6208
6361
|
cos node upload test_file.bin
|
6209
6362
|
```
|
6210
6363
|
|
6211
|
-
## <a id="async"></a>IBM Aspera Sync
|
6212
|
-
|
6213
|
-
An interface for the `async` utility is provided in the following plugins:
|
6214
|
-
|
6215
|
-
- server sync
|
6216
|
-
- node sync
|
6217
|
-
- aoc files sync (uses node)
|
6218
|
-
- shares files sync (uses node)
|
6219
|
-
|
6220
|
-
The main advantage over the `async` command line when using `server` is the possibility to use a configuration file, using standard options of `ascli`.
|
6221
|
-
|
6222
|
-
In this case, some of the `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
|
6223
|
-
|
6224
|
-
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
6225
|
-
|
6226
|
-
Two JSON syntax are supported for option `sync_info`.
|
6227
|
-
|
6228
|
-
### async JSON: API format
|
6229
|
-
|
6230
|
-
It is the same payload as specified on the option `--conf` of `async` or in node API `/asyncs`.
|
6231
|
-
This is the preferred syntax and allows a single session definition.
|
6232
|
-
But there is no progress output nor error messages.
|
6233
|
-
|
6234
|
-
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
6235
|
-
|
6236
|
-
### async JSON: Options mapping
|
6237
|
-
|
6238
|
-
`ascli` defines a JSON equivalent to regular `async`options.
|
6239
|
-
It is based on a JSON representation of `async` command line options.
|
6240
|
-
It allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
6241
|
-
|
6242
6364
|
## <a id="preview"></a>Plugin: `preview`: Preview generator for AoC
|
6243
6365
|
|
6244
6366
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
@@ -6249,6 +6371,8 @@ Several parameters can be used to tune several aspects:
|
|
6249
6371
|
- Methods for generation of video preview
|
6250
6372
|
- Parameters for video handling
|
6251
6373
|
|
6374
|
+
See also <https://github.com/IBM/aspera-on-cloud-file-previews>
|
6375
|
+
|
6252
6376
|
### Aspera Server configuration
|
6253
6377
|
|
6254
6378
|
Specify the previews folder as shown in:
|
@@ -6264,7 +6388,7 @@ asconfigurator -x "server;preview_dir,previews"
|
|
6264
6388
|
asnodeadmin --reload
|
6265
6389
|
```
|
6266
6390
|
|
6267
|
-
> **Note:**
|
6391
|
+
> **Note:** The configuration `preview_dir` is **relative** to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
|
6268
6392
|
|
6269
6393
|
If another folder is configured on the HSTS, then specify it to `ascli` using the option `previews_folder`.
|
6270
6394
|
|
@@ -6283,16 +6407,16 @@ asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
|
6283
6407
|
|
6284
6408
|
If you use a value different than 16777216, then specify it using option `max_size`.
|
6285
6409
|
|
6286
|
-
> **Note:**
|
6410
|
+
> **Note:** The HSTS parameter (`max_request_file_create_size_kb`) is in **kiloBytes** while the generator parameter is in **Bytes** (factor of 1024).
|
6287
6411
|
|
6288
6412
|
### <a id="prev_ext"></a>External tools: Linux
|
6289
6413
|
|
6290
6414
|
`ascli` requires the following external tools available in the `PATH`:
|
6291
6415
|
|
6292
|
-
- ImageMagick : `convert` `composite`
|
6293
|
-
-
|
6294
|
-
-
|
6295
|
-
-
|
6416
|
+
- **ImageMagick** : `convert` `composite`
|
6417
|
+
- **OptiPNG** : `optipng`
|
6418
|
+
- **FFmpeg** : `ffmpeg` `ffprobe`
|
6419
|
+
- **Libreoffice** : `libreoffice`
|
6296
6420
|
|
6297
6421
|
Here shown on Redhat/CentOS.
|
6298
6422
|
|
@@ -6421,7 +6545,7 @@ case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
|
6421
6545
|
|
6422
6546
|
- `trevents` : only recently uploaded files will be tested (transfer events)
|
6423
6547
|
- `events` : only recently uploaded files will be tested (file events: not working)
|
6424
|
-
- `scan` : recursively scan all files under the access key's
|
6548
|
+
- `scan` : recursively scan all files under the access key's **storage root**
|
6425
6549
|
- `test` : test using a local file
|
6426
6550
|
|
6427
6551
|
Once candidate are selected, once candidates are selected,
|
@@ -6460,11 +6584,11 @@ Use option `skip_format` to skip generation of a format.
|
|
6460
6584
|
|
6461
6585
|
The preview generator supports rendering of those file categories:
|
6462
6586
|
|
6463
|
-
- image
|
6464
|
-
- pdf
|
6465
|
-
- plaintext
|
6466
|
-
- office
|
6467
|
-
- video
|
6587
|
+
- `image`
|
6588
|
+
- `pdf`
|
6589
|
+
- `plaintext`
|
6590
|
+
- `office`
|
6591
|
+
- `video`
|
6468
6592
|
|
6469
6593
|
To avoid generation for some categories, specify a list using option `skip_types`.
|
6470
6594
|
|
@@ -6487,7 +6611,7 @@ In this case the `preview` command will first analyze the file content using `mi
|
|
6487
6611
|
|
6488
6612
|
If the `mimemagic` gem complains about missing mime info file:
|
6489
6613
|
|
6490
|
-
-
|
6614
|
+
- Any OS:
|
6491
6615
|
|
6492
6616
|
- Examine the error message
|
6493
6617
|
- Download the file: [`freedesktop.org.xml.in`](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
@@ -6511,7 +6635,7 @@ If the `mimemagic` gem complains about missing mime info file:
|
|
6511
6635
|
dnf install shared-mime-info
|
6512
6636
|
```
|
6513
6637
|
|
6514
|
-
- macOS
|
6638
|
+
- **macOS**:
|
6515
6639
|
|
6516
6640
|
```bash
|
6517
6641
|
brew install shared-mime-info
|
@@ -6542,9 +6666,153 @@ preview show --base=test my_mpg mp4 --video-conversion=reencode
|
|
6542
6666
|
preview show --base=test my_pdf
|
6543
6667
|
preview test --base=test my_dcm
|
6544
6668
|
preview test --base=test my_mxf mp4 --video-conversion=blend --query=@json:'{"text":true,"double":true}'
|
6669
|
+
preview test --mimemagic=yes --base=test my_dcm
|
6545
6670
|
preview trevents --once-only=yes --skip-types=office --log-level=info
|
6546
6671
|
```
|
6547
6672
|
|
6673
|
+
## <a id="async"></a>IBM Aspera Sync
|
6674
|
+
|
6675
|
+
An interface for the `async` utility is provided in the following plugins:
|
6676
|
+
|
6677
|
+
- `server sync`
|
6678
|
+
- `node sync`
|
6679
|
+
- `aoc files sync` (uses node)
|
6680
|
+
- `shares files sync` (uses node)
|
6681
|
+
|
6682
|
+
The main advantage over the `async` command line when using `server` is the possibility to use a configuration file, using standard options of `ascli`.
|
6683
|
+
|
6684
|
+
In this case, some of the `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
|
6685
|
+
|
6686
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
6687
|
+
|
6688
|
+
Two JSON syntax are supported for option `sync_info`.
|
6689
|
+
|
6690
|
+
### `async` JSON: API format
|
6691
|
+
|
6692
|
+
It is the same payload as specified on the option `--conf` of `async` or in node API `/asyncs`.
|
6693
|
+
This is the preferred syntax and allows a single session definition.
|
6694
|
+
But there is no progress output nor error messages.
|
6695
|
+
|
6696
|
+
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
6697
|
+
|
6698
|
+
### `async` JSON: Options mapping
|
6699
|
+
|
6700
|
+
`ascli` defines a JSON equivalent to regular `async`options.
|
6701
|
+
It is based on a JSON representation of `async` command line options.
|
6702
|
+
It allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
6703
|
+
|
6704
|
+
## Hot folder
|
6705
|
+
|
6706
|
+
### Requirements
|
6707
|
+
|
6708
|
+
`ascli` maybe used as a simple hot folder engine.
|
6709
|
+
A hot folder being defined as a tool that:
|
6710
|
+
|
6711
|
+
- Locally (or remotely) detects new files in a top folder
|
6712
|
+
- Send detected files to a remote (respectively, local) repository
|
6713
|
+
- Only sends new files, do not re-send already sent files
|
6714
|
+
- Optionally: sends only files that are not still **growing**
|
6715
|
+
- Optionally: after transfer of files, deletes or moves to an archive
|
6716
|
+
|
6717
|
+
In addition: the detection should be made **continuously** or on specific time/date.
|
6718
|
+
|
6719
|
+
### Setup procedure
|
6720
|
+
|
6721
|
+
The general idea is to rely on :
|
6722
|
+
|
6723
|
+
- Existing `ascp` features for detection and transfer
|
6724
|
+
- Take advantage of `ascli` configuration capabilities and server side knowledge
|
6725
|
+
- The OS scheduler for reliability and continuous operation
|
6726
|
+
|
6727
|
+
#### `ascp` features
|
6728
|
+
|
6729
|
+
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
6730
|
+
|
6731
|
+
- `ascp` already takes care of sending only **new** files: option `-k 1,2,3` (`resume_policy`)
|
6732
|
+
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
6733
|
+
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
6734
|
+
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
6735
|
+
|
6736
|
+
> **Note:** `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `ts` option.
|
6737
|
+
>
|
6738
|
+
> **Note:** Most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters.
|
6739
|
+
>
|
6740
|
+
> **Note:** Only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info` .
|
6741
|
+
|
6742
|
+
#### Server side and configuration
|
6743
|
+
|
6744
|
+
Virtually any transfer on a **repository** on a regular basis might emulate a hot folder.
|
6745
|
+
|
6746
|
+
> **Note:** File detection is not based on events (`inotify`, etc...), but on a simple folder scan on source side.
|
6747
|
+
>
|
6748
|
+
> **Note:** Parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
6749
|
+
|
6750
|
+
#### Scheduling
|
6751
|
+
|
6752
|
+
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
6753
|
+
Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
6754
|
+
|
6755
|
+
### Example: Upload hot folder
|
6756
|
+
|
6757
|
+
```bash
|
6758
|
+
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
6759
|
+
```
|
6760
|
+
|
6761
|
+
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
6762
|
+
Source files are deleted after transfer.
|
6763
|
+
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
6764
|
+
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
6765
|
+
|
6766
|
+
### Example: Unidirectional synchronization (upload) to server
|
6767
|
+
|
6768
|
+
```bash
|
6769
|
+
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
|
6770
|
+
```
|
6771
|
+
|
6772
|
+
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node.
|
6773
|
+
|
6774
|
+
### Example: Unidirectional synchronization (download) from Aspera on Cloud Files
|
6775
|
+
|
6776
|
+
```bash
|
6777
|
+
ascli aoc files download . --to-folder=. --lock-port=12345 --progress-bar=no --display=data --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
|
6778
|
+
```
|
6779
|
+
|
6780
|
+
> **Note:** Option `delete_before_transfer` will delete files locally, if they are not present on remote side.
|
6781
|
+
>
|
6782
|
+
> **Note:** Options `progress` and `display` limit output for headless operation (e.g. cron job)
|
6783
|
+
|
6784
|
+
## Health check and Nagios
|
6785
|
+
|
6786
|
+
Most plugin provide a `health` command that will check the health status of the application. Example:
|
6787
|
+
|
6788
|
+
```bash
|
6789
|
+
ascli console health
|
6790
|
+
```
|
6791
|
+
|
6792
|
+
```output
|
6793
|
+
+--------+-------------+------------+
|
6794
|
+
| status | component | message |
|
6795
|
+
+--------+-------------+------------+
|
6796
|
+
| ok | console api | accessible |
|
6797
|
+
+--------+-------------+------------+
|
6798
|
+
```
|
6799
|
+
|
6800
|
+
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
6801
|
+
|
6802
|
+
- Issuing a transfer to the server
|
6803
|
+
- Checking web app status with `asctl all:status`
|
6804
|
+
- Checking daemons process status
|
6805
|
+
|
6806
|
+
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
6807
|
+
|
6808
|
+
```bash
|
6809
|
+
ascli server health transfer --to-folder=/Upload --format=nagios --progress-bar=no
|
6810
|
+
```
|
6811
|
+
|
6812
|
+
```output
|
6813
|
+
OK - [transfer:ok]
|
6814
|
+
```
|
6815
|
+
|
6548
6816
|
## SMTP for email notifications
|
6549
6817
|
|
6550
6818
|
`ascli` can send email, for that setup SMTP configuration. This is done with option `smtp`.
|
@@ -6555,14 +6823,14 @@ The `smtp` option is a `Hash` (extended value) with the following fields:
|
|
6555
6823
|
| field | default | example | description |
|
6556
6824
|
|--------------|---------------------|----------------------------|----------------------------------|
|
6557
6825
|
| `server` | - | smtp.gmail.com | SMTP server address |
|
6558
|
-
| `tls` | true | true |
|
6559
|
-
| `ssl` | false | false |
|
6560
|
-
| `port` | 587 or 465 or 25 | 587 |
|
6561
|
-
| `domain` | domain of server | gmail.com |
|
6562
|
-
| `username` | - | john@example.com |
|
6563
|
-
| `password` | - | my_password_here |
|
6564
|
-
| `from_email` | username if defined | johnny@example.com |
|
6565
|
-
| `from_name` | same as email | John Wayne |
|
6826
|
+
| `tls` | true | true | Enable STARTTLS (port 587) |
|
6827
|
+
| `ssl` | false | false | Enable TLS (port 465) |
|
6828
|
+
| `port` | 587 or 465 or 25 | 587 | Port for service |
|
6829
|
+
| `domain` | domain of server | gmail.com | Email domain of user |
|
6830
|
+
| `username` | - | john@example.com | User to authenticate on SMTP server, leave empty for open auth. |
|
6831
|
+
| `password` | - | my_password_here | Password for above username |
|
6832
|
+
| `from_email` | username if defined | johnny@example.com | Address used if receiver replies |
|
6833
|
+
| `from_name` | same as email | John Wayne | Display name of sender |
|
6566
6834
|
<!-- markdownlint-enable MD034 -->
|
6567
6835
|
|
6568
6836
|
### Example of configuration
|
@@ -6625,10 +6893,10 @@ A default e-mail template is used, but it can be overridden with option `notify_
|
|
6625
6893
|
|
6626
6894
|
The environment provided contains the following additional variables:
|
6627
6895
|
|
6628
|
-
- subject
|
6629
|
-
- body
|
6630
|
-
- global_transfer_status
|
6631
|
-
- ts
|
6896
|
+
- `subject`
|
6897
|
+
- `body`
|
6898
|
+
- `global_transfer_status`
|
6899
|
+
- `ts`
|
6632
6900
|
|
6633
6901
|
Example of template:
|
6634
6902
|
|
@@ -6646,9 +6914,9 @@ This gem comes with a second executable tool providing a simplified standardized
|
|
6646
6914
|
|
6647
6915
|
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [*transfer-spec*](#transferspec) is:
|
6648
6916
|
|
6649
|
-
-
|
6650
|
-
-
|
6651
|
-
-
|
6917
|
+
- Common to Aspera Node API (HTTP POST /ops/transfer)
|
6918
|
+
- Common to Aspera Connect API (browser javascript startTransfer)
|
6919
|
+
- Easy to generate by using any third party language specific JSON library
|
6652
6920
|
|
6653
6921
|
Hopefully, IBM integrates this directly in `ascp`, and this tool is made redundant.
|
6654
6922
|
|
@@ -6662,22 +6930,22 @@ If no argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON
|
|
6662
6930
|
|
6663
6931
|
During execution, it generates all low level events, one per line, in JSON format on stdout.
|
6664
6932
|
|
6665
|
-
There are special
|
6933
|
+
There are special **extended** [*transfer-spec*](#transferspec) parameters supported by `asession`:
|
6666
6934
|
|
6667
6935
|
- `EX_loglevel` to change log level of `ascli`
|
6668
6936
|
- `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
6669
6937
|
|
6670
|
-
> **Note:** In addition, many
|
6938
|
+
> **Note:** In addition, many (deprecated) `EX_` [*transfer-spec*](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [*transfer-spec*](#transferspec).
|
6671
6939
|
|
6672
6940
|
### Comparison of interfaces
|
6673
6941
|
|
6674
|
-
| feature/tool
|
6675
|
-
|
6676
|
-
| language integration
|
6677
|
-
| required additional components to `ascp` |
|
6678
|
-
| startup | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) |
|
6679
|
-
| events |
|
6680
|
-
| platforms
|
6942
|
+
| feature/tool | Transfer SDK | FaspManager | `ascp` | `asession` |
|
6943
|
+
|-----------------------|--------------|---------------------------------|--------|------------|
|
6944
|
+
| language integration | Many | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | Any | Any |
|
6945
|
+
| required additional components to `ascp` | Daemon | Library<br/>(+headers) | - | Ruby<br/>Aspera gem |
|
6946
|
+
| startup | Daemon | API | Command line arguments | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) |
|
6947
|
+
| events | Poll | Callback | Possibility to open management port<br/>and proprietary text syntax | JSON on stdout |
|
6948
|
+
| platforms | Any with `ascp` and transfer daemon | Any with `ascp` (and SDK if compiled) | Any with `ascp` | Any with Ruby and `ascp` |
|
6681
6949
|
|
6682
6950
|
### Simple session
|
6683
6951
|
|
@@ -6695,7 +6963,7 @@ asession < session.json
|
|
6695
6963
|
|
6696
6964
|
### Asynchronous commands and Persistent session
|
6697
6965
|
|
6698
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be
|
6966
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter` -> `long_parameter`
|
6699
6967
|
|
6700
6968
|
This is particularly useful for a persistent session ( with the [*transfer-spec*](#transferspec) parameter: `"keepalive":true` )
|
6701
6969
|
|
@@ -6739,126 +7007,6 @@ EXAMPLES
|
|
6739
7007
|
|
6740
7008
|
```
|
6741
7009
|
|
6742
|
-
## Hot folder
|
6743
|
-
|
6744
|
-
### Requirements
|
6745
|
-
|
6746
|
-
`ascli` maybe used as a simple hot folder engine.
|
6747
|
-
A hot folder being defined as a tool that:
|
6748
|
-
|
6749
|
-
- locally (or remotely) detects new files in a top folder
|
6750
|
-
- send detected files to a remote (respectively, local) repository
|
6751
|
-
- only sends new files, do not re-send already sent files
|
6752
|
-
- optionally: sends only files that are not still "growing"
|
6753
|
-
- optionally: after transfer of files, deletes or moves to an archive
|
6754
|
-
|
6755
|
-
In addition: the detection should be made "continuously" or on specific time/date.
|
6756
|
-
|
6757
|
-
### Setup procedure
|
6758
|
-
|
6759
|
-
The general idea is to rely on :
|
6760
|
-
|
6761
|
-
- existing `ascp` features for detection and transfer
|
6762
|
-
- take advantage of `ascli` configuration capabilities and server side knowledge
|
6763
|
-
- the OS scheduler for reliability and continuous operation
|
6764
|
-
|
6765
|
-
#### `ascp` features
|
6766
|
-
|
6767
|
-
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
6768
|
-
|
6769
|
-
- `ascp` already takes care of sending only **new** files: option `-k 1,2,3` (`resume_policy`)
|
6770
|
-
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
6771
|
-
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
6772
|
-
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
6773
|
-
|
6774
|
-
> **Note:** `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `ts` option.
|
6775
|
-
>
|
6776
|
-
> **Note:** Most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters.
|
6777
|
-
>
|
6778
|
-
> **Note:** Only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info` .
|
6779
|
-
|
6780
|
-
#### server side and configuration
|
6781
|
-
|
6782
|
-
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
6783
|
-
|
6784
|
-
> **Note:** file detection is not based on events (`inotify`, etc...), but on a simple folder scan on source side.
|
6785
|
-
>
|
6786
|
-
> **Note:** parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
6787
|
-
|
6788
|
-
#### Scheduling
|
6789
|
-
|
6790
|
-
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
6791
|
-
Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
6792
|
-
|
6793
|
-
### Example: Upload hot folder
|
6794
|
-
|
6795
|
-
```bash
|
6796
|
-
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
6797
|
-
```
|
6798
|
-
|
6799
|
-
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
6800
|
-
Source files are deleted after transfer.
|
6801
|
-
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
6802
|
-
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
6803
|
-
|
6804
|
-
### Example: Unidirectional synchronization (upload) to server
|
6805
|
-
|
6806
|
-
```bash
|
6807
|
-
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
|
6808
|
-
```
|
6809
|
-
|
6810
|
-
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node:
|
6811
|
-
|
6812
|
-
### Example: Unidirectional synchronization (download) from Aspera on Cloud Files
|
6813
|
-
|
6814
|
-
```bash
|
6815
|
-
ascli aoc files download . --to-folder=. --lock-port=12345 --progress-bar=no --display=data --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
|
6816
|
-
```
|
6817
|
-
|
6818
|
-
> **Note:** option `delete_before_transfer` will delete files locally, if they are not present on remote side.
|
6819
|
-
>
|
6820
|
-
> **Note:** options `progress` and `display` limit output for headless operation (e.g. cron job)
|
6821
|
-
|
6822
|
-
## Health check and Nagios
|
6823
|
-
|
6824
|
-
Most plugin provide a `health` command that will check the health status of the application. Example:
|
6825
|
-
|
6826
|
-
```bash
|
6827
|
-
ascli console health
|
6828
|
-
```
|
6829
|
-
|
6830
|
-
```output
|
6831
|
-
+--------+-------------+------------+
|
6832
|
-
| status | component | message |
|
6833
|
-
+--------+-------------+------------+
|
6834
|
-
| ok | console api | accessible |
|
6835
|
-
+--------+-------------+------------+
|
6836
|
-
```
|
6837
|
-
|
6838
|
-
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
6839
|
-
|
6840
|
-
- issuing a transfer to the server
|
6841
|
-
- checking web app status with `asctl all:status`
|
6842
|
-
- checking daemons process status
|
6843
|
-
|
6844
|
-
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
6845
|
-
|
6846
|
-
```bash
|
6847
|
-
ascli server health transfer --to-folder=/Upload --format=nagios --progress-bar=no
|
6848
|
-
```
|
6849
|
-
|
6850
|
-
```output
|
6851
|
-
OK - [transfer:ok]
|
6852
|
-
```
|
6853
|
-
|
6854
|
-
```bash
|
6855
|
-
ascli server health asctl status --cmd_prefix='sudo ' --format=nagios
|
6856
|
-
```
|
6857
|
-
|
6858
|
-
```output
|
6859
|
-
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
6860
|
-
```
|
6861
|
-
|
6862
7010
|
## Ruby Module: `Aspera`
|
6863
7011
|
|
6864
7012
|
Main components:
|
@@ -6869,10 +7017,6 @@ Main components:
|
|
6869
7017
|
|
6870
7018
|
Working examples can be found in repo: <https://github.com/laurent-martin/aspera-api-examples> in Ruby examples.
|
6871
7019
|
|
6872
|
-
## Changes (Release notes)
|
6873
|
-
|
6874
|
-
See [CHANGELOG.md](CHANGELOG.md)
|
6875
|
-
|
6876
7020
|
## History
|
6877
7021
|
|
6878
7022
|
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares).
|
@@ -6887,9 +7031,9 @@ There were a few pitfalls:
|
|
6887
7031
|
|
6888
7032
|
So, it evolved into `ascli`:
|
6889
7033
|
|
6890
|
-
-
|
6891
|
-
-
|
6892
|
-
-
|
7034
|
+
- Portable: works on platforms supporting `ruby` (and `ascp`)
|
7035
|
+
- Easy to install with the `gem` utility
|
7036
|
+
- Supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from `ascp` command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
6893
7037
|
- `ruby` is consistent with other Aspera products
|
6894
7038
|
|
6895
7039
|
Over the time, a supported command line tool `aspera` was developed in C++, it was later on deprecated.
|
@@ -6898,17 +7042,20 @@ It had the advantage of being relatively easy to installed, as a single executab
|
|
6898
7042
|
Enjoy a coffee on me:
|
6899
7043
|
|
6900
7044
|
```bash
|
6901
|
-
ascli
|
6902
|
-
ascli
|
7045
|
+
ascli config coffee --ui=text
|
7046
|
+
ascli config coffee --ui=text --query=@json:'{"text":"true"}'
|
7047
|
+
ascli config coffee
|
6903
7048
|
```
|
6904
7049
|
|
6905
7050
|
## Common problems
|
6906
7051
|
|
7052
|
+
`ascli` detects common problems and provides hints to solve them.
|
7053
|
+
|
6907
7054
|
### Error: "Remote host is not who we expected"
|
6908
7055
|
|
6909
7056
|
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
6910
7057
|
|
6911
|
-
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the
|
7058
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the configuration file):
|
6912
7059
|
|
6913
7060
|
```bash
|
6914
7061
|
--ts=@json:'{"sshfp":null}'
|
@@ -6927,9 +7074,10 @@ If Ruby was installed as a Linux Packages, then also install Ruby development pa
|
|
6927
7074
|
|
6928
7075
|
### ED255519 key not supported
|
6929
7076
|
|
6930
|
-
ED25519 keys are deactivated since version 0.9.24
|
7077
|
+
ED25519 keys are deactivated since `ascli` version 0.9.24 as it requires additional gems that require native compilation and thus caused problems.
|
7078
|
+
This type of key will just be ignored.
|
6931
7079
|
|
6932
|
-
Without this deactivation, if such key was present the following error was generated:
|
7080
|
+
Without this deactivation, if such key was present in user's `.ssh` folder then the following error was generated:
|
6933
7081
|
|
6934
7082
|
```output
|
6935
7083
|
OpenSSH keys only supported if ED25519 is available
|
@@ -6937,3 +7085,5 @@ OpenSSH keys only supported if ED25519 is available
|
|
6937
7085
|
|
6938
7086
|
Which meant that you do not have Ruby support for ED25519 SSH keys.
|
6939
7087
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
7088
|
+
|
7089
|
+
To re-activate, set env var `ASCLI_ENABLE_ED25519` to `true`.
|