aspera-cli 4.10.0 → 4.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +19 -0
- data/CHANGELOG.md +528 -0
- data/CONTRIBUTING.md +143 -0
- data/README.md +977 -589
- data/bin/ascli +4 -4
- data/bin/asession +12 -12
- data/docs/test_env.conf +29 -19
- data/examples/aoc.rb +6 -6
- data/examples/dascli +18 -16
- data/examples/faspex4.rb +15 -15
- data/examples/node.rb +12 -12
- data/examples/proxy.pac +2 -2
- data/examples/server.rb +12 -12
- data/lib/aspera/aoc.rb +344 -272
- data/lib/aspera/ascmd.rb +56 -54
- data/lib/aspera/ats_api.rb +4 -4
- data/lib/aspera/cli/basic_auth_plugin.rb +15 -12
- data/lib/aspera/cli/extended_value.rb +9 -9
- data/lib/aspera/cli/{formater.rb → formatter.rb} +69 -69
- data/lib/aspera/cli/listener/line_dump.rb +1 -1
- data/lib/aspera/cli/listener/logger.rb +1 -1
- data/lib/aspera/cli/listener/progress.rb +5 -6
- data/lib/aspera/cli/listener/progress_multi.rb +16 -21
- data/lib/aspera/cli/main.rb +72 -73
- data/lib/aspera/cli/manager.rb +112 -112
- data/lib/aspera/cli/plugin.rb +68 -48
- data/lib/aspera/cli/plugins/alee.rb +4 -4
- data/lib/aspera/cli/plugins/aoc.rb +322 -720
- data/lib/aspera/cli/plugins/ats.rb +50 -52
- data/lib/aspera/cli/plugins/bss.rb +10 -10
- data/lib/aspera/cli/plugins/config.rb +514 -410
- data/lib/aspera/cli/plugins/console.rb +12 -12
- data/lib/aspera/cli/plugins/cos.rb +18 -20
- data/lib/aspera/cli/plugins/faspex.rb +134 -136
- data/lib/aspera/cli/plugins/faspex5.rb +235 -70
- data/lib/aspera/cli/plugins/node.rb +378 -309
- data/lib/aspera/cli/plugins/orchestrator.rb +52 -49
- data/lib/aspera/cli/plugins/preview.rb +129 -120
- data/lib/aspera/cli/plugins/server.rb +137 -83
- data/lib/aspera/cli/plugins/shares.rb +77 -52
- data/lib/aspera/cli/plugins/sync.rb +13 -33
- data/lib/aspera/cli/transfer_agent.rb +61 -61
- data/lib/aspera/cli/version.rb +2 -1
- data/lib/aspera/colors.rb +3 -3
- data/lib/aspera/command_line_builder.rb +78 -74
- data/lib/aspera/cos_node.rb +31 -29
- data/lib/aspera/data_repository.rb +1 -1
- data/lib/aspera/environment.rb +30 -28
- data/lib/aspera/fasp/agent_base.rb +17 -15
- data/lib/aspera/fasp/agent_connect.rb +34 -32
- data/lib/aspera/fasp/agent_direct.rb +70 -73
- data/lib/aspera/fasp/agent_httpgw.rb +79 -74
- data/lib/aspera/fasp/agent_node.rb +26 -26
- data/lib/aspera/fasp/agent_trsdk.rb +20 -20
- data/lib/aspera/fasp/error.rb +3 -2
- data/lib/aspera/fasp/error_info.rb +11 -8
- data/lib/aspera/fasp/installation.rb +80 -80
- data/lib/aspera/fasp/listener.rb +2 -2
- data/lib/aspera/fasp/parameters.rb +103 -92
- data/lib/aspera/fasp/parameters.yaml +313 -214
- data/lib/aspera/fasp/resume_policy.rb +10 -10
- data/lib/aspera/fasp/transfer_spec.rb +22 -2
- data/lib/aspera/fasp/uri.rb +7 -7
- data/lib/aspera/faspex_gw.rb +80 -159
- data/lib/aspera/faspex_postproc.rb +77 -0
- data/lib/aspera/hash_ext.rb +3 -3
- data/lib/aspera/id_generator.rb +5 -5
- data/lib/aspera/keychain/encrypted_hash.rb +23 -28
- data/lib/aspera/keychain/macos_security.rb +21 -20
- data/lib/aspera/log.rb +13 -13
- data/lib/aspera/nagios.rb +24 -23
- data/lib/aspera/node.rb +217 -38
- data/lib/aspera/oauth.rb +78 -74
- data/lib/aspera/open_application.rb +19 -11
- data/lib/aspera/persistency_action_once.rb +4 -4
- data/lib/aspera/persistency_folder.rb +13 -13
- data/lib/aspera/preview/file_types.rb +8 -8
- data/lib/aspera/preview/generator.rb +67 -67
- data/lib/aspera/preview/utils.rb +27 -27
- data/lib/aspera/proxy_auto_config.js +63 -63
- data/lib/aspera/proxy_auto_config.rb +19 -19
- data/lib/aspera/rest.rb +65 -67
- data/lib/aspera/rest_call_error.rb +2 -1
- data/lib/aspera/rest_error_analyzer.rb +22 -21
- data/lib/aspera/rest_errors_aspera.rb +16 -16
- data/lib/aspera/secret_hider.rb +17 -14
- data/lib/aspera/ssh.rb +15 -14
- data/lib/aspera/sync.rb +177 -62
- data/lib/aspera/temp_file_manager.rb +2 -2
- data/lib/aspera/uri_reader.rb +4 -4
- data/lib/aspera/web_auth.rb +13 -64
- data/lib/aspera/web_server_simple.rb +76 -0
- data.tar.gz.sig +0 -0
- metadata +11 -6
- metadata.gz.sig +0 -0
data/README.md
CHANGED
@@ -1,10 +1,15 @@
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
2
|
-
|
2
|
+
<!-- markdownlint-disable MD033 MD003 MD053 -->
|
3
|
+
<!-- cSpell:ignore devkit zcvf zxvf noded secondfile filesize sedemo eudemo webmail csum eascp loglevel cronfile magick keepalive inotify eastus bluemix trev sshfp struct genkey passout ibmaspera unpermitted -->
|
4
|
+
<!-- cSpell:ignoreRegExp /-P[a-z]+/g -->
|
5
|
+
<!-- cSpell:ignoreRegExp /my[a-z]+/g -->
|
3
6
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
4
7
|
|
5
|
-
|
8
|
+
##
|
9
|
+
|
10
|
+
Version : 4.12.0
|
6
11
|
|
7
|
-
Laurent/2016-
|
12
|
+
Laurent/2016-2023
|
8
13
|
|
9
14
|
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
10
15
|
|
@@ -14,9 +19,11 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
14
19
|
|
15
20
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
16
21
|
|
17
|
-
Minimum required Ruby version: >= 2.4.
|
22
|
+
Minimum required Ruby version: >= 2.4.
|
23
|
+
|
24
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
18
25
|
|
19
|
-
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
26
|
+
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs&sort=title_asc)
|
20
27
|
[Link 2](https://developer.ibm.com/apis/catalog/?search=aspera)
|
21
28
|
|
22
29
|
Release notes: see [CHANGELOG.md](CHANGELOG.md)
|
@@ -46,7 +53,7 @@ So it is designed for:
|
|
46
53
|
- A configuration file (config.yaml)
|
47
54
|
- Advanced command line options
|
48
55
|
- cURL (for REST calls)
|
49
|
-
- Aspera transfer (ascp)
|
56
|
+
- Aspera transfer (`ascp`)
|
50
57
|
|
51
58
|
If the need is to perform operations programmatically in languages such as: C, Go, Python, nodejs, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
52
59
|
|
@@ -59,10 +66,19 @@ For scripting and ad'hoc command line operations, `ascli` is perfect.
|
|
59
66
|
|
60
67
|
## Notations, Shell, Examples
|
61
68
|
|
62
|
-
|
69
|
+
Command line operations examples are shown using a shell such: `bash` or `zsh`.
|
63
70
|
|
64
71
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
65
72
|
|
73
|
+
`ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, etc...)
|
74
|
+
|
75
|
+
Some commands will start an Aspera-based transfer (e.g. `upload`).
|
76
|
+
The transfer is not directly implemented in `ascli`, rather `ascli` uses an external Aspera Client called **[Transfer Agents](#agents)**.
|
77
|
+
|
78
|
+
> **Note:** The transfer agent is a client for the remote Transfer Server (HSTS).
|
79
|
+
The transfer agent may be local or remote...
|
80
|
+
For example a remote Aspera Server may be used as a transfer agent (using node API).
|
81
|
+
i.e. using option `--transfer=node`
|
66
82
|
|
67
83
|
## Quick Start
|
68
84
|
|
@@ -77,7 +93,7 @@ ascli --version
|
|
77
93
|
```
|
78
94
|
|
79
95
|
```bash
|
80
|
-
4.
|
96
|
+
4.12.0
|
81
97
|
```
|
82
98
|
|
83
99
|
### First use
|
@@ -88,6 +104,8 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
88
104
|
|
89
105
|
To test with Aspera demo transfer server, setup the environment and then test:
|
90
106
|
|
107
|
+
<!-- spellchecker: disable -->
|
108
|
+
|
91
109
|
```bash
|
92
110
|
ascli config initdemo
|
93
111
|
```
|
@@ -107,6 +125,8 @@ ascli server browse /
|
|
107
125
|
:............:...........:......:........:...........................:.......................:
|
108
126
|
```
|
109
127
|
|
128
|
+
<!-- spellchecker: enable -->
|
129
|
+
|
110
130
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
111
131
|
|
112
132
|
- create a [option preset](#lprt)
|
@@ -115,7 +135,7 @@ If you want to use `ascli` with another server, and in order to make further cal
|
|
115
135
|
- download a file
|
116
136
|
|
117
137
|
```bash
|
118
|
-
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
138
|
+
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here
|
119
139
|
```
|
120
140
|
|
121
141
|
```output
|
@@ -127,7 +147,7 @@ ascli config preset set default server myserver
|
|
127
147
|
```
|
128
148
|
|
129
149
|
```output
|
130
|
-
updated: default→server to myserver
|
150
|
+
updated: default → server to myserver
|
131
151
|
```
|
132
152
|
|
133
153
|
```bash
|
@@ -138,15 +158,10 @@ ascli server browse /aspera-test-dir-large
|
|
138
158
|
:............:...........:......:..............:...........................:............................:
|
139
159
|
: zmode : zuid : zgid : size : mtime : name :
|
140
160
|
:............:...........:......:..............:...........................:............................:
|
141
|
-
: -rw-rw-rw- : asperaweb : fasp : 10133504 : 2018-05-04 14:16:24 +0200 : ctl_female_2.fastq.partial :
|
142
161
|
: -rw-r--r-- : asperaweb : fasp : 209715200 : 2014-04-10 19:49:27 +0200 : 200MB :
|
143
162
|
: -rw-r--r-- : asperaweb : fasp : 524288000 : 2014-04-10 19:44:15 +0200 : 500MB :
|
144
163
|
: -rw-r--r-- : asperaweb : fasp : 5368709120 : 2014-04-10 19:45:52 +0200 : 5GB :
|
145
164
|
: -rw-r--r-- : asperaweb : fasp : 500000000000 : 2017-06-14 20:09:57 +0200 : 500GB :
|
146
|
-
: -rw-rw-rw- : asperaweb : fasp : 13606912 : 2018-05-04 14:20:21 +0200 : ctl_male_2.fastq.partial :
|
147
|
-
: -rw-rw-rw- : asperaweb : fasp : 76 : 2018-05-04 14:13:18 +0200 : ctl_female_2.fastq.haspx :
|
148
|
-
: -rw-rw-rw- : asperaweb : fasp : 647348 : 2018-05-04 14:26:39 +0200 : ctl_female_2.gz :
|
149
|
-
: -rw-rw-rw- : asperaweb : fasp : 74 : 2018-05-04 14:16:00 +0200 : ctl_male_2.fastq.haspx :
|
150
165
|
: -rw-r--r-- : asperaweb : fasp : 1048576000 : 2014-04-10 19:49:23 +0200 : 1GB :
|
151
166
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
152
167
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
@@ -158,7 +173,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
158
173
|
```
|
159
174
|
|
160
175
|
```output
|
161
|
-
Time: 00:00:02
|
176
|
+
Time: 00:00:02 =========================================================== 100% 100 Mbps Time: 00:00:00
|
162
177
|
complete
|
163
178
|
```
|
164
179
|
|
@@ -174,9 +189,13 @@ It is possible to install *either* directly on the host operating system (Linux,
|
|
174
189
|
|
175
190
|
The direct installation is recommended and consists in installing:
|
176
191
|
|
177
|
-
- [Ruby](#ruby)
|
192
|
+
- [Ruby](#ruby)
|
178
193
|
- [aspera-cli](#the_gem)
|
179
|
-
- [Aspera SDK (ascp)](#fasp_prot)
|
194
|
+
- [Aspera SDK (`ascp`)](#fasp_prot)
|
195
|
+
|
196
|
+
Ruby version: >= 2.4.
|
197
|
+
|
198
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
180
199
|
|
181
200
|
The following sections provide information on the various installation methods.
|
182
201
|
|
@@ -184,33 +203,114 @@ An internet connection is required for the installation. If you don't have inter
|
|
184
203
|
|
185
204
|
### Docker container
|
186
205
|
|
187
|
-
|
206
|
+
The image is: [martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli).
|
207
|
+
The container contains: Ruby, `ascli` and the Aspera Transfer SDK.
|
208
|
+
To use the container, ensure that you have `docker` (or `podman`) installed.
|
188
209
|
|
189
|
-
|
190
|
-
|
210
|
+
```bash
|
211
|
+
docker --version
|
212
|
+
```
|
191
213
|
|
192
|
-
|
214
|
+
**Wanna start quickly ?** With an interactive shell ? Execute this:
|
193
215
|
|
194
216
|
```bash
|
195
|
-
docker --
|
217
|
+
docker run --tty --interactive --entrypoint bash martinlaurent/ascli:latest
|
218
|
+
```
|
219
|
+
|
220
|
+
Then, execute individual `ascli` commands such as:
|
221
|
+
|
222
|
+
```bash
|
223
|
+
ascli conf init
|
224
|
+
ascli conf preset overview
|
225
|
+
ascli conf ascp info
|
226
|
+
ascli server ls /
|
196
227
|
```
|
197
228
|
|
198
|
-
|
229
|
+
That is simple, but there are limitations:
|
199
230
|
|
200
|
-
|
231
|
+
- Everything happens in the container
|
232
|
+
- Any generated file in the container will be lost on container (shell) exit. Including configuration files and downloaded files.
|
233
|
+
- No possibility to upload files located on the host system
|
201
234
|
|
202
|
-
|
235
|
+
The container image is built from this [Dockerfile](Dockerfile): the entry point is `ascli` and the default command is `help`.
|
203
236
|
|
204
|
-
|
237
|
+
The container can also be execute for individual commands like this: (add `ascli` commands and options at the end of the command line, e.g. `-v` to display the version)
|
205
238
|
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
239
|
+
```bash
|
240
|
+
docker run --rm --tty --interactive martinlaurent/ascli:latest
|
241
|
+
```
|
242
|
+
|
243
|
+
For more convenience, you may define a shell alias:
|
244
|
+
|
245
|
+
```bash
|
246
|
+
alias ascli='docker run --rm --tty --interactive martinlaurent/ascli:latest'
|
247
|
+
```
|
248
|
+
|
249
|
+
Then, you can execute the container like a local command:
|
250
|
+
|
251
|
+
```bash
|
252
|
+
ascli -v
|
253
|
+
```
|
254
|
+
|
255
|
+
```text
|
256
|
+
4.12.0
|
257
|
+
```
|
258
|
+
|
259
|
+
In order to keep persistency of configuration on the host,
|
260
|
+
you should mount your user's config folder to the container.
|
261
|
+
To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`).
|
262
|
+
Add options:
|
263
|
+
|
264
|
+
```bash
|
265
|
+
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
266
|
+
```
|
212
267
|
|
213
|
-
|
268
|
+
> **Note:** if you are using a `podman machine`, e.g. on Macos , make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
269
|
+
|
270
|
+
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
271
|
+
you can change the entry point, add option:
|
272
|
+
|
273
|
+
```bash
|
274
|
+
--entrypoint bash
|
275
|
+
```
|
276
|
+
|
277
|
+
You may also probably want that files downloaded in the container are in fact placed on the host.
|
278
|
+
In this case you need also to mount the shared transfer folder:
|
279
|
+
|
280
|
+
```bash
|
281
|
+
--volume $HOME/xferdir:/xferfiles
|
282
|
+
```
|
283
|
+
|
284
|
+
> **Note:** ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
285
|
+
|
286
|
+
And if you want all the above, simply use all the options:
|
287
|
+
|
288
|
+
```bash
|
289
|
+
alias asclish="docker run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash martinlaurent/ascli:latest"
|
290
|
+
```
|
291
|
+
|
292
|
+
```bash
|
293
|
+
export xferdir=$HOME/xferdir
|
294
|
+
mkdir -p $xferdir
|
295
|
+
chmod -R 777 $xferdir
|
296
|
+
mkdir -p $HOME/.aspera/ascli
|
297
|
+
asclish
|
298
|
+
```
|
299
|
+
|
300
|
+
A convenience sample script is also provided: download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
301
|
+
|
302
|
+
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
303
|
+
|
304
|
+
Some environment variables can be set for this script to adapt its behavior:
|
305
|
+
|
306
|
+
| env var | description | default | example |
|
307
|
+
|--------------|------------------------------------|--------------------------|--------------------------|
|
308
|
+
| ASCLI_HOME | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascliconfig` |
|
309
|
+
| docker_args | additional options to `docker` | <empty> | `--volume /Users:/Users` |
|
310
|
+
| image | container image name | martinlaurent/ascli | |
|
311
|
+
| version | container image version | latest | `4.8.0.pre` |
|
312
|
+
|
313
|
+
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
214
314
|
(value expected in the container).
|
215
315
|
This allows having persistent configuration on the host.
|
216
316
|
|
@@ -221,21 +321,18 @@ Example of use:
|
|
221
321
|
```bash
|
222
322
|
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
223
323
|
chmod a+x ascli
|
224
|
-
export ASCLI__HOME=$HOME/.ascliconf
|
225
|
-
mkdir -p $ASCLI__HOME
|
226
|
-
chmod -R 777 $ASCLI__HOME
|
227
324
|
export xferdir=$HOME/xferdir
|
228
325
|
mkdir -p $xferdir
|
229
326
|
chmod -R 777 $xferdir
|
230
|
-
export docker_args="--volume $xferdir:/
|
327
|
+
export docker_args="--volume $xferdir:/xferfiles"
|
231
328
|
|
232
329
|
./ascli conf init
|
233
330
|
|
234
|
-
|
235
|
-
./ascli server upload /
|
331
|
+
echo 'Local file to transfer' > $xferdir/samplefile.txt
|
332
|
+
./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload
|
236
333
|
```
|
237
334
|
|
238
|
-
> The local file (`samplefile`) is specified relative to storage view from container (`/
|
335
|
+
> **Note:** The local file (`samplefile.txt`) is specified relative to storage view from container (`/xferfiles`) mapped to the host folder `$HOME/xferdir`
|
239
336
|
|
240
337
|
### <a id="ruby"></a>Ruby
|
241
338
|
|
@@ -243,7 +340,9 @@ Use this method to install on the native host.
|
|
243
340
|
|
244
341
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
245
342
|
|
246
|
-
Required Ruby version: >= 2.4.
|
343
|
+
Required Ruby version: >= 2.4.
|
344
|
+
|
345
|
+
> **Deprecation notice**: the minimum Ruby version will be 2.7 in a future version.
|
247
346
|
|
248
347
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
249
348
|
|
@@ -251,7 +350,7 @@ Refer to the following sections for a proposed method for specific operating sys
|
|
251
350
|
|
252
351
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
253
352
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
254
|
-
If you have a simpler better way to install Ruby : use it !
|
353
|
+
If you have a simpler better way to install Ruby : use it !
|
255
354
|
|
256
355
|
#### Generic: RVM: single user installation (not root)
|
257
356
|
|
@@ -318,7 +417,7 @@ Install Latest stable Ruby:
|
|
318
417
|
- Download the latest Ruby installer **with devkit**. (Msys2 is needed to install some native extensions, such as `grpc`)
|
319
418
|
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
320
419
|
- At the end of the installation procedure, the Msys2 installer is automatically executed, select option 3 (msys and mingw)
|
321
|
-
|
420
|
+
|
322
421
|
#### macOS: pre-installed or `brew`
|
323
422
|
|
324
423
|
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
@@ -337,15 +436,20 @@ brew install ruby
|
|
337
436
|
|
338
437
|
If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
|
339
438
|
|
340
|
-
Example:
|
439
|
+
Example: RHEL 8 and 9: basic installation
|
440
|
+
|
441
|
+
```bash
|
442
|
+
yum module install ruby:3.1
|
443
|
+
```
|
444
|
+
|
445
|
+
Example: RHEL 8, Centos 8 Stream: with extensions to compile native gems
|
341
446
|
|
342
447
|
```bash
|
343
448
|
yum install make automake gcc gcc-c++ kernel-devel
|
344
449
|
yum install redhat-rpm-config
|
345
450
|
dnf module reset ruby
|
346
|
-
dnf module enable ruby:3.
|
347
|
-
dnf module -y install ruby:3.
|
348
|
-
gem install aspera-cli
|
451
|
+
dnf module enable ruby:3.1
|
452
|
+
dnf module -y install ruby:3.1/common
|
349
453
|
```
|
350
454
|
|
351
455
|
Other examples:
|
@@ -398,7 +502,7 @@ If you already have a Java JVM on your system (`java`), it is possible to use `j
|
|
398
502
|
|
399
503
|
<https://www.jruby.org/download>
|
400
504
|
|
401
|
-
Note
|
505
|
+
> **Note:** Using jruby the startup time is longer than the native ruby, but the transfer speed is not impacted (executed by `ascp` binary).
|
402
506
|
|
403
507
|
### <a id="the_gem"></a>`aspera-cli` gem
|
404
508
|
|
@@ -427,7 +531,7 @@ ascli conf check_update
|
|
427
531
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
428
532
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
429
533
|
|
430
|
-
- ascp
|
534
|
+
- `ascp`
|
431
535
|
- aspera-license (in same folder, or ../etc)
|
432
536
|
|
433
537
|
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
@@ -467,7 +571,8 @@ Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, b
|
|
467
571
|
|
468
572
|
### <a id="offline_install"></a>Installation in air gapped environment
|
469
573
|
|
470
|
-
Note
|
574
|
+
> **Note:** no pre-packaged version is provided.
|
575
|
+
|
471
576
|
A method to build one is provided here:
|
472
577
|
|
473
578
|
The procedure:
|
@@ -528,7 +633,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
528
633
|
- Supports most Aspera server products (on-premise and SaaS)
|
529
634
|
- Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
530
635
|
- Supports Commands, Option values and Parameters shortcuts
|
531
|
-
- FASP [Transfer Agents](#agents) can be: local ascp
|
636
|
+
- FASP [Transfer Agents](#agents) can be: local `ascp`, or Connect Client, or any transfer node
|
532
637
|
- Transfer parameters can be altered by modification of [*transfer-spec*](#transferspec), this includes requiring multi-session
|
533
638
|
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
534
639
|
- Supports FaspStream creation (using Node API)
|
@@ -551,14 +656,14 @@ Not all `ascli` features are fully documented here, the user may explore command
|
|
551
656
|
|
552
657
|
If you want to use `ascp` directly as a command line, refer to IBM Aspera documentation of either [Desktop Client](https://www.ibm.com/docs/en/asdc), [Endpoint](https://www.ibm.com/docs/en/ahte) or [Transfer Server](https://www.ibm.com/docs/en/ahts) where [a section on `ascp` can be found](https://www.ibm.com/docs/en/ahts/4.4?topic=linux-ascp-transferring-from-command-line).
|
553
658
|
|
554
|
-
Using `ascli` with plugin `server` for command line gives advantages over ascp
|
659
|
+
Using `ascli` with plugin `server` for command line gives advantages over `ascp`:
|
555
660
|
|
556
661
|
- automatic resume on error
|
557
662
|
- configuration file
|
558
663
|
- choice of transfer agents
|
559
664
|
- integrated support of multi-session
|
560
665
|
|
561
|
-
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`
|
666
|
+
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args`).
|
562
667
|
|
563
668
|
### <a id="parsing"></a>Command line parsing, Special Characters
|
564
669
|
|
@@ -604,16 +709,17 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
604
709
|
```
|
605
710
|
|
606
711
|
`config echo` displays the value of the first argument using Ruby syntax: it surrounds a string with `"` and add `\` before special characters.
|
607
|
-
|
712
|
+
|
713
|
+
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
608
714
|
|
609
715
|
In the following examples (using a POSIX shell, such as `bash`), several sample commands are provided when equivalent.
|
610
|
-
For all example, most of special character handling is not specific to `ascli`: It
|
716
|
+
For all example, most of special character handling is not specific to `ascli`: It depends on the underlying syntax: shell , JSON, etc...
|
611
717
|
Depending on the case, a different `format` is used to display the actual value.
|
612
718
|
|
613
719
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
614
720
|
|
615
721
|
Double quotes are processed by the shell to create a single string argument.
|
616
|
-
For POSIX shells, single quotes can also be used in this case, or
|
722
|
+
For POSIX shells, single quotes can also be used in this case, or protect the special character ` ` (space) with a backslash. <!-- markdownlint-disable-line -->
|
617
723
|
|
618
724
|
```bash
|
619
725
|
ascli conf echo "Hello World" --format=text
|
@@ -628,8 +734,9 @@ Hello World
|
|
628
734
|
#### Using a shell variable, parsed by shell, in an extended value
|
629
735
|
|
630
736
|
To be evaluated by shell, the shell variable must not be in single quotes.
|
631
|
-
|
632
|
-
|
737
|
+
Even if the variable contains spaces it makes only one argument to `ascli` because word parsing is made before variable expansion by shell.
|
738
|
+
|
739
|
+
> **Note:** we use a simple variable here: the variable is not necessarily an environment variable.
|
633
740
|
|
634
741
|
```bash
|
635
742
|
MYVAR="Hello World"
|
@@ -702,9 +809,9 @@ ascli conf echo @ruby:"{'title'=>%q{Test \" ' & \\\\}}" --format=json
|
|
702
809
|
{"title":"Test \" ' & \\"}
|
703
810
|
```
|
704
811
|
|
705
|
-
#### Reading special characters
|
812
|
+
#### Reading special characters interactively
|
706
813
|
|
707
|
-
If `ascli` is used
|
814
|
+
If `ascli` is used interactively (a user typing on terminal), it is easy to require the user to type values:
|
708
815
|
|
709
816
|
```bash
|
710
817
|
ascli conf echo @ruby:"{'title'=>gets.chomp}" --format=json
|
@@ -787,7 +894,7 @@ ascli config echo -- --sample
|
|
787
894
|
"--sample"
|
788
895
|
```
|
789
896
|
|
790
|
-
Note
|
897
|
+
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
791
898
|
|
792
899
|
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
793
900
|
|
@@ -860,7 +967,7 @@ The style of output can be set using the `format` parameter, supporting:
|
|
860
967
|
|
861
968
|
Table output can be filtered using the `select` parameter. Example:
|
862
969
|
|
863
|
-
```
|
970
|
+
```bash
|
864
971
|
ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
865
972
|
```
|
866
973
|
|
@@ -873,7 +980,7 @@ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sor
|
|
873
980
|
:...............................:..................................:...........:
|
874
981
|
```
|
875
982
|
|
876
|
-
Note
|
983
|
+
> **Note:** `select` filters selected elements from the result of API calls, while the `query` parameters gives filtering parameters to the API when listing elements.
|
877
984
|
|
878
985
|
#### Verbosity of output
|
879
986
|
|
@@ -913,7 +1020,9 @@ The extended value syntax is:
|
|
913
1020
|
<0 or more decoders><0 or 1 reader><nothing or some text value>
|
914
1021
|
```
|
915
1022
|
|
916
|
-
The difference between reader and decoder is order and
|
1023
|
+
The difference between reader and decoder is order and cardinality.
|
1024
|
+
Both act like a function of value on right hand side.
|
1025
|
+
Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
|
917
1026
|
|
918
1027
|
The following "readers" are supported (returns value in []):
|
919
1028
|
|
@@ -923,13 +1032,13 @@ The following "readers" are supported (returns value in []):
|
|
923
1032
|
- @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
|
924
1033
|
- @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
925
1034
|
- @stdin: : [String] read from stdin (no value on right)
|
926
|
-
- @preset:NAME : [Hash] get whole option preset value by name.
|
1035
|
+
- @preset:NAME : [Hash] get whole option preset value by name. Sub-values can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]`
|
927
1036
|
|
928
1037
|
In addition it is possible to decode a value, using one or multiple decoders :
|
929
1038
|
|
930
1039
|
- @base64: [String] decode a base64 encoded string
|
931
1040
|
- @json: [any] decode JSON values (convenient to provide complex structures)
|
932
|
-
- @zlib: [String]
|
1041
|
+
- @zlib: [String] un-compress data
|
933
1042
|
- @ruby: [any] execute ruby code
|
934
1043
|
- @csvt: [Array] decode a titled CSV value
|
935
1044
|
- @lines: [Array] split a string in multiple lines and return an array
|
@@ -944,10 +1053,10 @@ Example: read the content of the specified file, then, base64 decode, then unzip
|
|
944
1053
|
ascli config echo @zlib:@base64:@file:myfile.dat
|
945
1054
|
```
|
946
1055
|
|
947
|
-
Example:
|
1056
|
+
Example: Create a value as a hash, with one key and the value is read from a file:
|
948
1057
|
|
949
1058
|
```bash
|
950
|
-
ascli config echo @ruby:'{"token_verification_key"=>File.read("
|
1059
|
+
ascli config echo @ruby:'{"token_verification_key"=>File.read("mykey.txt")}'
|
951
1060
|
```
|
952
1061
|
|
953
1062
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
@@ -977,7 +1086,7 @@ ascli config echo @csvt:@file:test.csv
|
|
977
1086
|
|
978
1087
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
979
1088
|
|
980
|
-
```
|
1089
|
+
```bash
|
981
1090
|
ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
982
1091
|
```
|
983
1092
|
|
@@ -985,7 +1094,7 @@ ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
|
985
1094
|
{"version"=>"0.9", "hello"=>true}
|
986
1095
|
```
|
987
1096
|
|
988
|
-
Note
|
1097
|
+
> **Note:** `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` are equivalent to: `@preset:config`
|
989
1098
|
|
990
1099
|
### <a id="native"></a>Structured Value
|
991
1100
|
|
@@ -1027,7 +1136,7 @@ ascli config folder
|
|
1027
1136
|
C:\Users\Kenji\.aspera\ascli
|
1028
1137
|
```
|
1029
1138
|
|
1030
|
-
When OAuth is used (AoC, Faspex4
|
1139
|
+
When OAuth is used (AoC, Faspex4 api v4, Faspex5) `ascli` keeps a cache of generated bearer tokens in `[config folder]/persist_store` by default.
|
1031
1140
|
Option `cache_tokens` (**yes**/no) allows to control if Oauth tokens are cached on file system, or generated for each request.
|
1032
1141
|
The command `config flush_tokens` deletes all existing tokens.
|
1033
1142
|
Tokens are kept on disk for a maximum of 30 minutes (`TOKEN_CACHE_EXPIRY_SEC`) and garbage collected after that.
|
@@ -1061,7 +1170,7 @@ ascli config preset set|delete|show|initialize|update <option preset>
|
|
1061
1170
|
The command `update` allows the easy creation of [option preset](#lprt) by simply providing the options in their command line format, e.g. :
|
1062
1171
|
|
1063
1172
|
```bash
|
1064
|
-
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
1173
|
+
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here --ts=@json:'{"precalculate_job_size":true}'
|
1065
1174
|
```
|
1066
1175
|
|
1067
1176
|
- This creates a [option preset](#lprt) `demo_server` with all provided options.
|
@@ -1069,13 +1178,13 @@ ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --u
|
|
1069
1178
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
1070
1179
|
|
1071
1180
|
```bash
|
1072
|
-
ascli config preset set demo_server password
|
1181
|
+
ascli config preset set demo_server password my_password_here
|
1073
1182
|
```
|
1074
1183
|
|
1075
1184
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a [*Structured Value*](#native).
|
1076
1185
|
|
1077
|
-
```
|
1078
|
-
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"
|
1186
|
+
```bash
|
1187
|
+
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"my_pass_here","ts":{"precalculate_job_size":true}}'
|
1079
1188
|
```
|
1080
1189
|
|
1081
1190
|
A full terminal based overview of the configuration can be displayed using:
|
@@ -1097,6 +1206,8 @@ If necessary, the configuration file can opened in a text editor with:
|
|
1097
1206
|
ascli config open
|
1098
1207
|
```
|
1099
1208
|
|
1209
|
+
> **Note:** this starts the editor specified by env var `EDITOR` if defined.
|
1210
|
+
|
1100
1211
|
Older format for commands are still supported:
|
1101
1212
|
|
1102
1213
|
```bash
|
@@ -1115,7 +1226,7 @@ This preset name is reserved and contains an array of key-value , where the key
|
|
1115
1226
|
|
1116
1227
|
When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
|
1117
1228
|
|
1118
|
-
Note
|
1229
|
+
> **Note:** Special plugin name: `config` can be associated with a preset that is loaded initially, typically used for default values.
|
1119
1230
|
|
1120
1231
|
Operations on this preset are done using regular `config` operations:
|
1121
1232
|
|
@@ -1127,13 +1238,13 @@ ascli config preset set default _plugin_name_ _default_preset_for_plugin_
|
|
1127
1238
|
ascli config preset get default _plugin_name_
|
1128
1239
|
```
|
1129
1240
|
|
1130
|
-
```
|
1241
|
+
```json
|
1131
1242
|
"_default_preset_for_plugin_"
|
1132
1243
|
```
|
1133
1244
|
|
1134
|
-
#### <a id="
|
1245
|
+
#### <a id="config"></a>Plugin: `config`: CLI Configuration
|
1135
1246
|
|
1136
|
-
Plugin `config`
|
1247
|
+
Plugin `config` is used to configure `ascli` and also contains global options.
|
1137
1248
|
|
1138
1249
|
When `ascli` starts, it looks for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
1139
1250
|
|
@@ -1197,7 +1308,7 @@ cli_default:
|
|
1197
1308
|
demo_server:
|
1198
1309
|
url: ssh://demo.asperasoft.com:33001
|
1199
1310
|
username: asperaweb
|
1200
|
-
password:
|
1311
|
+
password: my_password_here
|
1201
1312
|
```
|
1202
1313
|
|
1203
1314
|
We can see here:
|
@@ -1272,13 +1383,13 @@ ascli config preset unset cli_default interactive
|
|
1272
1383
|
Example: Define options using command line:
|
1273
1384
|
|
1274
1385
|
```bash
|
1275
|
-
ascli -N --url=_url_here_ --password=
|
1386
|
+
ascli -N --url=_url_here_ --password=my_password_here --username=_name_here_ node --show-config
|
1276
1387
|
```
|
1277
1388
|
|
1278
1389
|
Example: Define options using a hash:
|
1279
1390
|
|
1280
|
-
```
|
1281
|
-
ascli -N --preset=@json:'{"url":"_url_here_","password":"
|
1391
|
+
```bash
|
1392
|
+
ascli -N --preset=@json:'{"url":"_url_here_","password":"my_password_here","username":"_name_here_"}' node --show-config
|
1282
1393
|
```
|
1283
1394
|
|
1284
1395
|
#### Shares Examples
|
@@ -1288,7 +1399,7 @@ only username/password and url are required (either on command line, or from con
|
|
1288
1399
|
Those can usually be provided on the command line:
|
1289
1400
|
|
1290
1401
|
```bash
|
1291
|
-
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=
|
1402
|
+
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=my_password_here
|
1292
1403
|
```
|
1293
1404
|
|
1294
1405
|
This can also be provisioned in a config file:
|
@@ -1298,19 +1409,19 @@ This can also be provisioned in a config file:
|
|
1298
1409
|
```bash
|
1299
1410
|
ascli config preset set shares06 url https://10.25.0.6
|
1300
1411
|
ascli config preset set shares06 username john
|
1301
|
-
ascli config preset set shares06 password
|
1412
|
+
ascli config preset set shares06 password my_password_here
|
1302
1413
|
```
|
1303
1414
|
|
1304
1415
|
This can also be done with one single command:
|
1305
1416
|
|
1306
|
-
```
|
1307
|
-
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"
|
1417
|
+
```bash
|
1418
|
+
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"my_password_here"}'
|
1308
1419
|
```
|
1309
1420
|
|
1310
1421
|
or
|
1311
1422
|
|
1312
1423
|
```bash
|
1313
|
-
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=
|
1424
|
+
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=my_password_here
|
1314
1425
|
```
|
1315
1426
|
|
1316
1427
|
- Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
@@ -1347,20 +1458,20 @@ The vault is used with options `vault` and `vault_password`.
|
|
1347
1458
|
|
1348
1459
|
`vault_password` specifies the password for the vault.
|
1349
1460
|
Although it can be specified on command line, for security reason you can hide the value.
|
1350
|
-
For example it can be specified on command line like this:
|
1461
|
+
For example it can be securely specified on command line like this:
|
1351
1462
|
|
1352
1463
|
```bash
|
1353
1464
|
export ASCLI_VAULT_PASSWORD
|
1354
1465
|
read -s ASCLI_VAULT_PASSWORD
|
1355
1466
|
```
|
1356
1467
|
|
1357
|
-
#### Vault: System
|
1468
|
+
#### Vault: System key chain
|
1358
1469
|
|
1359
|
-
> **macOS only**
|
1470
|
+
> **Note:** **macOS only**
|
1360
1471
|
|
1361
|
-
It is possible to manage secrets in macOS
|
1472
|
+
It is possible to manage secrets in macOS key chain (only read supported currently).
|
1362
1473
|
|
1363
|
-
```
|
1474
|
+
```bash
|
1364
1475
|
--vault=@json:'{"type":"system","name":"ascli"}'
|
1365
1476
|
```
|
1366
1477
|
|
@@ -1368,11 +1479,11 @@ It is possible to manage secrets in macOS keychain (only read supported currentl
|
|
1368
1479
|
|
1369
1480
|
It is possible to store and use secrets encrypted in a file.
|
1370
1481
|
|
1371
|
-
```
|
1482
|
+
```bash
|
1372
1483
|
--vault=@json:'{"type":"file","name":"vault.bin"}'
|
1373
1484
|
```
|
1374
1485
|
|
1375
|
-
`name` is the file path, absolute or relative to the config folder `
|
1486
|
+
`name` is the file path, absolute or relative to the config folder `ASCLI_HOME`.
|
1376
1487
|
|
1377
1488
|
#### Vault: Operations
|
1378
1489
|
|
@@ -1386,7 +1497,7 @@ Then secrets can be manipulated using commands:
|
|
1386
1497
|
- `delete`
|
1387
1498
|
|
1388
1499
|
```bash
|
1389
|
-
ascli conf vault create mylabel @json:'{"password":"
|
1500
|
+
ascli conf vault create mylabel @json:'{"password":"my_password_here","description":"for this account"}'
|
1390
1501
|
```
|
1391
1502
|
|
1392
1503
|
#### <a id="config_finder"></a>Configuration Finder
|
@@ -1395,6 +1506,24 @@ When a secret is needed by a sub command, the command can search for existing co
|
|
1395
1506
|
|
1396
1507
|
The lookup is done by comparing the service URL and username (or access key).
|
1397
1508
|
|
1509
|
+
#### Securing passwords and secrets
|
1510
|
+
|
1511
|
+
A passwords can be saved in clear in a [option preset](#lprt) together with other account information (URL, username, etc...).
|
1512
|
+
Example:
|
1513
|
+
|
1514
|
+
```bash
|
1515
|
+
`ascli` conf preset update myconf --url=... --username=... --password=...
|
1516
|
+
```
|
1517
|
+
|
1518
|
+
For a more secure storage one can do:
|
1519
|
+
|
1520
|
+
```bash
|
1521
|
+
`ascli` conf preset update myconf --url=... --username=... --password=@val:@vault:myconf.password
|
1522
|
+
`ascli` conf vault create myconf @json:'{"password":"my_password_here"}'
|
1523
|
+
```
|
1524
|
+
|
1525
|
+
> **Note:** use `@val:` in front of `@vault:` so that the extended value is not evaluated.
|
1526
|
+
|
1398
1527
|
### <a id="private_key"></a>Private Key
|
1399
1528
|
|
1400
1529
|
Some applications allow the user to be authenticated using a private key (Server, AoC, Faspex5, ...).
|
@@ -1413,9 +1542,9 @@ Typically, in `$HOME/.ssh` or `$HOME/.aspera/ascli`:
|
|
1413
1542
|
PRIVKEYFILE=~/.aspera/ascli/my_private_key
|
1414
1543
|
```
|
1415
1544
|
|
1416
|
-
Several methods can be used to generate a key pair
|
1545
|
+
Several methods can be used to generate a key pair.
|
1417
1546
|
|
1418
|
-
|
1547
|
+
#### `ascli` for key generation
|
1419
1548
|
|
1420
1549
|
The generated key is of type RSA, by default: 4096 bit.
|
1421
1550
|
For convenience, the public key is also extracted with extension `.pub`.
|
@@ -1425,7 +1554,7 @@ The key is not passphrase protected.
|
|
1425
1554
|
ascli config genkey ${PRIVKEYFILE} 4096
|
1426
1555
|
```
|
1427
1556
|
|
1428
|
-
|
1557
|
+
#### `ssh-keygen`
|
1429
1558
|
|
1430
1559
|
Both private and public keys are generated, option `-N` is for passphrase.
|
1431
1560
|
|
@@ -1433,24 +1562,27 @@ Both private and public keys are generated, option `-N` is for passphrase.
|
|
1433
1562
|
ssh-keygen -t rsa -b 4096 -m PEM -N '' -f ${PRIVKEYFILE}
|
1434
1563
|
```
|
1435
1564
|
|
1436
|
-
|
1565
|
+
#### `openssl`
|
1437
1566
|
|
1438
1567
|
To generate a private key pair with a passphrase the following can be used on any system:
|
1439
|
-
|
1568
|
+
<!-- spellchecker: disable -->
|
1440
1569
|
```bash
|
1441
1570
|
openssl genrsa -passout pass:_passphrase_here_ -out ${PRIVKEYFILE}.protected 4096
|
1442
1571
|
openssl rsa -pubout -in ${PRIVKEYFILE} -out ${PRIVKEYFILE}.pub
|
1443
1572
|
```
|
1573
|
+
<!-- spellchecker: enable -->
|
1444
1574
|
|
1445
1575
|
`openssl` is sometimes compiled to support option `-nodes` (no DES, i.e. no passphrase, e.g. on macOS).
|
1446
1576
|
In that case, add option `-nodes` instead of `-passout pass:_passphrase_here_` to generate a key without passphrase.
|
1447
1577
|
|
1448
1578
|
If option `-nodes` is not available, the passphrase can be removed using this method:
|
1449
1579
|
|
1580
|
+
<!-- spellchecker: disable -->
|
1450
1581
|
```bash
|
1451
1582
|
openssl rsa -passin pass:_passphrase_here_ -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1452
1583
|
rm -f ${PRIVKEYFILE}.protected
|
1453
1584
|
```
|
1585
|
+
<!-- spellchecker: enable -->
|
1454
1586
|
|
1455
1587
|
To change (or add) the passphrase for a key do:
|
1456
1588
|
|
@@ -1494,9 +1626,9 @@ ascli conf plugin list
|
|
1494
1626
|
|
1495
1627
|
#### <a id="createownplugin"></a>Create your own plugin
|
1496
1628
|
|
1497
|
-
By default plugins are looked-up in folders
|
1629
|
+
By default plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
1498
1630
|
|
1499
|
-
```
|
1631
|
+
```bash
|
1500
1632
|
ascli --show-config --select=@json:'{"key":"plugin_folder"}'
|
1501
1633
|
```
|
1502
1634
|
|
@@ -1545,7 +1677,7 @@ Available loggers: `stdout`, `stderr`, `syslog`.
|
|
1545
1677
|
|
1546
1678
|
Available levels: `debug`, `info`, `warn`, `error`.
|
1547
1679
|
|
1548
|
-
Note
|
1680
|
+
> **Note:** When using the `direct` agent (`ascp`), additional transfer logs can be activated using `ascp` options and `ascp_args`, see [`direct`](#agt_direct).
|
1549
1681
|
|
1550
1682
|
Examples:
|
1551
1683
|
|
@@ -1591,7 +1723,7 @@ Like any other option, those can be set either on command line, or in config fil
|
|
1591
1723
|
|
1592
1724
|
Example:
|
1593
1725
|
|
1594
|
-
```
|
1726
|
+
```bash
|
1595
1727
|
ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
1596
1728
|
```
|
1597
1729
|
|
@@ -1623,9 +1755,9 @@ Refer to the following sections.
|
|
1623
1755
|
There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is used.
|
1624
1756
|
|
1625
1757
|
The `http_proxy` environment variable (**lower case**, preferred) can be set to the URL of the proxy, e.g. `http://myproxy.org.net:3128`.
|
1626
|
-
Refer to [Ruby
|
1758
|
+
Refer to [Ruby find proxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1627
1759
|
|
1628
|
-
> Note
|
1760
|
+
> **Note:** Ruby expects a URL and `myproxy.org.net:3128` alone is **not** accepted.
|
1629
1761
|
|
1630
1762
|
```bash
|
1631
1763
|
export http_proxy=http://proxy.example.com:3128
|
@@ -1678,8 +1810,7 @@ ascli --proxy-credentials=@list::__username_here__:__password_here__ ...
|
|
1678
1810
|
|
1679
1811
|
### Proxy for Legacy Aspera HTTP/S Fallback
|
1680
1812
|
|
1681
|
-
To specify a proxy for legacy HTTP fallback, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url
|
1682
|
-
(It is also possible to use `EX_ascp_args` and native options in `direct`)
|
1813
|
+
Only supported with the `direct` agent: To specify a proxy for legacy HTTP fallback, use `ascp` native option `-x` and `ascp_args`: `--transfer-info=@json:'{"ascp_args":["-x","url_here"]}'`. Alternatively, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url`.
|
1683
1814
|
|
1684
1815
|
### FASP proxy (forward) for transfers
|
1685
1816
|
|
@@ -1687,9 +1818,10 @@ To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) para
|
|
1687
1818
|
|
1688
1819
|
### <a id="client"></a>FASP configuration
|
1689
1820
|
|
1690
|
-
The `config` plugin also allows specification for the use of a local FASP client
|
1821
|
+
The `config` plugin also allows specification for the use of a local FASP **client**.
|
1822
|
+
It provides the following commands for `ascp` subcommand:
|
1691
1823
|
|
1692
|
-
- `show` : shows the path of ascp used
|
1824
|
+
- `show` : shows the path of `ascp` used
|
1693
1825
|
- `use` : list,download connect client versions available on internet
|
1694
1826
|
- `products` : list Aspera transfer products available locally
|
1695
1827
|
- `connect` : list,download connect client versions available on internet
|
@@ -1718,9 +1850,9 @@ ascli config ascp info
|
|
1718
1850
|
|
1719
1851
|
#### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
1720
1852
|
|
1721
|
-
By default, `ascli` uses any found local product with ascp
|
1853
|
+
By default, `ascli` uses any found local product with `ascp`, including SDK.
|
1722
1854
|
|
1723
|
-
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
1855
|
+
To temporarily use an alternate `ascp` path use option `ascp_path` (`--ascp-path=`)
|
1724
1856
|
|
1725
1857
|
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
1726
1858
|
|
@@ -1771,11 +1903,11 @@ ascli config ascp products list
|
|
1771
1903
|
|
1772
1904
|
#### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
1773
1905
|
|
1774
|
-
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1906
|
+
If no `ascp` is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1775
1907
|
|
1776
|
-
Using the option use_product finds the ascp binary of the selected product.
|
1908
|
+
Using the option use_product finds the `ascp` binary of the selected product.
|
1777
1909
|
|
1778
|
-
To permanently use the ascp of a product:
|
1910
|
+
To permanently use the `ascp` of a product:
|
1779
1911
|
|
1780
1912
|
```bash
|
1781
1913
|
ascli config ascp products use 'Aspera Connect'
|
@@ -1821,16 +1953,16 @@ ascli config ascp connect version 'Aspera Connect for Mac Intel' download enclos
|
|
1821
1953
|
```
|
1822
1954
|
|
1823
1955
|
```output
|
1824
|
-
Time: 00:00:02
|
1956
|
+
Time: 00:00:02 =========================================================== 100% 27766 KB/sec Time: 00:00:02
|
1825
1957
|
Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
1826
1958
|
```
|
1827
1959
|
|
1828
|
-
### <a id="agents"></a>Transfer Agents
|
1960
|
+
### <a id="agents"></a>Transfer Clients: Agents
|
1829
1961
|
|
1830
1962
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
1831
1963
|
|
1832
1964
|
When a transfer needs to be started, a [*transfer-spec*](#transferspec) has been internally prepared.
|
1833
|
-
This [*transfer-spec*](#transferspec) will be executed by a transfer client, here called
|
1965
|
+
This [*transfer-spec*](#transferspec) will be executed by a transfer client, here called **Transfer Agent**.
|
1834
1966
|
|
1835
1967
|
There are currently 3 agents:
|
1836
1968
|
|
@@ -1840,15 +1972,15 @@ There are currently 3 agents:
|
|
1840
1972
|
- [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1841
1973
|
- [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1842
1974
|
|
1843
|
-
Note
|
1844
|
-
For
|
1975
|
+
> **Note:** All transfer operations are seen from the point of view of the agent.
|
1976
|
+
For example, a node agent executing an "upload", or "package send" operation
|
1845
1977
|
will effectively push files to the related server from the agent node.
|
1846
1978
|
|
1847
|
-
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *
|
1979
|
+
`ascli` standardizes on the use of a [*transfer-spec*](#transferspec) instead of *native* `ascp` options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1848
1980
|
|
1849
1981
|
#### <a id="agt_direct"></a>Direct
|
1850
1982
|
|
1851
|
-
The `direct` agent directly executes a local ascp
|
1983
|
+
The `direct` agent directly executes a local `ascp`.
|
1852
1984
|
This is the default agent for `ascli`.
|
1853
1985
|
This is equivalent to option `--transfer=direct`.
|
1854
1986
|
`ascli` will detect locally installed Aspera products, including SDK, and use `ascp` from that component.
|
@@ -1856,18 +1988,18 @@ Refer to section [FASP](#client).
|
|
1856
1988
|
|
1857
1989
|
The `transfer_info` option accepts the following optional parameters to control multi-session, Web Socket Session and Resume policy:
|
1858
1990
|
|
1859
|
-
|
1860
|
-
|
1861
|
-
|
1862
|
-
|
1863
|
-
|
1864
|
-
|
1865
|
-
<
|
1866
|
-
|
1867
|
-
|
1868
|
-
|
1869
|
-
|
1870
|
-
|
1991
|
+
| Name | Type | Description |
|
1992
|
+
|----------------------|-------|-------------|
|
1993
|
+
| wss | Bool | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
1994
|
+
| ascp_args | Array | Array of strings with native ascp arguments<br/>Use this instead of deprecated `EX_ascp_args`.<br/>Default: [] |
|
1995
|
+
| spawn_timeout_sec | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
|
1996
|
+
| spawn_delay_sec | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
|
1997
|
+
| multi_incr_udp | Bool | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
|
1998
|
+
| resume | Hash | Resume<br/>parameters<br/>See below |
|
1999
|
+
| resume.iter_max | int | Resume<br/>Max number of retry on error<br/>Default: 7 |
|
2000
|
+
| resume.sleep_initial | int | Resume<br/>First Sleep before retry<br/>Default: 2 |
|
2001
|
+
| resume.sleep_factor | int | Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2 |
|
2002
|
+
| resume.sleep_max | int | Resume<br/>Default: 60 |
|
1871
2003
|
|
1872
2004
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
1873
2005
|
Sleep between iterations is:
|
@@ -1885,40 +2017,40 @@ ascli config ascp errors
|
|
1885
2017
|
|
1886
2018
|
Examples:
|
1887
2019
|
|
1888
|
-
```
|
2020
|
+
```bash
|
1889
2021
|
ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":20}}'
|
1890
2022
|
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
1891
2023
|
```
|
1892
2024
|
|
1893
|
-
Note
|
1894
|
-
|
2025
|
+
> **Note:** The `direct` agent supports additional `transfer_spec` parameters starting with `EX_` (extended).
|
2026
|
+
But it is preferred to use the option `transfer_info` with parameter `ascp_args`.
|
1895
2027
|
|
1896
|
-
This can be useful to activate logging using option `-L` of ascp
|
1897
|
-
For example the option `--
|
2028
|
+
This can be useful to activate logging using option `-L` of `ascp`.
|
2029
|
+
For example the option `--transfer-info=@json:'{"ascp_args":["-DDL-"]}'` will activate debug level 2 for `ascp` (`DD`), and display those logs on the terminal (`-`).
|
1898
2030
|
This is useful if the transfer fails.
|
1899
|
-
To store ascp logs in file `aspera-scp-transfer.log` in a folder, use `--
|
2031
|
+
To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--transfer-info=@json:'{"ascp_args":["-L","/path/to/folder"]}'`.
|
1900
2032
|
|
1901
|
-
>
|
2033
|
+
> **Note:** When transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided using `transfer_info` parameter `ascp_args`).
|
1902
2034
|
|
1903
2035
|
In addition to standard methods described in section [File List](#file_list), it is possible to specify the list of file using those additional methods:
|
1904
2036
|
|
1905
2037
|
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_file_list`
|
1906
2038
|
|
1907
|
-
```
|
1908
|
-
--sources=@ts --ts=@json:'{"EX_file_list":"
|
2039
|
+
```bash
|
2040
|
+
--sources=@ts --ts=@json:'{"EX_file_list":"file_list.txt"}'
|
1909
2041
|
```
|
1910
2042
|
|
1911
|
-
- Using
|
2043
|
+
- Using option `transfer_info` parameter `ascp_args`
|
1912
2044
|
|
1913
|
-
```
|
1914
|
-
--sources=@ts --
|
2045
|
+
```bash
|
2046
|
+
--sources=@ts --transfer-info=@json:'{"ascp_args":["--file-list","myfilelist"]}'
|
1915
2047
|
```
|
1916
2048
|
|
1917
|
-
> File lists is shown here, there are also similar options for file pair lists.
|
1918
|
-
|
1919
|
-
> Those 2 additional methods avoid the creation of a copy of the file list: if the standard options `--sources=@lines:@file:... --src-type=...` are used, then the file is list read and parsed, and a new file list is created in a temporary folder.
|
1920
|
-
|
1921
|
-
> Those methods have limitations: they apply **only** to the [`direct`](#agt_direct) transfer agent (i.e. local `ascp`) and not for Aspera on Cloud.
|
2049
|
+
> **Note:** File lists is shown here, there are also similar options for file pair lists.
|
2050
|
+
>
|
2051
|
+
> **Note:** Those 2 additional methods avoid the creation of a copy of the file list: if the standard options `--sources=@lines:@file:... --src-type=...` are used, then the file is list read and parsed, and a new file list is created in a temporary folder.
|
2052
|
+
>
|
2053
|
+
> **Note:** Those methods have limitations: they apply **only** to the [`direct`](#agt_direct) transfer agent (i.e. local `ascp`) and not for Aspera on Cloud.
|
1922
2054
|
|
1923
2055
|
#### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
1924
2056
|
|
@@ -1930,17 +2062,16 @@ By specifying option: `--transfer=node`, the CLI will start transfers in an Aspe
|
|
1930
2062
|
Transfer Server using the Node API, either on a local or remote node.
|
1931
2063
|
Parameters provided in option `transfer_info` are:
|
1932
2064
|
|
1933
|
-
|
1934
|
-
|
1935
|
-
|
1936
|
-
|
1937
|
-
|
1938
|
-
|
1939
|
-
</table>
|
2065
|
+
| Name | Type | Description |
|
2066
|
+
|------|------|-------------|
|
2067
|
+
| url | string | URL of the node API</br>Mandatory |
|
2068
|
+
| username | string | node api user or access key</br>Mandatory |
|
2069
|
+
| password | string | password, secret or bearer token</br>Mandatory |
|
2070
|
+
| root_id | string | password or secret</br>Mandatory only for bearer token |
|
1940
2071
|
|
1941
2072
|
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
1942
|
-
`--transfer-info=@preset
|
1943
|
-
`--transfer-info=@json:'{"url":"https://...","username":"
|
2073
|
+
`--transfer-info=@preset:_name_here_` or be specified using the extended value syntax :
|
2074
|
+
`--transfer-info=@json:'{"url":"https://...","username":"_user_here_","password":"my_password_here"}'`
|
1944
2075
|
|
1945
2076
|
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
1946
2077
|
|
@@ -1952,24 +2083,23 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
1952
2083
|
|
1953
2084
|
Parameters provided in option `transfer_info` are:
|
1954
2085
|
|
1955
|
-
|
1956
|
-
|
1957
|
-
|
1958
|
-
|
1959
|
-
|
1960
|
-
</table>
|
2086
|
+
| Name | Type | Description |
|
2087
|
+
|------------------------|--------|---------------------------------------|
|
2088
|
+
| url | string | URL of the HTTP GW</br>Mandatory |
|
2089
|
+
| upload_bar_refresh_sec | float | Refresh rate for upload progress bar |
|
2090
|
+
| upload_chunk_size | int | Size in bytes of chunks for upload |
|
1961
2091
|
|
1962
2092
|
Example:
|
1963
2093
|
|
1964
|
-
```
|
2094
|
+
```bash
|
1965
2095
|
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1966
2096
|
```
|
1967
2097
|
|
1968
|
-
> The gateway only supports transfers authorized with a token.
|
2098
|
+
> **Note:** The gateway only supports transfers authorized with a token.
|
1969
2099
|
|
1970
2100
|
#### <a id="agt_trsdk"></a>Transfer SDK
|
1971
2101
|
|
1972
|
-
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
2102
|
+
Another possibility is to use the Transfer SDK daemon (`asperatransferd`).
|
1973
2103
|
|
1974
2104
|
By default it will listen on local port `55002` on `127.0.0.1`.
|
1975
2105
|
|
@@ -1981,11 +2111,12 @@ gem install grpc
|
|
1981
2111
|
```
|
1982
2112
|
|
1983
2113
|
On Windows the compilation may fail for various reasons (3.1.1):
|
1984
|
-
|
2114
|
+
<!-- spellchecker: disable -->
|
1985
2115
|
- `cannot find -lx64-ucrt-ruby310`
|
1986
2116
|
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
1987
2117
|
- `conflicting types for 'gettimeofday'`
|
1988
2118
|
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)` ,i.e. change `struct timezone` to `void`
|
2119
|
+
<!-- spellchecker: enable -->
|
1989
2120
|
|
1990
2121
|
### <a id="transferspec"></a>Transfer Specification
|
1991
2122
|
|
@@ -2002,7 +2133,9 @@ is described in a [*transfer-spec*](#transferspec) (Transfer Specification), suc
|
|
2002
2133
|
|
2003
2134
|
If needed, it is possible to modify or add any of the supported [*transfer-spec*](#transferspec) parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several [*transfer-spec*](#transferspec) parameters. Multiple `ts` options on command line are cumulative.
|
2004
2135
|
|
2005
|
-
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using
|
2136
|
+
It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agt_direct) using `transfer_info` option parameter: `ascp_args`.
|
2137
|
+
Example: `--transfer-info=@json:'{"ascp_args":["-l","100m"]}'`.
|
2138
|
+
This is especially useful for `ascp` command line parameters not supported in the transfer spec.
|
2006
2139
|
|
2007
2140
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
2008
2141
|
|
@@ -2018,13 +2151,13 @@ All standard [*transfer-spec*](#transferspec) parameters can be specified.
|
|
2018
2151
|
|
2019
2152
|
References:
|
2020
2153
|
|
2021
|
-
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr
|
2022
|
-
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
2154
|
+
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22) → /opt/transfers
|
2155
|
+
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22) → Guides → API Ref → Transfer Spec V1
|
2023
2156
|
- [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
2024
2157
|
|
2025
2158
|
Parameters can be displayed with commands:
|
2026
2159
|
|
2027
|
-
```
|
2160
|
+
```bash
|
2028
2161
|
ascli config ascp spec
|
2029
2162
|
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
2030
2163
|
```
|
@@ -2039,91 +2172,91 @@ Columns:
|
|
2039
2172
|
|
2040
2173
|
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in `ascli`).
|
2041
2174
|
|
2042
|
-
|
2043
|
-
|
2044
|
-
|
2045
|
-
|
2046
|
-
|
2047
|
-
|
2048
|
-
|
2049
|
-
|
2050
|
-
|
2051
|
-
|
2052
|
-
|
2053
|
-
|
2054
|
-
|
2055
|
-
|
2056
|
-
|
2057
|
-
|
2058
|
-
|
2059
|
-
|
2060
|
-
|
2061
|
-
|
2062
|
-
|
2063
|
-
|
2064
|
-
|
2065
|
-
|
2066
|
-
|
2067
|
-
|
2068
|
-
|
2069
|
-
|
2070
|
-
|
2071
|
-
|
2072
|
-
<
|
2073
|
-
|
2074
|
-
|
2075
|
-
|
2076
|
-
|
2077
|
-
|
2078
|
-
|
2079
|
-
|
2080
|
-
|
2081
|
-
|
2082
|
-
|
2083
|
-
|
2084
|
-
|
2085
|
-
|
2086
|
-
|
2087
|
-
|
2088
|
-
|
2089
|
-
|
2090
|
-
<
|
2091
|
-
|
2092
|
-
|
2093
|
-
|
2094
|
-
|
2095
|
-
|
2096
|
-
|
2097
|
-
|
2098
|
-
|
2099
|
-
|
2100
|
-
|
2101
|
-
|
2102
|
-
|
2103
|
-
|
2104
|
-
|
2105
|
-
|
2106
|
-
|
2107
|
-
|
2108
|
-
|
2109
|
-
|
2110
|
-
|
2111
|
-
|
2112
|
-
|
2113
|
-
|
2114
|
-
|
2115
|
-
|
2116
|
-
|
2117
|
-
|
2118
|
-
|
2119
|
-
|
2120
|
-
|
2121
|
-
|
2122
|
-
|
2123
|
-
|
2124
|
-
|
2125
|
-
|
2126
|
-
|
2175
|
+
| Field | Type | D | N | C | Description |
|
2176
|
+
|-------|------|---|---|---|-------------|
|
2177
|
+
| EX_ascp_args | array | Y | | | Add native command line arguments to ascp |
|
2178
|
+
| EX_at_rest_password | string | Y | | | DEPRECATED: Prefer to use standard parameter: content_protection_password<br/>(env:ASPERA_SCP_FILEPASS) |
|
2179
|
+
| EX_file_list | string | Y | | | source file list |
|
2180
|
+
| EX_file_pair_list | string | Y | | | source file pair list |
|
2181
|
+
| EX_http_proxy_url | string | Y | | | Specify the proxy server address used by HTTP Fallback<br/>(-x {string}) |
|
2182
|
+
| EX_http_transfer_jpeg | int | Y | | | HTTP transfers as JPEG file<br/>(-j {int}) |
|
2183
|
+
| EX_license_text | string | Y | | | License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE) |
|
2184
|
+
| EX_no_read | bool | Y | | | no read source<br/>(--no-read) |
|
2185
|
+
| EX_no_write | bool | Y | | | no write on destination<br/>(--no-write) |
|
2186
|
+
| EX_proxy_password | string | Y | | | Password used for Aspera proxy server authentication.<br/>May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS) |
|
2187
|
+
| EX_ssh_key_paths | array | Y | | | Use public key authentication for SSH and specify the private key file paths<br/>(-i {array}) |
|
2188
|
+
| apply_local_docroot | bool | Y | | | (--apply-local-docroot) |
|
2189
|
+
| authentication | string | | | Y | value=token for SSH bypass keys, else password asked if not provided. |
|
2190
|
+
| cipher | string | Y | Y | Y | In transit encryption type.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum}) |
|
2191
|
+
| cipher_allowed | string | Y | Y | Y | returned by node API. Valid literals include "aes-128" and "none". |
|
2192
|
+
| content_protection | string | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt {enum}) |
|
2193
|
+
| content_protection_password | string | Y | Y | Y | Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS) |
|
2194
|
+
| cookie | string | Y | Y | Y | Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE) |
|
2195
|
+
| create_dir | bool | Y | Y | Y | Specifies whether to create new directories.<br/>(-d) |
|
2196
|
+
| delete_before_transfer | bool | Y | Y | Y | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects<br/>on the source are not deleted.<br/>(--delete-before-transfer) |
|
2197
|
+
| delete_source | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
2198
|
+
| destination_root | string | Y | Y | Y | Destination root directory. |
|
2199
|
+
| dgram_size | int | Y | Y | Y | UDP datagram size in bytes<br/>(-Z {int}) |
|
2200
|
+
| direction | string | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode (conversion){enum}) |
|
2201
|
+
| exclude_newer_than | int | Y | | | skip src files with mtime > arg<br/>(--exclude-newer-than {int}) |
|
2202
|
+
| exclude_older_than | int | Y | | | skip src files with mtime < arg<br/>(--exclude-older-than {int}) |
|
2203
|
+
| fasp_port | int | Y | Y | Y | Specifies fasp (UDP) port.<br/>(-O {int}) |
|
2204
|
+
| file_checksum | string | Y | Y | | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none |
|
2205
|
+
| http_fallback | bool<br/>string | Y | Y | Y | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){bool}|{string}) |
|
2206
|
+
| http_fallback_port | int | Y | | | Specifies http port when no cipher is used<br/>(-t {int}) |
|
2207
|
+
| https_fallback_port | int | Y | Y | Y | Specifies https port when cipher is used<br/>(-t {int}) |
|
2208
|
+
| lock_min_rate | bool | Y | Y | Y | |
|
2209
|
+
| lock_min_rate_kbps | bool | Y | Y | Y | |
|
2210
|
+
| lock_rate_policy | bool | Y | Y | Y | |
|
2211
|
+
| lock_target_rate | bool | Y | Y | Y | |
|
2212
|
+
| lock_target_rate_kbps | bool | Y | Y | Y | |
|
2213
|
+
| min_rate_cap_kbps | int | Y | Y | Y | |
|
2214
|
+
| min_rate_kbps | int | Y | Y | Y | Set the minimum transfer rate in kilobits per second.<br/>(-m {int}) |
|
2215
|
+
| move_after_transfer | string | Y | Y | | The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer {string}) |
|
2216
|
+
| multi_session | int | Y | Y | Y | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/> |
|
2217
|
+
| multi_session_threshold | int | Y | Y | | Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold {int}) |
|
2218
|
+
| overwrite | string | Y | Y | Y | Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite {enum}) |
|
2219
|
+
| password | string | | Y | | Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only. |
|
2220
|
+
| paths | array | Y | Y | Y | Array of path to the source (required) and a path to the destination (optional). |
|
2221
|
+
| precalculate_job_size | bool | Y | Y | Y | Specifies whether to precalculate the job size.<br/>(--precalculate-job-size) |
|
2222
|
+
| preserve_access_time | bool | Y | Y | Y | (--preserve-access-time) |
|
2223
|
+
| preserve_acls | string | Y | | | Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls {enum}) |
|
2224
|
+
| preserve_creation_time | bool | Y | Y | Y | (--preserve-creation-time) |
|
2225
|
+
| preserve_file_owner_gid | bool | Y | | | Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid) |
|
2226
|
+
| preserve_file_owner_uid | bool | Y | | | Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid) |
|
2227
|
+
| preserve_modification_time | bool | Y | Y | Y | (--preserve-modification-time) |
|
2228
|
+
| preserve_remote_acls | string | Y | | | Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls {enum}) |
|
2229
|
+
| preserve_source_access_time | bool | Y | | | Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time) |
|
2230
|
+
| preserve_times | bool | Y | Y | Y | (--preserve-times) |
|
2231
|
+
| proxy | string | Y | | | Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy {string}) |
|
2232
|
+
| rate_policy | string | Y | Y | Y | The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy {enum}) |
|
2233
|
+
| rate_policy_allowed | string | | | Y | Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed |
|
2234
|
+
| remote_host | string | Y | Y | Y | IP or fully qualified domain name of the remote server<br/>(--host {string}) |
|
2235
|
+
| remote_password | string | Y | Y | Y | SSH session password<br/>(env:ASPERA_SCP_PASS) |
|
2236
|
+
| remote_user | string | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user {string}) |
|
2237
|
+
| remove_after_transfer | bool | Y | Y | | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
2238
|
+
| remove_empty_directories | bool | Y | Y | | Specifies whether to remove empty directories.<br/>(--remove-empty-directories) |
|
2239
|
+
| remove_empty_source_directory | bool | Y | | | Remove empty source subdirectories and remove the source directory itself, if empty<br/>(--remove-empty-source-directory) |
|
2240
|
+
| remove_skipped | bool | Y | Y | Y | Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped) |
|
2241
|
+
| resume_policy | string | Y | Y | Y | If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum}) |
|
2242
|
+
| retry_duration | string<br/>int | | Y | Y | Specifies how long to wait before retrying transfer. (e.g. "5min") |
|
2243
|
+
| source_root | string | Y | Y | Y | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64 (conversion){string}) |
|
2244
|
+
| source_root_id | string | | Y | | The file ID of the source root directory. Required when using Bearer token auth for the source node. |
|
2245
|
+
| src_base | string | Y | Y | | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64 (conversion){string}) |
|
2246
|
+
| ssh_port | int | Y | Y | Y | Specifies SSH (TCP) port. Default: local:22, other:33001<br/>(-P {int}) |
|
2247
|
+
| ssh_private_key | string | Y | | | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----/nMII...<br/>Note the JSON encoding: /n for newlines.<br/>(env:ASPERA_SCP_KEY) |
|
2248
|
+
| ssh_private_key_passphrase | string | Y | | | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS) |
|
2249
|
+
| sshfp | string | Y | Y | Y | Check it against server SSH host key fingerprint<br/>(--check-sshfp {string}) |
|
2250
|
+
| symlink_policy | string | Y | Y | Y | Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links {enum}) |
|
2251
|
+
| tags | hash | Y | Y | Y | Metadata for transfer as JSON<br/>(--tags64 (conversion){hash}) |
|
2252
|
+
| target_rate_cap_kbps | int | | | Y | Returned by upload/download_setup node API. |
|
2253
|
+
| target_rate_kbps | int | Y | Y | Y | Specifies desired speed for the transfer.<br/>(-l {int}) |
|
2254
|
+
| target_rate_percentage | string | Y | Y | Y | |
|
2255
|
+
| title | string | | Y | Y | Title of the transfer |
|
2256
|
+
| token | string | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
2257
|
+
| use_ascp4 | bool | Y | Y | | specify version of protocol |
|
2258
|
+
| wss_enabled | bool | Y | Y | Y | Server has Web Socket service enabled |
|
2259
|
+
| wss_port | int | Y | Y | Y | TCP port used for websocket service feed |
|
2127
2260
|
|
2128
2261
|
#### Destination folder for transfers
|
2129
2262
|
|
@@ -2167,7 +2300,7 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2167
2300
|
|
2168
2301
|
- an [Extended Value](#extended) with type **Array of String**
|
2169
2302
|
|
2170
|
-
> Note
|
2303
|
+
> **Note:** extended values can be tested with the command `conf echo`
|
2171
2304
|
|
2172
2305
|
Examples:
|
2173
2306
|
|
@@ -2188,7 +2321,7 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2188
2321
|
|
2189
2322
|
- Using JSON array
|
2190
2323
|
|
2191
|
-
```
|
2324
|
+
```bash
|
2192
2325
|
--sources=@json:'["file1","file2"]'
|
2193
2326
|
```
|
2194
2327
|
|
@@ -2209,13 +2342,13 @@ Examples:
|
|
2209
2342
|
|
2210
2343
|
- Using transfer spec
|
2211
2344
|
|
2212
|
-
```
|
2345
|
+
```bash
|
2213
2346
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
2214
2347
|
```
|
2215
2348
|
|
2216
2349
|
The option `src_type` allows specifying if the list specified in option `sources` is a simple file list or if it is a file pair list.
|
2217
2350
|
|
2218
|
-
> Note
|
2351
|
+
> **Note:** Option `src_type` is not used if option `sources` is set to `@ts`
|
2219
2352
|
|
2220
2353
|
Supported values for `src_type` are:
|
2221
2354
|
|
@@ -2228,15 +2361,15 @@ Example: Source file `200KB.1` is renamed `sample1` on destination:
|
|
2228
2361
|
ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
2229
2362
|
```
|
2230
2363
|
|
2231
|
-
> Note
|
2364
|
+
> **Note:** There are some specific rules to specify a file list when using **Aspera on Cloud**, refer to the AoC plugin section.
|
2232
2365
|
|
2233
2366
|
#### <a id="multisession"></a>Support of multi-session
|
2234
2367
|
|
2235
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
2368
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one `ascp` process per session) is supported on "direct" and "node" agents, not yet on connect.
|
2236
2369
|
|
2237
2370
|
- when agent=node :
|
2238
2371
|
|
2239
|
-
```
|
2372
|
+
```bash
|
2240
2373
|
--ts=@json:'{"multi_session":10,"multi_session_threshold":1}'
|
2241
2374
|
```
|
2242
2375
|
|
@@ -2244,7 +2377,7 @@ Multi-session is directly supported by the node daemon.
|
|
2244
2377
|
|
2245
2378
|
- when agent=direct :
|
2246
2379
|
|
2247
|
-
```
|
2380
|
+
```bash
|
2248
2381
|
--ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'
|
2249
2382
|
```
|
2250
2383
|
|
@@ -2256,7 +2389,7 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
2256
2389
|
|
2257
2390
|
#### Content protection
|
2258
2391
|
|
2259
|
-
Also known as Client-side encryption at
|
2392
|
+
Also known as Client-side encryption at rest (CSEAR), content protection allows a client to send files to a server
|
2260
2393
|
which will store them encrypted (upload), and decrypt files as they are being downloaded from a server, both
|
2261
2394
|
using a passphrase, only known by users sharing files. Files stay encrypted on server side.
|
2262
2395
|
|
@@ -2267,45 +2400,43 @@ activating CSEAR consists in using transfer spec parameters:
|
|
2267
2400
|
|
2268
2401
|
Example: parameter to download a faspex package and decrypt on the fly
|
2269
2402
|
|
2270
|
-
```
|
2271
|
-
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"
|
2272
|
-
```
|
2273
|
-
|
2274
|
-
Note that up to version 4.6.0, the following parameters should be used for agent `direct`:
|
2275
|
-
|
2276
|
-
```javascript
|
2277
|
-
--ts=@json:'{"EX_ascp_args":["--file-crypt=decrypt"],"EX_at_rest_password":"_secret_here_"}'
|
2403
|
+
```bash
|
2404
|
+
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"my_password_here"}'
|
2278
2405
|
```
|
2279
2406
|
|
2280
2407
|
#### Transfer Spec Examples
|
2281
2408
|
|
2282
2409
|
- Change target rate
|
2283
2410
|
|
2284
|
-
```
|
2411
|
+
```bash
|
2285
2412
|
--ts=@json:'{"target_rate_kbps":500000}'
|
2286
2413
|
```
|
2287
2414
|
|
2288
2415
|
- Override the FASP SSH port to a specific TCP port:
|
2289
2416
|
|
2290
|
-
```
|
2417
|
+
```bash
|
2291
2418
|
--ts=@json:'{"ssh_port":33002}'
|
2292
2419
|
```
|
2293
2420
|
|
2294
2421
|
- Force http fallback mode:
|
2295
2422
|
|
2296
|
-
```
|
2423
|
+
```bash
|
2297
2424
|
--ts=@json:'{"http_fallback":"force"}'
|
2298
2425
|
```
|
2299
2426
|
|
2300
2427
|
- Activate progress when not activated by default on server
|
2301
2428
|
|
2302
|
-
```
|
2429
|
+
```bash
|
2303
2430
|
--ts=@json:'{"precalculate_job_size":true}'
|
2304
2431
|
```
|
2305
2432
|
|
2306
|
-
### <a id="scheduling"></a>
|
2433
|
+
### <a id="scheduling"></a>Scheduling
|
2434
|
+
|
2435
|
+
It is useful to configure automated scheduled execution.
|
2436
|
+
|
2437
|
+
#### <a id="locking"></a>Locking for exclusive execution
|
2307
2438
|
|
2308
|
-
|
2439
|
+
It is also useful to ensure that `ascli` is not executed several times in parallel.
|
2309
2440
|
|
2310
2441
|
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
2311
2442
|
|
@@ -2339,6 +2470,18 @@ The first instance will sleep 30 seconds, the second one will immediately exit l
|
|
2339
2470
|
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
2340
2471
|
```
|
2341
2472
|
|
2473
|
+
#### <a id="scheduler"></a>Scheduler
|
2474
|
+
|
2475
|
+
`ascli` does not provide an internal scheduler.
|
2476
|
+
|
2477
|
+
Instead, use the service provided by the Operating system:
|
2478
|
+
|
2479
|
+
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2480
|
+
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2481
|
+
- etc...
|
2482
|
+
|
2483
|
+
Linux also provides `anacron`, if tasks are hourly or daily.
|
2484
|
+
|
2342
2485
|
### "Provençale"
|
2343
2486
|
|
2344
2487
|
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
@@ -2396,7 +2539,7 @@ where:
|
|
2396
2539
|
- `filename` is the name that will be assigned to the file on the destination
|
2397
2540
|
- `filesize` is the number of bytes that will be sent (in decimal).
|
2398
2541
|
|
2399
|
-
Note: characters `?` and `&` are shell special characters (wildcard and
|
2542
|
+
Note: characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
2400
2543
|
|
2401
2544
|
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
2402
2545
|
|
@@ -2431,7 +2574,7 @@ The sequence parameter is applied as follows:
|
|
2431
2574
|
|
2432
2575
|
- If `seq` is `sequential` then each file size is:
|
2433
2576
|
|
2434
|
-
- `size + ((
|
2577
|
+
- `size + ((file_index - 1) * inc)`
|
2435
2578
|
- Where first file is index 1
|
2436
2579
|
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
2437
2580
|
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
@@ -2465,7 +2608,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
2465
2608
|
```text
|
2466
2609
|
ascli -h
|
2467
2610
|
NAME
|
2468
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
2611
|
+
ascli -- a command line tool for Aspera Applications (v4.12.0)
|
2469
2612
|
|
2470
2613
|
SYNOPSIS
|
2471
2614
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -2513,9 +2656,8 @@ OPTIONS: global
|
|
2513
2656
|
-w, --warnings check for language warnings
|
2514
2657
|
--ui=ENUM method to start browser: text, [graphical]
|
2515
2658
|
--log-level=ENUM Log level: debug, info, [warn], error, fatal, unknown
|
2516
|
-
--logger=ENUM
|
2659
|
+
--logger=ENUM logging method: [stderr], stdout, syslog
|
2517
2660
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
2518
|
-
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2519
2661
|
--http-options=VALUE options for http socket (extended value)
|
2520
2662
|
--insecure=ENUM do not validate HTTPS certificate: no, [yes]
|
2521
2663
|
--once-only=ENUM process only new items (some commands): [no], yes
|
@@ -2523,8 +2665,9 @@ OPTIONS: global
|
|
2523
2665
|
--cache-tokens=ENUM save and reuse Oauth tokens: no, [yes]
|
2524
2666
|
|
2525
2667
|
COMMAND: config
|
2526
|
-
SUBCOMMANDS:
|
2668
|
+
SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test export_to_cli file flush_tokens folder gem genkey id initdemo list lookup open overview plugin preset proxy_check secure smtp_settings vault wizard
|
2527
2669
|
OPTIONS:
|
2670
|
+
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2528
2671
|
--value=VALUE extended value for create, update, list filter
|
2529
2672
|
--property=VALUE name of property to set
|
2530
2673
|
--id=VALUE resource identifier (modify,delete,show)
|
@@ -2552,26 +2695,26 @@ OPTIONS:
|
|
2552
2695
|
--notif-template=VALUE Email ERB template for notification of transfers
|
2553
2696
|
--version-check-days=VALUE Period in days to check new version (zero to disable)
|
2554
2697
|
--plugin-folder=VALUE Folder where to find additional plugins
|
2555
|
-
--ts=VALUE
|
2556
|
-
--
|
2557
|
-
--
|
2558
|
-
--
|
2559
|
-
--
|
2560
|
-
--transfer=
|
2561
|
-
--
|
2562
|
-
--progress=ENUM type of progress bar: none, native, multi
|
2698
|
+
--ts=VALUE Override transfer spec values (Hash, e.g. use @json: prefix), current={"create_dir"=>true}
|
2699
|
+
--to-folder=VALUE Destination folder for transferred files
|
2700
|
+
--sources=VALUE How list of transferred files is provided (@args,@ts,Array)
|
2701
|
+
--src-type=ENUM Type of file list: list, pair
|
2702
|
+
--transfer=ENUM Type of transfer agent: direct, node, connect, httpgw, trsdk
|
2703
|
+
--transfer-info=VALUE Parameters for transfer agent
|
2704
|
+
--progress=ENUM Type of progress bar: none, native, multi
|
2563
2705
|
|
2564
2706
|
|
2565
2707
|
COMMAND: shares
|
2566
|
-
SUBCOMMANDS: health repository
|
2708
|
+
SUBCOMMANDS: admin health repository
|
2567
2709
|
OPTIONS:
|
2568
2710
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2569
2711
|
--username=VALUE username to log in
|
2570
2712
|
--password=VALUE user's password
|
2713
|
+
--type=ENUM Type of user/group for operations: any, local, ldap, saml
|
2571
2714
|
|
2572
2715
|
|
2573
2716
|
COMMAND: node
|
2574
|
-
SUBCOMMANDS:
|
2717
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
2575
2718
|
OPTIONS:
|
2576
2719
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2577
2720
|
--username=VALUE username to log in
|
@@ -2579,11 +2722,13 @@ OPTIONS:
|
|
2579
2722
|
--validator=VALUE identifier of validator (optional for central)
|
2580
2723
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
2581
2724
|
--sync-name=VALUE sync name
|
2725
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
2582
2726
|
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
2727
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
2583
2728
|
|
2584
2729
|
|
2585
2730
|
COMMAND: orchestrator
|
2586
|
-
SUBCOMMANDS: health info
|
2731
|
+
SUBCOMMANDS: health info plugins processes workflow
|
2587
2732
|
OPTIONS:
|
2588
2733
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2589
2734
|
--username=VALUE username to log in
|
@@ -2612,7 +2757,7 @@ OPTIONS:
|
|
2612
2757
|
|
2613
2758
|
|
2614
2759
|
COMMAND: ats
|
2615
|
-
SUBCOMMANDS:
|
2760
|
+
SUBCOMMANDS: access_key api_key aws_trust_policy cluster
|
2616
2761
|
OPTIONS:
|
2617
2762
|
--ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
|
2618
2763
|
--instance=VALUE ATS instance in ibm cloud
|
@@ -2624,7 +2769,7 @@ OPTIONS:
|
|
2624
2769
|
|
2625
2770
|
|
2626
2771
|
COMMAND: faspex5
|
2627
|
-
SUBCOMMANDS:
|
2772
|
+
SUBCOMMANDS: admin bearer_token gateway health package postprocessing shared_folders user version
|
2628
2773
|
OPTIONS:
|
2629
2774
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2630
2775
|
--username=VALUE username to log in
|
@@ -2632,9 +2777,10 @@ OPTIONS:
|
|
2632
2777
|
--client-id=VALUE OAuth client identifier
|
2633
2778
|
--client-secret=VALUE OAuth client secret
|
2634
2779
|
--redirect-uri=VALUE OAuth redirect URI for web authentication
|
2635
|
-
--auth=ENUM OAuth type of authentication: web, jwt
|
2780
|
+
--auth=ENUM OAuth type of authentication: boot, web, jwt
|
2636
2781
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2637
2782
|
--passphrase=VALUE RSA private key passphrase
|
2783
|
+
--shared-folder=VALUE Shared folder source for package files
|
2638
2784
|
|
2639
2785
|
|
2640
2786
|
COMMAND: cos
|
@@ -2643,14 +2789,14 @@ OPTIONS:
|
|
2643
2789
|
--bucket=VALUE Bucket name
|
2644
2790
|
--endpoint=VALUE Storage endpoint url
|
2645
2791
|
--apikey=VALUE Storage API key
|
2646
|
-
--crn=VALUE
|
2792
|
+
--crn=VALUE Resource instance id
|
2647
2793
|
--service-credentials=VALUE IBM Cloud service credentials (Hash)
|
2648
2794
|
--region=VALUE Storage region
|
2649
2795
|
--identity=VALUE Authentication url (https://iam.cloud.ibm.com/identity)
|
2650
2796
|
|
2651
2797
|
|
2652
2798
|
COMMAND: faspex
|
2653
|
-
SUBCOMMANDS: health package source
|
2799
|
+
SUBCOMMANDS: address_book dropbox health login_methods me package source v4
|
2654
2800
|
OPTIONS:
|
2655
2801
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2656
2802
|
--username=VALUE username to log in
|
@@ -2664,7 +2810,7 @@ OPTIONS:
|
|
2664
2810
|
|
2665
2811
|
|
2666
2812
|
COMMAND: preview
|
2667
|
-
SUBCOMMANDS:
|
2813
|
+
SUBCOMMANDS: check events scan test trevents
|
2668
2814
|
OPTIONS:
|
2669
2815
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2670
2816
|
--username=VALUE username to log in
|
@@ -2677,7 +2823,7 @@ OPTIONS:
|
|
2677
2823
|
--skip-folders=VALUE list of folder to skip
|
2678
2824
|
--case=VALUE basename of output for for test
|
2679
2825
|
--scan-path=VALUE subpath in folder id to start scan in (default=/)
|
2680
|
-
--scan-id=VALUE
|
2826
|
+
--scan-id=VALUE folder id in storage to start scan in, default is access key main folder id
|
2681
2827
|
--mimemagic=ENUM use Mime type detection of gem mimemagic: [no], yes
|
2682
2828
|
--overwrite=ENUM when to overwrite result file: always, never, [mtime]
|
2683
2829
|
--file-access=ENUM how to read and write files in repository: [local], remote
|
@@ -2699,38 +2845,46 @@ OPTIONS:
|
|
2699
2845
|
|
2700
2846
|
|
2701
2847
|
COMMAND: sync
|
2702
|
-
SUBCOMMANDS: start
|
2848
|
+
SUBCOMMANDS: admin start
|
2703
2849
|
OPTIONS:
|
2704
|
-
--
|
2705
|
-
--session
|
2850
|
+
--sync-info=VALUE Information for sync instance and sessions (Hash)
|
2851
|
+
--sync-session=VALUE Name of session to use for admin commands. default: first in parameters
|
2706
2852
|
|
2707
2853
|
|
2708
2854
|
COMMAND: aoc
|
2709
|
-
SUBCOMMANDS:
|
2855
|
+
SUBCOMMANDS: admin automation bearer_token files gateway organization packages reminder servers tier_restrictions user
|
2710
2856
|
OPTIONS:
|
2711
2857
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2712
2858
|
--username=VALUE username to log in
|
2713
2859
|
--password=VALUE user's password
|
2714
2860
|
--auth=ENUM OAuth type of authentication: web, jwt
|
2715
2861
|
--operation=ENUM client operation for transfers: push, pull
|
2716
|
-
--client-id=VALUE OAuth API client identifier
|
2717
|
-
--client-secret=VALUE OAuth API client
|
2862
|
+
--client-id=VALUE OAuth API client identifier
|
2863
|
+
--client-secret=VALUE OAuth API client secret
|
2718
2864
|
--redirect-uri=VALUE OAuth API client redirect URI
|
2719
2865
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2720
|
-
--passphrase=VALUE RSA private key passphrase
|
2721
|
-
--workspace=VALUE name of workspace
|
2722
|
-
--name=VALUE resource name
|
2723
|
-
--path=VALUE file or folder path
|
2724
|
-
--link=VALUE public link to shared resource
|
2725
|
-
--new-user-option=VALUE new user creation option for unknown package recipients
|
2726
|
-
--from-folder=VALUE share to share source folder
|
2727
2866
|
--scope=VALUE OAuth scope for AoC API calls
|
2728
|
-
--
|
2729
|
-
--
|
2867
|
+
--passphrase=VALUE RSA private key passphrase
|
2868
|
+
--workspace=VALUE Name of workspace
|
2869
|
+
--name=VALUE Resource name (prefer to use keyword name)
|
2870
|
+
--link=VALUE Public link to shared resource
|
2871
|
+
--new-user-option=VALUE New user creation option for unknown package recipients
|
2872
|
+
--from-folder=VALUE Source folder for Folder-to-Folder transfer
|
2873
|
+
--validate-metadata=ENUM Validate shared inbox metadata: [no], yes
|
2874
|
+
|
2875
|
+
COMMAND: node
|
2876
|
+
SUBCOMMANDS: access_key api_details asperabrowser async basic_token browse central delete download events health info license mkdir mkfile mklink rename search service space stream sync transfer upload watch_folder
|
2877
|
+
OPTIONS:
|
2878
|
+
--validator=VALUE identifier of validator (optional for central)
|
2879
|
+
--asperabrowserurl=VALUE URL for simple aspera web ui
|
2880
|
+
--sync-name=VALUE sync name
|
2881
|
+
--path=VALUE file or folder path for gen4 operation "file"
|
2882
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
2883
|
+
--default-ports=ENUM use standard FASP ports or get from node api (gen4): [no], yes
|
2730
2884
|
|
2731
2885
|
|
2732
2886
|
COMMAND: server
|
2733
|
-
SUBCOMMANDS:
|
2887
|
+
SUBCOMMANDS: browse cp delete df download du health info ls md5sum mkdir mv rename rm sync upload
|
2734
2888
|
OPTIONS:
|
2735
2889
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2736
2890
|
--username=VALUE username to log in
|
@@ -2740,7 +2894,7 @@ OPTIONS:
|
|
2740
2894
|
|
2741
2895
|
|
2742
2896
|
COMMAND: console
|
2743
|
-
SUBCOMMANDS: transfer
|
2897
|
+
SUBCOMMANDS: health transfer
|
2744
2898
|
OPTIONS:
|
2745
2899
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2746
2900
|
--username=VALUE username to log in
|
@@ -2751,7 +2905,7 @@ OPTIONS:
|
|
2751
2905
|
|
2752
2906
|
```
|
2753
2907
|
|
2754
|
-
Note
|
2908
|
+
> **Note:** commands and parameter values can be written in short form.
|
2755
2909
|
|
2756
2910
|
### Bulk creation and deletion of resources
|
2757
2911
|
|
@@ -2759,7 +2913,7 @@ Bulk creation and deletion of resources are possible using option `bulk` (yes,no
|
|
2759
2913
|
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2760
2914
|
This option is available only for some of the resources: if you need it: try and see if the entities you try to create or delete support this option.
|
2761
2915
|
|
2762
|
-
## <a id="aoc"></a>Plugin: Aspera on Cloud
|
2916
|
+
## <a id="aoc"></a>Plugin: `aoc`: IBM Aspera on Cloud
|
2763
2917
|
|
2764
2918
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
2765
2919
|
|
@@ -2802,7 +2956,7 @@ ascli config wizard --value=aoc
|
|
2802
2956
|
|
2803
2957
|
### <a id="aocmanual"></a>Configuration: using manual setup
|
2804
2958
|
|
2805
|
-
> If you used the wizard (recommended): skip this section.
|
2959
|
+
> **Note:** If you used the wizard (recommended): skip this section.
|
2806
2960
|
|
2807
2961
|
#### Configuration details
|
2808
2962
|
|
@@ -2826,12 +2980,12 @@ If you use the built-in client_id and client_secret, skip this and do not set th
|
|
2826
2980
|
|
2827
2981
|
Else you can use a specific OAuth API client_id, the first step is to declare `ascli` in Aspera on Cloud using the admin interface.
|
2828
2982
|
|
2829
|
-
(
|
2983
|
+
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client) ).
|
2830
2984
|
|
2831
2985
|
Let's start by a registration with web based authentication (auth=web):
|
2832
2986
|
|
2833
2987
|
- Open a web browser, log to your instance: e.g. `https://myorg.ibmaspera.com/`
|
2834
|
-
- Go to Apps→Admin→Organization→Integrations
|
2988
|
+
- Go to Apps → Admin → Organization → Integrations
|
2835
2989
|
- Click "Create New"
|
2836
2990
|
- Client Name: `ascli`
|
2837
2991
|
- Redirect URIs: `http://localhost:12345`
|
@@ -2850,13 +3004,15 @@ If you did not use the wizard, you can also manually create a [option preset](#l
|
|
2850
3004
|
|
2851
3005
|
Lets create an [option preset](#lprt) called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
2852
3006
|
|
3007
|
+
<!-- spellchecker: disable -->
|
2853
3008
|
```bash
|
2854
3009
|
ascli config preset ask my_aoc_org url client_id client_secret
|
2855
3010
|
option: url> https://myorg.ibmaspera.com/
|
2856
|
-
option: client_id>
|
2857
|
-
option: client_secret>
|
3011
|
+
option: client_id> my_client_id_here
|
3012
|
+
option: client_secret> my_client_secret_here
|
2858
3013
|
updated: my_aoc_org
|
2859
3014
|
```
|
3015
|
+
<!-- spellchecker: enable -->
|
2860
3016
|
|
2861
3017
|
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
2862
3018
|
|
@@ -2882,7 +3038,7 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
2882
3038
|
- Graphically
|
2883
3039
|
|
2884
3040
|
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
2885
|
-
- Go to Apps→Admin→Organization→Integrations
|
3041
|
+
- Go to Apps → Admin → Organization → Integrations
|
2886
3042
|
- Click on the previously created application
|
2887
3043
|
- select tab : "JSON Web Token Auth"
|
2888
3044
|
- Modify options if necessary, for instance: activate both options in section "Settings"
|
@@ -2902,7 +3058,7 @@ ascli aoc admin res client list
|
|
2902
3058
|
:............:...............:
|
2903
3059
|
```
|
2904
3060
|
|
2905
|
-
```
|
3061
|
+
```bash
|
2906
3062
|
ascli aoc admin res client modify my_BJbQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
2907
3063
|
```
|
2908
3064
|
|
@@ -3037,25 +3193,25 @@ Examples:
|
|
3037
3193
|
|
3038
3194
|
- List users with `laurent` in name:
|
3039
3195
|
|
3040
|
-
```
|
3196
|
+
```bash
|
3041
3197
|
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
3042
3198
|
```
|
3043
3199
|
|
3044
3200
|
- List users who logged-in before a date:
|
3045
3201
|
|
3046
|
-
```
|
3202
|
+
```bash
|
3047
3203
|
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
3048
3204
|
```
|
3049
3205
|
|
3050
3206
|
- List external users and sort in reverse alphabetical order using name:
|
3051
3207
|
|
3052
|
-
```
|
3208
|
+
```bash
|
3053
3209
|
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
3054
3210
|
```
|
3055
3211
|
|
3056
3212
|
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
3057
3213
|
|
3058
|
-
Note
|
3214
|
+
> **Note:** The option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
3059
3215
|
|
3060
3216
|
#### <a id="res_select"></a>Selecting a resource
|
3061
3217
|
|
@@ -3111,7 +3267,7 @@ If the command returns an error, example:
|
|
3111
3267
|
|
3112
3268
|
Well, remove the offending parameters and try again.
|
3113
3269
|
|
3114
|
-
Note
|
3270
|
+
> **Note:** Some properties that are shown in the web UI, such as membership, are not listed directly in the resource, but instead another resource is created to link a user and its group: `group_membership`
|
3115
3271
|
|
3116
3272
|
#### Access Key secrets
|
3117
3273
|
|
@@ -3120,7 +3276,7 @@ The secret is provided using the `secret` option.
|
|
3120
3276
|
For example in a command like:
|
3121
3277
|
|
3122
3278
|
```bash
|
3123
|
-
ascli aoc admin res node --id=123 --secret="
|
3279
|
+
ascli aoc admin res node --id=123 --secret="my_secret_here" v3 info
|
3124
3280
|
```
|
3125
3281
|
|
3126
3282
|
It is also possible to store secrets in the [secret vault](#vault) and then automatically find the related secret using the [config finder](#config_finder).
|
@@ -3155,7 +3311,7 @@ The environment provided contains the following additional variable:
|
|
3155
3311
|
|
3156
3312
|
Example:
|
3157
3313
|
|
3158
|
-
```
|
3314
|
+
```bash
|
3159
3315
|
ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
3160
3316
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
3161
3317
|
--notif-to=active --notif-template=@file:mytemplate.erb
|
@@ -3167,7 +3323,7 @@ Options:
|
|
3167
3323
|
- `query` filter (on API call)
|
3168
3324
|
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
3169
3325
|
|
3170
|
-
Note
|
3326
|
+
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
3171
3327
|
|
3172
3328
|
#### Transfer: Using specific transfer ports
|
3173
3329
|
|
@@ -3180,7 +3336,7 @@ Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `ao
|
|
3180
3336
|
|
3181
3337
|
#### Example: Bulk creation of users
|
3182
3338
|
|
3183
|
-
```
|
3339
|
+
```bash
|
3184
3340
|
ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
3185
3341
|
```
|
3186
3342
|
|
@@ -3195,7 +3351,9 @@ ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.
|
|
3195
3351
|
|
3196
3352
|
#### Example: Find with filter and delete
|
3197
3353
|
|
3198
|
-
|
3354
|
+
<!-- spellchecker: disable -->
|
3355
|
+
|
3356
|
+
```bash
|
3199
3357
|
ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
3200
3358
|
```
|
3201
3359
|
|
@@ -3216,7 +3374,7 @@ thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fiel
|
|
3216
3374
|
echo $thelist
|
3217
3375
|
```
|
3218
3376
|
|
3219
|
-
```
|
3377
|
+
```json
|
3220
3378
|
["113501","354061"]
|
3221
3379
|
```
|
3222
3380
|
|
@@ -3233,6 +3391,8 @@ ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
3233
3391
|
:.......:.........:
|
3234
3392
|
```
|
3235
3393
|
|
3394
|
+
<!-- spellchecker: enable -->
|
3395
|
+
|
3236
3396
|
#### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
3237
3397
|
|
3238
3398
|
```ruby
|
@@ -3273,11 +3433,11 @@ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q"
|
|
3273
3433
|
|
3274
3434
|
Examples of query (TODO: cleanup):
|
3275
3435
|
|
3276
|
-
```
|
3436
|
+
```json
|
3277
3437
|
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
3278
3438
|
```
|
3279
3439
|
|
3280
|
-
```
|
3440
|
+
```json
|
3281
3441
|
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
3282
3442
|
```
|
3283
3443
|
|
@@ -3289,7 +3449,7 @@ ascli aoc admin res node --secret=_secret_ v3 events
|
|
3289
3449
|
|
3290
3450
|
#### Example: Display members of a workspace
|
3291
3451
|
|
3292
|
-
```
|
3452
|
+
```bash
|
3293
3453
|
ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
3294
3454
|
```
|
3295
3455
|
|
@@ -3308,7 +3468,7 @@ ascli aoc admin res workspace_membership list --fields=member_type,manager,membe
|
|
3308
3468
|
|
3309
3469
|
Other query parameters:
|
3310
3470
|
|
3311
|
-
```
|
3471
|
+
```json
|
3312
3472
|
{"workspace_membership_through":true,"include_indirect":true}
|
3313
3473
|
```
|
3314
3474
|
|
@@ -3354,7 +3514,7 @@ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_membe
|
|
3354
3514
|
|
3355
3515
|
#### Example: Get users who did not log since a date
|
3356
3516
|
|
3357
|
-
```
|
3517
|
+
```bash
|
3358
3518
|
ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
3359
3519
|
```
|
3360
3520
|
|
@@ -3369,7 +3529,7 @@ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:
|
|
3369
3529
|
|
3370
3530
|
#### Example: List "Limited" users
|
3371
3531
|
|
3372
|
-
```
|
3532
|
+
```bash
|
3373
3533
|
ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
3374
3534
|
```
|
3375
3535
|
|
@@ -3449,13 +3609,15 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
3449
3609
|
|
3450
3610
|
Then, transfer between those:
|
3451
3611
|
|
3452
|
-
```
|
3612
|
+
```bash
|
3453
3613
|
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
3454
3614
|
```
|
3455
3615
|
|
3456
3616
|
#### Example: create registration key to register a node
|
3457
3617
|
|
3458
|
-
|
3618
|
+
<!-- spellchecker: disable -->
|
3619
|
+
|
3620
|
+
```bash
|
3459
3621
|
ascli aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
3460
3622
|
```
|
3461
3623
|
|
@@ -3480,6 +3642,8 @@ ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res cli
|
|
3480
3642
|
+-----+---------+
|
3481
3643
|
```
|
3482
3644
|
|
3645
|
+
<!-- spellchecker: enable -->
|
3646
|
+
|
3483
3647
|
#### Example: Create a Node
|
3484
3648
|
|
3485
3649
|
AoC nodes as actually composed with two related entities:
|
@@ -3489,7 +3653,8 @@ AoC nodes as actually composed with two related entities:
|
|
3489
3653
|
|
3490
3654
|
The web UI allows creation of both entities in one shot.
|
3491
3655
|
For more flexibility, `ascli` allows this in two separate steps.
|
3492
|
-
|
3656
|
+
|
3657
|
+
> **Note:** When selecting "Use existing access key" in the web UI, this actually skips access key creation (first step).
|
3493
3658
|
|
3494
3659
|
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
3495
3660
|
|
@@ -3497,13 +3662,13 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3497
3662
|
|
3498
3663
|
The creation options are the ones of ATS API, refer to the [section on ATS](#ats_params) for more details and examples.
|
3499
3664
|
|
3500
|
-
```
|
3665
|
+
```bash
|
3501
3666
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
3502
3667
|
```
|
3503
3668
|
|
3504
3669
|
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed.
|
3505
3670
|
|
3506
|
-
> Note
|
3671
|
+
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking to IBM support.
|
3507
3672
|
|
3508
3673
|
- Create the AoC node entity
|
3509
3674
|
|
@@ -3515,17 +3680,18 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3515
3680
|
|
3516
3681
|
Then use the returned address for the `url` key to actually create the AoC Node entity:
|
3517
3682
|
|
3518
|
-
```
|
3519
|
-
ascli aoc admin res node create @json:'{"name":"myname","access_key":"
|
3683
|
+
```bash
|
3684
|
+
ascli aoc admin res node create @json:'{"name":"myname","access_key":"myaccesskeyid","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
3520
3685
|
```
|
3521
3686
|
|
3522
3687
|
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
3523
3688
|
|
3524
3689
|
### List of files to transfer
|
3525
3690
|
|
3526
|
-
Source files are provided as a list with the `sources` option.
|
3691
|
+
Source files are provided as a list with the `sources` option.
|
3692
|
+
Refer to section [File list](#file_list)
|
3527
3693
|
|
3528
|
-
Note
|
3694
|
+
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file id` API).
|
3529
3695
|
|
3530
3696
|
Source files are located on "Aspera on cloud", when :
|
3531
3697
|
|
@@ -3565,46 +3731,54 @@ Notes:
|
|
3565
3731
|
|
3566
3732
|
#### Example: Send a package with one file to two users, using their email
|
3567
3733
|
|
3568
|
-
```
|
3569
|
-
ascli aoc
|
3734
|
+
```bash
|
3735
|
+
ascli aoc packages send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
3570
3736
|
```
|
3571
3737
|
|
3572
3738
|
#### Example: Send a package to a shared inbox with metadata
|
3573
3739
|
|
3574
|
-
```
|
3575
|
-
ascli aoc
|
3740
|
+
```bash
|
3741
|
+
ascli aoc packages send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox With Meta"],"metadata":{"Project Id":"123","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' ~/Documents/Samples/200KB.1
|
3576
3742
|
```
|
3577
3743
|
|
3578
3744
|
It is also possible to use identifiers and API parameters:
|
3579
3745
|
|
3580
|
-
```
|
3581
|
-
ascli aoc
|
3746
|
+
```bash
|
3747
|
+
ascli aoc packages send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":[{"type":"dropbox","id":"12345"}],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
3582
3748
|
```
|
3583
3749
|
|
3584
3750
|
#### Example: List packages in a given shared inbox
|
3585
3751
|
|
3586
3752
|
When user packages are listed, the following query is used:
|
3587
3753
|
|
3588
|
-
```
|
3754
|
+
```json
|
3589
3755
|
{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}
|
3590
3756
|
```
|
3591
3757
|
|
3592
|
-
To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier.
|
3758
|
+
To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier.
|
3759
|
+
Additional parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode).
|
3760
|
+
The current workspace is added unless specified in the query.
|
3593
3761
|
|
3594
3762
|
Using shared inbox name:
|
3595
3763
|
|
3596
|
-
```
|
3764
|
+
```bash
|
3597
3765
|
ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
3598
3766
|
```
|
3599
3767
|
|
3600
3768
|
Using shared inbox identifier: first retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
3601
3769
|
|
3602
3770
|
```bash
|
3603
|
-
|
3771
|
+
shared_box_id=$(ascli aoc packages shared_inboxes show name 'My Shared Inbox' --format=csv --display=data --fields=id --transpose-single=no)
|
3604
3772
|
```
|
3605
3773
|
|
3606
|
-
```
|
3607
|
-
ascli aoc packages list --query=@json:'{"dropbox_id":"'$
|
3774
|
+
```bash
|
3775
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shared_box_id'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
3776
|
+
```
|
3777
|
+
|
3778
|
+
#### Example: Receive all packages from a given shared inbox
|
3779
|
+
|
3780
|
+
```bash
|
3781
|
+
ascli aoc packages recv ALL --workspace=_workspace_ --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"_shared_inbox_name_","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
3608
3782
|
```
|
3609
3783
|
|
3610
3784
|
#### Example: Send a package with files from the Files app
|
@@ -3629,7 +3803,7 @@ ascli aoc files browse /src_folder
|
|
3629
3803
|
Let's send a package with the file `10M.dat` from subfolder /src_folder in a package:
|
3630
3804
|
|
3631
3805
|
```bash
|
3632
|
-
ascli aoc files node_info /src_folder --format=json --display=data | ascli aoc
|
3806
|
+
ascli aoc files node_info /src_folder --format=json --display=data | ascli aoc packages send --value=@json:'{"name":"test","recipients":["laurent.martin.aspera@fr.ibm.com"]}' 10M.dat --transfer=node --transfer-info=@json:@stdin:
|
3633
3807
|
```
|
3634
3808
|
|
3635
3809
|
#### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
@@ -3644,19 +3818,28 @@ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
3644
3818
|
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
3645
3819
|
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
3646
3820
|
|
3647
|
-
Typically, one would execute this command on a regular basis, using the method of your choice:
|
3648
|
-
|
3649
|
-
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
3650
|
-
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
3651
|
-
- etc...
|
3821
|
+
Typically, one would execute this command on a regular basis, using the method of your choice: see [Scheduler](#scheduler).
|
3652
3822
|
|
3653
3823
|
### Files
|
3654
3824
|
|
3655
|
-
|
3825
|
+
The Files application presents a **Home** folder to users in a given workspace.
|
3826
|
+
Files located here are either user's files, or shared folders.
|
3656
3827
|
|
3657
3828
|
#### Download Files
|
3658
3829
|
|
3659
|
-
|
3830
|
+
The general download command is:
|
3831
|
+
|
3832
|
+
```bash
|
3833
|
+
ascli aoc files download <source folder path> <source filename 1> ...
|
3834
|
+
```
|
3835
|
+
|
3836
|
+
I.e. the first argument is the source folder, and the following arguments are the source file names in this folder.
|
3837
|
+
|
3838
|
+
If a single file or folder is to be downloaded, then a single argument can be provided.
|
3839
|
+
|
3840
|
+
```bash
|
3841
|
+
ascli aoc files download <single file path>
|
3842
|
+
```
|
3660
3843
|
|
3661
3844
|
#### Shared folders
|
3662
3845
|
|
@@ -3756,36 +3939,35 @@ f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
|
3756
3939
|
- Find files older than 1 year on a given node and store in file list
|
3757
3940
|
|
3758
3941
|
```bash
|
3759
|
-
ascli aoc admin res node --name='my node name' --secret='
|
3942
|
+
ascli aoc admin res node --name='my node name' --secret='my_secret_here' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
3760
3943
|
```
|
3761
3944
|
|
3762
3945
|
- Delete the files, one by one
|
3763
3946
|
|
3764
3947
|
```bash
|
3765
|
-
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='
|
3948
|
+
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my_secret_here' v4 delete "$path" ;done
|
3766
3949
|
```
|
3767
3950
|
|
3768
3951
|
- Delete the files in bulk
|
3769
3952
|
|
3770
3953
|
```bash
|
3771
|
-
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='
|
3954
|
+
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my_secret_here' v3 delete @lines:@stdin:
|
3772
3955
|
```
|
3773
3956
|
|
3774
3957
|
### AoC sample commands
|
3775
3958
|
|
3776
3959
|
```bash
|
3777
|
-
aoc -N remind --username=my_aoc_user_email
|
3778
|
-
aoc -N servers
|
3779
3960
|
aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
3780
3961
|
aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
3781
|
-
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
3782
|
-
aoc admin ats access_key delete
|
3962
|
+
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"my_secret_here","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
3963
|
+
aoc admin ats access_key delete ak1ibmcloud
|
3783
3964
|
aoc admin ats access_key list --fields=name,id
|
3784
|
-
aoc admin ats access_key node
|
3965
|
+
aoc admin ats access_key node ak1ibmcloud --secret=my_secret_here browse /
|
3785
3966
|
aoc admin ats cluster clouds
|
3786
3967
|
aoc admin ats cluster list
|
3787
3968
|
aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
3788
3969
|
aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
3970
|
+
aoc admin auth_providers list
|
3789
3971
|
aoc admin res application list
|
3790
3972
|
aoc admin res client list
|
3791
3973
|
aoc admin res client_access_key list
|
@@ -3806,14 +3988,15 @@ aoc admin res self show
|
|
3806
3988
|
aoc admin res short_link list
|
3807
3989
|
aoc admin res user list
|
3808
3990
|
aoc admin res workspace_membership list
|
3809
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3810
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3811
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3812
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3813
|
-
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret
|
3814
|
-
aoc admin resource node
|
3991
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do browse /
|
3992
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do delete /folder1
|
3993
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do mkdir /folder1
|
3994
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
|
3995
|
+
aoc admin resource node --name=my_aoc_ak_name --secret=my_aoc_ak_secret do v3 events
|
3996
|
+
aoc admin resource node do name my_aoc_ak_name --secret=my_aoc_ak_secret v3 access_key delete testsub1
|
3815
3997
|
aoc admin resource workspace list
|
3816
3998
|
aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
3999
|
+
aoc admin subscription
|
3817
4000
|
aoc automation workflow action my_wf_id create --value=@json:'{"name":"toto"}'
|
3818
4001
|
aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
3819
4002
|
aoc automation workflow delete my_wf_id
|
@@ -3825,10 +4008,11 @@ aoc faspex
|
|
3825
4008
|
aoc files bearer /
|
3826
4009
|
aoc files bearer_token_node / --cache-tokens=no
|
3827
4010
|
aoc files browse /
|
3828
|
-
aoc files browse /
|
4011
|
+
aoc files browse / --link=my_aoc_publink_folder
|
3829
4012
|
aoc files delete /testsrc
|
3830
4013
|
aoc files download --transfer=connect /200KB.1
|
3831
|
-
aoc files file
|
4014
|
+
aoc files file modify --path=my_aoc_test_folder
|
4015
|
+
aoc files file permission --path=my_aoc_test_folder list
|
3832
4016
|
aoc files file show --path=/200KB.1
|
3833
4017
|
aoc files file show my_file_id
|
3834
4018
|
aoc files find / --value='\.partial$'
|
@@ -3838,33 +4022,39 @@ aoc files rename /somefolder testdst
|
|
3838
4022
|
aoc files short_link create --to-folder=/testdst --value=private
|
3839
4023
|
aoc files short_link create --to-folder=/testdst --value=public
|
3840
4024
|
aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_link"}'
|
4025
|
+
aoc files sync admin status --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
4026
|
+
aoc files sync start --to-folder=/testdst --sync-info=@json:'{"sessions":[{"name":"s2","direction":"pull","local_dir":"syncdir","reset":true}]}'
|
3841
4027
|
aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
|
4028
|
+
aoc files upload --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
3842
4029
|
aoc files upload --to-folder=/testsrc testfile.bin
|
3843
|
-
aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
3844
4030
|
aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
3845
4031
|
aoc files v3 info
|
3846
|
-
aoc
|
4032
|
+
aoc gateway --value=https://localhost:12345/aspera/faspex & jobs -p
|
4033
|
+
aoc org --link=my_aoc_publink_recv_from_aocuser
|
3847
4034
|
aoc organization
|
3848
4035
|
aoc packages browse "my_package_id" /contents
|
3849
4036
|
aoc packages list
|
3850
4037
|
aoc packages list --query=@json:'{"dropbox_name":"my_aoc_shbx_name","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
3851
4038
|
aoc packages recv "my_package_id" --to-folder=.
|
3852
4039
|
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
4040
|
+
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"my_aoc_shbx_name","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
3853
4041
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_external_user"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
3854
4042
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_internal_user"],"note":"my note"}' testfile.bin
|
4043
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
4044
|
+
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
3855
4045
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' testfile.bin
|
3856
4046
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' testfile.bin
|
3857
4047
|
aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
3858
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
3859
|
-
aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
3860
4048
|
aoc packages shared_inboxes list
|
4049
|
+
aoc remind --username=my_aoc_user_email
|
4050
|
+
aoc servers
|
3861
4051
|
aoc user profile modify @json:'{"name":"dummy change"}'
|
3862
4052
|
aoc user profile show
|
3863
4053
|
aoc user workspaces current
|
3864
4054
|
aoc user workspaces list
|
3865
4055
|
```
|
3866
4056
|
|
3867
|
-
## <a id="ats"></a>Plugin: Aspera Transfer Service
|
4057
|
+
## <a id="ats"></a>Plugin: `ats`: IBM Aspera Transfer Service
|
3868
4058
|
|
3869
4059
|
ATS is usable either :
|
3870
4060
|
|
@@ -3875,10 +4065,12 @@ ATS is usable either :
|
|
3875
4065
|
### IBM Cloud ATS : creation of api key
|
3876
4066
|
|
3877
4067
|
This section is about using ATS with an IBM cloud subscription.
|
3878
|
-
If you are using ATS as part of AoC, then authentication is
|
4068
|
+
If you are using ATS as part of AoC, then authentication is through AoC, not IBM Cloud.
|
3879
4069
|
|
3880
4070
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
3881
4071
|
|
4072
|
+
<!-- spellchecker: disable -->
|
4073
|
+
|
3882
4074
|
```bash
|
3883
4075
|
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
3884
4076
|
```
|
@@ -3892,7 +4084,7 @@ Please preserve the API key! It cannot be retrieved after it's created.
|
|
3892
4084
|
Name mykeyname
|
3893
4085
|
Description my sample key
|
3894
4086
|
Created At 2019-09-30T12:17+0000
|
3895
|
-
API Key
|
4087
|
+
API Key my_secret_api_key_here
|
3896
4088
|
Locked false
|
3897
4089
|
UUID ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f
|
3898
4090
|
```
|
@@ -3902,10 +4094,14 @@ References:
|
|
3902
4094
|
- [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
3903
4095
|
- [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
3904
4096
|
|
4097
|
+
<!-- spellchecker: enable -->
|
4098
|
+
|
3905
4099
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
3906
4100
|
|
4101
|
+
<!-- spellchecker: disable -->
|
4102
|
+
|
3907
4103
|
```bash
|
3908
|
-
ascli config preset update my_ibm_ats --ibm-api-key=
|
4104
|
+
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here
|
3909
4105
|
```
|
3910
4106
|
|
3911
4107
|
```bash
|
@@ -3942,36 +4138,39 @@ ascli ats api_key create
|
|
3942
4138
|
ascli config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
3943
4139
|
```
|
3944
4140
|
|
4141
|
+
<!-- spellchecker: enable -->
|
4142
|
+
|
3945
4143
|
### <a id="ats_params"></a>ATS Access key creation parameters
|
3946
4144
|
|
3947
|
-
When creating an ATS access key, the option `params` must contain an extended value with the creation parameters.
|
4145
|
+
When creating an ATS access key, the option `params` must contain an extended value with the creation parameters.
|
4146
|
+
Those are directly the parameters expected by the [ATS API](https://developer.ibm.com/apis/catalog?search=%22Aspera%20ATS%20API%22).
|
3948
4147
|
|
3949
4148
|
### Misc. Examples
|
3950
4149
|
|
3951
4150
|
Example: create access key on IBM Cloud (softlayer):
|
3952
4151
|
|
3953
|
-
```
|
3954
|
-
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"
|
4152
|
+
```bash
|
4153
|
+
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"my_secret_here","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
3955
4154
|
```
|
3956
4155
|
|
3957
4156
|
Example: create access key on AWS:
|
3958
4157
|
|
3959
|
-
```
|
3960
|
-
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"
|
4158
|
+
```bash
|
4159
|
+
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"myaccesskey","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"_access_key_id_here_","secret_access_key":"my_secret_here"},"path":"/laurent"}}'
|
3961
4160
|
```
|
3962
4161
|
|
3963
4162
|
Example: create access key on Azure SAS:
|
3964
4163
|
|
3965
|
-
```
|
3966
|
-
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"
|
4164
|
+
```bash
|
4165
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
3967
4166
|
```
|
3968
4167
|
|
3969
4168
|
(Note that the blob name is mandatory after server address and before parameters. and that parameter sr=c is mandatory.)
|
3970
4169
|
|
3971
4170
|
Example: create access key on Azure:
|
3972
4171
|
|
3973
|
-
```
|
3974
|
-
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"
|
4172
|
+
```bash
|
4173
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
3975
4174
|
```
|
3976
4175
|
|
3977
4176
|
delete all my access keys:
|
@@ -3985,13 +4184,15 @@ The parameters provided to ATS for access key creation are the ones of [ATS API]
|
|
3985
4184
|
### ATS sample commands
|
3986
4185
|
|
3987
4186
|
```bash
|
3988
|
-
ats access_key cluster
|
4187
|
+
ats access_key cluster ak2ibmcloud --secret=my_secret_here
|
3989
4188
|
ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
3990
|
-
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"
|
4189
|
+
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak2ibmcloud","secret":"my_secret_here","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
4190
|
+
ats access_key delete ak2ibmcloud
|
3991
4191
|
ats access_key delete ak_aws
|
3992
|
-
ats access_key
|
4192
|
+
ats access_key entitlement ak2ibmcloud
|
3993
4193
|
ats access_key list --fields=name,id
|
3994
|
-
ats access_key node
|
4194
|
+
ats access_key node ak2ibmcloud browse / --secret=my_secret_here
|
4195
|
+
ats access_key show ak2ibmcloud
|
3995
4196
|
ats api_key create
|
3996
4197
|
ats api_key instances
|
3997
4198
|
ats api_key list
|
@@ -4001,17 +4202,21 @@ ats cluster show --cloud=aws --region=eu-west-1
|
|
4001
4202
|
ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
4002
4203
|
```
|
4003
4204
|
|
4004
|
-
## Plugin: IBM Aspera High Speed Transfer Server (
|
4205
|
+
## <a id="server"></a>Plugin: `server`: IBM Aspera High Speed Transfer Server (SSH)
|
4005
4206
|
|
4006
|
-
|
4207
|
+
The `server` plugin is used for operations on Aspera HSTS using SSH authentication.
|
4007
4208
|
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
4008
|
-
|
4209
|
+
An SSH session is established, authenticated with either a password or an SSH private key,
|
4210
|
+
then commands `ascp` (for transfers) and `ascmd` (for file operations) are executed.
|
4211
|
+
|
4212
|
+
> **Note:** The URL to be provided is usually: `ssh://_server_address_:33001`
|
4009
4213
|
|
4010
4214
|
### Server sample commands
|
4011
4215
|
|
4012
4216
|
```bash
|
4013
|
-
server
|
4217
|
+
server br /
|
4014
4218
|
server browse /
|
4219
|
+
server browse NEW_SERVER_FOLDER/testfile.bin
|
4015
4220
|
server browse folder_1/target_hot
|
4016
4221
|
server cp NEW_SERVER_FOLDER/testfile.bin folder_1/200KB.2
|
4017
4222
|
server delete NEW_SERVER_FOLDER
|
@@ -4027,34 +4232,39 @@ server md5sum NEW_SERVER_FOLDER/testfile.bin
|
|
4027
4232
|
server mkdir NEW_SERVER_FOLDER --logger=stdout
|
4028
4233
|
server mkdir folder_1/target_hot
|
4029
4234
|
server mv folder_1/200KB.2 folder_1/to.delete
|
4235
|
+
server sync admin status --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"}}'
|
4236
|
+
server sync admin status --sync-session=mysync --sync-info=@json:'{"sessions":[{"name":"mysync","local_dir":"syncdir"}]}'
|
4237
|
+
server sync start --sync-info=@json:'{"name":"sync2","local":{"path":"syncdir"},"remote":{"path":"'"NEW_SERVER_FOLDER"'"},"reset":true,"quiet":false}'
|
4238
|
+
server sync start --sync-info=@json:'{"sessions":[{"name":"mysync","direction":"pull","remote_dir":"'"NEW_SERVER_FOLDER"'","local_dir":"syncdir","reset":true}]}'
|
4030
4239
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","'"filelist.txt"'"]}' --to-folder=NEW_SERVER_FOLDER
|
4031
4240
|
server upload --sources=@ts --ts=@json:'{"EX_ascp_args":["--file-pair-list","'"filepairlist.txt"'"]}'
|
4032
4241
|
server upload --sources=@ts --ts=@json:'{"EX_file_list":"'"filelist.txt"'"}' --to-folder=NEW_SERVER_FOLDER
|
4033
4242
|
server upload --sources=@ts --ts=@json:'{"EX_file_pair_list":"'"filepairlist.txt"'"}'
|
4034
4243
|
server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
4035
4244
|
server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
4036
|
-
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
4037
|
-
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"
|
4245
|
+
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email --transfer-info=@json:'{"ascp_args":["-l","10m"]}'
|
4246
|
+
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"my_secret_here","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
|
4038
4247
|
server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
4039
|
-
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
4248
|
+
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}' --progress=multi
|
4040
4249
|
```
|
4041
4250
|
|
4042
4251
|
### Authentication on Server with SSH session
|
4043
4252
|
|
4044
|
-
If SSH is the session
|
4253
|
+
If SSH is the session protocol (by default i.e. not WSS), then following session authentication methods are supported:
|
4045
4254
|
|
4046
|
-
- SSH password
|
4047
|
-
- SSH keys (Multiple SSH key paths can be provided.)
|
4255
|
+
- `password`: SSH password
|
4256
|
+
- `ssh_keys`: SSH keys (Multiple SSH key paths can be provided.)
|
4048
4257
|
|
4049
|
-
If username is not provided
|
4258
|
+
If `username` is not provided then the default transfer user `xfer` is used.
|
4050
4259
|
|
4051
|
-
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
4260
|
+
If no SSH password or key is provided and a transfer token is provided in transfer spec (option `ts`), then standard SSH bypass keys are used.
|
4261
|
+
Example:
|
4052
4262
|
|
4053
|
-
```
|
4054
|
-
ascli server --url=ssh
|
4263
|
+
```bash
|
4264
|
+
ascli server --url=ssh://_server_address_:33001 ... --ts=@json:'{"token":"Basic _token_here_"}'
|
4055
4265
|
```
|
4056
4266
|
|
4057
|
-
> Note
|
4267
|
+
> **Note:** If you need to use the Aspera public keys, then specify an empty token: `--ts=@json:'{"token":""}'` : Aspera public SSH keys will be used, but the protocol will ignore the empty token.
|
4058
4268
|
|
4059
4269
|
The value of the `ssh_keys` option can be a single value or an array.
|
4060
4270
|
Each value is a **path** to a private key and is expanded (`~` is replaced with the user's home folder).
|
@@ -4067,10 +4277,17 @@ ascli server --ssh-keys=@list:,~/.ssh/id_rsa
|
|
4067
4277
|
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
4068
4278
|
```
|
4069
4279
|
|
4070
|
-
For
|
4071
|
-
|
4280
|
+
For file operation command (browse, delete), the ruby SSH client library `Net::SSH` is used and provides several options settable using option `ssh_options`.
|
4281
|
+
|
4282
|
+
For a list of SSH client options, refer to the ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html#method-c-start).
|
4283
|
+
|
4284
|
+
Some of the 50 available SSH options:
|
4072
4285
|
|
4073
|
-
|
4286
|
+
- `verbose`
|
4287
|
+
- `use_agent`
|
4288
|
+
- `passphrase`
|
4289
|
+
|
4290
|
+
By default the SSH library will check if a local `ssh-agent` is running.
|
4074
4291
|
|
4075
4292
|
On Linux, if you get an error message such as:
|
4076
4293
|
|
@@ -4084,20 +4301,31 @@ or on Windows:
|
|
4084
4301
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
4085
4302
|
```
|
4086
4303
|
|
4087
|
-
This means that you don't have such an SSH agent running, then:
|
4304
|
+
This means that your environment suggessts to use an agent but you don't have such an SSH agent running, then:
|
4088
4305
|
|
4089
|
-
-
|
4090
|
-
-
|
4091
|
-
-
|
4306
|
+
- Check env var: `SSH_AGENT_SOCK`
|
4307
|
+
- Check your file: `$HOME/.ssh/config`
|
4308
|
+
- Check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option)
|
4309
|
+
- [Check the Ruby SSH manual](https://www.rubydoc.info/github/net-ssh/net-ssh/Net%2FSSH.start)
|
4092
4310
|
- To disable the use of `ssh-agent`, use the option `ssh_options` like this:
|
4093
4311
|
|
4094
4312
|
```bash
|
4095
|
-
ascli server --ssh-options=@
|
4313
|
+
ascli server --ssh-options=@json:'{"use_agent": false}' ...
|
4096
4314
|
```
|
4097
4315
|
|
4098
|
-
This can also be set
|
4316
|
+
> **Note:** This can also be set using a preset.
|
4317
|
+
|
4318
|
+
### Other session channels for `server`
|
4319
|
+
|
4320
|
+
URL schemes `local` and `https` are also supported (mainly for testing purpose).
|
4321
|
+
(`--url=local:` , `--url=https://...`)
|
4322
|
+
|
4323
|
+
- `local` will execute `ascmd` locally, instead of using an SSH connection.
|
4324
|
+
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
4099
4325
|
|
4100
|
-
|
4326
|
+
As, most of the time, SSH is used, if an `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
4327
|
+
|
4328
|
+
### Examples: `server`
|
4101
4329
|
|
4102
4330
|
One can test the `server` application using the well known demo server:
|
4103
4331
|
|
@@ -4109,7 +4337,7 @@ ascli server download /aspera-test-dir-large/200MB
|
|
4109
4337
|
|
4110
4338
|
`initdemo` creates a [option preset](#lprt) `demoserver` and set it as default for plugin `server`.
|
4111
4339
|
|
4112
|
-
## Plugin: IBM Aspera High Speed Transfer Server
|
4340
|
+
## <a id="node"></a>Plugin: `node`: IBM Aspera High Speed Transfer Server Node
|
4113
4341
|
|
4114
4342
|
This plugin gives access to capabilities provided by HSTS node API.
|
4115
4343
|
|
@@ -4127,17 +4355,17 @@ For transfers, it is possible to control how transfer is authorized using option
|
|
4127
4355
|
- `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
4128
4356
|
- `basic` : transfer spec is created like this:
|
4129
4357
|
|
4130
|
-
```
|
4358
|
+
```json
|
4131
4359
|
{
|
4132
|
-
"remote_host": address of node url,
|
4360
|
+
"remote_host": "<address of node url>",
|
4133
4361
|
"remote_user": "xfer",
|
4134
4362
|
"ssh_port": 33001,
|
4135
4363
|
"token": "Basic <base 64 encoded user/pass>",
|
4136
|
-
"direction": send
|
4364
|
+
"direction": "[send|receive]"
|
4137
4365
|
}
|
4138
4366
|
```
|
4139
4367
|
|
4140
|
-
Note
|
4368
|
+
> **Note:** the port is assumed to be the default Aspera SSH port `33001` and transfer user is assumed to be `xfer`.
|
4141
4369
|
|
4142
4370
|
### Central
|
4143
4371
|
|
@@ -4157,8 +4385,8 @@ It is possible to start a FASPStream session using the node API:
|
|
4157
4385
|
|
4158
4386
|
Use the "node stream create" command, then arguments are provided as a [*transfer-spec*](#transferspec).
|
4159
4387
|
|
4160
|
-
```
|
4161
|
-
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"
|
4388
|
+
```bash
|
4389
|
+
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"my_pass_here"}' --preset=stream
|
4162
4390
|
```
|
4163
4391
|
|
4164
4392
|
### Watchfolder
|
@@ -4170,17 +4398,21 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
4170
4398
|
- Start watchd and watchfolderd services running as a system user having access to files
|
4171
4399
|
- configure a watchfolder to define automated transfers
|
4172
4400
|
|
4173
|
-
|
4401
|
+
<!-- spellchecker: disable -->
|
4402
|
+
|
4403
|
+
```bash
|
4174
4404
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
4175
4405
|
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
4176
4406
|
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
4177
4407
|
```
|
4178
4408
|
|
4409
|
+
<!-- spellchecker: enable -->
|
4410
|
+
|
4179
4411
|
### Out of Transfer File Validation
|
4180
4412
|
|
4181
4413
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
4182
4414
|
|
4183
|
-
```
|
4415
|
+
```bash
|
4184
4416
|
ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
4185
4417
|
```
|
4186
4418
|
|
@@ -4192,7 +4424,7 @@ ascli node central file list --validator=ascli --data=@json:'{"file_transfer_fil
|
|
4192
4424
|
:..............:..............:............:......................................:
|
4193
4425
|
```
|
4194
4426
|
|
4195
|
-
```
|
4427
|
+
```bash
|
4196
4428
|
ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
4197
4429
|
```
|
4198
4430
|
|
@@ -4202,35 +4434,42 @@ updated
|
|
4202
4434
|
|
4203
4435
|
### Example: SHOD to ATS
|
4204
4436
|
|
4205
|
-
Scenario: Access to a
|
4437
|
+
Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner.
|
4206
4438
|
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
|
4207
|
-
Simply create an
|
4208
|
-
Then create a configuration for the
|
4209
|
-
Create another configuration for the Azure ATS instance: in section "node", named
|
4439
|
+
Simply create an **Aspera Transfer Service** instance, which provides access to the node API.
|
4440
|
+
Then create a configuration for the **SHOD** instance in the configuration file: in section "shares", a configuration named: aws_shod.
|
4441
|
+
Create another configuration for the Azure ATS instance: in section "node", named azure_ats.
|
4210
4442
|
Then execute the following command:
|
4211
4443
|
|
4212
4444
|
```bash
|
4213
|
-
ascli node download /share/sourcefile --to-folder=/
|
4445
|
+
ascli node download /share/sourcefile --to-folder=/destination_folder --preset=aws_shod --transfer=node --transfer-info=@preset:azure_ats
|
4214
4446
|
```
|
4215
4447
|
|
4216
4448
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.
|
4217
4449
|
|
4218
4450
|
### Create access key
|
4219
4451
|
|
4220
|
-
```
|
4221
|
-
ascli node access_key create --value=@json:'{"id":"
|
4452
|
+
```bash
|
4453
|
+
ascli node access_key create --value=@json:'{"id":"myaccesskey","secret":"my_secret_here","storage":{"type":"local","path":"/data/mydir"}}'
|
4222
4454
|
```
|
4223
4455
|
|
4224
4456
|
### Node sample commands
|
4225
4457
|
|
4226
4458
|
```bash
|
4227
|
-
node
|
4228
|
-
node
|
4229
|
-
node
|
4230
|
-
node
|
4231
|
-
node
|
4232
|
-
node
|
4233
|
-
node
|
4459
|
+
node access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
4460
|
+
node access_key delete aoc_1
|
4461
|
+
node access_key do my_aoc_ak_name browse /
|
4462
|
+
node access_key do my_aoc_ak_name delete /folder2
|
4463
|
+
node access_key do my_aoc_ak_name delete testfile1
|
4464
|
+
node access_key do my_aoc_ak_name download testfile1 --to-folder=.
|
4465
|
+
node access_key do my_aoc_ak_name file show --path=/testfile1
|
4466
|
+
node access_key do my_aoc_ak_name file show 1
|
4467
|
+
node access_key do my_aoc_ak_name find /
|
4468
|
+
node access_key do my_aoc_ak_name mkdir /folder1
|
4469
|
+
node access_key do my_aoc_ak_name node_info /
|
4470
|
+
node access_key do my_aoc_ak_name rename /folder1 folder2
|
4471
|
+
node access_key do my_aoc_ak_name upload 'faux:///testfile1?1k' --default_ports=no
|
4472
|
+
node access_key list
|
4234
4473
|
node api_details
|
4235
4474
|
node async bandwidth 1
|
4236
4475
|
node async counters 1
|
@@ -4241,7 +4480,7 @@ node async show ALL
|
|
4241
4480
|
node basic_token
|
4242
4481
|
node browse / -r
|
4243
4482
|
node delete /todelete
|
4244
|
-
node delete @list
|
4483
|
+
node delete @list:,folder_1/todelete,folder_1/tdlink,folder_1/delfile
|
4245
4484
|
node delete folder_1/10MB.1
|
4246
4485
|
node delete testfile.bin
|
4247
4486
|
node download testfile.bin --to-folder=.
|
@@ -4249,11 +4488,14 @@ node download testfile.bin --to-folder=. --token-type=hybrid
|
|
4249
4488
|
node health
|
4250
4489
|
node info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
|
4251
4490
|
node license
|
4252
|
-
node mkdir /todelete
|
4253
|
-
node mkfile /delfile1 "hello world"
|
4254
|
-
node mklink /todelete /tdlink
|
4255
|
-
node rename
|
4491
|
+
node mkdir folder_1/todelete
|
4492
|
+
node mkfile folder_1/delfile1 "hello world"
|
4493
|
+
node mklink folder_1/todelete folder_1/tdlink
|
4494
|
+
node rename folder_1 delfile1 delfile
|
4256
4495
|
node search / --value=@json:'{"sort":"mtime"}'
|
4496
|
+
node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
4497
|
+
node service delete service1
|
4498
|
+
node service list
|
4257
4499
|
node space /
|
4258
4500
|
node sync bandwidth my_syncid
|
4259
4501
|
node sync counters my_syncid
|
@@ -4267,13 +4509,13 @@ node sync state my_syncid
|
|
4267
4509
|
node sync stop my_syncid
|
4268
4510
|
node sync summary my_syncid
|
4269
4511
|
node transfer list --value=@json:'{"active_only":true}'
|
4270
|
-
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"
|
4512
|
+
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass_here"}'
|
4271
4513
|
node upload --username=my_aoc_ak_name --password=my_aoc_ak_secret testfile.bin --token-type=basic
|
4272
4514
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}'
|
4273
4515
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}' --token-type=hybrid
|
4274
4516
|
```
|
4275
4517
|
|
4276
|
-
## Plugin: IBM Aspera
|
4518
|
+
## <a id="faspex5"></a>Plugin: `faspex5`: IBM Aspera Faspex v5
|
4277
4519
|
|
4278
4520
|
IBM Aspera's newer self-managed application.
|
4279
4521
|
|
@@ -4285,7 +4527,7 @@ IBM Aspera's newer self-managed application.
|
|
4285
4527
|
|
4286
4528
|
### Faspex 5 JWT authentication
|
4287
4529
|
|
4288
|
-
This is the **
|
4530
|
+
This is the **recommended** method to use.
|
4289
4531
|
|
4290
4532
|
For `jwt`, create an API client in Faspex with JWT support:
|
4291
4533
|
|
@@ -4301,12 +4543,12 @@ Then use these options:
|
|
4301
4543
|
```text
|
4302
4544
|
--auth=jwt
|
4303
4545
|
--client-id=_client_id_here_
|
4304
|
-
--client-secret=
|
4546
|
+
--client-secret=my_secret_here
|
4305
4547
|
--username=_username_here_
|
4306
4548
|
--private-key=@file:.../path/to/key.pem
|
4307
4549
|
```
|
4308
4550
|
|
4309
|
-
> The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
4551
|
+
> **Note:** The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
4310
4552
|
|
4311
4553
|
### Faspex 5 web authentication
|
4312
4554
|
|
@@ -4323,7 +4565,7 @@ Then use options:
|
|
4323
4565
|
```text
|
4324
4566
|
--auth=web
|
4325
4567
|
--client-id=_client_id_here_
|
4326
|
-
--client-secret=
|
4568
|
+
--client-secret=my_secret_here
|
4327
4569
|
--redirect-uri=https://127.0.0.1:8888
|
4328
4570
|
```
|
4329
4571
|
|
@@ -4342,18 +4584,28 @@ Use this token as password and use `--auth=boot`.
|
|
4342
4584
|
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=_token_here_
|
4343
4585
|
```
|
4344
4586
|
|
4587
|
+
### Faspex 5 packages
|
4588
|
+
|
4589
|
+
The `value` option provided to command `faspex5 package send` is the same as for the Faspex 5 API: `POST /packages`.
|
4590
|
+
|
4591
|
+
In addition, `ascli` adds some convenience: the field `recipients` is normally an Array of Hash, each with field `name` and optionally `recipient_type`, but it is also possible to provide an Array of String, with simply a recipient name.
|
4592
|
+
Then `ascli` will lookup existing contacts, and if a single match is found will use it, and set the `name` and `recipient_type` accordingly.
|
4593
|
+
|
4594
|
+
> **Note:** The lookup is case insensitive and on partial matches.
|
4595
|
+
|
4345
4596
|
### Faspex 5 sample commands
|
4346
4597
|
|
4347
4598
|
Most commands are directly REST API calls.
|
4348
|
-
Parameters to
|
4599
|
+
Parameters to commands are carried through option `value`, as extended value.
|
4349
4600
|
Usually using JSON format with prefix `@json:`.
|
4350
4601
|
|
4351
|
-
> The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
4602
|
+
> **Note:** The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
4352
4603
|
|
4353
4604
|
```bash
|
4354
4605
|
faspex5 admin res accounts list
|
4355
4606
|
faspex5 admin res contacts list
|
4356
4607
|
faspex5 admin res jobs list
|
4608
|
+
faspex5 admin res metadata_profiles list
|
4357
4609
|
faspex5 admin res node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
4358
4610
|
faspex5 admin res oauth_clients list
|
4359
4611
|
faspex5 admin res registrations list
|
@@ -4361,37 +4613,114 @@ faspex5 admin res saml_configs list
|
|
4361
4613
|
faspex5 admin res shared_inboxes list
|
4362
4614
|
faspex5 admin res workgroups list
|
4363
4615
|
faspex5 bearer_token
|
4616
|
+
faspex5 gateway --value=https://localhost:12345/aspera/faspex &\
|
4364
4617
|
faspex5 health
|
4365
4618
|
faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
4366
4619
|
faspex5 package receive "my_package_id" --to-folder=. --ts=@json:'{"content_protection_password":"abc123_yo"}'
|
4367
|
-
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"
|
4620
|
+
faspex5 package send --value=@json:'{"title":"test title","recipients":["my_shinbox"],"metadata":{"Options":"Opt1","TextInput":"example text"}}' testfile.bin
|
4621
|
+
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin --ts=@json:'{"content_protection_password":"my_passphrase_here"}'
|
4622
|
+
faspex5 package show "my_package_id"
|
4623
|
+
faspex5 postprocessing --value=@json:'{"url":"https://localhost:8443/domain","processing":{"script_folder":"tests"},"certificate":{"key":"../local/k","cert":"../local/c","chain":"../local/ch"}}' &\
|
4624
|
+
faspex5 user profile modify @json:'{"preference":{"connect_disabled":false}}'
|
4625
|
+
faspex5 user profile show
|
4368
4626
|
```
|
4369
4627
|
|
4370
4628
|
Other examples:
|
4371
4629
|
|
4630
|
+
- Send a package with metadata
|
4631
|
+
|
4632
|
+
The interface is the one of the API (Refer to API documentation, or look at request in browser):
|
4633
|
+
|
4634
|
+
```bash
|
4635
|
+
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":["ascli shared inbox"],"metadata":{"Confidential":"Yes","Drop menu":"Option 1"}}' 'faux:///test1?k1'
|
4636
|
+
```
|
4637
|
+
|
4638
|
+
Basically, add the field `metadata`, with one key per metadata and the value is directly the metadata value.
|
4639
|
+
|
4372
4640
|
- List all shared inboxes
|
4373
4641
|
|
4374
|
-
```
|
4642
|
+
```bash
|
4375
4643
|
ascli faspex5 admin res shared list --value=@json:'{"all":true}' --fields=id,name
|
4376
4644
|
```
|
4377
4645
|
|
4378
4646
|
- Create Metadata profile
|
4379
4647
|
|
4380
|
-
```
|
4648
|
+
```bash
|
4381
4649
|
ascli faspex5 admin res metadata_profiles create --value=@json:'{"name":"the profile","default":false,"title":{"max_length":200,"illegal_chars":[]},"note":{"max_length":400,"illegal_chars":[],"enabled":false},"fields":[{"ordering":0,"name":"field1","type":"text_area","require":true,"illegal_chars":[],"max_length":100},{"ordering":1,"name":"fff2","type":"option_list","require":false,"choices":["opt1","opt2"]}]}'
|
4382
4650
|
```
|
4383
4651
|
|
4384
4652
|
- Create a Shared inbox with specific metadata profile
|
4385
4653
|
|
4386
|
-
```
|
4654
|
+
```bash
|
4387
4655
|
ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox","metadata_profile_id":1}'
|
4388
4656
|
```
|
4389
4657
|
|
4390
|
-
|
4658
|
+
- List content in Shared folder and send package from remote source
|
4659
|
+
|
4660
|
+
```bash
|
4661
|
+
ascli faspex5 shared_folders list
|
4662
|
+
```
|
4663
|
+
|
4664
|
+
```bash
|
4665
|
+
ascli faspex5 shared_folders br 3 /folder
|
4666
|
+
```
|
4667
|
+
|
4668
|
+
```bash
|
4669
|
+
ascli faspex5 package send --value=@json:'{"title":"hello","recipients":[{"name":"_recipient_here_"}]}' --shared-folder=3 /folder/file
|
4670
|
+
```
|
4671
|
+
|
4672
|
+
### Faspex 4-style postprocessing script with Faspex 5
|
4673
|
+
|
4674
|
+
`ascli` provides command `postprocessing` in plugin `faspex5` to emulate Faspex 4 postprocessing.
|
4675
|
+
It implements Faspex 5 web hooks, and calls a local script with the same environment as Faspex 4.
|
4676
|
+
|
4677
|
+
It is invoked like this:
|
4678
|
+
|
4679
|
+
```bash
|
4680
|
+
ascli faspex5 postprocessing --value=@json:'{"url":"http://localhost:8080/processing"}'
|
4681
|
+
```
|
4682
|
+
|
4683
|
+
The following parameters are supported:
|
4684
|
+
|
4685
|
+
| parameter | type | default | description |
|
4686
|
+
|----------------------------|---------|------------------------|-----------------------------------------------------|
|
4687
|
+
| url | string | http://localhost:8080 | Defines the base url on which requests are listened |
|
4688
|
+
| certificate | hash | nil | used to define certificate if https is used |
|
4689
|
+
| certificate.key | string | nil | path to private key file |
|
4690
|
+
| certificate.cert | string | nil | path to certificate |
|
4691
|
+
| certificate.chain | string | nil | path to intermediary certificates |
|
4692
|
+
| processing | hash | nil | behaviour of post processing |
|
4693
|
+
| processing.script_folder | string | . | prefix added to script path |
|
4694
|
+
| processing.fail_on_error | bool | false | if true and process exit with non zero, then fail |
|
4695
|
+
| processing.timeout_seconds | integer | 60 | processing script is killed if takes more time |
|
4696
|
+
|
4697
|
+
Parameter `url` defines:
|
4698
|
+
|
4699
|
+
- if http or https is used
|
4700
|
+
- the local port
|
4701
|
+
- the "domain", i.e. main path of url
|
4702
|
+
|
4703
|
+
When a request is received the following happens:
|
4704
|
+
|
4705
|
+
- the processor get the path of the url called
|
4706
|
+
- it removes the "domain
|
4707
|
+
- it pre-prends it with the value of `script_folder`
|
4708
|
+
- it executes the script
|
4709
|
+
- upon success, a success code is returned
|
4710
|
+
|
4711
|
+
In Faspex 5, configure like this:
|
4712
|
+
|
4713
|
+
`Webhook endpoint URI` : `http://localhost:8080/processing/script1.sh`
|
4714
|
+
|
4715
|
+
Then, the postprocessing script executed will be `script1.sh`.
|
4716
|
+
|
4717
|
+
Environment variables at set to the values provided by the web hook which are the same as Faspex 4 postprocessing.
|
4718
|
+
|
4719
|
+
## <a id="faspex"></a>Plugin: `faspex`: IBM Aspera Faspex v4
|
4391
4720
|
|
4392
4721
|
Notes:
|
4393
4722
|
|
4394
|
-
- The command
|
4723
|
+
- The command `v4` requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
4395
4724
|
- For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
4396
4725
|
|
4397
4726
|
### Listing Packages
|
@@ -4427,7 +4756,7 @@ If no parameter `max` or `pmax` is provided, then all packages will be listed in
|
|
4427
4756
|
|
4428
4757
|
#### Example: list packages in dropbox
|
4429
4758
|
|
4430
|
-
```
|
4759
|
+
```bash
|
4431
4760
|
ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
4432
4761
|
```
|
4433
4762
|
|
@@ -4449,8 +4778,8 @@ ascli faspex package recv --link=faspe://...
|
|
4449
4778
|
If the package is in a specific **dropbox**/**workgroup**, add option `recipient` for both the `list` and `recv` commands.
|
4450
4779
|
|
4451
4780
|
```bash
|
4452
|
-
ascli faspex package list --recipient='*
|
4453
|
-
ascli faspex package recv 125 --recipient='*
|
4781
|
+
ascli faspex package list --recipient='*dropbox_name'
|
4782
|
+
ascli faspex package recv 125 --recipient='*dropbox_name'
|
4454
4783
|
```
|
4455
4784
|
|
4456
4785
|
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
@@ -4462,7 +4791,7 @@ The contents of `delivery_info` is directly the contents of the `send` v3 [API o
|
|
4462
4791
|
|
4463
4792
|
Example:
|
4464
4793
|
|
4465
|
-
```
|
4794
|
+
```bash
|
4466
4795
|
ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
4467
4796
|
```
|
4468
4797
|
|
@@ -4480,7 +4809,7 @@ Like for any transfer, a notification can be sent by email using parameters: `no
|
|
4480
4809
|
|
4481
4810
|
Example:
|
4482
4811
|
|
4483
|
-
```
|
4812
|
+
```bash
|
4484
4813
|
ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
4485
4814
|
```
|
4486
4815
|
|
@@ -4490,7 +4819,7 @@ In this example the notification template is directly provided on command line.
|
|
4490
4819
|
|
4491
4820
|
Example:
|
4492
4821
|
|
4493
|
-
```
|
4822
|
+
```bash
|
4494
4823
|
ascli faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
4495
4824
|
ascli faspex v4 dropbox list
|
4496
4825
|
ascli faspex v4 dropbox delete --id=36
|
@@ -4510,17 +4839,17 @@ my_faspex_conf:
|
|
4510
4839
|
username: admin
|
4511
4840
|
password: MyUserPassword
|
4512
4841
|
storage:
|
4513
|
-
|
4842
|
+
my_storage:
|
4514
4843
|
node: "@preset:my_faspex_node"
|
4515
|
-
path: /
|
4844
|
+
path: /mydir
|
4516
4845
|
my_faspex_node:
|
4517
4846
|
url: https://10.25.0.3:9092
|
4518
4847
|
username: node_faspex
|
4519
4848
|
password: MyNodePassword
|
4520
4849
|
```
|
4521
4850
|
|
4522
|
-
In this example, a faspex storage named
|
4523
|
-
under the docroot in
|
4851
|
+
In this example, a faspex storage named `my_storage` exists in Faspex, and is located
|
4852
|
+
under the docroot in `/mydir` (this must be the same as configured in Faspex).
|
4524
4853
|
The node configuration name is "my_faspex_node" here.
|
4525
4854
|
|
4526
4855
|
Note: the v4 API provides an API for nodes and shares.
|
@@ -4570,18 +4899,25 @@ faspex v4 wmembership list
|
|
4570
4899
|
faspex v4 workgroup list
|
4571
4900
|
```
|
4572
4901
|
|
4573
|
-
## Plugin: IBM Aspera Shares
|
4902
|
+
## <a id="shares"></a>Plugin: `shares`: IBM Aspera Shares v1
|
4574
4903
|
|
4575
4904
|
Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)
|
4576
4905
|
|
4577
4906
|
### Shares 1 sample commands
|
4578
4907
|
|
4579
4908
|
```bash
|
4580
|
-
shares admin
|
4581
|
-
shares admin
|
4582
|
-
shares admin
|
4909
|
+
shares admin group list
|
4910
|
+
shares admin node list
|
4911
|
+
shares admin share list --fields=-status,status_message
|
4912
|
+
shares admin share user_permissions 1 list
|
4913
|
+
shares admin user add --type=ldap --value=the_name
|
4914
|
+
shares admin user app_authorizations 1 modify --value=@json:'{"app_login":true}'
|
4915
|
+
shares admin user app_authorizations 1 show
|
4916
|
+
shares admin user import --type=saml --value=@json:'{"id":"the_id","name_id":"the_name"}'
|
4583
4917
|
shares admin user list
|
4584
|
-
shares admin user share_permissions 1
|
4918
|
+
shares admin user share_permissions 1 list
|
4919
|
+
shares admin user share_permissions 1 show 1
|
4920
|
+
shares health
|
4585
4921
|
shares repository browse /
|
4586
4922
|
shares repository delete my_shares_upload/testfile.bin
|
4587
4923
|
shares repository download --to-folder=. my_shares_upload/testfile.bin
|
@@ -4590,7 +4926,7 @@ shares repository upload --to-folder=my_shares_upload testfile.bin
|
|
4590
4926
|
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
4591
4927
|
```
|
4592
4928
|
|
4593
|
-
## Plugin: Console
|
4929
|
+
## <a id="console"></a>Plugin: `console`: IBM Aspera Console
|
4594
4930
|
|
4595
4931
|
### Console sample commands
|
4596
4932
|
|
@@ -4601,11 +4937,12 @@ console transfer smart list
|
|
4601
4937
|
console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
4602
4938
|
```
|
4603
4939
|
|
4604
|
-
## Plugin: Orchestrator
|
4940
|
+
## <a id="orchestrator"></a>Plugin: `orchestrator`:IBM Aspera Orchestrator
|
4605
4941
|
|
4606
4942
|
### Orchestrator sample commands
|
4607
4943
|
|
4608
4944
|
```bash
|
4945
|
+
orchestrator health
|
4609
4946
|
orchestrator info
|
4610
4947
|
orchestrator plugins
|
4611
4948
|
orchestrator processes
|
@@ -4619,13 +4956,15 @@ orchestrator workflow status ALL
|
|
4619
4956
|
orchestrator workflow status my_orch_workflow_id
|
4620
4957
|
```
|
4621
4958
|
|
4622
|
-
## Plugin: IBM Cloud Object Storage
|
4959
|
+
## <a id="cos"></a>Plugin: `cos`: IBM Cloud Object Storage
|
4623
4960
|
|
4624
4961
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
4625
4962
|
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
4626
4963
|
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
4627
4964
|
|
4628
|
-
There are two possibilities to provide credentials.
|
4965
|
+
There are two possibilities to provide credentials.
|
4966
|
+
If you already have the endpoint, apikey and CRN, use the first method.
|
4967
|
+
If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
4629
4968
|
|
4630
4969
|
### Using endpoint, apikey and Resource Instance ID (CRN)
|
4631
4970
|
|
@@ -4647,9 +4986,17 @@ Then, jump to the transfer example.
|
|
4647
4986
|
|
4648
4987
|
### Using service credential file
|
4649
4988
|
|
4650
|
-
If you are the COS administrator and don't have yet the credential:
|
4989
|
+
If you are the COS administrator and don't have yet the credential:
|
4990
|
+
Service credentials are directly created using the IBM cloud Console (web UI).
|
4991
|
+
Navigate to:
|
4651
4992
|
|
4652
|
-
|
4993
|
+
- → Navigation Menu
|
4994
|
+
- → [Resource List](https://cloud.ibm.com/resources)
|
4995
|
+
- → [Storage](https://cloud.ibm.com/objectstorage)
|
4996
|
+
- → Select your storage instance
|
4997
|
+
- → Service Credentials
|
4998
|
+
- → New credentials (Leave default role: Writer, no special options)
|
4999
|
+
- → Copy to clipboard
|
4653
5000
|
|
4654
5001
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
4655
5002
|
|
@@ -4657,29 +5004,33 @@ or using the IBM Cloud CLI:
|
|
4657
5004
|
|
4658
5005
|
```bash
|
4659
5006
|
ibmcloud resource service-keys
|
4660
|
-
ibmcloud resource service-key
|
5007
|
+
ibmcloud resource service-key _service_key_name_here_ --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
4661
5008
|
```
|
4662
5009
|
|
4663
5010
|
(if you don't have `jq` installed, extract the structure as follows)
|
4664
5011
|
|
4665
5012
|
It consists in the following structure:
|
4666
5013
|
|
4667
|
-
|
5014
|
+
<!-- spellchecker: disable -->
|
5015
|
+
|
5016
|
+
```json
|
4668
5017
|
{
|
4669
|
-
"apikey": "
|
5018
|
+
"apikey": "my_api_key_here",
|
4670
5019
|
"cos_hmac_keys": {
|
4671
|
-
"access_key_id": "
|
4672
|
-
"secret_access_key": "
|
5020
|
+
"access_key_id": "my_access_key_here",
|
5021
|
+
"secret_access_key": "my_secret_here"
|
4673
5022
|
},
|
4674
5023
|
"endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
|
4675
|
-
"iam_apikey_description": "
|
4676
|
-
"iam_apikey_name": "
|
5024
|
+
"iam_apikey_description": "my_description_here",
|
5025
|
+
"iam_apikey_name": "my_key_name_here",
|
4677
5026
|
"iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
|
4678
5027
|
"iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....",
|
4679
5028
|
"resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....."
|
4680
5029
|
}
|
4681
5030
|
```
|
4682
5031
|
|
5032
|
+
<!-- spellchecker: enable -->
|
5033
|
+
|
4683
5034
|
The field `resource_instance_id` is for option `crn`
|
4684
5035
|
|
4685
5036
|
The field `apikey` is for option `apikey`
|
@@ -4710,31 +5061,43 @@ ascli cos node info
|
|
4710
5061
|
ascli cos node upload 'faux:///sample1G?1g'
|
4711
5062
|
```
|
4712
5063
|
|
4713
|
-
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
5064
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man `ascp` and section above), but you can of course send a real file by specifying a real file instead.
|
4714
5065
|
|
4715
5066
|
### COS sample commands
|
4716
5067
|
|
4717
5068
|
```bash
|
4718
|
-
cos
|
4719
|
-
cos
|
5069
|
+
cos --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
5070
|
+
cos --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
4720
5071
|
cos node access_key show self
|
4721
5072
|
cos node download testfile.bin --to-folder=.
|
4722
5073
|
cos node info
|
4723
5074
|
cos node upload testfile.bin
|
4724
5075
|
```
|
4725
5076
|
|
4726
|
-
## Plugin: IBM Aspera Sync
|
5077
|
+
## <a id="async"></a>Plugin: `async`: IBM Aspera Sync
|
4727
5078
|
|
4728
5079
|
A basic plugin to start an "async" using `ascli`.
|
4729
|
-
The main advantage is the possibility to
|
5080
|
+
The main advantage over bare `async` command line is the possibility to use a configuration file, using `ascli` standard options.
|
5081
|
+
|
5082
|
+
Also, the `sync` command is also made available through the `server sync` and `aoc files sync` commands.
|
5083
|
+
In this case, some of the `sync` parameters are fill from parameters of the related plugin.
|
5084
|
+
|
5085
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
5086
|
+
>
|
5087
|
+
> **Note:** Two JSON syntax are supported for option `sync_info`.
|
5088
|
+
> The first is same sync payload as specified on the `async` option `--conf` or in the latest node API, this is the preferred syntax and allows a single session definition.
|
5089
|
+
> The second (legacy) is specific to `ascli` and allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
5090
|
+
|
5091
|
+
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
4730
5092
|
|
4731
5093
|
### Sync sample commands
|
4732
5094
|
|
4733
5095
|
```bash
|
4734
|
-
sync
|
5096
|
+
sync admin status --sync-info=@json:'{"sessions":[{"name":"test","local_dir":"contents"}]}'
|
5097
|
+
sync start --sync-info=@json:'{"instance":{"quiet":true},"sessions":[{"name":"test","reset":true,"remote_dir":"/sync_test","local_dir":"contents","host":"my_remote_host","tcp_port":33001,"user":"my_remote_user","private_key_paths":["my_local_user_key"]}]}'
|
4735
5098
|
```
|
4736
5099
|
|
4737
|
-
## Plugin: Preview
|
5100
|
+
## <a id="preview"></a>Plugin: `preview`: Preview generator for AoC
|
4738
5101
|
|
4739
5102
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
4740
5103
|
It uses the **node API** of Aspera HSTS and requires use of Access Keys and it's **storage root**.
|
@@ -4831,6 +5194,7 @@ dnf install unoconv
|
|
4831
5194
|
|
4832
5195
|
- Amazon Linux
|
4833
5196
|
|
5197
|
+
<!-- spellchecker: disable -->
|
4834
5198
|
```bash
|
4835
5199
|
amazon-linux-extras enable libreoffice
|
4836
5200
|
yum clean metadata
|
@@ -4839,6 +5203,7 @@ wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
|
4839
5203
|
mv unoconv /usr/bin
|
4840
5204
|
chmod a+x /usr/bin/unoconv
|
4841
5205
|
```
|
5206
|
+
<!-- spellchecker: enable -->
|
4842
5207
|
|
4843
5208
|
### Configuration
|
4844
5209
|
|
@@ -4846,14 +5211,14 @@ The preview generator is run as a user, preferably a regular user (not root). Wh
|
|
4846
5211
|
|
4847
5212
|
Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). The configuration file must be created with the same user used to run so that it is properly used on runtime.
|
4848
5213
|
|
4849
|
-
|
5214
|
+
The `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
4850
5215
|
|
4851
5216
|
```bash
|
4852
5217
|
su -s /bin/bash - xfer
|
4853
5218
|
|
4854
|
-
ascli config preset update
|
5219
|
+
ascli config preset update mypreviewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
4855
5220
|
|
4856
|
-
ascli config preset set default preview
|
5221
|
+
ascli config preset set default preview mypreviewconf
|
4857
5222
|
```
|
4858
5223
|
|
4859
5224
|
Here we assume that Office file generation is disabled, else remove this option.
|
@@ -4869,11 +5234,11 @@ This shall list the contents of the storage root of the access key.
|
|
4869
5234
|
|
4870
5235
|
### Execution
|
4871
5236
|
|
4872
|
-
The tool intentionally supports only a
|
4873
|
-
It needs to be run on a regular basis to create or update preview files.
|
4874
|
-
|
5237
|
+
The tool intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
|
5238
|
+
It needs to be run on a regular basis to create or update preview files.
|
5239
|
+
For that use your best reliable scheduler, see [Scheduling](#scheduling).
|
4875
5240
|
|
4876
|
-
Typically, for
|
5241
|
+
Typically, for **Access key** access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
|
4877
5242
|
|
4878
5243
|
Lets do a one shot test, using the configuration previously created:
|
4879
5244
|
|
@@ -4889,15 +5254,16 @@ On subsequent run it reads this file and check that previews are generated for t
|
|
4889
5254
|
|
4890
5255
|
### Configuration for Execution in scheduler
|
4891
5256
|
|
4892
|
-
Here is an example of configuration for use with cron on Linux.
|
4893
|
-
Adapt the scripts to your own
|
5257
|
+
Here is an example of configuration for use with `cron` on Linux.
|
5258
|
+
Adapt the scripts to your own needs.
|
4894
5259
|
|
4895
5260
|
We assume here that a configuration preset was created as shown previously.
|
4896
5261
|
|
4897
|
-
Lets first setup a script that will be used in the scheduler and
|
5262
|
+
Lets first setup a script that will be used in the scheduler and set up the environment.
|
4898
5263
|
|
4899
5264
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
4900
5265
|
|
5266
|
+
<!-- spellchecker: disable -->
|
4901
5267
|
```bash
|
4902
5268
|
#!/bin/bash
|
4903
5269
|
# set a timeout protection, just in case
|
@@ -4906,6 +5272,7 @@ case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
|
4906
5272
|
rvm use 2.6 --quiet
|
4907
5273
|
exec timeout ${tmout} ascli "${@}"
|
4908
5274
|
```
|
5275
|
+
<!-- spellchecker: enable -->
|
4909
5276
|
|
4910
5277
|
Here the cronjob is created for user `xfer`.
|
4911
5278
|
|
@@ -4916,7 +5283,7 @@ crontab<<EOF
|
|
4916
5283
|
EOF
|
4917
5284
|
```
|
4918
5285
|
|
4919
|
-
Note
|
5286
|
+
> **Note:** The logging options are kept here in the cronfile instead of conf file to allow execution on command line with output on command line.
|
4920
5287
|
|
4921
5288
|
### Candidate detection for creation or update (or deletion)
|
4922
5289
|
|
@@ -4997,12 +5364,12 @@ If the `mimemagic` gem complains about missing mime info file:
|
|
4997
5364
|
- any OS:
|
4998
5365
|
|
4999
5366
|
- Examine the error message
|
5000
|
-
- Download the file:
|
5367
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
5001
5368
|
- move and rename this file to one of the locations expected by mimemagic as specified in the error message
|
5002
5369
|
|
5003
5370
|
- Windows:
|
5004
5371
|
|
5005
|
-
- Download the file:
|
5372
|
+
- Download the file: [freedesktop.org.xml.in](https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in)
|
5006
5373
|
- Place this file in the root of Ruby (or elsewhere): `C:\RubyVV-x64\freedesktop.org.xml.in`
|
5007
5374
|
- Set a global variable using `SystemPropertiesAdvanced.exe` or using `cmd` (replace `VV` with version) to the exact path of this file:
|
5008
5375
|
|
@@ -5057,36 +5424,36 @@ preview trevents --once-only=yes --skip-types=office --log-level=info
|
|
5057
5424
|
Aspera CLI can send email, for that setup SMTP configuration. This is done with option `smtp`.
|
5058
5425
|
|
5059
5426
|
The `smtp` option is a hash table (extended value) with the following fields:
|
5060
|
-
|
5061
|
-
|
5062
|
-
|
5063
|
-
|
5064
|
-
|
5065
|
-
<
|
5066
|
-
|
5067
|
-
|
5068
|
-
|
5069
|
-
|
5070
|
-
|
5427
|
+
|
5428
|
+
| field | default | example | description |
|
5429
|
+
|--------------|---------------------|----------------------------|----------------------------------|
|
5430
|
+
| `server` | - | smtp.gmail.com | SMTP server address |
|
5431
|
+
| `tls` | true | false | use of TLS |
|
5432
|
+
| `port` | TLS: 587<br/>25 | 587 | port for service |
|
5433
|
+
| `domain` | domain of server | gmail.com | email domain of user |
|
5434
|
+
| `username` | - | john@example.com | user to authenticate on SMTP server, leave empty for open auth. |
|
5435
|
+
| `password` | - | my_password_here | password for above username |
|
5436
|
+
| `from_email` | username if defined | johnny@example.com | address used if receiver replies |
|
5437
|
+
| `from_name` | same as email | John Wayne | display name of sender |
|
5071
5438
|
|
5072
5439
|
### Example of configuration
|
5073
5440
|
|
5074
5441
|
```bash
|
5075
5442
|
ascli config preset set smtp_google server smtp.google.com
|
5076
5443
|
ascli config preset set smtp_google username john@gmail.com
|
5077
|
-
ascli config preset set smtp_google password
|
5444
|
+
ascli config preset set smtp_google password my_password_here
|
5078
5445
|
```
|
5079
5446
|
|
5080
5447
|
or
|
5081
5448
|
|
5082
|
-
```
|
5083
|
-
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"
|
5449
|
+
```bash
|
5450
|
+
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"my_password_here"}'
|
5084
5451
|
```
|
5085
5452
|
|
5086
5453
|
or
|
5087
5454
|
|
5088
5455
|
```bash
|
5089
|
-
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=
|
5456
|
+
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=my_password_here
|
5090
5457
|
```
|
5091
5458
|
|
5092
5459
|
Set this configuration as global default, for instance:
|
@@ -5104,9 +5471,9 @@ The template is the full SMTP message, including headers.
|
|
5104
5471
|
|
5105
5472
|
The following variables are defined by default:
|
5106
5473
|
|
5107
|
-
- from_name
|
5108
|
-
- from_email
|
5109
|
-
- to
|
5474
|
+
- `from_name`
|
5475
|
+
- `from_email`
|
5476
|
+
- `to`
|
5110
5477
|
|
5111
5478
|
Other variables are defined depending on context.
|
5112
5479
|
|
@@ -5154,7 +5521,7 @@ It aims at simplifying the startup of a FASP session from a programmatic stand p
|
|
5154
5521
|
- common to Aspera Connect API (browser javascript startTransfer)
|
5155
5522
|
- easy to generate by using any third party language specific JSON library
|
5156
5523
|
|
5157
|
-
Hopefully, IBM integrates this
|
5524
|
+
Hopefully, IBM integrates this directly in `ascp`, and this tool is made redundant.
|
5158
5525
|
|
5159
5526
|
This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.
|
5160
5527
|
|
@@ -5162,50 +5529,50 @@ The tool expect one single argument: a [*transfer-spec*](#transferspec).
|
|
5162
5529
|
|
5163
5530
|
If no argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [*transfer-spec*](#transferspec) on stdin.
|
5164
5531
|
|
5165
|
-
Note
|
5532
|
+
> **Note:** If JSON is the format, specify `@json:` to tell `ascli` to decode the hash using JSON syntax.
|
5166
5533
|
|
5167
5534
|
During execution, it generates all low level events, one per line, in JSON format on stdout.
|
5168
5535
|
|
5169
|
-
|
5536
|
+
There are special "extended" [*transfer-spec*](#transferspec) parameters supported by `asession`:
|
5170
5537
|
|
5171
5538
|
- `EX_loglevel` to change log level of the tool
|
5172
5539
|
- `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
5173
5540
|
|
5174
|
-
Note
|
5541
|
+
> **Note:** In addition, many "EX_" [*transfer-spec*](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [*transfer-spec*](#transferspec).
|
5175
5542
|
|
5176
5543
|
### Comparison of interfaces
|
5177
5544
|
|
5178
|
-
|
5179
|
-
|
5180
|
-
|
5181
|
-
|
5182
|
-
|
5183
|
-
|
5184
|
-
|
5545
|
+
| feature/tool | asession | `ascp` | FaspManager | Transfer SDK |
|
5546
|
+
|--------------|----------|--------|-------------|--------------|
|
5547
|
+
| language integration | any | any | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | many |
|
5548
|
+
| required additional components to `ascp` | Ruby<br/>Aspera | - | library<br/>(headers) | daemon |
|
5549
|
+
| startup | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) | command line arguments | API | daemon |
|
5550
|
+
| events | JSON on stdout | none by default<br/>or need to open management port<br/>and proprietary text syntax | callback | callback |
|
5551
|
+
| platforms | any with ruby and `ascp` | any with `ascp` (and SDK if compiled) | any with `ascp` | any with `ascp` and transfer daemon |
|
5185
5552
|
|
5186
5553
|
### Simple session
|
5187
5554
|
|
5188
5555
|
Create a file `session.json` with:
|
5189
5556
|
|
5190
5557
|
```json
|
5191
|
-
{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"
|
5558
|
+
{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"my_password_here","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}
|
5192
5559
|
````
|
5193
5560
|
|
5194
5561
|
Then start the session:
|
5195
5562
|
|
5196
|
-
```
|
5563
|
+
```bash
|
5197
5564
|
asession < session.json
|
5198
5565
|
```
|
5199
5566
|
|
5200
5567
|
### Asynchronous commands and Persistent session
|
5201
5568
|
|
5202
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
5569
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -> `long_parameter`
|
5203
5570
|
|
5204
5571
|
This is particularly useful for a persistent session ( with the [*transfer-spec*](#transferspec) parameter: `"keepalive":true` )
|
5205
5572
|
|
5206
|
-
```
|
5573
|
+
```json
|
5207
5574
|
asession
|
5208
|
-
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"
|
5575
|
+
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"my_password_here","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
5209
5576
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
5210
5577
|
{"type":"DONE"}
|
5211
5578
|
```
|
@@ -5266,27 +5633,28 @@ The general idea is to rely on :
|
|
5266
5633
|
- take advantage of `ascli` configuration capabilities and server side knowledge
|
5267
5634
|
- the OS scheduler for reliability and continuous operation
|
5268
5635
|
|
5269
|
-
#### ascp features
|
5636
|
+
#### `ascp` features
|
5270
5637
|
|
5271
|
-
Interesting ascp features are found in its arguments: (see ascp manual):
|
5638
|
+
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
5272
5639
|
|
5273
|
-
- `ascp` already takes care of sending only
|
5640
|
+
- `ascp` already takes care of sending only **new** files: option `-k 1,2,3` (`resume_policy`)
|
5274
5641
|
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
5275
5642
|
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
5276
5643
|
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
5277
5644
|
|
5278
|
-
Note
|
5279
|
-
|
5280
|
-
|
5281
|
-
|
5282
|
-
|
5645
|
+
> **Note:** `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `--ts` parameter.
|
5646
|
+
>
|
5647
|
+
> **Note:** Most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters.
|
5648
|
+
>
|
5649
|
+
> **Note:** Only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided option `transfer_info` parameter `ascp_args`.
|
5283
5650
|
|
5284
5651
|
#### server side and configuration
|
5285
5652
|
|
5286
5653
|
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
5287
|
-
> file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
5288
5654
|
|
5289
|
-
>
|
5655
|
+
> **Note:** file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
5656
|
+
>
|
5657
|
+
> **Note:** parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
5290
5658
|
|
5291
5659
|
#### Scheduling
|
5292
5660
|
|
@@ -5295,20 +5663,34 @@ Refer to section [Scheduling](#scheduling). (on use of option `lock_port`)
|
|
5295
5663
|
|
5296
5664
|
### Example: upload hot folder
|
5297
5665
|
|
5298
|
-
```
|
5666
|
+
```bash
|
5299
5667
|
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
5300
5668
|
```
|
5301
5669
|
|
5302
5670
|
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
5303
5671
|
Source files are deleted after transfer.
|
5304
|
-
Growing files will be sent only once they don't grow anymore (based on an 8-second
|
5672
|
+
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
5305
5673
|
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
5306
5674
|
|
5307
|
-
### Example: unidirectional synchronization
|
5675
|
+
### Example: unidirectional synchronization (upload) to server
|
5308
5676
|
|
5309
|
-
```
|
5310
|
-
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than:-8,"src_base":"source_sync"}'
|
5677
|
+
```bash
|
5678
|
+
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
|
5311
5679
|
```
|
5680
|
+
|
5681
|
+
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node:
|
5682
|
+
|
5683
|
+
### Example: unidirectional synchronization (download) from Aspera on Cloud Files
|
5684
|
+
|
5685
|
+
```bash
|
5686
|
+
ascli aoc files download . --to-folder=. --lock-port=12345 --progress=none --display=data \
|
5687
|
+
--ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
|
5688
|
+
```
|
5689
|
+
|
5690
|
+
> **Note:** option `delete_before_transfer` will delete files locally, if they are not present on remote side.
|
5691
|
+
>
|
5692
|
+
> **Note:** options `progress` and `display` limit output for headless operation (e.g. cron job)
|
5693
|
+
|
5312
5694
|
## Health check and Nagios
|
5313
5695
|
|
5314
5696
|
Most plugin provide a `health` command that will check the health status of the application. Example:
|
@@ -5342,7 +5724,7 @@ OK - [transfer:ok]
|
|
5342
5724
|
```
|
5343
5725
|
|
5344
5726
|
```bash
|
5345
|
-
ascli server health
|
5727
|
+
ascli server health asctl status --cmd_prefix='sudo ' --format=nagios
|
5346
5728
|
```
|
5347
5729
|
|
5348
5730
|
```output
|
@@ -5384,24 +5766,30 @@ aoc.read('self')
|
|
5384
5766
|
|
5385
5767
|
## Changes (Release notes)
|
5386
5768
|
|
5387
|
-
See
|
5769
|
+
See [CHANGELOG.md](CHANGELOG.md)
|
5388
5770
|
|
5389
5771
|
## History
|
5390
5772
|
|
5391
|
-
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares).
|
5773
|
+
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares).
|
5774
|
+
Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API.
|
5775
|
+
Since all products relate to file transfers using FASP (`ascp`), I thought it would be interesting to have a unified CLI for transfers using FASP.
|
5776
|
+
Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
5392
5777
|
|
5393
5778
|
There were a few pitfalls:
|
5394
5779
|
|
5395
|
-
- The tool was written in the aging `perl` language while most Aspera application products (but the Transfer Server) are written in `ruby`.
|
5780
|
+
- The tool was written in the aging `perl` language while most Aspera web application products (but the Transfer Server) are written in `ruby`.
|
5396
5781
|
- The tool was only for transfers, but not able to call other products APIs
|
5397
5782
|
|
5398
5783
|
So, it evolved into `ascli`:
|
5399
5784
|
|
5400
5785
|
- portable: works on platforms supporting `ruby` (and `ascp`)
|
5401
5786
|
- easy to install with the `gem` utility
|
5402
|
-
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from ascp command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
5787
|
+
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from `ascp` command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
5403
5788
|
- `ruby` is consistent with other Aspera products
|
5404
5789
|
|
5790
|
+
Over the time, a supported command line tool `aspera` was developed in C++, it was later on deprecated.
|
5791
|
+
It had the advantage of being relatively easy to installed, as a single executable (well, still using `ascp`), but it was too limited IMHO, and lacked a lot of the features of this CLI.
|
5792
|
+
|
5405
5793
|
## Common problems
|
5406
5794
|
|
5407
5795
|
### Error: "Remote host is not who we expected"
|
@@ -5410,7 +5798,7 @@ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including EC
|
|
5410
5798
|
|
5411
5799
|
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
5412
5800
|
|
5413
|
-
```
|
5801
|
+
```bash
|
5414
5802
|
--ts=@json:'{"sshfp":null}'
|
5415
5803
|
```
|
5416
5804
|
|
@@ -5422,7 +5810,7 @@ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "C
|
|
5422
5810
|
|
5423
5811
|
Some Ruby gems dependencies require compilation of native parts (C).
|
5424
5812
|
This also requires Ruby header files.
|
5425
|
-
If Ruby was installed as a Linux Packages, then also install ruby
|
5813
|
+
If Ruby was installed as a Linux Packages, then also install ruby development package:
|
5426
5814
|
`ruby-dev` ir `ruby-devel`, depending on distribution.
|
5427
5815
|
|
5428
5816
|
### ED255519 key not supported
|