aspera-cli 4.21.1 → 4.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +1 -1
- data/CHANGELOG.md +52 -22
- data/CONTRIBUTING.md +69 -148
- data/README.md +929 -668
- data/bin/ascli +5 -14
- data/bin/asession +1 -3
- data/examples/get_proto_file.rb +4 -3
- data/examples/proxy.pac +20 -20
- data/lib/aspera/agent/base.rb +11 -5
- data/lib/aspera/agent/connect.rb +30 -28
- data/lib/aspera/agent/{alpha.rb → desktop.rb} +35 -31
- data/lib/aspera/agent/direct.rb +141 -121
- data/lib/aspera/agent/httpgw.rb +22 -26
- data/lib/aspera/agent/node.rb +14 -11
- data/lib/aspera/agent/transferd.rb +30 -19
- data/lib/aspera/api/alee.rb +1 -1
- data/lib/aspera/api/aoc.rb +6 -6
- data/lib/aspera/api/cos_node.rb +2 -2
- data/lib/aspera/api/httpgw.rb +7 -3
- data/lib/aspera/api/node.rb +10 -8
- data/lib/aspera/ascmd.rb +3 -3
- data/lib/aspera/ascp/installation.rb +53 -72
- data/lib/aspera/ascp/management.rb +1 -1
- data/lib/aspera/assert.rb +11 -2
- data/lib/aspera/cli/error.rb +2 -2
- data/lib/aspera/cli/extended_value.rb +46 -21
- data/lib/aspera/cli/formatter.rb +55 -48
- data/lib/aspera/cli/hints.rb +1 -1
- data/lib/aspera/cli/info.rb +1 -0
- data/lib/aspera/cli/main.rb +192 -170
- data/lib/aspera/cli/manager.rb +18 -18
- data/lib/aspera/cli/plugin.rb +23 -20
- data/lib/aspera/cli/plugin_factory.rb +1 -1
- data/lib/aspera/cli/plugins/alee.rb +1 -1
- data/lib/aspera/cli/plugins/aoc.rb +247 -159
- data/lib/aspera/cli/plugins/ats.rb +19 -17
- data/lib/aspera/cli/plugins/config.rb +76 -113
- data/lib/aspera/cli/plugins/console.rb +5 -3
- data/lib/aspera/cli/plugins/faspex.rb +39 -35
- data/lib/aspera/cli/plugins/faspex5.rb +111 -84
- data/lib/aspera/cli/plugins/faspio.rb +13 -1
- data/lib/aspera/cli/plugins/httpgw.rb +13 -1
- data/lib/aspera/cli/plugins/node.rb +312 -182
- data/lib/aspera/cli/plugins/orchestrator.rb +34 -40
- data/lib/aspera/cli/plugins/preview.rb +3 -3
- data/lib/aspera/cli/plugins/server.rb +6 -6
- data/lib/aspera/cli/plugins/shares.rb +5 -5
- data/lib/aspera/cli/sync_actions.rb +19 -18
- data/lib/aspera/cli/transfer_agent.rb +5 -5
- data/lib/aspera/cli/transfer_progress.rb +2 -2
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +116 -95
- data/lib/aspera/coverage.rb +8 -5
- data/lib/aspera/environment.rb +26 -17
- data/lib/aspera/faspex_gw.rb +14 -14
- data/lib/aspera/faspex_postproc.rb +10 -11
- data/lib/aspera/hash_ext.rb +4 -14
- data/lib/aspera/json_rpc.rb +1 -1
- data/lib/aspera/keychain/encrypted_hash.rb +47 -34
- data/lib/aspera/keychain/factory.rb +41 -0
- data/lib/aspera/keychain/hashicorp_vault.rb +71 -0
- data/lib/aspera/keychain/macos_security.rb +19 -11
- data/lib/aspera/log.rb +28 -34
- data/lib/aspera/nagios.rb +6 -6
- data/lib/aspera/node_simulator.rb +8 -8
- data/lib/aspera/oauth/base.rb +14 -7
- data/lib/aspera/oauth/factory.rb +5 -6
- data/lib/aspera/oauth/url_json.rb +6 -6
- data/lib/aspera/persistency_action_once.rb +6 -4
- data/lib/aspera/persistency_folder.rb +2 -2
- data/lib/aspera/preview/generator.rb +13 -10
- data/lib/aspera/preview/options.rb +16 -16
- data/lib/aspera/preview/terminal.rb +4 -4
- data/lib/aspera/preview/utils.rb +15 -17
- data/lib/aspera/products/connect.rb +35 -1
- data/lib/aspera/products/{alpha.rb → desktop.rb} +3 -3
- data/lib/aspera/products/transferd.rb +9 -2
- data/lib/aspera/proxy_auto_config.rb +2 -2
- data/lib/aspera/rest.rb +56 -47
- data/lib/aspera/rest_errors_aspera.rb +1 -1
- data/lib/aspera/secret_hider.rb +12 -5
- data/lib/aspera/ssh.rb +4 -4
- data/lib/aspera/temp_file_manager.rb +5 -4
- data/lib/aspera/transfer/convert.rb +29 -0
- data/lib/aspera/transfer/error_info.rb +66 -66
- data/lib/aspera/transfer/parameters.rb +13 -68
- data/lib/aspera/transfer/spec.rb +5 -6
- data/lib/aspera/transfer/spec.schema.yaml +753 -0
- data/lib/aspera/transfer/spec_doc.rb +62 -0
- data/lib/aspera/transfer/sync.rb +23 -72
- data/lib/aspera/transfer/sync_instance.schema.yaml +13 -0
- data/lib/aspera/transfer/sync_session.schema.yaml +79 -0
- data/lib/aspera/transfer/uri.rb +6 -6
- data/lib/aspera/uri_reader.rb +18 -1
- data/lib/aspera/web_auth.rb +1 -1
- data/lib/aspera/web_server_simple.rb +53 -44
- data.tar.gz.sig +0 -0
- metadata +28 -165
- metadata.gz.sig +0 -0
- data/examples/build_exec +0 -74
- data/examples/build_exec_rubyc +0 -40
- data/examples/build_package.sh +0 -28
- data/lib/aspera/transfer/spec.yaml +0 -718
data/README.md
CHANGED
@@ -1,11 +1,12 @@
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
2
2
|
<!--
|
3
|
-
|
3
|
+
DO NOT EDIT: THIS FILE IS GENERATED, edit docs/README.erb.md, for details, read docs/README.md
|
4
4
|
|
5
5
|
markdownlint-disable MD033 MD003 MD053
|
6
6
|
cspell:ignore Serban Antipolis
|
7
7
|
PANDOC_META_BEGIN
|
8
|
-
subtitle: "ascli 4.
|
8
|
+
subtitle: "ascli 4.22.0.pre"
|
9
|
+
author: "Laurent MARTIN"
|
9
10
|
PANDOC_META_END
|
10
11
|
-->
|
11
12
|
|
@@ -15,13 +16,13 @@ PANDOC_META_END
|
|
15
16
|
|
16
17
|
## Introduction
|
17
18
|
|
18
|
-
Version : 4.
|
19
|
+
Version : 4.22.0.pre
|
19
20
|
|
20
21
|
Laurent/2016-2025
|
21
22
|
|
22
23
|
This gem provides the `ascli` CLI (Command Line Interface) to IBM Aspera software.
|
23
24
|
|
24
|
-
`ascli` is
|
25
|
+
`ascli` is also great tool to learn Aspera APIs.
|
25
26
|
|
26
27
|
Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
|
27
28
|
|
@@ -66,7 +67,7 @@ It is designed for:
|
|
66
67
|
- `curl` (for REST calls)
|
67
68
|
- Aspera transfer (`ascp`)
|
68
69
|
|
69
|
-
If the need is to perform operations programmatically in languages such as: C, Go, Python,
|
70
|
+
If the need is to perform operations programmatically in languages such as: C, Go, Python, NodeJS, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
70
71
|
|
71
72
|
- Product APIs (REST) : e.g. AoC, Faspex, node
|
72
73
|
- Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, Go, Ruby, Rust, etc...)
|
@@ -76,15 +77,18 @@ Code examples here: <https://github.com/laurent-martin/aspera-api-examples>
|
|
76
77
|
|
77
78
|
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
78
79
|
|
80
|
+
`ascli` is developer-friendly, designed for quickly testing APIs and learning how to work with Aspera APIs.
|
81
|
+
Refer to section: [Logging, Debugging](#logging-debugging).
|
82
|
+
|
79
83
|
So, which is Aspera's CLI ? `ascli` or `ascp`
|
80
84
|
|
81
85
|
`ascp` is the low level Aspera command line for **transfers**.
|
82
86
|
It is in fact the implementation of the FASP protocol.
|
83
|
-
So, ANY Aspera transfer leads to one ascp process running on client side and another on server side.
|
84
|
-
`ascp` can be used as a command line, but it is very low level, and practically it can be used on command line only if there is no Aspera web
|
87
|
+
So, ANY Aspera transfer leads to one `ascp` process running on client side and another on server side.
|
88
|
+
`ascp` can be used as a command line, but it is very low level, and practically it can be used on command line only if there is no Aspera web app (AoC, Faspex, etc...) and ONLY to do a transfer (send/receive), not for any operation on Aspera apps (e.g. listing remote files).
|
85
89
|
`ascp` does not provide a configuration file to store credentials or options, it does not resume automatically on transfer error.
|
86
90
|
|
87
|
-
In fact, `ascli` can do everything that `ascp` does, and much more,
|
91
|
+
In fact, `ascli` can do everything that `ascp` does, and much more, in an easier way.
|
88
92
|
|
89
93
|
### Notations, Shell, Examples
|
90
94
|
|
@@ -94,12 +98,12 @@ Command line arguments beginning with `my_` in examples, e.g. `my_param_value`,
|
|
94
98
|
|
95
99
|
`ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, etc...)
|
96
100
|
|
97
|
-
Some commands will start an Aspera
|
101
|
+
Some commands will start an Aspera transfer (e.g. `upload`).
|
98
102
|
The transfer is not directly implemented in `ascli`, rather `ascli` uses one of the external Aspera Transfer Clients called **[Transfer Agents](#transfer-clients-agents)**.
|
99
103
|
|
100
|
-
> **Note:** A **[Transfer
|
101
|
-
|
102
|
-
For example a remote Aspera Server may be used as a transfer agent (using
|
104
|
+
> **Note:** A **[Transfer Agent](#transfer-clients-agents)** is a client for the remote Transfer Server (HSTS).
|
105
|
+
A **[Transfer Agent](#transfer-clients-agents)** can be local or remote...
|
106
|
+
For example a remote Aspera Server may be used as a transfer agent (using Node API).
|
103
107
|
i.e. using option `--transfer=node`
|
104
108
|
|
105
109
|
## Quick Start
|
@@ -112,7 +116,7 @@ Once the gem is installed, `ascli` shall be accessible:
|
|
112
116
|
|
113
117
|
```console
|
114
118
|
$ ascli --version
|
115
|
-
4.
|
119
|
+
4.22.0.pre
|
116
120
|
```
|
117
121
|
|
118
122
|
### First use
|
@@ -121,7 +125,7 @@ Once installation is completed, you can proceed to the first use with a demo ser
|
|
121
125
|
|
122
126
|
If you want to test with Aspera on Cloud, jump to section: [Wizard](#configuration-using-wizard).
|
123
127
|
|
124
|
-
To test with Aspera demo transfer server,
|
128
|
+
To test with Aspera demo transfer server, set up the environment and then test:
|
125
129
|
|
126
130
|
```bash
|
127
131
|
ascli config initdemo
|
@@ -132,20 +136,20 @@ ascli server browse /
|
|
132
136
|
```
|
133
137
|
|
134
138
|
```output
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
139
|
+
╭────────────┬──────┬───────────┬───────┬───────────────────────────┬───────────────────────╮
|
140
|
+
│ zmode │ zuid │ zgid │ size │ mtime │ name │
|
141
|
+
╞════════════╪══════╪═══════════╪═══════╪═══════════════════════════╪═══════════════════════╡
|
142
|
+
│ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-large │
|
143
|
+
│ drwxrwxr-x │ xfer │ demousers │ 94208 │ 2025-03-31 11:27:33 +0200 │ Upload │
|
144
|
+
│ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-small │
|
145
|
+
│ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-tiny │
|
146
|
+
╰────────────┴──────┴───────────┴───────┴───────────────────────────┴───────────────────────╯
|
143
147
|
```
|
144
148
|
|
145
|
-
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define
|
149
|
+
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define an [Option Preset](#option-preset) for the server's authentication options.
|
146
150
|
The following example will:
|
147
151
|
|
148
|
-
- Create
|
152
|
+
- Create an [Option Preset](#option-preset)
|
149
153
|
- Define it as default for the `server` plugin
|
150
154
|
- List files in a folder
|
151
155
|
- Download a file
|
@@ -173,17 +177,16 @@ ascli server browse /aspera-test-dir-large
|
|
173
177
|
```
|
174
178
|
|
175
179
|
```output
|
176
|
-
|
177
|
-
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
+------------+-----------+-----------+--------------+---------------------------+-------+
|
180
|
+
╭────────────┬──────┬───────────┬─────────────┬───────────────────────────┬───────╮
|
181
|
+
│ zmode │ zuid │ zgid │ size │ mtime │ name │
|
182
|
+
╞════════════╪══════╪═══════════╪═════════════╪═══════════════════════════╪═══════╡
|
183
|
+
│ -rw-r--r-- │ xfer │ demousers │ 5368709120 │ 2014-11-05 16:01:56 +0100 │ 5GB │
|
184
|
+
│ -rw-r--r-- │ xfer │ demousers │ 524288000 │ 2014-11-05 16:01:56 +0100 │ 500MB │
|
185
|
+
│ -rw-r--r-- │ xfer │ demousers │ 209715200 │ 2014-11-05 16:01:56 +0100 │ 200MB │
|
186
|
+
│ -rw-r--r-- │ xfer │ demousers │ 1048576000 │ 2014-11-05 16:01:56 +0100 │ 1GB │
|
187
|
+
│ -rw-r--r-- │ xfer │ demousers │ 104857600 │ 2014-11-05 16:01:56 +0100 │ 100MB │
|
188
|
+
│ -rw-r--r-- │ xfer │ demousers │ 10737418240 │ 2014-11-05 16:01:56 +0100 │ 10GB │
|
189
|
+
╰────────────┴──────┴───────────┴─────────────┴───────────────────────────┴───────╯
|
187
190
|
```
|
188
191
|
|
189
192
|
```bash
|
@@ -199,7 +202,7 @@ complete
|
|
199
202
|
|
200
203
|
Get familiar with configuration, options, commands : [Command Line Interface](#command-line-interface).
|
201
204
|
|
202
|
-
Then, follow the section relative to the product you want to interact with (
|
205
|
+
Then, follow the section relative to the product you want to interact with (Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
203
206
|
|
204
207
|
## Installation
|
205
208
|
|
@@ -224,10 +227,34 @@ A package with pre-installed Ruby, gem and `ascp` may also be provided.
|
|
224
227
|
|
225
228
|
### `ascli` executable
|
226
229
|
|
227
|
-
**Note:** This is an Alpha feature. The binary depends on certain GLIBC version for Linux.
|
228
|
-
|
229
230
|
It is planned to provide `ascli` as a single platform-dependent executable.
|
230
|
-
[
|
231
|
+
[Beta releases can be found here](https://ibm.biz/aspera-cli-exe).
|
232
|
+
|
233
|
+
**Note:** This is a Beta feature. On Linux, the executable requires a minimum GLIBC version. Installation of `ascp` is still required separately. Refer to [Install `ascp`](#installation-of-ascp-through-transferd).
|
234
|
+
|
235
|
+
On Linux, check the minimum required GLIBC on this site: [repology.org](https://repology.org/project/glibc/versions), or check your GLIBC version with `ldd`:
|
236
|
+
|
237
|
+
```bash
|
238
|
+
ldd --version | head -n1
|
239
|
+
```
|
240
|
+
|
241
|
+
```console
|
242
|
+
ldd (GNU libc) 2.34
|
243
|
+
```
|
244
|
+
|
245
|
+
Check an executable's (`ascli`, `ascp`) minimum required GLIBC version:
|
246
|
+
|
247
|
+
```bash
|
248
|
+
objdump -p /bin/bash | sed -n 's/^.*GLIBC_//p' | sort -V | tail -n1
|
249
|
+
```
|
250
|
+
|
251
|
+
```console
|
252
|
+
2.34
|
253
|
+
```
|
254
|
+
|
255
|
+
> **Note:** if `objdump` is not available, then use `strings` or `grep -z 'GLIBC_'|tr \\0 \\n`
|
256
|
+
|
257
|
+
The required GLIBC version for `ascp` can be found in the [Release Notes of HSTS](https://www.ibm.com/docs/en/ahts) or [here](https://eudemo.asperademo.com/download/sdk.html).
|
231
258
|
|
232
259
|
### Ruby
|
233
260
|
|
@@ -237,7 +264,7 @@ Required Ruby version: >= 3.1.
|
|
237
264
|
|
238
265
|
> **Deprecation notice**: the minimum Ruby version will be 3.2 in a future version.
|
239
266
|
|
240
|
-
**Ruby can be installed using any method** : rpm
|
267
|
+
**Ruby can be installed using any method** : `rpm`, `yum`, `dnf`, `rvm`, `brew`, Windows installer, ...
|
241
268
|
|
242
269
|
**In priority**, refer to the official Ruby documentation:
|
243
270
|
|
@@ -246,7 +273,7 @@ Required Ruby version: >= 3.1.
|
|
246
273
|
|
247
274
|
For convenience, you may refer to the following sections for a proposed method for specific operating systems.
|
248
275
|
|
249
|
-
Latest version of `ascli` requires a
|
276
|
+
Latest version of `ascli` requires a Ruby version [at least under maintenance support](https://www.ruby-lang.org/en/downloads/branches/).
|
250
277
|
If an older Ruby version is needed, then use an older version of `ascli` that supports it.
|
251
278
|
|
252
279
|
#### Windows: Installer
|
@@ -255,9 +282,9 @@ Manual installation:
|
|
255
282
|
|
256
283
|
- Navigate to [https://rubyinstaller.org/](https://rubyinstaller.org/) → **Downloads**.
|
257
284
|
- Download the latest Ruby installer **"with devkit"**. (`Msys2` is needed to install some native extensions, such as `grpc`)
|
258
|
-
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
259
|
-
- At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and mingw)
|
260
|
-
- Then install the aspera-cli gem and Aspera SDK (see next sections)
|
285
|
+
- Execute the installer which installs by default in: `C:\RubyVV-x64` (`VV` is the version number)
|
286
|
+
- At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and `mingw`)
|
287
|
+
- Then install the `aspera-cli` gem and Aspera SDK (see next sections)
|
261
288
|
|
262
289
|
Automated installation, with internet access:
|
263
290
|
|
@@ -272,18 +299,41 @@ rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\asper
|
|
272
299
|
#### macOS: `brew`
|
273
300
|
|
274
301
|
**macOS** comes with Ruby 2.6.
|
275
|
-
It is an old unsupported version and [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes)
|
302
|
+
It is an old unsupported version and [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes).
|
303
|
+
It will be removed from macOS in the future.
|
276
304
|
Do not use it.
|
277
305
|
|
278
|
-
The recommended way is to
|
306
|
+
The recommended way is to use [Homebrew](https://brew.sh/).
|
279
307
|
|
280
308
|
```bash
|
281
309
|
brew install ruby
|
282
310
|
```
|
283
311
|
|
284
|
-
This installs a recent
|
312
|
+
This installs a recent Ruby suitable for `ascli`.
|
313
|
+
|
314
|
+
To add PATH to Ruby on Apple Silicon, add this in your shell configuration file (e.g. `~/.bash_profile` or `~/.zshrc`):
|
285
315
|
|
286
|
-
|
316
|
+
```bash
|
317
|
+
use_ruby(){
|
318
|
+
version=$1
|
319
|
+
local prefix=$(brew --prefix ruby${version:+@}$version)
|
320
|
+
if ! test -d $prefix;then
|
321
|
+
echo "No such ruby version: $version"
|
322
|
+
brew list|grep ruby
|
323
|
+
return 1
|
324
|
+
fi
|
325
|
+
export PATH="$prefix/bin:$PATH"
|
326
|
+
export PATH="$(gem env gemdir)/bin:$PATH"
|
327
|
+
export LDFLAGS="-L$prefix/lib"
|
328
|
+
export CPPFLAGS="-I$prefix/include"
|
329
|
+
export PKG_CONFIG_PATH="$prefix/lib/pkgconfig"
|
330
|
+
echo "Using: $prefix"
|
331
|
+
ruby -v
|
332
|
+
}
|
333
|
+
use_ruby
|
334
|
+
```
|
335
|
+
|
336
|
+
If using [RVM](https://rvm.io/), one way to force use of OpenSSL 3.0 is:
|
287
337
|
|
288
338
|
```bash
|
289
339
|
RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@3.0)" rvm install 3.4.0
|
@@ -293,7 +343,7 @@ RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@3.0)" rvm instal
|
|
293
343
|
|
294
344
|
If your Linux distribution provides a standard Ruby package, you can use it provided that the version supported.
|
295
345
|
|
296
|
-
**Example:** RHEL 8+, Rocky Linux 8
|
346
|
+
**Example:** RHEL 8+, Rocky Linux 8+: with extensions to compile native gems
|
297
347
|
|
298
348
|
- Check available Ruby versions:
|
299
349
|
|
@@ -348,7 +398,7 @@ Install `rvm`.
|
|
348
398
|
Follow [https://rvm.io/](https://rvm.io/).
|
349
399
|
|
350
400
|
Execute the shell/curl command.
|
351
|
-
As regular user, it installs in the user's home: `~/.rvm
|
401
|
+
As regular user, it installs in the user's home: `~/.rvm`.
|
352
402
|
|
353
403
|
```bash
|
354
404
|
\curl -sSL https://get.rvm.io | bash -s stable
|
@@ -403,7 +453,7 @@ rvm version
|
|
403
453
|
|
404
454
|
#### Linux as non-root
|
405
455
|
|
406
|
-
If you don't have root access, you can install Ruby in your home directory using `rbenv` see [rbenv-installer](https://github.com/rbenv/rbenv-installer#rbenv-installer):
|
456
|
+
If you don't have root access, you can install Ruby in your home directory using `rbenv` see [`rbenv-installer`](https://github.com/rbenv/rbenv-installer#rbenv-installer):
|
407
457
|
|
408
458
|
```bash
|
409
459
|
curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash
|
@@ -456,7 +506,7 @@ make install
|
|
456
506
|
All what is needed is a JVM (Java Virtual Machine) on your system (`java`).
|
457
507
|
The JRuby package comes pre-complied and does not require compilation of native extensions.
|
458
508
|
Use a version of JRuby compatible with Ruby version supported by `ascli`.
|
459
|
-
Refer to [the
|
509
|
+
Refer to [the Wikipedia page](https://en.wikipedia.org/wiki/JRuby) to match JRuby and Ruby versions.
|
460
510
|
Choose the latest version from:
|
461
511
|
|
462
512
|
<https://www.jruby.org/download>
|
@@ -481,22 +531,22 @@ JRUBY_OPTS=--dev ascli -v
|
|
481
531
|
|
482
532
|
#### Optional gems
|
483
533
|
|
484
|
-
Some additional gems are required for some specific features, see [Gemfile
|
534
|
+
Some additional gems are required for some specific features, see [Gemfile](Gemfile):
|
485
535
|
|
486
536
|
| name | version | comment |
|
487
537
|
| ---- | ------- | ------- |
|
488
|
-
| grpc | ~> 1.
|
538
|
+
| grpc | ~> 1.71 | for Aspera Transfer Daemon |
|
489
539
|
| mimemagic | ~> 0.4 | for preview |
|
490
|
-
| rmagick | ~>
|
491
|
-
| symmetric-encryption | ~> 4.6 | for file
|
540
|
+
| rmagick | ~> 6.1 | for terminal view |
|
541
|
+
| symmetric-encryption | ~> 4.6 | for encrypted hash file secrets |
|
492
542
|
| bigdecimal | ~> 3.1.9 | if RUBY_VERSION >= '3.4' for symmetric-encryption ? |
|
493
543
|
|
494
544
|
Install like this:
|
495
545
|
|
496
546
|
```bash
|
497
|
-
gem install grpc -v '~> 1.
|
547
|
+
gem install grpc -v '~> 1.71'
|
498
548
|
gem install mimemagic -v '~> 0.4'
|
499
|
-
gem install rmagick -v '~>
|
549
|
+
gem install rmagick -v '~> 6.1'
|
500
550
|
gem install symmetric-encryption -v '~> 4.6'
|
501
551
|
gem install bigdecimal -v '~> 3.1.9'
|
502
552
|
```
|
@@ -508,7 +558,7 @@ gem install bigdecimal -v '~> 3.1.9'
|
|
508
558
|
Once you have Ruby and rights to install gems, install the `aspera-cli` gem and its dependencies:
|
509
559
|
|
510
560
|
```bash
|
511
|
-
gem install aspera-cli
|
561
|
+
gem install aspera-cli --pre
|
512
562
|
```
|
513
563
|
|
514
564
|
To upgrade to the latest version:
|
@@ -518,7 +568,7 @@ gem update aspera-cli
|
|
518
568
|
```
|
519
569
|
|
520
570
|
During its execution, `ascli` checks every week if a new version is available and notifies the user in a WARN log.
|
521
|
-
To
|
571
|
+
To deactivate this feature, globally set the option `version_check_days` to `0`, or specify a different period in days.
|
522
572
|
|
523
573
|
To check if a new version is available (independently of `version_check_days`):
|
524
574
|
|
@@ -528,18 +578,18 @@ ascli config check_update
|
|
528
578
|
|
529
579
|
#### Gem installation with signature verification
|
530
580
|
|
531
|
-
The gem is signed with a private key, and the public certificate is available in the
|
581
|
+
The gem is signed with a private key, and the public certificate is available in the GitHub repository (`certs/aspera-cli-public-cert.pem`).
|
532
582
|
When installing the gem, the signature can be optionally verified.
|
533
583
|
|
534
584
|
For [secure installation](https://guides.rubygems.org/security/#using-gems), one can install the gem with the public key:
|
535
585
|
|
536
|
-
|
586
|
+
Import the verification certificate:
|
537
587
|
|
538
588
|
```bash
|
539
589
|
gem cert --add <(curl -Ls https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem)
|
540
590
|
```
|
541
591
|
|
542
|
-
|
592
|
+
The user installs the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted:
|
543
593
|
|
544
594
|
```bash
|
545
595
|
gem install -P MediumSecurity aspera-cli
|
@@ -551,7 +601,7 @@ Most file transfers will be executed using the **FASP** protocol, using `ascp`.
|
|
551
601
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
552
602
|
|
553
603
|
- `ascp`
|
554
|
-
- `aspera-license` (in same folder, or
|
604
|
+
- `aspera-license` (in same folder, or `../etc`)
|
555
605
|
|
556
606
|
This can be installed either be installing an Aspera transfer software or using an `ascli` command.
|
557
607
|
|
@@ -561,7 +611,7 @@ The easiest option to install `ascp` is through the use of the IBM Aspera Transf
|
|
561
611
|
|
562
612
|
Supported platforms are listed in the [Release Notes](https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Release+notes) and archives can be downloaded from [Downloads](https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/downloads/downloads.json).
|
563
613
|
|
564
|
-
Install with:
|
614
|
+
Install using `ascli` for the current platform with:
|
565
615
|
|
566
616
|
```bash
|
567
617
|
ascli config ascp install
|
@@ -573,10 +623,16 @@ or
|
|
573
623
|
ascli config transferd install
|
574
624
|
```
|
575
625
|
|
576
|
-
|
577
|
-
|
626
|
+
The installation of the transfer binary follows those steps:
|
627
|
+
|
628
|
+
- Check the value of option `sdk_url`: if the value is the default value `DEF`, then the procedure follows, else it specified a URL where to take the archive from.
|
629
|
+
- The location of archives is retrieved from the URL specified by option `locations_url` whose default value is <https://ibm.biz/sdk_location>
|
630
|
+
- The archive for the current system architecture (CPU and OS) is selected and downloaded.
|
578
631
|
|
579
|
-
|
632
|
+
The option `locations_url` can be set to override the URL where the list of versions is located, in case of air-gap environment or for testing.
|
633
|
+
Option `sdk_url` can be set to specify a direct location for the transfer binaries.
|
634
|
+
|
635
|
+
Available Transfer Daemon versions available from `locations_url` can be listed with: `ascli config transferd list`
|
580
636
|
|
581
637
|
To install a specific version, e.g. 1.1.3:
|
582
638
|
|
@@ -611,7 +667,7 @@ If the embedded method is not used, the following packages are also suitable:
|
|
611
667
|
- IBM Aspera Connect Client (Free)
|
612
668
|
- IBM Aspera Desktop Client (Free)
|
613
669
|
- IBM Aspera High Speed Transfer Server (Licensed)
|
614
|
-
- IBM Aspera High Speed Transfer
|
670
|
+
- IBM Aspera High Speed Transfer Endpoint (Licensed)
|
615
671
|
|
616
672
|
For instance, Aspera Connect Client can be installed by visiting the page:
|
617
673
|
[https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
@@ -629,7 +685,7 @@ Refer to section: [Transfer Agents](#transfer-clients-agents)
|
|
629
685
|
|
630
686
|
#### Gem files and dependencies
|
631
687
|
|
632
|
-
The sample script: [examples/build_package.sh](examples/build_package.sh) can be used to download all necessary gems and dependencies in a tar
|
688
|
+
The sample script: [examples/build_package.sh](examples/build_package.sh) can be used to download all necessary gems and dependencies in a `tar.gz`.
|
633
689
|
|
634
690
|
```console
|
635
691
|
$ ./build_package.sh aspera-cli 4.18.0
|
@@ -651,17 +707,7 @@ The procedure:
|
|
651
707
|
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
652
708
|
```
|
653
709
|
|
654
|
-
-
|
655
|
-
|
656
|
-
```bash
|
657
|
-
ascli --show-config --fields=sdk_url
|
658
|
-
```
|
659
|
-
|
660
|
-
- Download the SDK archive from that URL
|
661
|
-
|
662
|
-
```bash
|
663
|
-
curl -Lso sdk.zip https://ibm.biz/aspera_transfer_sdk
|
664
|
-
```
|
710
|
+
- Download the SDK archive for the selected architecture, follow [Install `ascp`](#installation-of-ascp-through-transferd)
|
665
711
|
|
666
712
|
- Transfer those 2 files to the target system
|
667
713
|
|
@@ -674,10 +720,10 @@ tar zxvf rvm-ascli.tgz
|
|
674
720
|
|
675
721
|
source ~/.rvm/scripts/rvm
|
676
722
|
|
677
|
-
ascli config ascp install --sdk-url=file:///
|
723
|
+
ascli config ascp install --sdk-url=file:///[SDK archive file path]
|
678
724
|
```
|
679
725
|
|
680
|
-
- Add those lines to shell
|
726
|
+
- Add those lines to shell environment (`.profile`)
|
681
727
|
|
682
728
|
```bash
|
683
729
|
source ~/.rvm/scripts/rvm
|
@@ -689,18 +735,18 @@ Installation without network:
|
|
689
735
|
|
690
736
|
It is essentially the same procedure as installation for Windows with internet, but instead of retrieving files from internet, copy the files from a machine with internet access, and then install from those archives:
|
691
737
|
|
692
|
-
- Download the
|
738
|
+
- Download the Ruby installer from <https://rubyinstaller.org/downloads/>
|
693
739
|
|
694
740
|
```bash
|
695
|
-
v=$(curl -s https://rubyinstaller.org/downloads
|
741
|
+
v=$(curl -s https://rubyinstaller.org/downloads/ | sed -nEe 's|.*(https://.*/releases/download/.*exe).*|\1|p' | head -n 1)
|
696
742
|
curl -o ${v##*/} $v
|
697
743
|
```
|
698
744
|
|
699
745
|
- Create an archive with necessary gems like in previous section
|
700
746
|
|
701
|
-
- Download the SDK
|
747
|
+
- Download the SDK following: [Install `ascp`](#installation-of-ascp-through-transferd)
|
702
748
|
|
703
|
-
- Create a Zip with all those files
|
749
|
+
- Create a Zip with all those files and transfer to the target system.
|
704
750
|
|
705
751
|
Then, on the target system:
|
706
752
|
|
@@ -711,7 +757,7 @@ Then, on the target system:
|
|
711
757
|
rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\aspera-cli
|
712
758
|
```
|
713
759
|
|
714
|
-
- Install the gems: Extract the gem archive
|
760
|
+
- Install the gems: Extract the gem archive and then:
|
715
761
|
|
716
762
|
```bat
|
717
763
|
gem install --force --local *.gem
|
@@ -738,13 +784,14 @@ podman --version
|
|
738
784
|
#### Container: Quick start
|
739
785
|
|
740
786
|
**Wanna start quickly ?** With an interactive shell ?
|
787
|
+
|
741
788
|
Execute this:
|
742
789
|
|
743
790
|
```bash
|
744
791
|
podman run --rm --tty --interactive --entrypoint bash docker.io/martinlaurent/ascli:latest
|
745
792
|
```
|
746
793
|
|
747
|
-
> **Note:** This command changes the
|
794
|
+
> **Note:** This command changes the entry point to an interactive shell instead of direct execution of `ascli`.
|
748
795
|
|
749
796
|
Then, execute individual `ascli` commands such as:
|
750
797
|
|
@@ -785,7 +832,7 @@ ascli -v
|
|
785
832
|
```
|
786
833
|
|
787
834
|
```text
|
788
|
-
4.
|
835
|
+
4.22.0.pre
|
789
836
|
```
|
790
837
|
|
791
838
|
In order to keep persistency of configuration on the host, you should specify your user's configuration folder as a volume for the container.
|
@@ -796,7 +843,7 @@ Add options:
|
|
796
843
|
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
797
844
|
```
|
798
845
|
|
799
|
-
> **Note:** If you are using a `podman machine`, e.g. on macOS
|
846
|
+
> **Note:** If you are using a `podman machine`, e.g. on macOS, make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
800
847
|
|
801
848
|
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
802
849
|
you can change the entry point, add option:
|
@@ -840,7 +887,7 @@ Some environment variables can be set for this script to adapt its behavior:
|
|
840
887
|
|----------------|------------------------------------|--------------------------|--------------------------|
|
841
888
|
| `ASCLI_HOME` | Configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascli_config` |
|
842
889
|
| `docker_args` | Additional options to `podman` | <empty> | `--volume /Users:/Users` |
|
843
|
-
| `image` | Container image name | `docker.io/martinlaurent/ascli`
|
890
|
+
| `image` | Container image name | `docker.io/martinlaurent/ascli` | n/a |
|
844
891
|
| `version` | Container image version | Latest | `4.8.0.pre` |
|
845
892
|
|
846
893
|
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
@@ -886,8 +933,8 @@ podman load -i ascli_image_latest.tar.gz
|
|
886
933
|
|
887
934
|
#### Container: `aspera.conf`
|
888
935
|
|
889
|
-
`ascp`'s configuration file `aspera.conf` is located in the container at: `/
|
890
|
-
As the container is immutable, it is not recommended
|
936
|
+
`ascp`'s configuration file `aspera.conf` is located in the container at: `/ibm_aspera/aspera.conf` (see Dockerfile).
|
937
|
+
As the container is immutable, it is not recommended modifying this file.
|
891
938
|
If one wants to change the content, it is possible to tell `ascp` to use another file using `ascp` option `-f`, e.g. by locating it on the host folder `$HOME/.aspera/ascli` mapped to the container folder `/home/cliuser/.aspera/ascli`:
|
892
939
|
|
893
940
|
```bash
|
@@ -932,7 +979,7 @@ singularity shell ascli.sif
|
|
932
979
|
|
933
980
|
`ascli` uses the Ruby `openssl` gem which uses by default the system's `openssl` library and its CA certificate bundle.
|
934
981
|
|
935
|
-
To display the version of **
|
982
|
+
To display the version of **OpenSSL** used in `ascli`:
|
936
983
|
|
937
984
|
```bash
|
938
985
|
ascli config echo @ruby:OpenSSL::OPENSSL_VERSION --format=text
|
@@ -964,7 +1011,7 @@ To display trusted certificate store locations:
|
|
964
1011
|
ascli --show-config --fields=cert_stores
|
965
1012
|
```
|
966
1013
|
|
967
|
-
Certificates are checked against the [Ruby default certificate store](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html) `OpenSSL::X509::DEFAULT_CERT_FILE` and `OpenSSL::X509::DEFAULT_CERT_DIR`, which are typically the ones of `openssl` on Unix-like systems (Linux, macOS, etc
|
1014
|
+
Certificates are checked against the [Ruby default certificate store](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html) `OpenSSL::X509::DEFAULT_CERT_FILE` and `OpenSSL::X509::DEFAULT_CERT_DIR`, which are typically the ones of `openssl` on Unix-like systems (Linux, macOS, etc...).
|
968
1015
|
Ruby's default values can be overridden using env vars: `SSL_CERT_FILE` and `SSL_CERT_DIR`.
|
969
1016
|
|
970
1017
|
One can display those default values:
|
@@ -985,13 +1032,13 @@ update-ca-trust extract
|
|
985
1032
|
```
|
986
1033
|
|
987
1034
|
The SSL CA certificate bundle can be specified using option `cert_stores`, which is a list of files or folders.
|
988
|
-
By default it uses Ruby's default certificate store.
|
1035
|
+
By default, it uses Ruby's default certificate store.
|
989
1036
|
|
990
1037
|
If you use this option, then default locations are not used.
|
991
1038
|
Default locations can be added using special value `DEF`.
|
992
1039
|
The value can be either an `Array` or `String` (path).
|
993
1040
|
Successive options add paths incrementally.
|
994
|
-
All files of a
|
1041
|
+
All files of a folder are added.
|
995
1042
|
|
996
1043
|
JRuby uses its own implementation and CA bundles.
|
997
1044
|
|
@@ -1003,8 +1050,8 @@ For example, on Linux to force the use the system's certificate store:
|
|
1003
1050
|
|
1004
1051
|
`ascp` also needs to validate certificates when using **WSS** for transfer TCP part (instead of **SSH**).
|
1005
1052
|
|
1006
|
-
By default,`ascp` uses
|
1007
|
-
Original `ascp`'s
|
1053
|
+
By default,`ascp` uses a hard coded root location `OPENSSLDIR`.
|
1054
|
+
Original `ascp`'s hard coded locations can be found using:
|
1008
1055
|
|
1009
1056
|
```bash
|
1010
1057
|
ascli config ascp info --fields=openssldir
|
@@ -1012,7 +1059,7 @@ ascli config ascp info --fields=openssldir
|
|
1012
1059
|
|
1013
1060
|
E.g. on macOS: `/Library/Aspera/ssl`.
|
1014
1061
|
Then trusted certificates are taken from `[OPENSSLDIR]/cert.pem` and files in `[OPENSSLDIR]/certs`.
|
1015
|
-
`ascli` overrides the default
|
1062
|
+
`ascli` overrides the default hard coded location used by `ascp` for WSS and uses the same locations as specified in `cert_stores` (using the `-i` option of `ascp`).
|
1016
1063
|
|
1017
1064
|
To update trusted root certificates for `ascli`:
|
1018
1065
|
Display the trusted certificate store locations used by `ascli`.
|
@@ -1062,17 +1109,17 @@ The `aspera-cli` gem provides a command line interface (CLI) which interacts wit
|
|
1062
1109
|
|
1063
1110
|
`ascli` provides the following features:
|
1064
1111
|
|
1065
|
-
-
|
1112
|
+
- Commands to Aspera server products (on-premise and SaaS)
|
1066
1113
|
- Any command line **options** (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files, ...
|
1067
|
-
-
|
1114
|
+
- Commands, Options, and Option values shortcuts
|
1068
1115
|
- FASP [Transfer Agents](#transfer-clients-agents) can be: local `ascp`, or Connect Client, or any transfer node
|
1069
1116
|
- Transfer parameters can be altered by modification of [**transfer-spec**](#transfer-specification), this includes requiring multi-session
|
1070
1117
|
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
1071
|
-
-
|
1072
|
-
-
|
1118
|
+
- faspstream creation (using Node API)
|
1119
|
+
- **Watchfolder** creation (using Node API)
|
1073
1120
|
- Additional command plugins can be written by the user
|
1074
|
-
-
|
1075
|
-
-
|
1121
|
+
- Download of Faspex and Aspera on Cloud "external" links
|
1122
|
+
- **Legacy** SSH based FASP transfers and remote commands (`ascmd`)
|
1076
1123
|
|
1077
1124
|
Basic usage is displayed by executing:
|
1078
1125
|
|
@@ -1095,7 +1142,7 @@ Using `ascli` with plugin `server` for command line gives advantages over `ascp`
|
|
1095
1142
|
- Choice of transfer agents
|
1096
1143
|
- Integrated support of multi-session
|
1097
1144
|
|
1098
|
-
|
1145
|
+
All `ascp` options are supported either through transfer spec parameters (listed with `conf ascp spec`) and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args` in `transfer_info`).
|
1099
1146
|
|
1100
1147
|
### Command line parsing, Special Characters
|
1101
1148
|
|
@@ -1105,10 +1152,10 @@ The way arguments are parsed and provided to `ascli` depend on the Operating Sys
|
|
1105
1152
|
|
1106
1153
|
#### Shell parsing for Unix-like systems: Linux, macOS, AIX
|
1107
1154
|
|
1108
|
-
Linux command line parsing is well
|
1155
|
+
Linux command line parsing is well-defined:
|
1109
1156
|
It is fully documented in the shell's documentation.
|
1110
1157
|
|
1111
|
-
On Unix-like environments, this is typically a POSIX-like shell (bash
|
1158
|
+
On Unix-like environments, this is typically a POSIX-like shell (`bash`, `zsh`, `ksh`, `sh`).
|
1112
1159
|
A c-shell (`csh`, `tcsh`) or other shell can also be used.
|
1113
1160
|
In this environment the shell parses the command line, possibly replacing variables, etc...
|
1114
1161
|
See [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation).
|
@@ -1122,7 +1169,7 @@ Command line parsing first depends on the shell used.
|
|
1122
1169
|
MS Windows command line parsing is not like Unix-like systems simply because Windows does not provide a list of arguments to the executable (Ruby): it provides the whole command line as a single string, but the shell may interpret some special characters.
|
1123
1170
|
|
1124
1171
|
So command line parsing is not handled by the shell (`cmd.exe`), not handled by the operating system, but it is handled by the executable.
|
1125
|
-
Typically, Windows executables use the [
|
1172
|
+
Typically, Windows executables use the [Microsoft library for this parsing](https://learn.microsoft.com/en-us/cpp/cpp/main-function-command-line-args).
|
1126
1173
|
|
1127
1174
|
As far as `ascli` is concerned: the executable is Ruby.
|
1128
1175
|
It has its own parsing algorithm, close to a Linux shell parsing.
|
@@ -1139,7 +1186,7 @@ Hello World
|
|
1139
1186
|
2
|
1140
1187
|
```
|
1141
1188
|
|
1142
|
-
> **Note:** Use `pp` instead of `puts` to display as
|
1189
|
+
> **Note:** Use `pp` instead of `puts` to display as Ruby `Array`.
|
1143
1190
|
|
1144
1191
|
Once the shell has dealt with the command line "special" characters for it, the shell calls Windows' `CreateProcess` with just the whole command line as a single string.
|
1145
1192
|
(Unlike Unix-like systems where the command line is split into arguments by the shell.)
|
@@ -1212,11 +1259,11 @@ PS C:\> ascli config echo --% @json:'{"k":"v","x":"y"}'
|
|
1212
1259
|
PS C:\> ascli config echo @json:'{"""k""":"""v""","""x""":"""y"""}'
|
1213
1260
|
```
|
1214
1261
|
|
1215
|
-
> **Note:** The special
|
1262
|
+
> **Note:** The special Powershell argument `--%` places Powershell in legacy parsing mode.
|
1216
1263
|
|
1217
1264
|
#### Extended Values (JSON, Ruby, ...)
|
1218
1265
|
|
1219
|
-
Some
|
1266
|
+
Some values provided to `ascli` (options, **Command Parameters**) are expected to be [Extended Values](#extended-value-syntax), i.e. not a simple `String`, but a composite structure (`Hash`, `Array`).
|
1220
1267
|
Typically, the `@json:` modifier is used, it expects a [JSON](https://www.json.org/) string.
|
1221
1268
|
JSON itself has some special syntax: for example `"` is used to enclose a `String`.
|
1222
1269
|
|
@@ -1243,8 +1290,8 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
1243
1290
|
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
1244
1291
|
|
1245
1292
|
In the following examples (using a POSIX shell, such as `bash`), several equivalent commands are provided.
|
1246
|
-
For all example, most
|
1247
|
-
It depends on the underlying syntax: shell
|
1293
|
+
For all example, most special character handling is not specific to `ascli`:
|
1294
|
+
It depends on the underlying syntax: shell, JSON, etc...
|
1248
1295
|
Depending on the case, a different `format` option is used to display the actual value.
|
1249
1296
|
|
1250
1297
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
@@ -1294,7 +1341,7 @@ ascli config echo '"'
|
|
1294
1341
|
```
|
1295
1342
|
|
1296
1343
|
Double quote in JSON is a little tricky because `"` is special both for the shell and JSON.
|
1297
|
-
Both shell and JSON syntax allow
|
1344
|
+
Both shell and JSON syntax allow protecting `"`, but only the shell allows protection using single quote.
|
1298
1345
|
|
1299
1346
|
```bash
|
1300
1347
|
ascli config echo @json:'"\""' --format=text
|
@@ -1466,7 +1513,7 @@ The advantages of using a **Command Parameter** instead of an option for the sam
|
|
1466
1513
|
The disadvantage is that it is not possible to define a default value in a configuration file or environment variable using an option value.
|
1467
1514
|
Nevertheless, [Extended Values](#extended-value-syntax) syntax is supported, so it is possible to retrieve a value from the configuration file (using `@preset:`) or environment variable (using `@env:`).
|
1468
1515
|
|
1469
|
-
If a **Command
|
1516
|
+
If a **Command Parameter** begins with `-`, then either use the `@val:` syntax (see [Extended Values](#extended-value-syntax)), or use the `--` separator (see below).
|
1470
1517
|
|
1471
1518
|
A few **Command Parameters** are optional, they are always located at the end of the command line.
|
1472
1519
|
|
@@ -1475,7 +1522,7 @@ A few **Command Parameters** are optional, they are always located at the end of
|
|
1475
1522
|
All options, e.g. `--log-level=debug`, are command line arguments that:
|
1476
1523
|
|
1477
1524
|
- Start with `--`
|
1478
|
-
- Have a name, in lowercase, using `-` as word separator in name
|
1525
|
+
- Have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
|
1479
1526
|
- Have a value, separated from name with a `=`
|
1480
1527
|
- Can be used by prefix (not recommended), provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
|
1481
1528
|
- Is optional on command line (it has a default value or no value)
|
@@ -1499,7 +1546,7 @@ ascli config echo -- --sample
|
|
1499
1546
|
|
1500
1547
|
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
1501
1548
|
|
1502
|
-
Options may have a (
|
1549
|
+
Options may have a (hard coded) default value.
|
1503
1550
|
|
1504
1551
|
Options can be placed anywhere on command line and are evaluated in order.
|
1505
1552
|
Usually the last value evaluated overrides previous values, but some options are cumulative, e.g. `ts`.
|
@@ -1578,7 +1625,7 @@ By default, result of type `single_object` and `object_list` are displayed using
|
|
1578
1625
|
|
1579
1626
|
The table style can be customized with option: `table_style` which expects a `Hash`, options are the ones described in gem [`terminal-table`](https://github.com/tj/terminal-table).
|
1580
1627
|
|
1581
|
-
For example, to display a table with thick
|
1628
|
+
For example, to display a table with thick Unicode borders:
|
1582
1629
|
|
1583
1630
|
```bash
|
1584
1631
|
ascli config preset over --table-style=@ruby:'{border: :unicode_thick_edge}'
|
@@ -1588,13 +1635,13 @@ ascli config preset over --table-style=@ruby:'{border: :unicode_thick_edge}'
|
|
1588
1635
|
|
1589
1636
|
#### Option: `flat_hash`: `.`-join keys
|
1590
1637
|
|
1591
|
-
This
|
1638
|
+
This option controls how object fields are displayed for complex objects.
|
1592
1639
|
|
1593
1640
|
Effective only when `format` is `table` to display `single_object` or `object_list`.
|
1594
1641
|
|
1595
1642
|
If value is `no`, then object's `field` names are only the first level keys of the `Hash` result and values that are `Hash` are displayed as such in Ruby syntax.
|
1596
1643
|
|
1597
|
-
If value is `yes` (default), then object are flattened, i.e. deep `Hash` are transformed into 1-level Hash
|
1644
|
+
If value is `yes` (default), then object are flattened, i.e. deep `Hash` are transformed into 1-level `Hash`, where keys are `.`-junction of deep keys.
|
1598
1645
|
In this case, it is possible to filter fields using the option `fields` using the compound field name using `.` (dot) as separator.
|
1599
1646
|
|
1600
1647
|
Example: Result of command is a list of objects with a single object:
|
@@ -1622,7 +1669,7 @@ This option controls if object fields are displayed as columns or lines.
|
|
1622
1669
|
If value is `no` (default), `object_list` are displayed with one object per line, with fields as columns (see above).
|
1623
1670
|
`single_object` are displayed with one field per line (and columns are: `field`, `value`).
|
1624
1671
|
|
1625
|
-
If
|
1672
|
+
If a `object_list` has a single element, it is possible to have `ascli` display the object as a single object (one field per line instead of columns) with option: `multi_single` set to `single`.
|
1626
1673
|
|
1627
1674
|
This parameter can be set as a global default with:
|
1628
1675
|
|
@@ -1659,11 +1706,9 @@ The option `display` controls the level of output:
|
|
1659
1706
|
|
1660
1707
|
#### Option: `show_secrets`: Hide or show secrets in results
|
1661
1708
|
|
1662
|
-
If value is `no` (default), then secrets are redacted from command results.
|
1663
|
-
|
1664
|
-
If
|
1665
|
-
|
1666
|
-
If `display` is `data`, secrets are included to allows piping results.
|
1709
|
+
- If value is `no` (default), then secrets are redacted from command results.
|
1710
|
+
- If value is `yes`, then secrets shown in clear in results.
|
1711
|
+
- If `display` is `data`, secrets are included to allow piping results.
|
1667
1712
|
|
1668
1713
|
#### Option: `fields`: Selection of output object fields
|
1669
1714
|
|
@@ -1691,7 +1736,7 @@ Examples:
|
|
1691
1736
|
|
1692
1737
|
Table output (`object_list`) can be filtered using option `select`.
|
1693
1738
|
This option is either a `Hash` or `Proc`.
|
1694
|
-
The `Proc` takes as argument a line (`Hash`) in the table and is a Ruby lambda expression that shall
|
1739
|
+
The `Proc` takes as argument a line (`Hash`) in the table and is a Ruby lambda expression that shall return `true` to select or `false` to remove an entry.
|
1695
1740
|
|
1696
1741
|
Example:
|
1697
1742
|
|
@@ -1700,12 +1745,12 @@ ascli aoc admin user list --fields=name,email,ats_admin --query=@json:'{"sort":"
|
|
1700
1745
|
```
|
1701
1746
|
|
1702
1747
|
```output
|
1703
|
-
|
1704
|
-
|
1705
|
-
|
1706
|
-
|
1707
|
-
|
1708
|
-
|
1748
|
+
╭────────────────┬─────────────────────┬───────────╮
|
1749
|
+
│ name │ email │ ats_admin │
|
1750
|
+
╞════════════════╪═════════════════════╪═══════════╡
|
1751
|
+
│ John Curtis │ john@example.com │ true │
|
1752
|
+
│ Laurent Martin │ laurent@example.com │ true │
|
1753
|
+
╰────────────────┴─────────────────────┴───────────╯
|
1709
1754
|
```
|
1710
1755
|
|
1711
1756
|
> **Note:** Option `select` filters elements from the result of command, while the `query` option gives filtering parameters to the API when listing elements.
|
@@ -1740,7 +1785,7 @@ ascli aoc admin user show %name:john
|
|
1740
1785
|
### Extended Value Syntax
|
1741
1786
|
|
1742
1787
|
Most options and arguments are specified by a simple string (e.g. `username` or `url`).
|
1743
|
-
|
1788
|
+
Sometimes it is convenient to read a value from a file: for example read the PEM value of a private key, or a list of files.
|
1744
1789
|
Some options expect a more complex value such as `Hash` or `Array`.
|
1745
1790
|
|
1746
1791
|
The **Extended Value** Syntax allows to specify such values and even read values from other sources than the command line itself.
|
@@ -1751,7 +1796,7 @@ The syntax is:
|
|
1751
1796
|
<0 or more decoders><some text value or nothing>
|
1752
1797
|
```
|
1753
1798
|
|
1754
|
-
Decoders act like a function with its parameter on right
|
1799
|
+
Decoders act like a function with its parameter on right-hand side and are recognized by the prefix: `@` and suffix `:`
|
1755
1800
|
|
1756
1801
|
The following decoders are supported:
|
1757
1802
|
|
@@ -1766,7 +1811,7 @@ The following decoders are supported:
|
|
1766
1811
|
| `list` | `String` | `Array` | Split a string in multiple items taking first character as separator and return an array |
|
1767
1812
|
| `none` | None | Nil | A null value |
|
1768
1813
|
| `path` | `String` | `String` | Performs path expansion on specified path (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml` |
|
1769
|
-
| `preset` | `String` | `Hash` | Get whole option preset value by name. Sub-values can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]` |
|
1814
|
+
| `preset` | `String` | `Hash` | Get whole option preset value by name. Sub-values can also be used, using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]` |
|
1770
1815
|
| `extend` | `String` | `String` | Evaluates embedded extended value syntax in string |
|
1771
1816
|
| `re` | `String` | `Regexp` | Ruby Regular Expression (short for `@ruby:/.../`) |
|
1772
1817
|
| `ruby` | `String` | Any | Execute specified Ruby code |
|
@@ -1776,7 +1821,7 @@ The following decoders are supported:
|
|
1776
1821
|
| `uri` | `String` | `String` | Read value from specified URL, e.g. `--fpac=@uri:http://serv/f.pac` |
|
1777
1822
|
| `val` | `String` | `String` | Prevent decoders on the right to be decoded. e.g. `--key=@val:@file:foo` sets the option `key` to value `@file:foo`. |
|
1778
1823
|
| `yaml` | `String` | Any | Decode YAML |
|
1779
|
-
| `zlib` | `String` | `String` |
|
1824
|
+
| `zlib` | `String` | `String` | Decompress data using zlib |
|
1780
1825
|
|
1781
1826
|
> **Note:** A few commands support a value of type `Proc` (lambda expression).
|
1782
1827
|
For example, the **Extended Value** `@ruby:'->(i){i["attr"]}'` is a lambda expression that returns the value for key `attr` of the `Hash` `i`.
|
@@ -1804,7 +1849,7 @@ Example: Create a `Hash` value with one key and the value is read from a file:
|
|
1804
1849
|
ascli config echo @ruby:'{"token_verification_key"=>File.read("mykey.txt")}'
|
1805
1850
|
```
|
1806
1851
|
|
1807
|
-
Example: read a
|
1852
|
+
Example: read a CSV file and create an `Array` of `Hash` for bulk provisioning:
|
1808
1853
|
|
1809
1854
|
```bash
|
1810
1855
|
cat test.csv
|
@@ -1836,15 +1881,15 @@ ascli config echo @json:@extend:'{"hello":true,"version":"@preset:config.version
|
|
1836
1881
|
```
|
1837
1882
|
|
1838
1883
|
```output
|
1839
|
-
|
1840
|
-
|
1841
|
-
|
1842
|
-
|
1843
|
-
|
1844
|
-
|
1884
|
+
╭─────────┬────────╮
|
1885
|
+
│ field │ value │
|
1886
|
+
╞═════════╪════════╡
|
1887
|
+
│ hello │ true │
|
1888
|
+
│ version │ 4.21.1 │
|
1889
|
+
╰─────────┴────────╯
|
1845
1890
|
```
|
1846
1891
|
|
1847
|
-
Example: Create a `Hash` from YAML provided as shell **
|
1892
|
+
Example: Create a `Hash` from YAML provided as shell **Here document**:
|
1848
1893
|
|
1849
1894
|
```bash
|
1850
1895
|
ascli config echo @yaml:@stdin: --format=json<<EOF
|
@@ -1894,7 +1939,7 @@ C:\Users\Kenji\.aspera\ascli
|
|
1894
1939
|
```
|
1895
1940
|
|
1896
1941
|
When OAuth is used (AoC, Faspex5) `ascli` keeps a cache of generated bearer tokens in folder `persist_store` in configuration folder by default.
|
1897
|
-
Option `cache_tokens` (**yes**/no) allows
|
1942
|
+
Option `cache_tokens` (**yes**/no) allows controlling if OAuth tokens are cached on file system, or generated for each request.
|
1898
1943
|
The command `config tokens flush` clears that cache.
|
1899
1944
|
Tokens are kept on disk for a maximum of 30 minutes (`TOKEN_CACHE_EXPIRY_SEC`) and garbage collected after that.
|
1900
1945
|
When a token has expired, then a new token is generated, either using a refresh_token if it is available, or by the default method.
|
@@ -1920,11 +1965,11 @@ A configuration file provides a way to define default values, especially for aut
|
|
1920
1965
|
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overridden with option `--config-file=path` or its env var).
|
1921
1966
|
|
1922
1967
|
The configuration file is a catalog of named lists of options, called: [Option Preset](#option-preset).
|
1923
|
-
Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of
|
1968
|
+
Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of an [Option Preset](#option-preset) (e.g. `mypreset`) using the option `preset`: `--preset=mypreset` or its shortcut: `-Pmypreset`.
|
1924
1969
|
|
1925
1970
|
#### Option Preset
|
1926
1971
|
|
1927
|
-
|
1972
|
+
An [Option Preset](#option-preset) is a collection of options and their associated values in a named section in the configuration file.
|
1928
1973
|
|
1929
1974
|
A named [Option Preset](#option-preset) can be modified directly using `ascli`, which will update the configuration file :
|
1930
1975
|
|
@@ -1938,9 +1983,9 @@ The command `update` allows the easy creation of [Option Preset](#option-preset)
|
|
1938
1983
|
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here --ts=@json:'{"precalculate_job_size":true}'
|
1939
1984
|
```
|
1940
1985
|
|
1941
|
-
- This creates
|
1986
|
+
- This creates an [Option Preset](#option-preset) `demo_server` with all provided options.
|
1942
1987
|
|
1943
|
-
The command `set` allows setting individual options in
|
1988
|
+
The command `set` allows setting individual options in an [Option Preset](#option-preset).
|
1944
1989
|
|
1945
1990
|
```bash
|
1946
1991
|
ascli config preset set demo_server password my_password_here
|
@@ -1965,7 +2010,7 @@ ascli config preset list
|
|
1965
2010
|
```
|
1966
2011
|
|
1967
2012
|
A good practice is to not manually edit the configuration file and use modification commands instead.
|
1968
|
-
If necessary, the configuration file can opened in a text editor with:
|
2013
|
+
If necessary, the configuration file can be opened in a text editor with:
|
1969
2014
|
|
1970
2015
|
```bash
|
1971
2016
|
ascli config open
|
@@ -1973,7 +2018,7 @@ ascli config open
|
|
1973
2018
|
|
1974
2019
|
> **Note:** This starts the editor specified by env var `EDITOR` if defined.
|
1975
2020
|
|
1976
|
-
|
2021
|
+
The former format for commands is still supported:
|
1977
2022
|
|
1978
2023
|
```bash
|
1979
2024
|
ascli config preset set|delete|show|initialize|update <name>
|
@@ -1981,14 +2026,14 @@ ascli config preset over
|
|
1981
2026
|
ascli config preset list
|
1982
2027
|
```
|
1983
2028
|
|
1984
|
-
It is possible to load
|
2029
|
+
It is possible to load an [Option Preset](#option-preset) from within another [Option Preset](#option-preset) using the `preset` option.
|
1985
2030
|
For example if `pcommon` is a preset with common options, and `pspecific` is a preset with specific options, then `pspecific` can load `pcommon` using:
|
1986
2031
|
|
1987
2032
|
```bash
|
1988
2033
|
ascli config preset set pspecific preset pcommon
|
1989
2034
|
```
|
1990
2035
|
|
1991
|
-
When `pspecific` is loaded, then cumulative option `preset` will be set and it will also load `pcommon`.
|
2036
|
+
When `pspecific` is loaded, then cumulative option `preset` will be set, and it will also load `pcommon`.
|
1992
2037
|
|
1993
2038
|
#### Special Option Preset: `config`
|
1994
2039
|
|
@@ -1996,7 +2041,7 @@ This preset name is reserved and contains a single key: `version`. This is the v
|
|
1996
2041
|
|
1997
2042
|
#### Special Option Preset: `default`
|
1998
2043
|
|
1999
|
-
This preset name is reserved and contains an array of key-value
|
2044
|
+
This preset name is reserved and contains an array of key-value, where the key is the name of a plugin, and the value is the name of another preset.
|
2000
2045
|
|
2001
2046
|
When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
|
2002
2047
|
|
@@ -2068,6 +2113,7 @@ ascp install --sdk-folder=sdk_test_dir
|
|
2068
2113
|
ascp install 1.1.3
|
2069
2114
|
ascp products list
|
2070
2115
|
ascp products use 'IBM Aspera Connect'
|
2116
|
+
ascp schema --format=jsonpp
|
2071
2117
|
ascp show
|
2072
2118
|
ascp spec
|
2073
2119
|
check_update
|
@@ -2173,7 +2219,7 @@ We can see here:
|
|
2173
2219
|
- The [Option Preset](#option-preset) `demo_server` defines some options: the URL and credentials
|
2174
2220
|
- The default [Option Preset](#option-preset) to load in any case is : `cli_default`
|
2175
2221
|
|
2176
|
-
Two [Option
|
2222
|
+
Two [Option Presets](#option-preset) are reserved:
|
2177
2223
|
|
2178
2224
|
- `config` contains a single value: `version` showing the version used to create the configuration file.
|
2179
2225
|
It is used to check compatibility.
|
@@ -2228,7 +2274,7 @@ ascli config preset set cli_default interactive no
|
|
2228
2274
|
ascli config preset set default config cli_default
|
2229
2275
|
```
|
2230
2276
|
|
2231
|
-
|
2277
|
+
An [Option Preset](#option-preset) value can be removed with `unset`:
|
2232
2278
|
|
2233
2279
|
```bash
|
2234
2280
|
ascli config preset unset cli_default interactive
|
@@ -2248,9 +2294,25 @@ ascli -N --preset=@json:'{"url":"_url_here_","password":"my_password_here","user
|
|
2248
2294
|
|
2249
2295
|
#### Wizard
|
2250
2296
|
|
2251
|
-
The wizard is a command that asks the user for information and creates
|
2297
|
+
The wizard is a command that asks the user for information and creates an [Option Preset](#option-preset) with the provided information for a given application.
|
2298
|
+
|
2299
|
+
It takes three optional arguments:
|
2300
|
+
|
2301
|
+
- the URL of the application, else it will ask for it
|
2302
|
+
- the plugin name: it limits detection to a given plugin, else it will try to detect known plugins from the URL
|
2303
|
+
- the preset name: it will create a new [Option Preset](#option-preset) with this name, else it will use specific information to generate a unique preset name.
|
2304
|
+
|
2305
|
+
Special options are also available to the wizard:
|
2252
2306
|
|
2253
|
-
|
2307
|
+
| Option | Value | Description |
|
2308
|
+
|-------------|----------|-------------|
|
2309
|
+
| `default` | [yes]/no | Set as default configuration for specified plugin. |
|
2310
|
+
| `override` | yes/[no] | Override existing default preset name for the plugin, if it exists. |
|
2311
|
+
| `key-path` | path | Path to private key for JWT. |
|
2312
|
+
| `test-mode` | yes/[no] | Skip private key check step. |
|
2313
|
+
|
2314
|
+
Other options can be provided to the wizard, such as `--username`, etc...
|
2315
|
+
They will be added to the [Option Preset](#option-preset) created by the wizard.
|
2254
2316
|
|
2255
2317
|
The simplest invocation is:
|
2256
2318
|
|
@@ -2262,7 +2324,7 @@ ascli config wizard
|
|
2262
2324
|
|
2263
2325
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
2264
2326
|
only username/password and URL are required (either on command line, or from configuration file).
|
2265
|
-
Those can usually
|
2327
|
+
Those can be usually provided on the command line:
|
2266
2328
|
|
2267
2329
|
```bash
|
2268
2330
|
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=my_password_here
|
@@ -2302,7 +2364,7 @@ ascli config preset set default shares shares06
|
|
2302
2364
|
ascli config preset overview
|
2303
2365
|
```
|
2304
2366
|
|
2305
|
-
- Execute a command on the
|
2367
|
+
- Execute a command on the **Shares** application using default options
|
2306
2368
|
|
2307
2369
|
```bash
|
2308
2370
|
ascli shares repo browse /
|
@@ -2310,8 +2372,8 @@ ascli shares repo browse /
|
|
2310
2372
|
|
2311
2373
|
### Secret Vault
|
2312
2374
|
|
2313
|
-
Secrets
|
2314
|
-
|
2375
|
+
Secrets, e.g. passwords, keys, are needed when connecting to applications.
|
2376
|
+
Those secrets are usually provided as command options, on command line, env vars, files etc.
|
2315
2377
|
|
2316
2378
|
For security reasons, those secrets shall not be exposed in clear, either:
|
2317
2379
|
|
@@ -2319,13 +2381,13 @@ For security reasons, those secrets shall not be exposed in clear, either:
|
|
2319
2381
|
- In logs
|
2320
2382
|
- In command output
|
2321
2383
|
|
2322
|
-
Instead, they shall be hidden or encrypted.
|
2384
|
+
Instead, they shall be hidden (logs) or encrypted (configuration).
|
2323
2385
|
|
2324
2386
|
Terminal output secret removal is controlled by option `show_secrets` (default: `no`).
|
2325
2387
|
Log secret removal is controlled by option `log_secrets` (default: `no`).
|
2326
2388
|
Mandatory command line options can be requested interactively (e.g. password) using option `interactive`.
|
2327
2389
|
Or it is possible to use extended value `@secret:[name]` to ask for a secret interactively.
|
2328
|
-
It is also possible to enter an option as an environment variable, e.g. `ASCLI_PASSWORD` for option `password` and read the env var
|
2390
|
+
It is also possible to enter an option as an environment variable, e.g. `ASCLI_PASSWORD` for option `password` and read the env var like this:
|
2329
2391
|
|
2330
2392
|
```bash
|
2331
2393
|
read -s ASCLI_PASSWORD
|
@@ -2345,18 +2407,31 @@ The vault is used with options `vault` and `vault_password`.
|
|
2345
2407
|
`vault_password` specifies the password for the vault.
|
2346
2408
|
|
2347
2409
|
Although it can be specified on command line, for security reason you should avoid exposing the secret.
|
2348
|
-
For example it can be securely specified on command line like this:
|
2410
|
+
For example, it can be securely specified on command line like this:
|
2349
2411
|
|
2350
2412
|
```bash
|
2351
2413
|
read -s ASCLI_VAULT_PASSWORD
|
2352
2414
|
export ASCLI_VAULT_PASSWORD
|
2353
2415
|
```
|
2354
2416
|
|
2355
|
-
#### Vault:
|
2417
|
+
#### Vault: IBM HashiCorp Vault
|
2418
|
+
|
2419
|
+
<https://developer.hashicorp.com/vault>
|
2420
|
+
|
2421
|
+
Quick start macOS:
|
2422
|
+
|
2423
|
+
```bash
|
2424
|
+
gem install vault
|
2425
|
+
brew tap hashicorp/tap
|
2426
|
+
brew install hashicorp/tap/vault
|
2427
|
+
vault server -dev -dev-root-token-id=dev-only-token
|
2428
|
+
```
|
2429
|
+
|
2430
|
+
#### Vault: System keychain
|
2356
2431
|
|
2357
2432
|
> **Note:** **macOS only**
|
2358
2433
|
|
2359
|
-
It is possible to manage secrets in macOS
|
2434
|
+
It is possible to manage secrets in macOS keychain (only read supported currently).
|
2360
2435
|
|
2361
2436
|
```bash
|
2362
2437
|
--vault=@json:'{"type":"system","name":"ascli"}'
|
@@ -2395,7 +2470,7 @@ The lookup is done by comparing the service URL and username (or access key).
|
|
2395
2470
|
|
2396
2471
|
#### Securing passwords and secrets
|
2397
2472
|
|
2398
|
-
A
|
2473
|
+
A password can be saved in clear in an [Option Preset](#option-preset) together with other account information (URL, username, etc...).
|
2399
2474
|
Example:
|
2400
2475
|
|
2401
2476
|
```bash
|
@@ -2499,15 +2574,35 @@ For example: <https://cryptotools.net/rsagen>
|
|
2499
2574
|
|
2500
2575
|
> **Note:** Be careful that private keys are sensitive information, and shall be kept secret (like a password), so using online tools is risky.
|
2501
2576
|
|
2577
|
+
### Web service
|
2578
|
+
|
2579
|
+
Some plugins start a local web server.
|
2580
|
+
This server can server HTTP or HTTPS (with certificate):
|
2581
|
+
|
2582
|
+
The following parameters are supported:
|
2583
|
+
|
2584
|
+
| Parameter | Type | Default | Description |
|
2585
|
+
|-------------------|----------|---------|-----------------------------------------------------|
|
2586
|
+
| `url` | `String` | `http://localhost:8080` | Base URL on which requests are listened, a path can be provided. | <!-- markdownlint-disable-line -->
|
2587
|
+
| `cert` | `String` | - | (HTTPS) Path to certificate file (with ext. `.pfx` or `.p12` for PKCS12) |
|
2588
|
+
| `key` | `String` | - | (HTTPS) Path to private key file (PEM), or passphrase for PKCS12 |
|
2589
|
+
| `chain` | `String` | - | (HTTPS) Path to certificate chain (PEM only) |
|
2590
|
+
|
2591
|
+
Parameter `url` (base URL) defines:
|
2592
|
+
|
2593
|
+
- If `http` or `https` is used
|
2594
|
+
- The local port number (default 443 for HTTPS, 80 for HTTP)
|
2595
|
+
- The **base path**, i.e. the path under which requests are received, if a reverse proxy is used this can be used to route.
|
2596
|
+
|
2502
2597
|
### Image and video thumbnails
|
2503
2598
|
|
2504
2599
|
`ascli` can display thumbnails for images and videos in the terminal.
|
2505
2600
|
This is available:
|
2506
2601
|
|
2507
|
-
-
|
2508
|
-
-
|
2602
|
+
- In the `thumbnail` command of `node` when using **gen4/access key** API.
|
2603
|
+
- When using the `show` command of `preview` plugin.
|
2509
2604
|
- `coffee` and `image` commands of `config` plugin.
|
2510
|
-
-
|
2605
|
+
- Any displayed value which is a URL to image can be displayed with option `format` set to `image`
|
2511
2606
|
|
2512
2607
|
The following options can be specified in the option `image`:
|
2513
2608
|
|
@@ -2537,8 +2632,8 @@ Some actions may require the use of a graphical tool:
|
|
2537
2632
|
- A browser for Aspera on Cloud authentication (web auth method)
|
2538
2633
|
- A text editor for configuration file edition
|
2539
2634
|
|
2540
|
-
By default `ascli` assumes that a graphical environment is available on Windows, and on other systems, rely on the presence of the `DISPLAY` environment variable.
|
2541
|
-
It is also possible to force the graphical mode with option
|
2635
|
+
By default, `ascli` assumes that a graphical environment is available on Windows, and on other systems, rely on the presence of the `DISPLAY` environment variable.
|
2636
|
+
It is also possible to force the graphical mode with option `--ui` :
|
2542
2637
|
|
2543
2638
|
- `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
2544
2639
|
- `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
@@ -2546,10 +2641,10 @@ It is also possible to force the graphical mode with option --ui :
|
|
2546
2641
|
### Logging, Debugging
|
2547
2642
|
|
2548
2643
|
The gem is equipped with traces, mainly for debugging and learning APIs.
|
2549
|
-
By default logging level is `warn` and the output channel is `stderr`.
|
2644
|
+
By default, logging level is `warn` and the output channel is `stderr`.
|
2550
2645
|
To increase debug level, use option `log_level` (e.g. using command line `--log-level=xx`, env var `ASCLI_LOG_LEVEL`, or an [Option Preset](#option-preset)).
|
2551
2646
|
|
2552
|
-
By default passwords and secrets are redacted from logs.
|
2647
|
+
By default, passwords and secrets are redacted from logs.
|
2553
2648
|
Set option `log_secrets` to `yes` to include secrets in logs.
|
2554
2649
|
|
2555
2650
|
Option `logger`: `stdout`, `stderr`, `syslog`.
|
@@ -2588,24 +2683,27 @@ It will display the exact content of HTTP requests and responses.
|
|
2588
2683
|
To ignore SSL certificate for **any** address/port, use option: `insecure`, i.e. `--insecure=yes`.
|
2589
2684
|
To ignore SSL certificate for a list of specific address/port, use option `ignore_certificate`, set to an `Array` of URL for which certificate will be ignored (only the address and port are matched), e.g. `--ignore-certificate=@list:,https://127.0.0.1:9092`
|
2590
2685
|
|
2591
|
-
> **Note:** Ignoring certificate also applies to `ascp`
|
2686
|
+
> **Note:** Ignoring certificate also applies to `ascp` WSS.
|
2592
2687
|
|
2593
2688
|
Ignoring a certificate is not recommended, it is better to add the certificate to the trusted store.
|
2594
2689
|
So, a warning is displayed when a certificate is ignored.
|
2595
2690
|
To disable the warning, use option `silent_insecure` set to `no`.
|
2596
2691
|
|
2597
|
-
HTTP connection parameters (not `ascp`
|
2598
|
-
|
2599
|
-
| Parameter | Default
|
2600
|
-
|
2601
|
-
| `read_timeout` | 60 |
|
2602
|
-
| `write_timeout` | 60 |
|
2603
|
-
| `open_timeout` | 60 |
|
2604
|
-
| `keep_alive_timeout` | 2 |
|
2605
|
-
| `user_agent` | `ascli`
|
2606
|
-
| `download_partial_suffix` |
|
2607
|
-
| `retry_on_error` |
|
2608
|
-
| `
|
2692
|
+
HTTP connection parameters (not `ascp` WSS) can be adjusted using option `http_options`:
|
2693
|
+
|
2694
|
+
| Parameter | Type | Default | Handler |
|
2695
|
+
|---------------------------|------|-----------------|-----------|
|
2696
|
+
| `read_timeout` | int | `60` | Ruby |
|
2697
|
+
| `write_timeout` | int | `60` | Ruby |
|
2698
|
+
| `open_timeout` | int | `60` | Ruby |
|
2699
|
+
| `keep_alive_timeout` | int | `2` | Ruby |
|
2700
|
+
| `user_agent` | int | `ascli` | `ascli` |
|
2701
|
+
| `download_partial_suffix` | int | `.http_partial` | `ascli` |
|
2702
|
+
| `retry_on_error` | bool | `false` | `ascli` |
|
2703
|
+
| `retry_on_timeout` | bool | `true` | `ascli` |
|
2704
|
+
| `retry_on_unavailable` | bool | `true` | `ascli` |
|
2705
|
+
| `retry_max` | int | `1` | `ascli` |
|
2706
|
+
| `retry_sleep` | int | `4` | `ascli` |
|
2609
2707
|
|
2610
2708
|
Time values are in set **seconds** and can be of type either `Integer` or `Float`.
|
2611
2709
|
Default values are the ones of Ruby:
|
@@ -2634,7 +2732,7 @@ Refer to the following sections.
|
|
2634
2732
|
REST API calls and transfers based on HTTP Gateway both use Ruby's `Net::HTTP` class.
|
2635
2733
|
Refer to [Ruby find proxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
2636
2734
|
|
2637
|
-
When Ruby HTTP is used, there are two possibilities to define an HTTP proxy to be used
|
2735
|
+
When Ruby HTTP is used, there are two possibilities to define an HTTP proxy to be used.
|
2638
2736
|
|
2639
2737
|
The `http_proxy` environment variable (**lower case**) can be set to the **URL** of the proxy (with optional credentials).
|
2640
2738
|
Syntax is: `(http|https)://[user:password@]host:port`.
|
@@ -2654,7 +2752,7 @@ Option `http_proxy` does the same (set env var) but on command line:
|
|
2654
2752
|
ascli --http-proxy=http://username:password@host:port ...
|
2655
2753
|
```
|
2656
2754
|
|
2657
|
-
Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config)
|
2755
|
+
Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) JavaScript value.
|
2658
2756
|
|
2659
2757
|
Note that proxy credentials are not supported in PAC files.
|
2660
2758
|
|
@@ -2726,7 +2824,7 @@ Or, alternatively, (prefer transfer spec like above, generally):
|
|
2726
2824
|
|
2727
2825
|
`ascli` uses one of the transfer agents to execute transfers.
|
2728
2826
|
|
2729
|
-
By default it uses the `direct` agent, which is basically a local `ascp`.
|
2827
|
+
By default, it uses the `direct` agent, which is basically a local `ascp`.
|
2730
2828
|
Nevertheless, `ascli` does not come with `ascp` installed.
|
2731
2829
|
This is the reason why it is advised to install the Aspera Transfer SDK during installation (`ascli config ascp install`).
|
2732
2830
|
|
@@ -2737,11 +2835,11 @@ The way to specify the location of `ascp` is to use either options:
|
|
2737
2835
|
- `ascp_path`
|
2738
2836
|
- `use_product`
|
2739
2837
|
|
2740
|
-
The `config` plugin allows
|
2741
|
-
It provides the following commands for `ascp`
|
2838
|
+
The `config` plugin allows finding and specifying the location of `ascp`.
|
2839
|
+
It provides the following commands for `ascp` sub-command:
|
2742
2840
|
|
2743
2841
|
- `show` : shows the path of `ascp` used
|
2744
|
-
- `use` : specify the ascp path to use
|
2842
|
+
- `use` : specify the `ascp` path to use
|
2745
2843
|
- `products` : list Aspera transfer products available locally
|
2746
2844
|
- `connect` : list and download connect client versions available on internet
|
2747
2845
|
|
@@ -2760,11 +2858,12 @@ ascli config ascp info
|
|
2760
2858
|
```
|
2761
2859
|
|
2762
2860
|
```output
|
2763
|
-
|
2764
|
-
|
2765
|
-
|
2766
|
-
|
2861
|
+
╭─────────┬──────────────────────────────────────────────────╮
|
2862
|
+
│ field │ value │
|
2863
|
+
╞═════════╪══════════════════════════════════════════════════╡
|
2864
|
+
│ ascp │ /Users/john/.aspera/sdk/ascp │
|
2767
2865
|
...
|
2866
|
+
╰─────────┴──────────────────────────────────────────────────╯
|
2768
2867
|
```
|
2769
2868
|
|
2770
2869
|
#### Selection of `ascp` location for [`direct`](#agent-direct) agent
|
@@ -2877,7 +2976,7 @@ Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
|
2877
2976
|
|
2878
2977
|
### Transfer Clients: Agents
|
2879
2978
|
|
2880
|
-
Some
|
2979
|
+
Some actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
2881
2980
|
Transfers will be executed by a transfer client, here called **Transfer Agent**.
|
2882
2981
|
|
2883
2982
|
The following agents are supported and selected with option `transfer`:
|
@@ -2887,14 +2986,14 @@ The following agents are supported and selected with option `transfer`:
|
|
2887
2986
|
| [`direct`](#agent-direct) | local | direct execution of `ascp` |
|
2888
2987
|
| [`transferd`](#agent-transfer-sdk) | local | Aspera Transfer Daemon |
|
2889
2988
|
| [`connect`](#agent-connect-client) | local | Aspera Connect Client |
|
2890
|
-
| [`
|
2989
|
+
| [`desktop`](#agent-desktop-client) | local | Aspera for Desktop |
|
2891
2990
|
| [`node`](#agent-node-api) | remote | Aspera Transfer Node |
|
2892
2991
|
| [`httpgw`](#agent-http-gateway) | remote | Aspera HTTP Gateway |
|
2893
2992
|
|
2894
2993
|
> **Note:** All transfer operations are seen from the point of view of the agent.
|
2895
2994
|
For example, an agent executing an **upload**, or **package send** operation will effectively push files to the related server from the system where the agent runs.
|
2896
2995
|
|
2897
|
-
All
|
2996
|
+
All above agents (including `direct`) receive transfer parameters as a [**transfer-spec**](#transfer-specification).
|
2898
2997
|
Parameters in transfer-spec can be modified with option `ts`.
|
2899
2998
|
|
2900
2999
|
**Specific** options for agents are provided with option `transfer_info`.
|
@@ -2912,11 +3011,11 @@ The `transfer_info` option accepts the following optional parameters to control
|
|
2912
3011
|
|
2913
3012
|
| Name | Type | Description |
|
2914
3013
|
|------------------------|---------|-------------|
|
2915
|
-
| `wss` | `Bool` | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
3014
|
+
| `wss` | `Bool` | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: `true` |
|
2916
3015
|
| `quiet` | `Bool` | If `true`, then `ascp` progress bar is not shown.<br/>Default: `false` |
|
2917
3016
|
| `trusted_certs` | `Array` | List of repositories for trusted certificates. |
|
2918
3017
|
| `client_ssh_key` | `String` | SSH Keys to use for token-based transfers. One of: `dsa_rsa`, `rsa`, `per_client`. Default: `rsa` |
|
2919
|
-
| `ascp_args` | `Array` | Array of strings with native `ascp` arguments.<br/>Default: `[]` |
|
3018
|
+
| `ascp_args` | `Array` | `Array` of strings with native `ascp` arguments.<br/>Default: `[]` |
|
2920
3019
|
| `spawn_timeout_sec` | `Float` | Multi session<br/>Verification time that `ascp` is running<br/>Default: `3` |
|
2921
3020
|
| `spawn_delay_sec` | `Float` | Multi session<br/>Delay between startup of sessions<br/>Default: `2` |
|
2922
3021
|
| `multi_incr_udp` | `Bool` | Multi Session<br/>Increment UDP port on multi-session<br/>If `true`, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: `true` on Windows, else `false` |
|
@@ -2925,14 +3024,20 @@ The `transfer_info` option accepts the following optional parameters to control
|
|
2925
3024
|
| `resume.sleep_initial` | `Integer` | First Sleep before retry<br/>Default: `2` |
|
2926
3025
|
| `resume.sleep_factor` | `Integer` | Multiplier of sleep period between attempts<br/>Default: `2` |
|
2927
3026
|
| `resume.sleep_max` | `Integer` | Default: `60` |
|
3027
|
+
| `monitor` | `Bool` | Use management port.<br/>Default: `true` |
|
2928
3028
|
|
2929
3029
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
2930
3030
|
Sleep between iterations is given by the following formula where `iter_index` is the current iteration index, starting at 0:
|
2931
3031
|
|
2932
3032
|
```bash
|
2933
|
-
max( sleep_max
|
3033
|
+
max( sleep_max, sleep_initial * sleep_factor ^ iter_index )
|
2934
3034
|
```
|
2935
3035
|
|
3036
|
+
To display the native progress bar of `ascp`, use `--progress-bar=no --transfer-info=@json:'{"quiet":false}'`.
|
3037
|
+
|
3038
|
+
To skip usage of management port (which disables custom progress bar), set option `monitor` to `false`.
|
3039
|
+
In that, use the native progress bar: `--transfer-info=@json:'{"monitor":false,"quiet":false}'`
|
3040
|
+
|
2936
3041
|
By default, Ruby's root CA store is used to validate any HTTPS endpoint used by `ascp` (e.g. WSS).
|
2937
3042
|
In order to use a custom certificate store, use the `trusted_certs` option of direct agent's option `transfer_info`.
|
2938
3043
|
To use `ascp`'s default, use option: `--transfer-info=@json:'{"trusted_certs":null}'`.
|
@@ -3046,7 +3151,7 @@ There are no option for `transfer_info`.
|
|
3046
3151
|
|
3047
3152
|
#### Agent: Desktop Client
|
3048
3153
|
|
3049
|
-
By specifying option: `--transfer=
|
3154
|
+
By specifying option: `--transfer=desktop`, `ascli` will start transfers using the locally installed **IBM Aspera Desktop Client**.
|
3050
3155
|
There are no option for `transfer_info`.
|
3051
3156
|
|
3052
3157
|
#### Agent: Node API
|
@@ -3056,12 +3161,12 @@ This is especially useful for direct node-to-node transfers.
|
|
3056
3161
|
|
3057
3162
|
Parameters provided in option `transfer_info` are:
|
3058
3163
|
|
3059
|
-
|
|
3060
|
-
|
3061
|
-
| `url` | `String` | URL of the
|
3062
|
-
| `username` | `String` | Node
|
3063
|
-
| `password` | `String` | Password, secret or bearer token
|
3064
|
-
| `root_id` | `String` | Root file
|
3164
|
+
| Parameter | Type | Description |
|
3165
|
+
|------------|----------|----------------------------------------------------|
|
3166
|
+
| `url` | `String` | URL of the Node API<br/>Mandatory |
|
3167
|
+
| `username` | `String` | Node API user or access key<br/>Mandatory |
|
3168
|
+
| `password` | `String` | Password, secret or bearer token<br/>Mandatory |
|
3169
|
+
| `root_id` | `String` | Root file ID<br/>Mandatory only for bearer token |
|
3065
3170
|
|
3066
3171
|
Like any other option, `transfer_info` can get its value from a pre-configured [Option Preset](#option-preset) :
|
3067
3172
|
|
@@ -3077,8 +3182,8 @@ or be specified using the extended value syntax :
|
|
3077
3182
|
|
3078
3183
|
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
3079
3184
|
|
3080
|
-
If the `password` value begins with `Bearer` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the file
|
3081
|
-
It can be either the access key's root file
|
3185
|
+
If the `password` value begins with `Bearer` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the file ID of the top folder to use on the node using this access key.
|
3186
|
+
It can be either the access key's root file ID, or any authorized file ID underneath it.
|
3082
3187
|
|
3083
3188
|
#### Agent: HTTP Gateway
|
3084
3189
|
|
@@ -3088,12 +3193,12 @@ By specifying option: `--transfer=httpgw`, `ascli` will start transfers using th
|
|
3088
3193
|
|
3089
3194
|
Parameters provided in option `transfer_info` are:
|
3090
3195
|
|
3091
|
-
| Name
|
3092
|
-
|
3093
|
-
| url
|
3094
|
-
| upload_chunk_size
|
3095
|
-
| api_version
|
3096
|
-
| synchronous
|
3196
|
+
| Name | Type | Description |
|
3197
|
+
|---------------------|-----------|---------------------------------------|
|
3198
|
+
| `url` | `String` | URL of the HTTP GW<br/>Mandatory |
|
3199
|
+
| `upload_chunk_size` | `Integer` | Size in bytes of chunks for upload<br/>Default: `64000` |
|
3200
|
+
| `api_version` | `String` | Force use of version (`v1`, `v2`)<br/>Default: `v2` |
|
3201
|
+
| `synchronous` | `Bool` | Wait for each message acknowledgment<br/>Default: `false` |
|
3097
3202
|
|
3098
3203
|
Example:
|
3099
3204
|
|
@@ -3112,11 +3217,11 @@ Set option `transfer` to `transferd`.
|
|
3112
3217
|
|
3113
3218
|
Options for `transfer_info` are:
|
3114
3219
|
|
3115
|
-
| Name | Type
|
3116
|
-
|
3117
|
-
| `url`
|
3118
|
-
| `
|
3119
|
-
| `
|
3220
|
+
| Name | Type | Description |
|
3221
|
+
|----------|----------|-------------|
|
3222
|
+
| `url` | `String` | IP address and port listened by the daemon<br/>Mandatory<br/>Default: `:0` |
|
3223
|
+
| `start` | `Bool` | Start a new daemon.<br/>Default: `true` |
|
3224
|
+
| `stop` | `Bool` | Stop daemon when exiting `ascli`<br/>Default: `true` |
|
3120
3225
|
|
3121
3226
|
> **Note:** If port zero is specified in the URL, then the daemon will listen on a random available port. If no address is specified, then `127.0.0.1` is used.
|
3122
3227
|
|
@@ -3133,11 +3238,11 @@ On Windows the compilation may fail for various reasons (3.1.1):
|
|
3133
3238
|
|
3134
3239
|
- `cannot find -lx64-ucrt-ruby310`
|
3135
3240
|
|
3136
|
-
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
3241
|
+
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the `dll` extension)
|
3137
3242
|
|
3138
3243
|
- `conflicting types for 'gettimeofday'`
|
3139
3244
|
|
3140
|
-
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)
|
3245
|
+
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)`, i.e. change `struct timezone` to `void`
|
3141
3246
|
|
3142
3247
|
<!-- spellchecker: enable -->
|
3143
3248
|
|
@@ -3147,13 +3252,13 @@ Some commands lead to file transfer (upload/download).
|
|
3147
3252
|
All parameters necessary for this transfer are described in a [**transfer-spec**](#transfer-specification) (Transfer Specification), such as:
|
3148
3253
|
|
3149
3254
|
- Server address
|
3150
|
-
- Transfer
|
3255
|
+
- Transfer username
|
3151
3256
|
- Credentials
|
3152
3257
|
- File list
|
3153
3258
|
- Etc...
|
3154
3259
|
|
3155
3260
|
`ascli` builds the [**transfer-spec**](#transfer-specification) internally as a `Hash`.
|
3156
|
-
It is not necessary to provide additional parameters on the command line for
|
3261
|
+
It is not necessary to provide additional parameters on the command line for a transfer.
|
3157
3262
|
|
3158
3263
|
It is possible to modify or add any of the supported [**transfer-spec**](#transfer-specification) parameter using the `ts` option.
|
3159
3264
|
The `ts` option accepts a [`Hash` Extended Value](#extended-value-syntax) containing one or several [**transfer-spec**](#transfer-specification) parameters.
|
@@ -3174,7 +3279,7 @@ The use of a [**transfer-spec**](#transfer-specification) instead of `ascp` comm
|
|
3174
3279
|
### Transfer Parameters
|
3175
3280
|
|
3176
3281
|
All standard [**transfer-spec**](#transfer-specification) parameters can be specified.
|
3177
|
-
[**transfer-spec**](#transfer-specification) can also be saved/overridden in the configuration file.
|
3282
|
+
A [**transfer-spec**](#transfer-specification) can also be saved/overridden in the configuration file.
|
3178
3283
|
|
3179
3284
|
References:
|
3180
3285
|
|
@@ -3189,102 +3294,126 @@ ascli config ascp spec
|
|
3189
3294
|
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
3190
3295
|
```
|
3191
3296
|
|
3192
|
-
|
3297
|
+
A JSON Schema can be generated with command:
|
3298
|
+
|
3299
|
+
```bash
|
3300
|
+
ascli config ascp schema --format=jsonpp
|
3301
|
+
```
|
3302
|
+
|
3303
|
+
An optional parameter can be specified to display the schema for a specific transfer agent:
|
3193
3304
|
|
3194
|
-
|
3195
|
-
|
3196
|
-
|
3197
|
-
- T=Transfer SDK
|
3198
|
-
- H=HTTP Gateway
|
3305
|
+
```bash
|
3306
|
+
ascli config ascp schema transferd --format=jsonpp
|
3307
|
+
```
|
3199
3308
|
|
3200
3309
|
`ascp` argument or environment variable is provided in description.
|
3201
3310
|
|
3202
|
-
|
|
3311
|
+
| ID | Name |
|
3312
|
+
| -- | ---- |
|
3313
|
+
| A | Direct |
|
3314
|
+
| C | Connect |
|
3315
|
+
| D | Desktop |
|
3316
|
+
| H | Httpgw |
|
3317
|
+
| N | Node |
|
3318
|
+
| T | Transferd |
|
3319
|
+
|
3320
|
+
| Field | Type | A | C | D | H | N | T | Description |
|
3203
3321
|
| ----- | ---- | - | - | - | - | - | - | ----------- |
|
3204
|
-
| apply_local_docroot |
|
3205
|
-
| authentication | string | | | | | |
|
3206
|
-
| cipher | string | Y | Y | Y | Y | Y | Y | In transit encryption
|
3207
|
-
| cipher_allowed | string |
|
3208
|
-
| compression |
|
3209
|
-
| content_protection | string | Y | Y | Y | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt
|
3322
|
+
| apply_local_docroot | boolean | Y | | | | | Y | Apply local docroot to source paths.<br/>(--apply-local-docroot) |
|
3323
|
+
| authentication | string | | Y | | | | | value=token for SSH bypass keys, else password asked if not provided. |
|
3324
|
+
| cipher | string | Y | Y | Y | Y | Y | Y | In transit encryption algorithms.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum}) |
|
3325
|
+
| cipher_allowed | string | | Y | | | | | returned by node API. Valid literals include "aes-128" and "none". |
|
3326
|
+
| compression | integer | | | | | | | ascp4 only, 0 / 1? |
|
3327
|
+
| content_protection | string | Y | Y | Y | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt={enum}) |
|
3210
3328
|
| content_protection_password | string | Y | Y | Y | Y | Y | Y | Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS) |
|
3211
3329
|
| cookie | string | Y | Y | Y | Y | Y | Y | Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE) |
|
3212
|
-
| create_dir |
|
3213
|
-
| delete_before_transfer |
|
3214
|
-
| delete_source |
|
3330
|
+
| create_dir | boolean | Y | Y | Y | Y | Y | Y | Specifies whether to create new directories.<br/>(-d) |
|
3331
|
+
| delete_before_transfer | boolean | Y | Y | Y | Y | Y | Y | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects on the source are not deleted.<br/>(--delete-before-transfer) |
|
3332
|
+
| delete_source | boolean | Y | | | | Y | Y | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
3215
3333
|
| destination_root | string | Y | Y | Y | Y | Y | Y | Destination root directory.<br/>(<special>) |
|
3216
|
-
| destination_root_id | string | | | | | |
|
3217
|
-
| dgram_size |
|
3218
|
-
| direction | string | Y | Y | Y | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode
|
3219
|
-
| exclude_newer_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are newer than the specified number of seconds added to the source computer's epoch
|
3220
|
-
| exclude_older_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are older than the specified number of seconds added to the source computer's epoch
|
3221
|
-
|
|
3222
|
-
|
|
3223
|
-
|
|
3224
|
-
|
|
3225
|
-
|
|
3226
|
-
|
|
3227
|
-
|
|
3228
|
-
|
|
3229
|
-
|
|
3230
|
-
|
|
3231
|
-
|
|
3232
|
-
|
|
3233
|
-
|
|
3234
|
-
|
|
3235
|
-
|
|
3236
|
-
|
|
3237
|
-
|
|
3238
|
-
|
|
3239
|
-
|
|
3240
|
-
|
|
3334
|
+
| destination_root_id | string | | | | | | Y | The file ID of the destination root directory.<br/>Required when using Bearer token auth for the destination node. |
|
3335
|
+
| dgram_size | integer | Y | Y | Y | Y | Y | Y | UDP datagram size in bytes<br/>(-Z {integer}) |
|
3336
|
+
| direction | string | Y | Y | Y | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode=(conversion){enum}) |
|
3337
|
+
| exclude_newer_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are newer than the specified number of seconds added to the source computer's epoch.<br/>e.g. "-86400" for newer than a day back.<br/>(--exclude-newer-than={string}) |
|
3338
|
+
| exclude_older_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are older than the specified number of seconds added to the source computer's epoch.<br/>e.g. "-86400" for older than a day back.<br/>(--exclude-older-than={string}) |
|
3339
|
+
| fail_bad_filepass | boolean | Y | | | | | Y | Fail on bad file decryption passphrase.<br/>(--fail-bad-filepass) |
|
3340
|
+
| fasp_port | integer | Y | Y | Y | Y | Y | Y | Specifies fasp (UDP) port.<br/>(-O {integer}) |
|
3341
|
+
| fasp_proxy | object | | | | | | Y | Proxy for communications between the remote server and the (local) client. |
|
3342
|
+
| fasp_url | string | | | | | | | Only used in Faspex. |
|
3343
|
+
| file_checksum | string | Y | | | | Y | | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none<br/>(--file-checksum={enum}) |
|
3344
|
+
| http_fallback | boolean<br/>string | Y | Y | Y | Y | Y | Y | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){boolean\|string}) |
|
3345
|
+
| http_fallback_port | integer | Y | Y | Y | Y | Y | Y | Specifies http port when no cipher is used<br/>(-t {integer}) |
|
3346
|
+
| https_fallback_port | integer | Y | Y | Y | Y | Y | Y | Specifies https port when cipher is used<br/>(-t {integer}) |
|
3347
|
+
| icos | object | | | | | | Y | Configuration parameters for IBM Cloud Object Storage (ICOS). |
|
3348
|
+
| keepalive | boolean | Y | | | | | Y | The session is running in persistent session mode.<br/>(--keepalive) |
|
3349
|
+
| lock_min_rate | boolean | | Y | | | | | TODO: remove ? |
|
3350
|
+
| lock_min_rate_kbps | boolean | | Y | | | | Y | If true, lock the minimum transfer rate to the value set for min_rate_kbps.<br/>If false, users can adjust the transfer rate up to the value set for target_rate_cap_kbps. |
|
3351
|
+
| lock_rate_policy | boolean | | Y | | | | Y | If true, lock the rate policy to the default value. |
|
3352
|
+
| lock_target_rate | boolean | | Y | | | | | TODO: remove ? |
|
3353
|
+
| lock_target_rate_kbps | boolean | | Y | | | | Y | If true, lock the target transfer rate to the default value set for target_rate_kbps.<br/>If false, users can adjust the transfer rate up to the value set for target_rate_cap_kbps. |
|
3354
|
+
| min_rate_cap_kbps | integer | | Y | | | | Y | The highest minimum rate that an incoming transfer can request, in kilobits per second.<br/>Client minimum rate requests that exceed the minimum rate cap are ignored.<br/>The default value of unlimited applies no cap to the minimum rate. (Default: 0) |
|
3355
|
+
| min_rate_kbps | integer | Y | Y | Y | Y | Y | Y | Set the minimum transfer rate in kilobits per second.<br/>(-m {integer}) |
|
3356
|
+
| move_after_transfer | string | Y | | | | Y | Y | The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer={string}) |
|
3357
|
+
| multi_session | integer | Y | Y | Y | Y | Y | Y | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/>(<special>) |
|
3358
|
+
| multi_session_threshold | integer | Y | | | | Y | Y | Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold={integer}) |
|
3359
|
+
| obfuscate_file_names | boolean | | | | Y | | | HTTP Gateway obfuscates file names when set to true. |
|
3360
|
+
| overwrite | string | Y | Y | Y | Y | Y | Y | Overwrite files at the destination with source files of the same name based on the policy:<br/>- always – Always overwrite the file.<br/>- never – Never overwrite the file. If the destination contains partial files that are older or the same as the source files and resume is enabled, the partial files resume transfer. Partial files with checksums or sizes that differ from the source files are not overwritten.<br/>- diff – Overwrite the file if it is different from the source, depending on the compare method (default is size). If the destination is object storage, diff has the same effect as always. If resume is not enabled, partial files are overwritten if they are different from the source, otherwise they are skipped. If resume is enabled, only partial files with different sizes or checksums from the source are overwritten; otherwise, files resume. <br/>- diff+older – Overwrite the file if it is older and different from the source, depending on the compare method (default is size). If resume is not enabled, partial files are overwritten if they are older and different from the source, otherwise they are skipped. If resume is enabled, only partial files that are different and older than the source are overwritten, otherwise they are resumed. <br/>- older – Overwrite the file if its timestamp is older than the source timestamp. <br/>If you set an overwrite policy of diff or diff+older, difference is determined by the value set for resume_policy:<br/>"none" - the source and destination files are always considered different and the destination file is always overwritten<br/>"attributes" - the source and destination files are compared based on file attributes <br/>"sparse_checksum" - the source and destination files are compared based on sparse checksums, (currently file size)<br/>"full_checksum" - the source and destination files are compared based on full checksums <br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite={enum}) |
|
3361
|
+
| password | string | | | | | Y | | Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only. |
|
3241
3362
|
| paths | array | Y | Y | Y | Y | Y | Y | Array of path to the source (required) and a path to the destination (optional).<br/>(<special>) |
|
3242
|
-
| precalculate_job_size |
|
3243
|
-
| preserve_access_time |
|
3244
|
-
| preserve_acls | string |
|
3245
|
-
| preserve_creation_time |
|
3246
|
-
| preserve_extended_attrs | string |
|
3247
|
-
| preserve_file_owner_gid |
|
3248
|
-
| preserve_file_owner_uid |
|
3249
|
-
| preserve_modification_time |
|
3250
|
-
| preserve_remote_acls | string |
|
3251
|
-
|
|
3252
|
-
|
|
3253
|
-
|
|
3254
|
-
|
|
3255
|
-
|
|
3256
|
-
|
|
3257
|
-
|
|
3258
|
-
|
|
3363
|
+
| precalculate_job_size | boolean | Y | Y | Y | Y | Y | Y | Specifies whether to precalculate the job size.<br/>(--precalculate-job-size) |
|
3364
|
+
| preserve_access_time | boolean | Y | Y | Y | Y | Y | Y | Preserve the source-file access timestamps at the destination.<br/>Because source access times are updated by the transfer operation, the timestamp that is preserved is the one just before to the transfer.<br/>(--preserve-access-time) |
|
3365
|
+
| preserve_acls | string | Y | | | | | Y | Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls={enum}) |
|
3366
|
+
| preserve_creation_time | boolean | Y | Y | Y | Y | Y | Y | (Windows only) Preserve source-file creation timestamps at the destination.<br/>Only Windows systems retain information about creation time.<br/>If the destination is not a Windows computer, this option is ignored.<br/>(--preserve-creation-time) |
|
3367
|
+
| preserve_extended_attrs | string | Y | | | | | Y | Preserve the extended attributes.<br/>Allowed values: none, native, metafile<br/>(--preserve-xattrs={enum}) |
|
3368
|
+
| preserve_file_owner_gid | boolean | Y | | | | | Y | Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid) |
|
3369
|
+
| preserve_file_owner_uid | boolean | Y | | | | | Y | Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid) |
|
3370
|
+
| preserve_modification_time | boolean | Y | Y | Y | Y | Y | Y | Set the modification time, the last time a file or directory was modified (written), of a transferred file to the modification of the source file or directory.<br/>Preserve source-file modification timestamps at the destination.<br/>(--preserve-modification-time) |
|
3371
|
+
| preserve_remote_acls | string | Y | | | | | Y | Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls={enum}) |
|
3372
|
+
| preserve_remote_extended_attrs | string | Y | | | | | Y | Preserve remote extended attributes.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-xattrs={enum}) |
|
3373
|
+
| preserve_source_access_time | boolean | Y | | | | | Y | Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time) |
|
3374
|
+
| preserve_times | boolean | Y | | | | Y | Y | Preserve file timestamps.<br/>(-p {boolean}) |
|
3375
|
+
| proxy | string | Y | | | | | | Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy={string}) |
|
3376
|
+
| rate_policy | string | Y | Y | Y | Y | Y | Y | The transfer rate policy to use when sharing bandwidth. Allowable values:<br/>- high : When sharing bandwidth, transfer at twice the rate of a transfer using a fair policy.<br/>- fair : (Default) Share bandwidth equally with other traffic.<br/>- low : Use only unused bandwidth.<br/>- fixed : Transfer at the target rate, regardless of the actual network capacity. Do not share bandwidth. Aspera recommends that you do not use this setting except under special circumstances, otherwise the destination storage can be damaged. <br/>Allowed values: low, fair, high, fixed<br/>(--policy={enum}) |
|
3377
|
+
| rate_policy_allowed | string | | Y | | | | | Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed |
|
3378
|
+
| read_threads | integer | | | | | | | ascp4 only |
|
3379
|
+
| remote_access_key | string | | | | | | Y | The access key ID of the access key that was used to construct the bearer token that is used to authenticate to the remote node. |
|
3380
|
+
| remote_host | string | Y | Y | Y | Y | Y | Y | IP or fully qualified domain name of the remote server<br/>(--host={string}) |
|
3259
3381
|
| remote_password | string | Y | Y | Y | Y | Y | Y | SSH session password<br/>(env:ASPERA_SCP_PASS) |
|
3260
|
-
| remote_user | string | Y | Y | Y | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user
|
3261
|
-
| remove_after_transfer |
|
3262
|
-
| remove_empty_directories |
|
3263
|
-
|
|
3264
|
-
|
|
3265
|
-
|
|
3266
|
-
|
|
3267
|
-
|
|
3268
|
-
|
|
3269
|
-
|
|
3270
|
-
|
|
3271
|
-
|
|
3272
|
-
|
|
3273
|
-
|
|
3274
|
-
|
|
3275
|
-
|
|
3276
|
-
|
|
3277
|
-
|
|
3278
|
-
|
|
3279
|
-
|
|
3280
|
-
|
|
3382
|
+
| remote_user | string | Y | Y | Y | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user={string}) |
|
3383
|
+
| remove_after_transfer | boolean | Y | | | | Y | Y | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
3384
|
+
| remove_empty_directories | boolean | Y | | | | Y | Y | Specifies whether to remove empty directories.<br/>(--remove-empty-directories) |
|
3385
|
+
| remove_empty_source_dir | boolean | | | | | | Y | Remove empty source subdirectories and remove the source directory itself, if empty. |
|
3386
|
+
| remove_empty_source_directory | boolean | Y | | | | | | Remove empty source subdirectories and remove the source directory itself, if empty.<br/>(--remove-empty-source-directory) |
|
3387
|
+
| remove_skipped | boolean | Y | Y | | | Y | | Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped) |
|
3388
|
+
| resume_policy | string | Y | Y | Y | Y | Y | Y | If a transfer is interrupted or fails to finish, this policy directs the transfer to resume without retransferring the files. Allowable values:<br/>- none : Always re-transfer the entire file<br/>- attrs : Compare file attributes and resume if they match, and re-transfer if they do not<br/>- sparse_csum : Compare file attributes and the sparse file checksums; resume if they match, and re-transfer if they do not<br/>- full_csum : Compare file attributes and the full file checksums; resume if they match, and re-transfer if they do not. <br/>Note: transferd uses values: attributes, sparse_checksum, full_checksum.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum}) |
|
3389
|
+
| retry_duration | integer<br/>string | | | | | | Y | Specifies how long to wait before retrying transfer. (e.g. "5min") |
|
3390
|
+
| save_before_overwrite | boolean | Y | | | | Y | Y | If a transfer would result in an existing file <filename>.<ext> being overwritten, move that file to <filename>.yyyy.mm.dd.hh.mm.ss.index.<ext> (where index is set to 1 at the beginning of each new second and incremented for each file saved in this manner during the same second) in the same directory before writing the new file.<br/>File attributes are maintained in the renamed file.<br/>(--save-before-overwrite) |
|
3391
|
+
| skip_duplicate_check | boolean | Y | | | | | Y | Don't check for duplicate files at the destination.<br/>(--skip-dir-traversal-dupes) |
|
3392
|
+
| skip_special_files | boolean | Y | | | | | Y | All assets other than files, directories and symbolic links are considered special. A transfer will fail if the user attempts to transfer special assets. If true, ascp skips special assets and proceeds with the transfer of all other assets. <br/>(--skip-special-files) |
|
3393
|
+
| source_root | string | Y | Y | Y | Y | Y | Y | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64=(conversion){string}) |
|
3394
|
+
| source_root_id | string | | | | | Y | Y | The file ID of the source root directory. Required when using Bearer token auth for the source node. |
|
3395
|
+
| src_base | string | Y | | | | Y | Y | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64=(conversion){string}) |
|
3396
|
+
| src_base64 | string | Y | | | | | Y | The folder name below which the directory structure is preserved (base64 encoded).<br/>(--src-base64={string}) |
|
3397
|
+
| ssh_args | array | Y | | | | | Y | Array of arguments to pass to SSH. Use with caution.<br/>(-i {array}) |
|
3398
|
+
| ssh_port | integer | Y | Y | Y | Y | Y | Y | Specifies SSH (TCP) port.<br/>(-P {integer}) |
|
3399
|
+
| ssh_private_key | string | Y | | | | | Y | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----\nMII...<br/>Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY) |
|
3400
|
+
| ssh_private_key_passphrase | string | Y | | | | | Y | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS) |
|
3401
|
+
| ssh_private_key_path | string | Y | | | | | Y | Path to private key for SSH.<br/>(-i {string}) |
|
3402
|
+
| sshfp | string | Y | Y | Y | Y | Y | Y | Check it against server SSH host key fingerprint.<br/>(--check-sshfp={string}) |
|
3403
|
+
| symlink_policy | string | Y | Y | Y | Y | Y | Y | Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links={enum}) |
|
3404
|
+
| tags | object | Y | Y | Y | Y | Y | Y | Metadata for transfer as JSON. Key `aspera` is reserved. Key `aspera.xfer_retry` specifies a retry timeout for node api initiated transfers.<br/>(--tags64=(conversion){object}) |
|
3405
|
+
| tags64 | string | Y | | | | | Y | Metadata for transfer as JSON. Key `aspera` is reserved. Key `aspera.xfer_retry` specifies a retry timeout for node api initiated transfers.<br/>(--tags64={string}) |
|
3406
|
+
| target_rate_cap_kbps | integer | | Y | | | | Y | Maximum target rate for incoming transfers, in kilobits per second. Returned by upload/download_setup node API. |
|
3407
|
+
| target_rate_kbps | integer | Y | Y | Y | Y | Y | Y | Specifies desired speed for the transfer.<br/>(-l {integer}) |
|
3408
|
+
| target_rate_percentage | string | | | | | | | TODO: remove ? |
|
3409
|
+
| title | string | | Y | | | Y | Y | Title of the transfer. |
|
3281
3410
|
| token | string | Y | Y | Y | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
3282
|
-
| use_ascp4 |
|
3283
|
-
| use_system_ssh | string |
|
3284
|
-
| write_threads |
|
3285
|
-
| wss_enabled |
|
3286
|
-
| wss_port |
|
3287
|
-
| xfer_max_retries |
|
3411
|
+
| use_ascp4 | boolean | Y | | | | Y | Y | Specify version of protocol. Do not use ascp4.<br/>(<special>) |
|
3412
|
+
| use_system_ssh | string | Y | | | | | Y | TODO, comment...<br/>(-SSH {string}) |
|
3413
|
+
| write_threads | integer | | | | | | | ascp4 only |
|
3414
|
+
| wss_enabled | boolean | Y | Y | Y | Y | Y | Y | Server has Web Socket service enabled<br/>(<special>) |
|
3415
|
+
| wss_port | integer | Y | Y | Y | Y | Y | Y | TCP port used for websocket service feed<br/>(<special>) |
|
3416
|
+
| xfer_max_retries | integer | | | | | Y | | Maximum number of retries, for node API initiated transfers. Shall not exceed aspera.conf `transfer_manager_max_retries` (default 5). |
|
3288
3417
|
|
3289
3418
|
#### Destination folder for transfers
|
3290
3419
|
|
@@ -3296,18 +3425,18 @@ The destination folder is set by `ascli` by default to:
|
|
3296
3425
|
It is specified by the [**transfer-spec**](#transfer-specification) parameter `destination_root`.
|
3297
3426
|
As such, it can be modified with option: `--ts=@json:'{"destination_root":"<path>"}'`.
|
3298
3427
|
The option `to_folder` provides an equivalent and convenient way to change this parameter:
|
3299
|
-
`--to-folder=<path
|
3428
|
+
`--to-folder=<path>`.
|
3300
3429
|
|
3301
3430
|
#### List of files for transfers
|
3302
3431
|
|
3303
3432
|
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
3304
3433
|
|
3305
|
-
By default the list of files to transfer is simply provided on the command line.
|
3434
|
+
By default, the list of files to transfer is simply provided on the command line.
|
3306
3435
|
|
3307
3436
|
The list of (source) files to transfer is specified by (extended value) option `sources` (default: `@args`).
|
3308
3437
|
The list is either simply the list of source files, or a combined source/destination list (see below) depending on value of option `src_type` (default: `list`).
|
3309
3438
|
|
3310
|
-
In `ascli`, all transfer parameters, including file list, are provided to the transfer agent in a [**transfer-spec**](#transfer-specification) so that execution of a transfer is independent of the transfer agent (direct, connect, node, transfer
|
3439
|
+
In `ascli`, all transfer parameters, including file list, are provided to the transfer agent in a [**transfer-spec**](#transfer-specification) so that execution of a transfer is independent of the transfer agent (direct, connect, node, transfer daemon...).
|
3311
3440
|
So, eventually, the list of files to transfer is provided to the transfer agent using the [**transfer-spec**](#transfer-specification) field: `"paths"` which is a list (array) of pairs of `"source"` (mandatory) and `"destination"` (optional).
|
3312
3441
|
The `sources` and `src_type` options provide convenient ways to populate the transfer spec with the source file list.
|
3313
3442
|
|
@@ -3416,16 +3545,16 @@ In the following table source folder `d3` contains 2 files: `f1` and `d4/f2`.
|
|
3416
3545
|
|
3417
3546
|
| Source files | Destination | Folders on Dest. |`create_dir`| Destination Files |
|
3418
3547
|
|--------------|-------------|-------------------|------------|-----------------------------|
|
3419
|
-
| f1
|
3420
|
-
| f1
|
3421
|
-
| f1
|
3422
|
-
| f1
|
3423
|
-
| f1 f2
|
3424
|
-
| d3
|
3425
|
-
| f1
|
3426
|
-
| f1 f2
|
3427
|
-
| d1/f1 d2/f2
|
3428
|
-
| d3
|
3548
|
+
| `f1` | `d/f` | - | false | Error: `d` does not exist. |
|
3549
|
+
| `f1` | `d/f` | `d` | false | `d/f` (renamed) |
|
3550
|
+
| `f1` | `d/f/.` | `d` | false | `d/f` (renamed) |
|
3551
|
+
| `f1` | `d/f` | `d/f` | false | `d/f/f1` |
|
3552
|
+
| `f1` `f2` | `d` | `d` | false | `d/f1` `d/f2` |
|
3553
|
+
| `d3` | `d` | - | false | `d/f1` `d/f2` (renamed) |
|
3554
|
+
| `f1` | `d` | - | true | `d/f1` |
|
3555
|
+
| `f1` `f2` | `d` | - | true | `d/f1` `d/f2` |
|
3556
|
+
| `d1/f1` `d2/f2` | `d` | - | true | `d/f1` `d/f2` |
|
3557
|
+
| `d3` | `d` | - | true | `d/d3/f1` `d/d3/d4/f2` |
|
3429
3558
|
|
3430
3559
|
If a file par list is provided then it is possible to rename or specify a different destination folder for each source (relative to the destination).
|
3431
3560
|
|
@@ -3433,7 +3562,7 @@ If transfer spec has a `src_base`, it has the side effect that the simple source
|
|
3433
3562
|
|
3434
3563
|
| Source files | Destination |`src_base`| Destination Files |
|
3435
3564
|
|-------------------|-------------|----------|-----------------------------|
|
3436
|
-
| d1/d2/f2 d1/d3/f3 | d
|
3565
|
+
| `d1/d2/f2` `d1/d3/f3` | `d` | `d1` | `d/d2/f2` `d/d3/f3` |
|
3437
3566
|
|
3438
3567
|
Advanced Example: Send files `./file1` and `./folder2/files2` to server (e.g. `/Upload`) and keep the original file names and folders, i.e. send `file1` to `/Upload/file1` and `files2` to `/Upload/folder2/files2`.
|
3439
3568
|
|
@@ -3483,7 +3612,7 @@ Activating CSEAR consists in using transfer spec parameters:
|
|
3483
3612
|
- `content_protection` : activate encryption (`encrypt` for upload) or decryption (`decrypt` for download)
|
3484
3613
|
- `content_protection_password` : the passphrase to be used.
|
3485
3614
|
|
3486
|
-
Example: parameter to download a
|
3615
|
+
Example: parameter to download a Faspex package and decrypt on the fly
|
3487
3616
|
|
3488
3617
|
```bash
|
3489
3618
|
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"my_password_here"}'
|
@@ -3503,7 +3632,7 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
3503
3632
|
--ts=@json:'{"ssh_port":33002}'
|
3504
3633
|
```
|
3505
3634
|
|
3506
|
-
- Force
|
3635
|
+
- Force HTTP fallback mode:
|
3507
3636
|
|
3508
3637
|
```bash
|
3509
3638
|
--ts=@json:'{"http_fallback":"force"}'
|
@@ -3517,12 +3646,10 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
3517
3646
|
|
3518
3647
|
### Transfer progress bar
|
3519
3648
|
|
3520
|
-
File transfer operations are monitored and a progress bar is displayed on the terminal if option `progress_bar` (`Bool`) is set to `yes` (default if the output is a terminal).
|
3649
|
+
File transfer operations are monitored, and a progress bar is displayed on the terminal if option `progress_bar` (`Bool`) is set to `yes` (default if the output is a terminal).
|
3521
3650
|
|
3522
3651
|
The same progress bar is used for any type of transfer, using `ascp`, server to server, using HTTPS, etc...
|
3523
3652
|
|
3524
|
-
To display the native progress bar of `ascp`, use `--progress-bar=no --transfer-info=@json:'{"quiet":false}'`.
|
3525
|
-
|
3526
3653
|
### Scheduler
|
3527
3654
|
|
3528
3655
|
It is useful to configure automated scheduled execution.
|
@@ -3536,17 +3663,17 @@ It can be configured:
|
|
3536
3663
|
|
3537
3664
|
- Using utility [`schtasks.exe`](https://learn.microsoft.com/fr-fr/windows-server/administration/windows-commands/schtasks-create)
|
3538
3665
|
|
3539
|
-
- Using
|
3666
|
+
- Using Powershell function [`scheduletasks`](https://learn.microsoft.com/en-us/powershell/module/scheduledtasks)
|
3540
3667
|
|
3541
3668
|
- Using `taskschd.msc` (UI)
|
3542
3669
|
|
3543
3670
|
#### Unix-like Scheduler
|
3544
3671
|
|
3545
|
-
Unix-like systems (Linux, ...) provide cron
|
3672
|
+
Unix-like systems (Linux, ...) provide `cron`, configured using a [`crontab`](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
3546
3673
|
|
3547
3674
|
Linux also provides `anacron`, if tasks are hourly or daily.
|
3548
3675
|
|
3549
|
-
For example, on Linux it is convenient to create a wrapping script, e.g. `cron_ascli` that will
|
3676
|
+
For example, on Linux it is convenient to create a wrapping script, e.g. `cron_ascli` that will set up the environment (e.g. Ruby) to properly start `ascli`:
|
3550
3677
|
|
3551
3678
|
```bash
|
3552
3679
|
#!/bin/bash
|
@@ -3568,13 +3695,70 @@ crontab<<EOF
|
|
3568
3695
|
EOF
|
3569
3696
|
```
|
3570
3697
|
|
3571
|
-
> **Note:** Logging options are kept here in the cron file instead of configuration file to allow execution on command line with output on command line.
|
3698
|
+
> **Note:** Logging options are kept here in the `cron` file instead of configuration file to allow execution on command line with output on command line.
|
3699
|
+
|
3700
|
+
### Running as service
|
3701
|
+
|
3702
|
+
Some commands result in `ascli` running as a server, listening on a port.
|
3703
|
+
In this case, it is usually desirable to run `ascli` as a service.
|
3704
|
+
On Linux, typically, [SystemD](https://systemd.io/) is used.
|
3705
|
+
|
3706
|
+
A convenient way is to write a startup script, and run this script as a service.
|
3707
|
+
|
3708
|
+
Let's give a base name for our service: `my_ascli_svc`
|
3709
|
+
|
3710
|
+
The startup script can be simply the `ascli` command line, for example: `/usr/local/bin/start_my_ascli_svc.sh`:
|
3711
|
+
|
3712
|
+
```bash
|
3713
|
+
#!/bin/bash
|
3714
|
+
set -e
|
3715
|
+
echo "Starting my_ascli_svc at $(date)"
|
3716
|
+
# set PATH to find ascli, and other environment setup
|
3717
|
+
exec ascli .....
|
3718
|
+
```
|
3719
|
+
|
3720
|
+
And make this script executable:
|
3721
|
+
|
3722
|
+
```bash
|
3723
|
+
chmod a+x /usr/local/bin/start_my_ascli_svc.sh
|
3724
|
+
```
|
3725
|
+
|
3726
|
+
Create a startup file: `/etc/systemd/system/my_ascli_svc.service` :
|
3727
|
+
|
3728
|
+
```ini
|
3729
|
+
[Unit]
|
3730
|
+
Description=My ascli daemon
|
3731
|
+
After=network.target
|
3732
|
+
|
3733
|
+
[Service]
|
3734
|
+
ExecStart=/usr/local/bin/start_my_ascli_svc.sh
|
3735
|
+
Restart=on-failure
|
3736
|
+
RestartSec=15
|
3737
|
+
User=xfer
|
3738
|
+
# Optional, specify a working directory
|
3739
|
+
# WorkingDirectory=/path/to/working/dir
|
3740
|
+
# Optional, redirect output
|
3741
|
+
StandardOutput=journal
|
3742
|
+
StandardError=journal
|
3743
|
+
|
3744
|
+
[Install]
|
3745
|
+
WantedBy=multi-user.target
|
3746
|
+
```
|
3747
|
+
|
3748
|
+
Then enable and start with:
|
3749
|
+
|
3750
|
+
```bash
|
3751
|
+
sudo systemctl daemon-reload
|
3752
|
+
sudo systemctl enable --now my_ascli_svc.service
|
3753
|
+
systemctl status my_ascli_svc.service
|
3754
|
+
journalctl -u my_ascli_svc.service
|
3755
|
+
```
|
3572
3756
|
|
3573
3757
|
### Locking for exclusive execution
|
3574
3758
|
|
3575
3759
|
In some cases one needs to ensure that `ascli` is not executed several times in parallel.
|
3576
3760
|
|
3577
|
-
When `ascli` is executed automatically on a schedule basis, one generally desires that a new execution is not started if a previous execution is still running because an
|
3761
|
+
When `ascli` is executed automatically on a schedule basis, one generally desires that a new execution is not started if a previous execution is still running because an ongoing operation may last longer than the scheduling period:
|
3578
3762
|
|
3579
3763
|
- Executing instances may pile-up and kill the system
|
3580
3764
|
- The same file may be transferred by multiple instances at the same time.
|
@@ -3583,7 +3767,7 @@ When `ascli` is executed automatically on a schedule basis, one generally desire
|
|
3583
3767
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
3584
3768
|
|
3585
3769
|
- The Windows scheduler does this by default
|
3586
|
-
- Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
3770
|
+
- Linux `cron` can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
3587
3771
|
|
3588
3772
|
```bash
|
3589
3773
|
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
@@ -3644,10 +3828,9 @@ It is a pseudo file system allowing generation of file data without actual stora
|
|
3644
3828
|
|
3645
3829
|
### `faux:` for testing
|
3646
3830
|
|
3647
|
-
This is an extract of the man page of `ascp`.
|
3648
|
-
This feature is a feature of `ascp`, not `ascli`.
|
3649
|
-
|
3650
3831
|
This adapter can be used to simulate a file or a directory.
|
3832
|
+
This is a feature of `ascp`, not `ascli`.
|
3833
|
+
The following is an extract of the man page of `ascp`.
|
3651
3834
|
|
3652
3835
|
To discard data at the destination, the destination argument is set to `faux://`.
|
3653
3836
|
|
@@ -3664,7 +3847,7 @@ where:
|
|
3664
3847
|
|
3665
3848
|
> **Note:** Characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
3666
3849
|
|
3667
|
-
For all sizes, a suffix can be added (case
|
3850
|
+
For all sizes, a suffix can be added (case-insensitive) to the size: k, m, g, t, p, e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8\*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
3668
3851
|
|
3669
3852
|
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
3670
3853
|
|
@@ -3690,17 +3873,18 @@ The sequence parameter is applied as follows:
|
|
3690
3873
|
|
3691
3874
|
- If `seq` is `random` then each file size is:
|
3692
3875
|
|
3693
|
-
- size +/- (inc * rand())
|
3876
|
+
- `size +/- (inc * rand())`
|
3694
3877
|
- Where rand is a random number between 0 and 1
|
3695
3878
|
- Note that file size must not be negative, inc will be set to size if it is greater than size
|
3696
|
-
- Similarly, overall file size must be less than 8
|
3879
|
+
- Similarly, overall file size must be less than 8\*2<sup>60</sup>.
|
3880
|
+
If size + inc is greater, inc will be reduced to limit size + inc to 7\*2<sup>60</sup>.
|
3697
3881
|
|
3698
3882
|
- If `seq` is `sequential` then each file size is:
|
3699
3883
|
|
3700
3884
|
- `size + ((file_index - 1) * inc)`
|
3701
3885
|
- Where first file is index 1
|
3702
3886
|
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
3703
|
-
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less
|
3887
|
+
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less than 8\*2<sup>60</sup>.
|
3704
3888
|
|
3705
3889
|
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
3706
3890
|
|
@@ -3718,18 +3902,24 @@ ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
|
3718
3902
|
ascli server upload /tmp/sample --to-folder=faux://
|
3719
3903
|
```
|
3720
3904
|
|
3721
|
-
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the
|
3905
|
+
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the base name of each file being `testfile` to /Upload
|
3722
3906
|
|
3723
3907
|
```bash
|
3724
3908
|
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
3725
3909
|
```
|
3726
3910
|
|
3911
|
+
- Upload a faux directory `mydir` containing 1000 files, of size 1 byte, with the base name of each file being `testfile` to /Upload
|
3912
|
+
|
3913
|
+
```bash
|
3914
|
+
ascli server upload "faux:///mydir?file=testfile&count=1000&size=1" --to-folder=/Upload
|
3915
|
+
```
|
3916
|
+
|
3727
3917
|
### Usage
|
3728
3918
|
|
3729
3919
|
```text
|
3730
3920
|
ascli -h
|
3731
3921
|
NAME
|
3732
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
3922
|
+
ascli -- a command line tool for Aspera Applications (v4.22.0.pre)
|
3733
3923
|
|
3734
3924
|
SYNOPSIS
|
3735
3925
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -3802,7 +3992,8 @@ OPTIONS: global
|
|
3802
3992
|
--key-path=VALUE Wizard: path to private key for JWT
|
3803
3993
|
--ascp-path=VALUE Path to ascp
|
3804
3994
|
--use-product=VALUE Use ascp from specified product
|
3805
|
-
--sdk-url=VALUE URL to get
|
3995
|
+
--sdk-url=VALUE URL to get Aspera Transfer Daemon
|
3996
|
+
--locations-url=VALUE URL to get locations of Aspera Transfer Daemon
|
3806
3997
|
--sdk-folder=VALUE SDK folder path
|
3807
3998
|
--progress-bar=ENUM Display progress bar: [no], yes
|
3808
3999
|
--smtp=VALUE SMTP configuration (Hash)
|
@@ -3821,7 +4012,7 @@ OPTIONS: global
|
|
3821
4012
|
--to-folder=VALUE Destination folder for transferred files
|
3822
4013
|
--sources=VALUE How list of transferred files is provided (@args,@ts,Array)
|
3823
4014
|
--src-type=ENUM Type of file list: [list], pair
|
3824
|
-
--transfer=ENUM Type of transfer agent: node, [direct],
|
4015
|
+
--transfer=ENUM Type of transfer agent: desktop, node, [direct], transferd, httpgw, connect
|
3825
4016
|
--transfer-info=VALUE Parameters for transfer agent (Hash)
|
3826
4017
|
|
3827
4018
|
COMMAND: config
|
@@ -3837,7 +4028,7 @@ OPTIONS:
|
|
3837
4028
|
|
3838
4029
|
|
3839
4030
|
COMMAND: node
|
3840
|
-
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse central delete download events health
|
4031
|
+
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse cat central delete download events health info license mkdir mkfile mklink rename search service simulator slash space ssync stream sync telemetry transfer transport upload watch_folder
|
3841
4032
|
OPTIONS:
|
3842
4033
|
--url=VALUE URL of application, e.g. https://faspex.example.com/aspera/faspex
|
3843
4034
|
--username=VALUE User's name to log in
|
@@ -3966,10 +4157,10 @@ OPTIONS:
|
|
3966
4157
|
--thumb-vid-scale=VALUE Png: video: size (ffmpeg scale argument)
|
3967
4158
|
--thumb-vid-fraction=VALUE Png: video: time percent position of snapshot
|
3968
4159
|
--thumb-img-size=VALUE Png: non-video: height (and width)
|
3969
|
-
--thumb-text-font=VALUE Png: plaintext: font
|
4160
|
+
--thumb-text-font=VALUE Png: plaintext: font for text rendering: `magick identify -list font`
|
3970
4161
|
--video-conversion=ENUM Mp4: method for preview generation: [reencode], blend, clips
|
3971
4162
|
--video-png-conv=ENUM Mp4: method for thumbnail generation: [fixed], animated
|
3972
|
-
--video-scale=VALUE Mp4: all: video scale (ffmpeg)
|
4163
|
+
--video-scale=VALUE Mp4: all: video scale (ffmpeg scale argument)
|
3973
4164
|
--video-start-sec=VALUE Mp4: all: start offset (seconds) of video preview
|
3974
4165
|
--reencode-ffmpeg=VALUE Mp4: reencode: options to ffmpeg
|
3975
4166
|
--blend-keyframes=VALUE Mp4: blend: # key frames
|
@@ -4028,16 +4219,16 @@ OPTIONS:
|
|
4028
4219
|
|
4029
4220
|
Bulk creation and deletion of resources are possible using option `bulk` (`yes`,`no`(default)).
|
4030
4221
|
In that case, the operation expects an `Array` of `Hash` instead of a simple `Hash` using the [Extended Value Syntax](#extended-value-syntax).
|
4031
|
-
This option is available only for some
|
4222
|
+
This option is available only for some resources: if you need it: try and see if the entities you try to create or delete support this option.
|
4032
4223
|
|
4033
4224
|
### Option: `query`
|
4034
4225
|
|
4035
|
-
The `query` option can generally be used to add URL parameters to commands that list
|
4226
|
+
The `query` option can generally be used to add URL parameters to commands that list resources.
|
4036
4227
|
It takes either a `Hash` or an `Array`, corresponding to key/value pairs that appear in the query part of request.
|
4037
4228
|
|
4038
4229
|
For example: `--query=@json:'{"p1":"v1","p2":"v2"}'` leads to query: `?p1=v1&p2=v2`.
|
4039
4230
|
|
4040
|
-
If the same parameter needs to be provided several times, then it's possible as well to provide an Array or 2-element Array
|
4231
|
+
If the same parameter needs to be provided several times, then it's possible as well to provide an `Array` or 2-element `Array`: `--query=@json:'[["p1":,"v1"],["p2":"v2"]]'` leads to the same result as previously.
|
4041
4232
|
|
4042
4233
|
If PHP's style array is used, then one can use either:
|
4043
4234
|
|
@@ -4073,9 +4264,9 @@ Most plugins will take the URL option: `url` to identify their location.
|
|
4073
4264
|
|
4074
4265
|
REST APIs of Aspera legacy applications (Aspera Node, Faspex 4, Shares, Console, Orchestrator) use simple username/password authentication: HTTP Basic Authentication using options: `username` and `password`.
|
4075
4266
|
|
4076
|
-
Aspera on Cloud and Faspex 5 rely on
|
4267
|
+
Aspera on Cloud and Faspex 5 rely on OAuth.
|
4077
4268
|
|
4078
|
-
By default plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
4269
|
+
By default, plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
4079
4270
|
|
4080
4271
|
```bash
|
4081
4272
|
ascli --show-config --fields=plugin_folder
|
@@ -4097,17 +4288,18 @@ ascli --plugin-folder=. foo
|
|
4097
4288
|
|
4098
4289
|
## Plugin: `aoc`: IBM Aspera on Cloud
|
4099
4290
|
|
4100
|
-
Aspera on Cloud API requires the use of
|
4291
|
+
Aspera on Cloud API requires the use of OAuth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
4101
4292
|
|
4102
4293
|
It is recommended to use the wizard to set it up, although manual configuration is also possible.
|
4103
4294
|
|
4104
4295
|
### Configuration: Using Wizard
|
4105
4296
|
|
4106
|
-
`ascli` provides a configuration wizard.
|
4297
|
+
`ascli` provides a configuration [wizard](#wizard).
|
4107
4298
|
|
4108
4299
|
The wizard guides you through the steps to create a new configuration preset for Aspera on Cloud.
|
4109
4300
|
|
4110
|
-
The first
|
4301
|
+
The first optional argument is the URL of your Aspera on Cloud instance, e.g. `https://_your_instance_.ibmaspera.com` or simply the organization name, and a second optional argument can also be provided to specify the plugin name, e.g. `aoc` for Aspera on Cloud.
|
4302
|
+
If optional arguments are not provided, the wizard will ask interactively and try to detect the application.
|
4111
4303
|
|
4112
4304
|
Here is a sample invocation :
|
4113
4305
|
|
@@ -4144,6 +4336,13 @@ ascli config wizard _your_instance_ aoc
|
|
4144
4336
|
|
4145
4337
|
> **Note:** In above example, replace `_your_instance_` with the first part of your actual AoC URL: `https://_your_instance_.ibmaspera.com`.
|
4146
4338
|
|
4339
|
+
After successful completion of the wizard, a new configuration preset is created, and set as default for the `aoc` plugin.
|
4340
|
+
This can be verified with command:
|
4341
|
+
|
4342
|
+
```bash
|
4343
|
+
ascli config preset over
|
4344
|
+
```
|
4345
|
+
|
4147
4346
|
### Configuration: Using manual setup
|
4148
4347
|
|
4149
4348
|
> **Note:** If you used the wizard (recommended): skip this section.
|
@@ -4152,17 +4351,17 @@ ascli config wizard _your_instance_ aoc
|
|
4152
4351
|
|
4153
4352
|
Several types of OAuth authentication are supported:
|
4154
4353
|
|
4155
|
-
- JSON Web Token (JWT) : authentication is secured by a private key (recommended
|
4354
|
+
- JSON Web Token (JWT) : authentication is secured by a private key (recommended)
|
4156
4355
|
- Web based authentication : authentication is made by user using a browser
|
4157
|
-
- URL Token : external
|
4356
|
+
- URL Token : external user's authentication with URL tokens (public links)
|
4158
4357
|
|
4159
4358
|
The authentication method is controlled by option `auth`.
|
4160
4359
|
|
4161
4360
|
For a **quick start**, follow the mandatory and sufficient section: [API Client Registration](#api-client-registration) (auth=web) as well as [[Option Preset](#option-preset) for Aspera on Cloud](#configuration-for-aspera-on-cloud).
|
4162
4361
|
|
4163
|
-
For a more convenient, browser-less, experience follow the [JWT](#authentication-with-private-key) section (auth=jwt) in addition to Client Registration.
|
4362
|
+
For a more convenient, browser-less, experience follow the [JWT](#authentication-with-private-key) section (`auth=jwt`) in addition to Client Registration.
|
4164
4363
|
|
4165
|
-
In
|
4364
|
+
In OAuth, a **Bearer** token is generated to authenticate REST calls.
|
4166
4365
|
Bearer tokens are valid for a period of time defined (by the AoC app, configurable by admin) at its creation.
|
4167
4366
|
`ascli` saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
4168
4367
|
|
@@ -4174,7 +4373,7 @@ If you use the built-in client_id and client_secret, skip this and do not set th
|
|
4174
4373
|
|
4175
4374
|
Else you can use a specific OAuth API client_id, the first step is to declare `ascli` in Aspera on Cloud using the admin interface.
|
4176
4375
|
|
4177
|
-
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client)
|
4376
|
+
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client)).
|
4178
4377
|
|
4179
4378
|
Let's start by a registration with web based authentication (auth=web):
|
4180
4379
|
|
@@ -4195,9 +4394,9 @@ Once the client is registered, a **Client ID** and **Secret** are created, these
|
|
4195
4394
|
|
4196
4395
|
#### Configuration for Aspera on Cloud
|
4197
4396
|
|
4198
|
-
If you did not use the wizard, you can also manually create
|
4397
|
+
If you did not use the wizard, you can also manually create an [Option Preset](#option-preset) for `ascli` in its configuration file.
|
4199
4398
|
|
4200
|
-
|
4399
|
+
Let's create an [Option Preset](#option-preset) called: `my_aoc_org` using `ask` for interactive input (client info from previous step):
|
4201
4400
|
|
4202
4401
|
```bash
|
4203
4402
|
ascli config preset ask my_aoc_org url client_id client_secret
|
@@ -4247,11 +4446,15 @@ ascli aoc admin client list
|
|
4247
4446
|
```
|
4248
4447
|
|
4249
4448
|
```output
|
4250
|
-
|
4251
|
-
|
4252
|
-
|
4253
|
-
|
4254
|
-
|
4449
|
+
Items: 4/4
|
4450
|
+
╭────────────┬─────────────────────╮
|
4451
|
+
│ id │ name │
|
4452
|
+
╞════════════╪═════════════════════╡
|
4453
|
+
│ oXPUyJ7JpQ │ PRI Sydney │
|
4454
|
+
│ TaoAmAG8Rg │ ascli_test_web │
|
4455
|
+
│ TDNl2bLZqw │ ascli_web │
|
4456
|
+
│ VTh92i5OfQ │ shannon │
|
4457
|
+
╰────────────┴─────────────────────╯
|
4255
4458
|
```
|
4256
4459
|
|
4257
4460
|
```bash
|
@@ -4318,7 +4521,7 @@ ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.asp
|
|
4318
4521
|
|
4319
4522
|
> **Note:** The private key argument represents the actual PEM string.
|
4320
4523
|
In order to read the content from a file, use the `@file:` prefix.
|
4321
|
-
But if the
|
4524
|
+
But if the `@file:` argument is used as is, it will read the file and set in the configuration file.
|
4322
4525
|
So, to keep the `@file:` tag in the configuration file, the `@val:` prefix is added.
|
4323
4526
|
|
4324
4527
|
After this last step, commands do not require web login anymore.
|
@@ -4387,12 +4590,12 @@ Values are directly sent to the API call and used as a filter on server side.
|
|
4387
4590
|
|
4388
4591
|
The following parameters are supported:
|
4389
4592
|
|
4390
|
-
- `q` : a filter on name of resource (case
|
4593
|
+
- `q` : a filter on name of resource (case-insensitive, matches if value is contained in name)
|
4391
4594
|
- `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
4392
4595
|
- `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
4393
4596
|
- `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
4394
|
-
- `page` : native
|
4395
|
-
- `per_page` : native
|
4597
|
+
- `page` : native API parameter, in general do not use (added by `ascli`)
|
4598
|
+
- `per_page` : native API parameter, number of items par API call, in general do not use
|
4396
4599
|
- Other specific parameters depending on resource type.
|
4397
4600
|
|
4398
4601
|
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
@@ -4431,24 +4634,25 @@ Refer to the AoC API for full list of query parameters, or use the browser in de
|
|
4431
4634
|
|
4432
4635
|
#### Selecting a resource
|
4433
4636
|
|
4434
|
-
Resources are identified by a unique `id`, as well as a unique `name` (case
|
4637
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case-insensitive).
|
4435
4638
|
|
4436
4639
|
To execute an action on a specific resource, select it using one of those methods:
|
4437
4640
|
|
4438
|
-
- **recommended**: give
|
4641
|
+
- **recommended**: give ID directly on command line **after the action**: `aoc admin node show 123`
|
4439
4642
|
- Give name on command line **after the action**: `aoc admin node show name abc`
|
4440
4643
|
- Provide option `id` : `aoc admin node show 123`
|
4441
4644
|
- Provide option `name` : `aoc admin node show --name=abc`
|
4442
4645
|
|
4443
4646
|
#### Creating a resource
|
4444
4647
|
|
4445
|
-
New resources (users, groups, workspaces, etc
|
4648
|
+
New resources (users, groups, workspaces, etc...) can be created using a command like:
|
4446
4649
|
|
4447
4650
|
```bash
|
4448
4651
|
ascli aoc admin create <resource type> @json:'{<...parameters...>}'
|
4449
4652
|
```
|
4450
4653
|
|
4451
|
-
Some
|
4654
|
+
Some API endpoints are described in [IBM API Hub](https://developer.ibm.com/apis/catalog?search=%22aspera%20on%20cloud%20api%22).
|
4655
|
+
Sadly, not all.
|
4452
4656
|
|
4453
4657
|
Nevertheless, it is possible to guess the structure of the creation value by simply dumping an existing resource, and use the same parameters for the creation.
|
4454
4658
|
|
@@ -4505,7 +4709,8 @@ The activity app can be queried with:
|
|
4505
4709
|
ascli aoc admin analytics transfers
|
4506
4710
|
```
|
4507
4711
|
|
4508
|
-
It can also support filters and send notification using option `notify_to`.
|
4712
|
+
It can also support filters and send notification using option `notify_to`.
|
4713
|
+
A template is defined using option `notify_template` :
|
4509
4714
|
|
4510
4715
|
`mytemplate.erb`:
|
4511
4716
|
|
@@ -4537,11 +4742,11 @@ Options:
|
|
4537
4742
|
- `query` filter (on API call)
|
4538
4743
|
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
4539
4744
|
|
4540
|
-
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now]
|
4745
|
+
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is `[date of previous execution]..[now]`.
|
4541
4746
|
|
4542
4747
|
#### Transfer: Using specific transfer ports
|
4543
4748
|
|
4544
|
-
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
4749
|
+
By default, transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
4545
4750
|
The option `default_ports` ([yes]/no) allows `ascli` to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
4546
4751
|
|
4547
4752
|
#### Using ATS
|
@@ -4554,7 +4759,7 @@ Aspera on Cloud Shared folders are implemented through a special type of file: `
|
|
4554
4759
|
A `link` is the equivalent of a symbolic link on a file system: it points to another folder (not file).
|
4555
4760
|
|
4556
4761
|
Listing a link (in terminal position of path) will information on the link itself, not the content of the folder it points to.
|
4557
|
-
To list the target folder content, add a `/`
|
4762
|
+
To list the target folder content, add a `/` at the end of the path.
|
4558
4763
|
|
4559
4764
|
Example:
|
4560
4765
|
|
@@ -4647,7 +4852,7 @@ ascli aoc user workspaces list
|
|
4647
4852
|
|
4648
4853
|
#### Example: Create a sub access key in a `node`
|
4649
4854
|
|
4650
|
-
Creation of a sub-access key is like creation of access key with the following difference: authentication to
|
4855
|
+
Creation of a sub-access key is like creation of access key with the following difference: authentication to Node API is made with access key (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
4651
4856
|
|
4652
4857
|
```bash
|
4653
4858
|
ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create @json:'{"storage":{"path":"/folder1"}}'
|
@@ -4702,14 +4907,14 @@ Other query parameters:
|
|
4702
4907
|
|
4703
4908
|
#### Example: Add all members of a workspace to another workspace
|
4704
4909
|
|
4705
|
-
a- Get
|
4910
|
+
a- Get ID of first workspace
|
4706
4911
|
|
4707
4912
|
```bash
|
4708
4913
|
WS1='First Workspace'
|
4709
4914
|
WS1ID=$(ascli aoc admin workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
4710
4915
|
```
|
4711
4916
|
|
4712
|
-
b- Get
|
4917
|
+
b- Get ID of second workspace
|
4713
4918
|
|
4714
4919
|
```bash
|
4715
4920
|
WS2='Second Workspace'
|
@@ -4728,7 +4933,7 @@ d- Convert to creation data for second workspace:
|
|
4728
4933
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
4729
4934
|
```
|
4730
4935
|
|
4731
|
-
or, using jq
|
4936
|
+
or, using `jq`:
|
4732
4937
|
|
4733
4938
|
```bash
|
4734
4939
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
@@ -4771,7 +4976,7 @@ ascli aoc admin group create @json:'{"name":"group 1","description":"my super gr
|
|
4771
4976
|
|
4772
4977
|
Group: `11111`
|
4773
4978
|
|
4774
|
-
- Get the workspace
|
4979
|
+
- Get the workspace ID
|
4775
4980
|
|
4776
4981
|
```bash
|
4777
4982
|
ascli aoc admin workspace list --query=@json:'{"q":"myworkspace"}' --fields=id --format=csv --display=data
|
@@ -4785,7 +4990,7 @@ Workspace: 22222
|
|
4785
4990
|
ascli aoc admin workspace_membership create @json:'{"workspace_id":22222,"member_type":"user","member_id":11111}'
|
4786
4991
|
```
|
4787
4992
|
|
4788
|
-
- Get a user's
|
4993
|
+
- Get a user's ID
|
4789
4994
|
|
4790
4995
|
```bash
|
4791
4996
|
ascli aoc admin user list --query=@json:'{"q":"manu.macron@example.com"}' --fields=id --format=csv --display=data
|
@@ -4803,7 +5008,7 @@ ascli aoc admin group_membership create @json:'{"group_id":11111,"member_type":"
|
|
4803
5008
|
|
4804
5009
|
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
4805
5010
|
|
4806
|
-
First,
|
5011
|
+
First, set up the environment (skip if already done)
|
4807
5012
|
|
4808
5013
|
```bash
|
4809
5014
|
ascli config wizard --url=https://sedemo.ibmaspera.com --username=someuser@example.com
|
@@ -4831,7 +5036,7 @@ You can test with:
|
|
4831
5036
|
ascli aoc user profile show
|
4832
5037
|
```
|
4833
5038
|
|
4834
|
-
This creates the option preset `aoc_[org name]` to allow seamless command line access and sets it as default for
|
5039
|
+
This creates the option preset `aoc_[org name]` to allow seamless command line access and sets it as default for Aspera on Cloud.
|
4835
5040
|
|
4836
5041
|
Then, create two shared folders located in two regions, in your files home, in a workspace.
|
4837
5042
|
|
@@ -4841,20 +5046,18 @@ Then, transfer between those:
|
|
4841
5046
|
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
4842
5047
|
```
|
4843
5048
|
|
4844
|
-
#### Example: Create registration key to register a node
|
5049
|
+
#### Example: Create registration key to register a tethered node
|
4845
5050
|
|
4846
|
-
|
4847
|
-
ascli aoc admin client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
4848
|
-
```
|
5051
|
+
The following command will create and display a secret token to register a self-managed Aspera Transfer Server:
|
4849
5052
|
|
4850
|
-
```
|
4851
|
-
|
5053
|
+
```bash
|
5054
|
+
ascli aoc admin client_registration_token create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --show-secrets=yes
|
4852
5055
|
```
|
4853
5056
|
|
4854
5057
|
#### Example: Delete all registration keys
|
4855
5058
|
|
4856
5059
|
```bash
|
4857
|
-
ascli aoc admin
|
5060
|
+
ascli aoc admin client_registration_token list --fields=id --format=csv|ascli aoc admin client_registration_token delete @lines:@stdin: --bulk=yes
|
4858
5061
|
```
|
4859
5062
|
|
4860
5063
|
```output
|
@@ -4890,9 +5093,9 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
4890
5093
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
4891
5094
|
```
|
4892
5095
|
|
4893
|
-
Once executed, the access key `id` and `secret`, randomly generated by the
|
5096
|
+
Once executed, the access key `id` and `secret`, randomly generated by the Node API, is displayed.
|
4894
5097
|
|
4895
|
-
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking
|
5098
|
+
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking IBM support.
|
4896
5099
|
|
4897
5100
|
- Create the AoC node entity
|
4898
5101
|
|
@@ -4915,12 +5118,12 @@ Creation of a node with a self-managed node is similar, but the command `aoc adm
|
|
4915
5118
|
Source files are provided as a list with the `sources` option.
|
4916
5119
|
Refer to section [File list](#list-of-files-for-transfers)
|
4917
5120
|
|
4918
|
-
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file
|
5121
|
+
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file ID` API).
|
4919
5122
|
|
4920
5123
|
Source files are located on **Aspera on cloud**, when :
|
4921
5124
|
|
4922
|
-
- The server is Aspera on Cloud, and executing a download or recv
|
4923
|
-
- The agent is Aspera on Cloud, and executing an upload or send
|
5125
|
+
- The server is Aspera on Cloud, and executing a `download` or `recv`
|
5126
|
+
- The agent is Aspera on Cloud, and executing an `upload` or `send`
|
4924
5127
|
|
4925
5128
|
In this case:
|
4926
5129
|
|
@@ -4953,10 +5156,10 @@ ascli aoc packages shared_inboxes list
|
|
4953
5156
|
|
4954
5157
|
Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: **user** or **shared inbox**:
|
4955
5158
|
|
4956
|
-
- Provide either
|
5159
|
+
- Provide either IDs as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
4957
5160
|
- or just names: `"recipients":[{"The Dest"}]`.
|
4958
5161
|
|
4959
|
-
ascli will resolve the list of email addresses and dropbox names to the expected type/
|
5162
|
+
ascli will resolve the list of email addresses and dropbox names to the expected type/ID list, based on case-insensitive partial match.
|
4960
5163
|
|
4961
5164
|
If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and:
|
4962
5165
|
|
@@ -4965,17 +5168,20 @@ If a user recipient (email) is not already registered and the workspace allows e
|
|
4965
5168
|
|
4966
5169
|
#### List packages
|
4967
5170
|
|
4968
|
-
By default, when using `aoc packages list`, the following `query` is performed:
|
5171
|
+
By default, when using `aoc packages list` or `aoc packages receive ALL`, the following `query` is performed:
|
4969
5172
|
|
4970
|
-
|
|
4971
|
-
|
5173
|
+
| Query parameter | Value |
|
5174
|
+
|----------------------------|---------|
|
4972
5175
|
| `archived` | `false` |
|
4973
5176
|
| `has_content` | `true` |
|
4974
5177
|
| `received` | `true` |
|
4975
5178
|
| `completed` | `true` |
|
4976
|
-
| `workspace_id` | based on current workspace |
|
4977
|
-
| `
|
4978
|
-
| `
|
5179
|
+
| `workspace_id` | Set based on current workspace |
|
5180
|
+
| `dropbox_id` | Set according to `dropbox_name`, if provided |
|
5181
|
+
| `exclude_dropbox_packages` | `true` unless `dropbox_id` is provided |
|
5182
|
+
|
5183
|
+
Parameters provided using option `query` override this query.
|
5184
|
+
To remove a parameter, set it to `null`.
|
4979
5185
|
|
4980
5186
|
#### Example: Send a package with one file to two users, using their email
|
4981
5187
|
|
@@ -5003,11 +5209,11 @@ When user packages are listed, the following query is used:
|
|
5003
5209
|
{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}
|
5004
5210
|
```
|
5005
5211
|
|
5006
|
-
To list packages in a shared inbox, the query has to be specified with the
|
5212
|
+
To list packages in a shared inbox, the query has to be specified with the shared inbox by name or its identifier.
|
5007
5213
|
Additional parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode).
|
5008
5214
|
The current workspace is added unless specified in the query.
|
5009
5215
|
|
5010
|
-
> **Note:** By default, `exclude_dropbox_packages` is set to true for user packages, and to false for shared inbox packages. This can be overridden in the query.
|
5216
|
+
> **Note:** By default, `exclude_dropbox_packages` is set to `true` for user packages, and to false for shared inbox packages. This can be overridden in the query.
|
5011
5217
|
|
5012
5218
|
Using shared inbox name:
|
5013
5219
|
|
@@ -5015,7 +5221,7 @@ Using shared inbox name:
|
|
5015
5221
|
ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
5016
5222
|
```
|
5017
5223
|
|
5018
|
-
Using shared inbox identifier: first retrieve the
|
5224
|
+
Using shared inbox identifier: first retrieve the ID of the shared inbox, and then list packages with the appropriate filter.
|
5019
5225
|
|
5020
5226
|
```bash
|
5021
5227
|
shared_box_id=$(ascli aoc packages shared_inboxes show --name='My Shared Inbox' --format=csv --display=data --fields=id)
|
@@ -5031,6 +5237,8 @@ ascli aoc packages list --query=@json:'{"dropbox_id":"'$shared_box_id'","archive
|
|
5031
5237
|
ascli aoc packages recv ALL --workspace=_workspace_ --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"_shared_inbox_name_","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
5032
5238
|
```
|
5033
5239
|
|
5240
|
+
To list packages that would be downloaded, without actually downloading them, replace `recv ALL` with `list` (keep options `once_only` and `query`)
|
5241
|
+
|
5034
5242
|
#### Example: Send a package with files from the Files app
|
5035
5243
|
|
5036
5244
|
Find files in Files app:
|
@@ -5064,7 +5272,7 @@ It is possible to automatically download new packages, like using Aspera Cargo:
|
|
5064
5272
|
ascli aoc packages recv ALL --once-only=yes --lock-port=12345
|
5065
5273
|
```
|
5066
5274
|
|
5067
|
-
- `ALL` (case
|
5275
|
+
- `ALL` (case-sensitive) will download all packages
|
5068
5276
|
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
5069
5277
|
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
5070
5278
|
|
@@ -5082,12 +5290,12 @@ ascli aoc package browse my5CnbeWng /
|
|
5082
5290
|
|
5083
5291
|
To list recursively, use command `find`.
|
5084
5292
|
|
5085
|
-
To download only some
|
5293
|
+
To download only some files listed in the package, just add the path of the files on the command line.
|
5086
5294
|
|
5087
5295
|
For advanced users, it's also possible to pipe node information for the package and use node operations:
|
5088
5296
|
|
5089
5297
|
```bash
|
5090
|
-
ascli aoc package node_info <package
|
5298
|
+
ascli aoc package node_info <package ID here> / --format=json --show-secrets=yes --display=data | ascli node -N --preset=@json:@stdin: access_key do self browse /
|
5091
5299
|
```
|
5092
5300
|
|
5093
5301
|
### Files
|
@@ -5116,13 +5324,13 @@ ascli aoc files download <single file path>
|
|
5116
5324
|
#### Shared folders
|
5117
5325
|
|
5118
5326
|
Like in AoC web UI, "Shared Folders" can be created and shared with either **Private** or **Public** links.
|
5119
|
-
**Private** links require the collaborator to log
|
5327
|
+
**Private** links require the collaborator to log in to access the shared folder.
|
5120
5328
|
**Public** links include a passcode that enables the user to access the shared folder without login-in.
|
5121
5329
|
|
5122
5330
|
Shared folders can be created either:
|
5123
5331
|
|
5124
5332
|
- by users in a workspace: they can share personal folders with other users in the same workspace: `aoc files perm`
|
5125
|
-
- by administrators: they can share a folder with users in any workspace: `aoc admin node do [node
|
5333
|
+
- by administrators: they can share a folder with users in any workspace: `aoc admin node do [node ID] perm`
|
5126
5334
|
|
5127
5335
|
Technically (API), shared folder are managed through [permissions](https://developer.ibm.com/apis/catalog/aspera--aspera-node-api/Introduction) on node and an event is sent to AoC to create a **link** in the user's home folder to the shared folder.
|
5128
5336
|
In both cases, it is necessary to specify a workspace.
|
@@ -5142,18 +5350,18 @@ The basic payload (last argument at creation usually specified with `@json:`) is
|
|
5142
5350
|
`ascli` expects the same payload for creation.
|
5143
5351
|
`ascli` automatically populates this payload like this:
|
5144
5352
|
|
5145
|
-
- `file_id` : the
|
5353
|
+
- `file_id` : the ID of the folder to share whose path is specified in the command line
|
5146
5354
|
- `access_levels` : are set by default to full access.
|
5147
|
-
- `tags` : are set with expected values for AoC:
|
5355
|
+
- `tags` : are set with expected values for AoC: username who creates, and workspace in which the shared folder is created.
|
5148
5356
|
- `access_type` and `access_id` : need to be set by the user, or using special key as follows.
|
5149
5357
|
|
5150
5358
|
To change `access_levels`, just provide the list of levels in the `@json:` payload.
|
5151
5359
|
|
5152
|
-
In order to declare/create the shared folder in the workspace, a special value for `access_id` is used: `ASPERA_ACCESS_KEY_ADMIN_WS_[workspace
|
5360
|
+
In order to declare/create the shared folder in the workspace, a special value for `access_id` is used: `ASPERA_ACCESS_KEY_ADMIN_WS_[workspace ID]]`. This is conveniently set by `ascli` using an empty string for the pseudo key `with`. In order to share a folder with a different, special tags are set, but this is conveniently done by `ascli` using the `as` key.
|
5153
5361
|
|
5154
5362
|
The following optional additional helper keys are supported by `ascli`:
|
5155
5363
|
|
5156
|
-
- `with` : Recipient of shared folder. Can be a
|
5364
|
+
- `with` : Recipient of shared folder. Can be a username, a group name, or a workspace name. `ascli` will resolve the name to the proper type and ID in fields `access_type` and `access_id`. If the value is the empty string, then it declares the shared folder in the workspace (first action to do, see below).
|
5157
5365
|
- `link_name` : The name of the link file created in the user's home folder for private links.
|
5158
5366
|
- `as` : The name of the link file created in the user's home folder for admin shared folders.
|
5159
5367
|
|
@@ -5194,7 +5402,7 @@ ascli aoc admin node do 1234 perm folder_on_node create @json:'{"with":"group 1"
|
|
5194
5402
|
ascli aoc admin node do 1234 perm folder_on_node create @json:'{"with":"my ws","as":"folder_for_users"}' --workspace="my ws"
|
5195
5403
|
```
|
5196
5404
|
|
5197
|
-
> **Note:** The node can be conveniently identified using the **percent selector** instead of numerical
|
5405
|
+
> **Note:** The node can be conveniently identified using the **percent selector** instead of numerical ID.
|
5198
5406
|
|
5199
5407
|
#### Cross Organization transfers
|
5200
5408
|
|
@@ -5204,9 +5412,9 @@ Although optional, the creation of [Option Preset](#option-preset) is recommende
|
|
5204
5412
|
|
5205
5413
|
Procedure to send a file from org1 to org2:
|
5206
5414
|
|
5207
|
-
- Get access to Organization 1 and create
|
5415
|
+
- Get access to Organization 1 and create an [Option Preset](#option-preset): e.g. `org1`, for instance, use the [Wizard](#configuration-using-wizard)
|
5208
5416
|
- Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
5209
|
-
- Get access to Organization 2 and create
|
5417
|
+
- Get access to Organization 2 and create an [Option Preset](#option-preset): e.g. `org2`
|
5210
5418
|
- Check that access works and locate the destination folder `mydestfolder`
|
5211
5419
|
- Execute the following:
|
5212
5420
|
|
@@ -5217,18 +5425,18 @@ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | as
|
|
5217
5425
|
Explanation:
|
5218
5426
|
|
5219
5427
|
- `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
5220
|
-
- `files node_info /mydestfolder` generate transfer information including
|
5428
|
+
- `files node_info /mydestfolder` generate transfer information including Node API credential and root ID, suitable for the next command
|
5221
5429
|
- `--format=json` format the output in JSON (instead of default text table)
|
5222
5430
|
- `--display=data` display only the result, and remove other information, such as workspace name
|
5223
5431
|
- `|` the standard output of the first command is fed into the second one
|
5224
5432
|
- `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
5225
5433
|
- `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
5226
5434
|
- `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agent-direct)
|
5227
|
-
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e.
|
5435
|
+
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. Node API credentials, those are expected in JSON format and read from standard input
|
5228
5436
|
|
5229
5437
|
#### Find Files
|
5230
5438
|
|
5231
|
-
The command `aoc files find` allows
|
5439
|
+
The command `aoc files find` allows searching for files in a given workspace.
|
5232
5440
|
|
5233
5441
|
It works also on `node` resource using the `v4` command:
|
5234
5442
|
|
@@ -5268,6 +5476,7 @@ admin dropbox_membership list
|
|
5268
5476
|
admin group list
|
5269
5477
|
admin kms_profile list
|
5270
5478
|
admin node do %name:my_node_name --secret=my_ak_secret browse /
|
5479
|
+
admin node do %name:my_node_name --secret=my_ak_secret browse /folder_sub --node-cache=no
|
5271
5480
|
admin node do %name:my_node_name --secret=my_ak_secret delete /folder1
|
5272
5481
|
admin node do %name:my_node_name --secret=my_ak_secret delete /folder_sub
|
5273
5482
|
admin node do %name:my_node_name --secret=my_ak_secret mkdir /folder1
|
@@ -5287,7 +5496,9 @@ admin package list --http-options=@json:'{"read_timeout":120.0}'
|
|
5287
5496
|
admin saml_configuration list
|
5288
5497
|
admin self show
|
5289
5498
|
admin short_link list
|
5290
|
-
admin subscription
|
5499
|
+
admin subscription account
|
5500
|
+
admin subscription usage
|
5501
|
+
admin subscription usage MONTH
|
5291
5502
|
admin user list
|
5292
5503
|
admin user modify %name:my_user_email @json:'{"deactivated":false}'
|
5293
5504
|
admin workspace list
|
@@ -5309,13 +5520,13 @@ files browse / --url=my_public_link_folder_pass --password=my_public_link_passwo
|
|
5309
5520
|
files browse my_remote_file
|
5310
5521
|
files browse my_remote_folder
|
5311
5522
|
files browse my_remote_folder/
|
5523
|
+
files cat --to-folder=. testdst/test_file.bin
|
5312
5524
|
files delete /testsrc
|
5313
|
-
files download --transfer=alpha testdst/test_file.bin
|
5314
5525
|
files download --transfer=connect testdst/test_file.bin
|
5526
|
+
files download --transfer=desktop testdst/test_file.bin
|
5315
5527
|
files find /
|
5316
5528
|
files find / '\.partial$'
|
5317
5529
|
files find / @ruby:'->(f){f["type"].eql?("file")}'
|
5318
|
-
files http_node_download --to-folder=. testdst/test_file.bin
|
5319
5530
|
files mkdir /testsrc
|
5320
5531
|
files modify my_test_folder
|
5321
5532
|
files permission my_test_folder list
|
@@ -5331,8 +5542,8 @@ files sync admin status --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","
|
|
5331
5542
|
files sync start --sync-info=@json:'{"name":"my_aoc_sync1","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
|
5332
5543
|
files sync start --sync-info=@json:'{"name":"my_aoc_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
|
5333
5544
|
files sync start --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/testdst","reset":true}]}'
|
5334
|
-
files sync start pull /
|
5335
|
-
files sync start pull /
|
5545
|
+
files sync start pull /data/local_sync /testdst --sync-info=@json:'{"reset":true,"quiet":false}'
|
5546
|
+
files sync start pull /data/local_sync /testdst --sync-info=@json:'{"sessions":[{"reset":true}]}'
|
5336
5547
|
files thumbnail my_test_folder/video_file.mpg
|
5337
5548
|
files thumbnail my_test_folder/video_file.mpg --query=@json:'{"text":true,"double":true}'
|
5338
5549
|
files transfer push /testsrc --to-folder=/testdst test_file.bin
|
@@ -5340,7 +5551,7 @@ files upload --to-folder=/ test_file.bin --url=my_public_link_folder_no_pass
|
|
5340
5551
|
files upload --to-folder=/testsrc test_file.bin
|
5341
5552
|
files upload --workspace=my_other_workspace --to-folder=my_other_folder test_file.bin --transfer=node --transfer-info=@json:@stdin:
|
5342
5553
|
files v3 info
|
5343
|
-
gateway --pid-file=pid_aocfxgw https://localhost:12345/aspera/faspex &
|
5554
|
+
gateway --pid-file=pid_aocfxgw @json:'{"url":"https://localhost:12345/aspera/faspex"}' &
|
5344
5555
|
org --url=my_public_link_recv_from_aoc_user
|
5345
5556
|
organization
|
5346
5557
|
packages browse package_id3 /
|
@@ -5380,12 +5591,12 @@ ATS is usable either :
|
|
5380
5591
|
|
5381
5592
|
- Or from an IBM Cloud subscription : `ascli ats` : use IBM Cloud API key authentication
|
5382
5593
|
|
5383
|
-
### IBM Cloud ATS : Creation of
|
5594
|
+
### IBM Cloud ATS : Creation of API key
|
5384
5595
|
|
5385
5596
|
This section is about using ATS with an IBM cloud subscription.
|
5386
5597
|
If you are using ATS as part of AoC, then authentication is through AoC, not IBM Cloud.
|
5387
5598
|
|
5388
|
-
First get your IBM Cloud
|
5599
|
+
First get your IBM Cloud API key. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
5389
5600
|
|
5390
5601
|
```bash
|
5391
5602
|
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
@@ -5413,7 +5624,7 @@ References:
|
|
5413
5624
|
|
5414
5625
|
<!-- spellchecker: enable -->
|
5415
5626
|
|
5416
|
-
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
5627
|
+
Then, to register the key by default for the `ats` plugin, create a preset. Execute:
|
5417
5628
|
|
5418
5629
|
```bash
|
5419
5630
|
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here
|
@@ -5460,7 +5671,7 @@ Those are directly the parameters expected by the [ATS API](https://developer.ib
|
|
5460
5671
|
|
5461
5672
|
### Misc. Examples
|
5462
5673
|
|
5463
|
-
Example: create access key on IBM Cloud (
|
5674
|
+
Example: create access key on IBM Cloud (Softlayer):
|
5464
5675
|
|
5465
5676
|
```bash
|
5466
5677
|
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"my_secret_here","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
@@ -5478,7 +5689,7 @@ Example: create access key on Azure SAS:
|
|
5478
5689
|
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
5479
5690
|
```
|
5480
5691
|
|
5481
|
-
|
5692
|
+
> **Note:** The blob name is mandatory after server address and before parameters, and that parameter `sr=c` is mandatory.
|
5482
5693
|
|
5483
5694
|
Example: create access key on Azure:
|
5484
5695
|
|
@@ -5606,13 +5817,13 @@ For file operation command (browse, delete), the Ruby SSH client library `Net::S
|
|
5606
5817
|
|
5607
5818
|
For a list of SSH client options, refer to the Ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html#method-c-start).
|
5608
5819
|
|
5609
|
-
|
5820
|
+
Among the 50 available SSH options:
|
5610
5821
|
|
5611
5822
|
- `verbose`
|
5612
5823
|
- `use_agent`
|
5613
5824
|
- `passphrase`
|
5614
5825
|
|
5615
|
-
By default the SSH library will check if a local `ssh-agent` is running.
|
5826
|
+
By default, the SSH library will check if a local `ssh-agent` is running.
|
5616
5827
|
|
5617
5828
|
On Linux, if you get an error message such as:
|
5618
5829
|
|
@@ -5626,7 +5837,7 @@ or on Windows:
|
|
5626
5837
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
5627
5838
|
```
|
5628
5839
|
|
5629
|
-
This means that your environment suggests
|
5840
|
+
This means that your environment suggests using an agent, but you don't have such an SSH agent running, then:
|
5630
5841
|
|
5631
5842
|
- Check env var: `SSH_AGENT_SOCK`
|
5632
5843
|
- Check your file: `$HOME/.ssh/config`
|
@@ -5646,12 +5857,12 @@ It is equivalent to setting both options `ssh_options.passphrase` and `ts.ssh_pr
|
|
5646
5857
|
### Other session channels for `server`
|
5647
5858
|
|
5648
5859
|
URL schemes `local` and `https` are also supported (mainly for testing purpose).
|
5649
|
-
(`--url=local
|
5860
|
+
(`--url=local:`, `--url=https://...`)
|
5650
5861
|
|
5651
5862
|
- `local` will execute `ascmd` locally, instead of using an SSH connection.
|
5652
5863
|
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
5653
5864
|
|
5654
|
-
As, most of the time, SSH is used, if
|
5865
|
+
As, most of the time, SSH is used, if a `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
5655
5866
|
|
5656
5867
|
### Examples: `server`
|
5657
5868
|
|
@@ -5663,9 +5874,9 @@ ascli server browse /aspera-test-dir-large
|
|
5663
5874
|
ascli server download /aspera-test-dir-large/200MB
|
5664
5875
|
```
|
5665
5876
|
|
5666
|
-
`initdemo` creates
|
5877
|
+
`initdemo` creates an [Option Preset](#option-preset) `demoserver` and set it as default for plugin `server`.
|
5667
5878
|
|
5668
|
-
If an SSH private key is used for authentication with a passphrase, the passphrase needs to be provided to both options: `ssh_options
|
5879
|
+
If an SSH private key is used for authentication with a passphrase, the passphrase needs to be provided to both options: `ssh_options` (for browsing) and `ts` (for transfers):
|
5669
5880
|
|
5670
5881
|
```bash
|
5671
5882
|
ascli server --url=ssh://_server_address_here_:33001 --username=_user_here_ --ssh_keys=_private_key_path_here_ --passphrase=_passphrase_here_
|
@@ -5673,13 +5884,13 @@ ascli server --url=ssh://_server_address_here_:33001 --username=_user_here_ --ss
|
|
5673
5884
|
|
5674
5885
|
## Plugin: `node`: IBM Aspera High Speed Transfer Server Node
|
5675
5886
|
|
5676
|
-
This plugin gives access to capabilities provided by the HSTS
|
5887
|
+
This plugin gives access to capabilities provided by the HSTS Node API.
|
5677
5888
|
|
5678
5889
|
The authentication is `username` and `password` or `access_key` and `secret` through options: `username` and `password`.
|
5679
5890
|
|
5680
|
-
> **Note:** Capabilities of this plugin are used in other plugins which access to the
|
5891
|
+
> **Note:** Capabilities of this plugin are used in other plugins which access to the Node API, such as `aoc`, `ats`, `shares`.
|
5681
5892
|
>
|
5682
|
-
> **Note:** This plugin can be used with any type of **Aspera Node**, either on
|
5893
|
+
> **Note:** This plugin can be used with any type of **Aspera Node**, either on premise or ATS, provided that you have Node API credentials. Those credentials can be either Node API user or Access Key (e.g. on ATS).
|
5683
5894
|
|
5684
5895
|
### File Operations
|
5685
5896
|
|
@@ -5718,7 +5929,7 @@ The command `find <folder> [filter_expr]` is available for **gen4/access key**,
|
|
5718
5929
|
The argument `<folder>` is mandatory and is the root from which search is performed.
|
5719
5930
|
The argument `[filter_expr]` is optional and represent the matching criteria.
|
5720
5931
|
|
5721
|
-
It recursively scans storage to find files/folders matching
|
5932
|
+
It recursively scans storage to find files/folders matching criteria and then returns a list of matching entries.
|
5722
5933
|
|
5723
5934
|
`[filter_expr]` is either:
|
5724
5935
|
|
@@ -5740,10 +5951,10 @@ Examples of expressions:
|
|
5740
5951
|
ascli node access_keys do self find /Documents '*.txt'
|
5741
5952
|
```
|
5742
5953
|
|
5743
|
-
The following are examples of
|
5954
|
+
The following are examples of Ruby lambda code to be provided in the following template command:
|
5744
5955
|
|
5745
5956
|
```bash
|
5746
|
-
ascli node access_keys do self find / @ruby:'
|
5957
|
+
ascli node access_keys do self find / @ruby:'->(f){[code here]}'
|
5747
5958
|
```
|
5748
5959
|
|
5749
5960
|
> **Note:** Single quotes are used here above to protect the whole **Ruby** expression from the shell. Then double quotes are used for strings in the **Ruby** expression to not mix with the shell.
|
@@ -5760,7 +5971,7 @@ The following are examples of `ruby_lambda` to be provided in the following temp
|
|
5760
5971
|
->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}
|
5761
5972
|
```
|
5762
5973
|
|
5763
|
-
- Find files larger than
|
5974
|
+
- Find files larger than 1 MB
|
5764
5975
|
|
5765
5976
|
```ruby
|
5766
5977
|
->(f){f["type"].eql?("file") and f["size"].to_i>1000000}
|
@@ -5788,10 +5999,10 @@ ascli node access_keys do self find / @ruby:'->(f){f["type"].eql?("file") and (D
|
|
5788
5999
|
|
5789
6000
|
### Listing transfer events
|
5790
6001
|
|
5791
|
-
When a transfer is run, its information is stored (
|
6002
|
+
When a transfer is run, its information is stored (typically, 1 day) in the HSTS database (Redis).
|
5792
6003
|
This information can be retrieved with command: `transfer list`.
|
5793
6004
|
|
5794
|
-
If the number of
|
6005
|
+
If the number of transfers is too large, then the list will be retrieved using several API calls.
|
5795
6006
|
|
5796
6007
|
In addition, it is possible to list "only new information" using option `once_only`.
|
5797
6008
|
|
@@ -5799,14 +6010,14 @@ In addition, it is possible to list "only new information" using option `once_on
|
|
5799
6010
|
ascli node transfer list --once-only=yes
|
5800
6011
|
```
|
5801
6012
|
|
5802
|
-
The `
|
6013
|
+
The `iteration_token` that keeps memory of the latest event is stored in the persistence repository of `ascli`.
|
5803
6014
|
To reset it, add option: `--query=@json:'{"reset": true}'`.
|
5804
6015
|
To list only a number of events, use the `max` parameter in query.
|
5805
6016
|
Other parameters are directly transmitted to the underlying API (`GET /ops/transfers`).
|
5806
6017
|
|
5807
6018
|
### Central
|
5808
6019
|
|
5809
|
-
The central
|
6020
|
+
The central sub-command uses the **reliable query** API (session and file).
|
5810
6021
|
It allows listing transfer sessions and transferred files.
|
5811
6022
|
|
5812
6023
|
Filtering can be applied:
|
@@ -5824,12 +6035,12 @@ By providing the `validator` option, offline transfer validation can be done.
|
|
5824
6035
|
There are three sync types of operations:
|
5825
6036
|
|
5826
6037
|
- `sync`: perform a local sync, by executing `async` locally
|
5827
|
-
- `async`: calls legacy async API on node : `/async`
|
5828
|
-
- `ssync` : calls newer async API on node : `/asyncs`
|
6038
|
+
- `async`: calls legacy `async` API on node : `/async`
|
6039
|
+
- `ssync` : calls newer `async` API on node : `/asyncs`
|
5829
6040
|
|
5830
6041
|
### FASP Stream
|
5831
6042
|
|
5832
|
-
It is possible to start a
|
6043
|
+
It is possible to start a faspstream session using the Node API:
|
5833
6044
|
|
5834
6045
|
Use the command `ascli node stream create --ts=@json:<value>`, with [**transfer-spec**](#transfer-specification):
|
5835
6046
|
|
@@ -5841,10 +6052,10 @@ Use the command `ascli node stream create --ts=@json:<value>`, with [**transfer-
|
|
5841
6052
|
|
5842
6053
|
Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/entsrv/3.7.4/es_admin_linux/webhelp/index.html#watchfolder_external/dita/json_conf.html) for watch folder creation.
|
5843
6054
|
|
5844
|
-
`ascli` supports remote operations through the
|
6055
|
+
`ascli` supports remote operations through the Node API. Operations are:
|
5845
6056
|
|
5846
6057
|
- Start `watchd` and `watchfolderd` services running as a system user having access to files
|
5847
|
-
- Configure a **
|
6058
|
+
- Configure a **Watchfolder** to define automated transfers
|
5848
6059
|
|
5849
6060
|
```bash
|
5850
6061
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -5884,7 +6095,7 @@ updated
|
|
5884
6095
|
|
5885
6096
|
Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner.
|
5886
6097
|
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
|
5887
|
-
Simply create an **Aspera Transfer Service** instance, which provides access to the
|
6098
|
+
Simply create an **Aspera Transfer Service** instance, which provides access to the Node API.
|
5888
6099
|
Then create a configuration for the **SHOD** instance in the configuration file: in section **shares**, a configuration named: `aws_shod`.
|
5889
6100
|
Create another configuration for the Azure ATS instance: in section **node**, named `azure_ats`.
|
5890
6101
|
Then execute the following command:
|
@@ -5897,7 +6108,7 @@ This will get transfer information from the SHOD instance and tell the Azure ATS
|
|
5897
6108
|
|
5898
6109
|
### Node file information
|
5899
6110
|
|
5900
|
-
When
|
6111
|
+
When Node API is used with an **Access key**, extra information can be retrieved, such as preview.
|
5901
6112
|
|
5902
6113
|
> **Note:** Display of preview on terminal requires installation of extra gem: `rmagick`
|
5903
6114
|
|
@@ -5918,7 +6129,7 @@ Previews are mainly used in AoC, this also works with AoC:
|
|
5918
6129
|
ascli aoc files thumbnail /preview_samples/Aspera.mpg
|
5919
6130
|
```
|
5920
6131
|
|
5921
|
-
> **Note:** To specify the file by its file
|
6132
|
+
> **Note:** To specify the file by its file ID, use the selector syntax: `%id:_file_id_here_`
|
5922
6133
|
>
|
5923
6134
|
> **Note:** To force textual display of the preview on **iTerm**, prefix command with: `env -u TERM_PROGRAM -u LC_TERMINAL` or use option: ``
|
5924
6135
|
|
@@ -5929,7 +6140,7 @@ ascli node access_key create @json:'{"id":"myaccesskey","secret":"my_secret_here
|
|
5929
6140
|
```
|
5930
6141
|
|
5931
6142
|
> **Note:** The `id` and `secret` are optional.
|
5932
|
-
> If not provided, they will be generated and returned
|
6143
|
+
> If not provided, they will be generated and returned into the result.
|
5933
6144
|
|
5934
6145
|
Access keys support extra overriding parameters using parameter: `configuration` and sub keys `transfer` and `server`. For example, an access key can be modified or created with the following options:
|
5935
6146
|
|
@@ -5949,36 +6160,36 @@ Bearer tokens are part of the **gen4/access key** API.
|
|
5949
6160
|
It follows the model of OAuth 2.
|
5950
6161
|
For example, they are used in Aspera on Cloud.
|
5951
6162
|
This is also available for developers for any application integrating Aspera.
|
5952
|
-
In this API, files, users and groups are identified by an
|
6163
|
+
In this API, files, users and groups are identified by an ID (a `String`, e.g. `"125"`, not necessarily numerical).
|
5953
6164
|
|
5954
|
-
Bearer tokens are typically generated by the
|
6165
|
+
Bearer tokens are typically generated by the authenticating application and then recognized by the Node API.
|
5955
6166
|
A bearer token is authorized on the node by creating `permissions` on a **folder**.
|
5956
6167
|
|
5957
|
-
Bearer tokens can be generated using command `bearer_token`: it takes two arguments:
|
6168
|
+
Bearer tokens can be generated using `ascli` command `bearer_token`: it takes two arguments:
|
5958
6169
|
|
5959
|
-
- The private key used to sign the token
|
6170
|
+
- The private key used to sign the token.
|
5960
6171
|
- The token information, which is a `Hash` containing the following elements:
|
5961
6172
|
|
5962
|
-
|
|
6173
|
+
| Parameter | Default | Type | Description |
|
5963
6174
|
| ------------------------ |-----------------------------|-----------|----------------------------------------------------------|
|
5964
|
-
|
|
5965
|
-
|
|
5966
|
-
|
|
5967
|
-
|
|
5968
|
-
|
|
5969
|
-
|
|
5970
|
-
|
|
5971
|
-
|
|
5972
|
-
|
|
6175
|
+
| `_scope` | `user:all` | Special | Either `user:all` or `admin:all` |
|
6176
|
+
| `_validity` | 86400 | Special | Validity in seconds from now. |
|
6177
|
+
| `user_id` | - | Mandatory | Identifier of user |
|
6178
|
+
| `scope` | `node.<access_key>:<_scope>`| Mandatory | API scope, e.g. `node.<access_key>:<node scope>` |
|
6179
|
+
| `expires_at` | `now+<_validity>` | Mandatory | Format: `%Y-%m-%dT%H:%M:%SZ` e.g. `2021-12-31T23:59:59Z` |
|
6180
|
+
| `auth_type` | `access_key` | Optional | `access_key`, `node_user` |
|
6181
|
+
| `group_ids` | - | Optional | List of group IDs |
|
6182
|
+
| `organization_id` | - | Optional | Organization ID |
|
6183
|
+
| `watermarking_json_base64` | - | Optional | Watermarking information (not used) |
|
5973
6184
|
|
5974
6185
|
> **Note:** For convenience, `ascli` provides additional parameters `_scope` and `_validity`.
|
5975
6186
|
> They are not part of the API and are removed from the final payload.
|
5976
|
-
> They are used respectively to
|
6187
|
+
> They are used respectively to easily set a value for `scope` and `expires_at`.
|
5977
6188
|
|
5978
6189
|
#### Bearer token: Environment
|
5979
6190
|
|
5980
6191
|
An access key shall be created to grant access for transfers to its storage.
|
5981
|
-
The
|
6192
|
+
The access key and its secret represent administrative access to the storage as it has access rights to the whole storage of the access key.
|
5982
6193
|
|
5983
6194
|
They way to create access keys depend slightly on the type of HSTS:
|
5984
6195
|
|
@@ -5990,36 +6201,40 @@ They way to create access keys depend slightly on the type of HSTS:
|
|
5990
6201
|
|
5991
6202
|
- If Aspera on Cloud or ATS is used, then the SaaS API for access key creation is used.
|
5992
6203
|
|
6204
|
+
> **Note:** Refer to [HSTS manual](https://www.ibm.com/docs/en/ahts): `Access key authentication` section for more details on access key creation.
|
6205
|
+
|
6206
|
+
In the next sections, we will assume that an access key has been created and that `ascli` is configured to use this access key by default using `node`.
|
6207
|
+
|
5993
6208
|
#### Bearer token: Preparation
|
5994
6209
|
|
5995
6210
|
Let's assume that the access key was created, and a default configuration is set to use this **access key**.
|
5996
6211
|
Using `ascli`, an access key can be created using the `access_key create` on the node (using main node credentials) or ATS.
|
5997
6212
|
|
5998
|
-
|
6213
|
+
Create a private key (organization key) that will be used to sign bearer tokens:
|
5999
6214
|
|
6000
|
-
|
6001
|
-
|
6002
|
-
|
6003
|
-
|
6215
|
+
```bash
|
6216
|
+
my_private_pem=./myorgkey.pem
|
6217
|
+
ascli config genkey $my_private_pem
|
6218
|
+
```
|
6004
6219
|
|
6005
|
-
|
6006
|
-
|
6220
|
+
> **Note:** This key is not used for authentication, it is used to sign bearer tokens.
|
6221
|
+
> Refer to section [private key](#private-key) for more details on generation.
|
6007
6222
|
|
6008
|
-
|
6223
|
+
The corresponding public key shall be placed as an attribute of the **access key** (done with `PUT /access_keys/<id>`):
|
6009
6224
|
|
6010
|
-
|
6011
|
-
|
6012
|
-
|
6225
|
+
```bash
|
6226
|
+
ascli node access_key set_bearer_key self @file:$my_private_pem
|
6227
|
+
```
|
6013
6228
|
|
6014
|
-
|
6015
|
-
|
6016
|
-
|
6229
|
+
> **Note:** Either the public or private key can be provided, and only the public key is used.
|
6230
|
+
> This will enable to check the signature of the bearer token.
|
6231
|
+
> Above command is executed with access key credentials.
|
6017
6232
|
|
6018
|
-
|
6233
|
+
Alternatively, use the following equivalent command, as `ascli` kindly extracts the public key with extension `.pub`:
|
6019
6234
|
|
6020
|
-
|
6021
|
-
|
6022
|
-
|
6235
|
+
```bash
|
6236
|
+
ascli node access_key modify %id:self @ruby:'{token_verification_key: File.read("'$my_private_pem'.pub")}'
|
6237
|
+
```
|
6023
6238
|
|
6024
6239
|
#### Bearer token: Configuration for user
|
6025
6240
|
|
@@ -6031,14 +6246,14 @@ Using `ascli`, an access key can be created using the `access_key create` on the
|
|
6031
6246
|
|
6032
6247
|
> **Note:** Here we simply select `/`, but any folder can be selected in the access key storage.
|
6033
6248
|
|
6034
|
-
- Let's designate a user by its
|
6249
|
+
- Let's designate a user by its ID:
|
6035
6250
|
|
6036
6251
|
```bash
|
6037
6252
|
my_user_id=777
|
6038
6253
|
```
|
6039
6254
|
|
6040
6255
|
> **Note:** This is an arbitrary identifier, typically managed by the web application.
|
6041
|
-
> Not related to Linux user
|
6256
|
+
> Not related to Linux user IDs or anything else.
|
6042
6257
|
|
6043
6258
|
- Grant this user access to the selected folder:
|
6044
6259
|
|
@@ -6052,18 +6267,20 @@ Using `ascli`, an access key can be created using the `access_key create` on the
|
|
6052
6267
|
ascli node bearer_token @file:./myorgkey.pem @json:'{"user_id":"'$my_user_id'","_validity":3600}' --output=bearer.txt
|
6053
6268
|
```
|
6054
6269
|
|
6270
|
+
> **Note:** The Bearer token can also be created using command `asnodeadmin` on HSTS. Refer to the [HSTS manual](https://www.ibm.com/docs/en/ahts): `Bearer tokens` section. Code for token generation is provided in [lib/aspera/api/node.rb](lib/aspera/api/node.rb)
|
6271
|
+
|
6055
6272
|
#### Bearer token: User side
|
6056
6273
|
|
6057
6274
|
Now, let's assume we are the user, the only information received are:
|
6058
6275
|
|
6059
|
-
- The URL of the
|
6276
|
+
- The URL of the Node API
|
6060
6277
|
- A Bearer token
|
6061
|
-
- A file
|
6278
|
+
- A file ID for which we have access
|
6062
6279
|
|
6063
6280
|
Let's use it:
|
6064
6281
|
|
6065
6282
|
```bash
|
6066
|
-
ascli node -N --url
|
6283
|
+
ascli node -N --url=https://... --password="Bearer $(cat bearer.txt)" --root-id=$my_folder_id access_key do self br /
|
6067
6284
|
```
|
6068
6285
|
|
6069
6286
|
### Node sample commands
|
@@ -6087,6 +6304,7 @@ access_key do my_ak_name show %id:1
|
|
6087
6304
|
access_key do my_ak_name show /test_nd_ak3
|
6088
6305
|
access_key do my_ak_name upload 'faux:///test_nd_ak3?100k' --default-ports=no
|
6089
6306
|
access_key do self permission %id:root_id create @json:'{"access_type":"user","access_id":"666"}'
|
6307
|
+
access_key do self permission / show 1
|
6090
6308
|
access_key do self show / --fields=id --output=root_id
|
6091
6309
|
access_key list
|
6092
6310
|
access_key set_bearer_key self @file:my_private_key
|
@@ -6102,6 +6320,7 @@ async show ALL
|
|
6102
6320
|
basic_token
|
6103
6321
|
bearer_token @file:my_private_key @json:'{"user_id":"666"}' --output=bearer_666
|
6104
6322
|
browse / --log-level=trace2
|
6323
|
+
cat my_upload_folder/test_file.bin --to-folder=.
|
6105
6324
|
central file list
|
6106
6325
|
central file modify --validator=1 @json:'{"files":[]}'
|
6107
6326
|
central session list
|
@@ -6109,7 +6328,6 @@ delete @list:,my_upload_folder/a_folder,my_upload_folder/tdlink,my_upload_folder
|
|
6109
6328
|
delete my_upload_folder/test_file.bin
|
6110
6329
|
download my_upload_folder/test_file.bin --to-folder=.
|
6111
6330
|
health
|
6112
|
-
http_node_download my_upload_folder/test_file.bin --to-folder=.
|
6113
6331
|
info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
|
6114
6332
|
license
|
6115
6333
|
mkdir my_upload_folder/a_folder
|
@@ -6149,6 +6367,32 @@ upload test_file.bin --to-folder=my_upload_folder --ts=@json:'{"target_rate_cap_
|
|
6149
6367
|
watch_folder list
|
6150
6368
|
```
|
6151
6369
|
|
6370
|
+
### Open Telemetry
|
6371
|
+
|
6372
|
+
The Node plugin supports Open Telemetry (OTel) for monitoring and tracing.
|
6373
|
+
|
6374
|
+
`ascli` can poll the Node API for transfer events and send them to an OTel collector.
|
6375
|
+
|
6376
|
+
The command expects the following parameters provided as a `Hash` positional parameter:
|
6377
|
+
|
6378
|
+
| Parameter | Type | Default | Description |
|
6379
|
+
|-------------|----------|---------|---------------------------------|
|
6380
|
+
| `url` | `String` | - | URL of the Instana backend. |
|
6381
|
+
| `apikey` | `String` | - | Token for the OTel collector. |
|
6382
|
+
| `interval` | `Float` | 10 | Polling interval in seconds. |
|
6383
|
+
|
6384
|
+
For convenience, those parameters can be provided in a preset, e.g. `otel_default`.
|
6385
|
+
|
6386
|
+
```bash
|
6387
|
+
ascli config preset init otel_default @json:'{"url":"https://otlp-orange-saas.instana.io:4318","apikey":"*********","interval":1.1}'
|
6388
|
+
```
|
6389
|
+
|
6390
|
+
Then it is invoked like this (assuming a default node is configured):
|
6391
|
+
|
6392
|
+
```bash
|
6393
|
+
ascli node telemetry @preset:otel_default
|
6394
|
+
```
|
6395
|
+
|
6152
6396
|
## Plugin: `faspex5`: IBM Aspera Faspex v5
|
6153
6397
|
|
6154
6398
|
IBM Aspera's newer self-managed application.
|
@@ -6164,7 +6408,7 @@ IBM Aspera's newer self-managed application.
|
|
6164
6408
|
|
6165
6409
|
> **Note:** If you have a Faspex 5 public link, provide it, unchanged, through the option `url`
|
6166
6410
|
|
6167
|
-
For a quick start, one can use the wizard, which will help
|
6411
|
+
For a quick start, one can use the wizard, which will help to create an [Option Preset](#option-preset):
|
6168
6412
|
|
6169
6413
|
```bash
|
6170
6414
|
ascli config wizard
|
@@ -6186,7 +6430,7 @@ option: key_path>
|
|
6186
6430
|
Using existing key:
|
6187
6431
|
/Users/someuser/.aspera/ascli/my_key
|
6188
6432
|
option: username> someuser@example.com
|
6189
|
-
Ask the ascli client
|
6433
|
+
Ask the ascli client ID and secret to your Administrator.
|
6190
6434
|
Admin should login to: https://faspex5.example.com/aspera/faspex
|
6191
6435
|
Navigate to: :: → Admin → Configurations → API clients
|
6192
6436
|
Create an API client with:
|
@@ -6225,11 +6469,11 @@ Activation is in two steps:
|
|
6225
6469
|
- As Admin, Navigate to the web UI: Admin → Configurations → API Clients → Create
|
6226
6470
|
- Give a name, like `ascli`
|
6227
6471
|
- Activate JWT
|
6228
|
-
- There is an option to set a
|
6472
|
+
- There is an option to set a global public key allowing the owner of the private key to impersonate any user. Unless you want to do this, leave this field empty.
|
6229
6473
|
- Click on `Create` Button
|
6230
|
-
- Take note of Client
|
6474
|
+
- Take note of Client ID (and Client Secret, but not used in current version)
|
6231
6475
|
|
6232
|
-
- The user will authenticate with a private key and set the public key in his
|
6476
|
+
- The user will authenticate with a private key and set the public key in his Faspex 5 profile.
|
6233
6477
|
|
6234
6478
|
**Note:** If you don't have a private key refer to section [Private Key](#private-key) to generate one.
|
6235
6479
|
|
@@ -6271,7 +6515,7 @@ The administrator must create an API client in Faspex for an external web app su
|
|
6271
6515
|
- Do not Activate JWT
|
6272
6516
|
- Set **Redirect URI** to `https://127.0.0.1:8888`
|
6273
6517
|
- Click on `Create` Button
|
6274
|
-
- Take note of Client Id (and Client Secret, but not used in current version)
|
6518
|
+
- Take note of the Client Id (and Client Secret, but not used in current version)
|
6275
6519
|
|
6276
6520
|
The user will use the following options:
|
6277
6521
|
|
@@ -6303,6 +6547,7 @@ ascli config preset update f5boot --url=https://localhost/aspera/faspex --auth=b
|
|
6303
6547
|
|
6304
6548
|
```bash
|
6305
6549
|
admin accounts list
|
6550
|
+
admin alternate_addresses list
|
6306
6551
|
admin clean_deleted
|
6307
6552
|
admin contacts list
|
6308
6553
|
admin distribution_lists create @json:'{"name":"test4","contacts":[{"name":"john@example.com"}]}'
|
@@ -6312,7 +6557,12 @@ admin event app --query=@json:'{"max":20}'
|
|
6312
6557
|
admin event web
|
6313
6558
|
admin jobs list --query=@json:'{"job_type":"email","status":"failed"}' --fields=id,error_desc
|
6314
6559
|
admin metadata_profiles list
|
6560
|
+
admin node browse %name:Local
|
6315
6561
|
admin node list
|
6562
|
+
admin node shared_folders %name:Local list
|
6563
|
+
admin node shared_folders %name:Local show %name:Main
|
6564
|
+
admin node shared_folders %name:Local user %name:Main list
|
6565
|
+
admin node show %name:Local
|
6316
6566
|
admin oauth_clients list --query=@json:'[["client_types[]","public"]]'
|
6317
6567
|
admin registrations list
|
6318
6568
|
admin saml_configs list
|
@@ -6327,10 +6577,12 @@ admin smtp show
|
|
6327
6577
|
admin smtp test my_email_external
|
6328
6578
|
admin workgroups list
|
6329
6579
|
bearer_token
|
6330
|
-
gateway --pid-file=pid_f5_fxgw https://localhost:12346/aspera/faspex &
|
6580
|
+
gateway --pid-file=pid_f5_fxgw @json:'{"url":"https://localhost:12346/aspera/faspex"}' &
|
6331
6581
|
health
|
6332
6582
|
invitation list
|
6333
6583
|
invitations create @json:'{"email_address":"aspera.user1+u@gmail.com"}'
|
6584
|
+
packages browse f5_pack_id --query=@json:'{"recursive":true}'
|
6585
|
+
packages delete f5_pack_id
|
6334
6586
|
packages list --box=my_shared_box_name
|
6335
6587
|
packages list --box=my_workgroup --group-type=workgroups
|
6336
6588
|
packages list --query=@json:'{"mailbox":"inbox","status":"completed"}'
|
@@ -6338,8 +6590,8 @@ packages receive --box=my_shared_box_name package_box_id1 --to-folder=.
|
|
6338
6590
|
packages receive --box=my_workgroup --group-type=workgroups workgroup_package_id1 --to-folder=.
|
6339
6591
|
packages receive ALL --once-only=yes --to-folder=.
|
6340
6592
|
packages receive INIT --once-only=yes
|
6341
|
-
packages receive
|
6342
|
-
packages send --shared-folder=%name:my_shared_folder_name @json:'{"title":"test title","recipients":["my_email_internal"]}' my_shared_folder_file
|
6593
|
+
packages receive f5_pack_id --to-folder=. --ts=@json:'{"content_protection_password":"my_secret_here"}'
|
6594
|
+
packages send --shared-folder=%name:my_shared_folder_name @json:'{"title":"test title","recipients":["my_email_internal"]}' my_shared_folder_file --fields=id --display=data --output=f5_pack_id
|
6343
6595
|
packages send --url=my_public_link_send_f5_user @json:'{"title":"test title"}' test_file.bin
|
6344
6596
|
packages send --url=my_public_link_send_shared_box @json:'{"title":"test title"}' test_file.bin
|
6345
6597
|
packages send @json:'{"title":"test title","recipients":["my_shared_box_name"],"metadata":{"Options":"Opt1","TextInput":"example text"}}' test_file.bin
|
@@ -6347,9 +6599,9 @@ packages send @json:'{"title":"test title","recipients":["my_workgroup"]}' test_
|
|
6347
6599
|
packages send @json:'{"title":"test title","recipients":[{"name":"my_username"}]my_meta}' test_file.bin --ts=@json:'{"content_protection_password":"my_secret_here"}'
|
6348
6600
|
packages show --box=my_shared_box_name package_box_id1
|
6349
6601
|
packages show --box=my_workgroup --group-type=workgroups workgroup_package_id1
|
6350
|
-
packages show
|
6351
|
-
packages status
|
6352
|
-
postprocessing --pid-file=pid_f5_postproc @json:'{"url":"
|
6602
|
+
packages show f5_pack_id
|
6603
|
+
packages status f5_pack_id
|
6604
|
+
postprocessing --pid-file=pid_f5_postproc @json:'{"url":"https://localhost:8553/asclihook","script_folder":"","cert":"localhost.p12","key":"changeit"}' &
|
6353
6605
|
shared browse %name:my_src
|
6354
6606
|
shared list
|
6355
6607
|
shared_folders browse %name:my_shared_folder_name
|
@@ -6402,7 +6654,7 @@ Example (assuming a default preset is created for the connection information):
|
|
6402
6654
|
ascli faspex5 packages send @json:'{"title":"some title","recipients":["user@example.com"]}' mybygfile1
|
6403
6655
|
```
|
6404
6656
|
|
6405
|
-
Longer example for the
|
6657
|
+
Longer example for the payload of `@json:`:
|
6406
6658
|
|
6407
6659
|
```json
|
6408
6660
|
{"title":"some title","recipients":[{"recipient_type":"user","name":"user@example.com"}]}
|
@@ -6419,10 +6671,10 @@ Longer example for the ayload of `@json:`:
|
|
6419
6671
|
`ascli` adds some convenience:
|
6420
6672
|
The API expects the field `recipients` to be an `Array` of `Hash`, each with field `name` and optionally `recipient_type`.
|
6421
6673
|
`ascli` also accepts an `Array` of `String`, with simply a recipient name.
|
6422
|
-
Then, `ascli` will
|
6674
|
+
Then, `ascli` will look up existing contacts among all possible types, use it if a single match is found, and set the `name` and `recipient_type` accordingly.
|
6423
6675
|
Else an exception is sent.
|
6424
6676
|
|
6425
|
-
> **Note:** The lookup is case
|
6677
|
+
> **Note:** The lookup is case-insensitive and on partial matches.
|
6426
6678
|
|
6427
6679
|
```json
|
6428
6680
|
{"title":"some title","recipients":["user@example.com"]}
|
@@ -6451,11 +6703,11 @@ Option `box` can be used to list packages from a specific box (see [Inbox Select
|
|
6451
6703
|
|
6452
6704
|
Option `query` can be used to filter the list of packages, based on native API parameters, directly sent to [Faspex 5 API `GET /packages`](https://developer.ibm.com/apis/catalog/aspera--ibm-aspera-faspex-5-0-api/api/API--aspera--ibm-aspera-faspex-api#getAllPackages).
|
6453
6705
|
|
6454
|
-
|
|
6706
|
+
| Parameter | Type | Description |
|
6455
6707
|
|-----------|---------|-------------|
|
6456
6708
|
| `offset` | Native | Managed by `ascli`: Offset of first package. Default: 0 |
|
6457
6709
|
| `limit` | Native | Managed by `ascli`: # of packages per API call. Default: 100 |
|
6458
|
-
| `q` | Native | General search string (case
|
6710
|
+
| `q` | Native | General search string (case-insensitive, matches if value is contained in several fields) |
|
6459
6711
|
| ... | Native | Other native parameters are supported (Refer to API documentation) |
|
6460
6712
|
| `max` | Special | Maximum number of items to retrieve (stop pages when the maximum is passed) |
|
6461
6713
|
| `pmax` | Special | Maximum number of pages to request (stop pages when the maximum is passed) |
|
@@ -6478,32 +6730,43 @@ Examples:
|
|
6478
6730
|
ascli faspex5 packages list @ruby:'->(p){p["state"].eql?("released")}'
|
6479
6731
|
```
|
6480
6732
|
|
6481
|
-
### Faspex 5:
|
6733
|
+
### Faspex 5: Browsing folder content
|
6482
6734
|
|
6483
|
-
|
6735
|
+
Several entities support folder browsing: Packages, Nodes, Shared Folders.
|
6736
|
+
All support two modes: paging and legacy API.
|
6737
|
+
By default, paging is used.
|
6484
6738
|
|
6485
|
-
|
6739
|
+
Option `query` is available with parameters supported by the API and `ascli` :
|
6486
6740
|
|
6487
|
-
|
6741
|
+
| Parameter | Evaluation | Description |
|
6742
|
+
|--------------|--------------|-------------|
|
6743
|
+
| `paging` | `ascli` | Use paging API. Default: `true` |
|
6744
|
+
| `recursive` | `ascli` | List inside folders. Default: `false` |
|
6745
|
+
| `max` | `ascli` | Maximum number of items. |
|
6746
|
+
| `filter` | API | Refer to API doc. Default: `{"basenames":[]}` |
|
6747
|
+
| `offset` | API (legacy) | Index of first item. Default: `0` |
|
6748
|
+
| `limit` | API (legacy) | Number of items in one API call result. Default: `500` |
|
6749
|
+
| `per_page` | API (paging) | Number of items in one API call result. Default: `500` |
|
6488
6750
|
|
6489
|
-
|
6751
|
+
### Faspex 5: Content of a received Package
|
6490
6752
|
|
6491
|
-
> **Note:**
|
6753
|
+
> **Note:** Listing content also applies to sent packages using `--box=outbox`.
|
6492
6754
|
|
6493
|
-
To
|
6755
|
+
To list the content of a received package, use command `faspex5 packages browse <package id>`.
|
6756
|
+
Optionally, provide a folder path.
|
6494
6757
|
|
6495
6758
|
### Faspex 5: Receive a package
|
6496
6759
|
|
6497
6760
|
To receive one, or several packages at once, use command `faspex5 packages receive`.
|
6498
|
-
Provide either a single package
|
6761
|
+
Provide either a single package ID, or an extended value `Array` of package IDs, e.g. `@list:,1,2,3` as argument.
|
6499
6762
|
|
6500
6763
|
The same options as for `faspex5 packages list` can be used to select the box and filter the packages to download.
|
6501
6764
|
I.e. options `box` and `query`, as well as last **Command Parameter** `Proc` (filter).
|
6502
6765
|
|
6503
6766
|
Option `--once-only=yes` can be used, for "cargo-like" behavior.
|
6504
|
-
Special package
|
6767
|
+
Special package ID `INIT` initializes the persistency of already received packages when option `--once-only=yes` is used.
|
6505
6768
|
|
6506
|
-
Special package
|
6769
|
+
Special package ID `ALL` selects all packages (of the selected box).
|
6507
6770
|
In this case, typically, only `completed` packages should be downloaded, so use option `--query=@json:'{"status":"completed"}'`.
|
6508
6771
|
|
6509
6772
|
If a package is password protected, then the content protection password is asked interactively.
|
@@ -6628,7 +6891,7 @@ To start package purge, i.e. permanently remove packages marked for deletion old
|
|
6628
6891
|
ascli faspex5 admin clean_deleted
|
6629
6892
|
```
|
6630
6893
|
|
6631
|
-
> **Note:** The expiration
|
6894
|
+
> **Note:** The expiration period taken by default is the one from `admin configuration show`. To use a different period than the default, specify it on command line with: `@json:'{"days_before_deleting_package_records":15}'`
|
6632
6895
|
|
6633
6896
|
To delete all packages, one can use the following command:
|
6634
6897
|
|
@@ -6641,7 +6904,7 @@ ascli faspex5 packages list --box=ALL --format=yaml --fields=id | ascli faspex5
|
|
6641
6904
|
|
6642
6905
|
### Faspex 5: Admin: Unlock user
|
6643
6906
|
|
6644
|
-
To unlock a user, you can
|
6907
|
+
To unlock a user, you can deactivate and then re-activate the user:
|
6645
6908
|
|
6646
6909
|
```bash
|
6647
6910
|
ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":false}'
|
@@ -6651,89 +6914,83 @@ ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account
|
|
6651
6914
|
ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":true}'
|
6652
6915
|
```
|
6653
6916
|
|
6654
|
-
> **Note:**
|
6917
|
+
> **Note:** Here we use the convenient percent selector, but the numerical ID can be used as well.
|
6655
6918
|
|
6656
6919
|
To send a password reset link to a user, use command `reset_password` on the `account`.
|
6657
6920
|
|
6658
|
-
### Faspex 5: Faspex 4-style
|
6921
|
+
### Faspex 5: Faspex 4-style post-processing
|
6659
6922
|
|
6660
|
-
The command
|
6661
|
-
It implements Faspex 5
|
6662
|
-
Environment variables at set to the values provided by the web hook which are the same as Faspex 4
|
6923
|
+
The command `ascli faspex5 postprocessing` emulates Faspex 4 post-processing script execution in Faspex 5.
|
6924
|
+
It implements a web hook for Faspex 5 and calls a script with the same environment variables as set by Faspex 4.
|
6925
|
+
Environment variables at set to the values provided by the web hook which are the same as Faspex 4 post-processing.
|
6663
6926
|
|
6664
|
-
It allows to quickly migrate workflows to Faspex 5 while preserving scripts.
|
6665
|
-
Nevertheless, on long term, a native approach shall be considered, such as using Aspera Orchestrator or other workflow engine.
|
6927
|
+
It allows to quickly migrate workflows from Faspex 4 to Faspex 5 while preserving scripts.
|
6928
|
+
Nevertheless, on long term, a native approach shall be considered, such as using Aspera Orchestrator or other workflow engine, using Faspex 5 native web hooks or File Processing.
|
6666
6929
|
|
6667
6930
|
It is invoked like this:
|
6668
6931
|
|
6669
6932
|
```bash
|
6670
|
-
ascli faspex5 postprocessing
|
6933
|
+
ascli faspex5 postprocessing
|
6671
6934
|
```
|
6672
6935
|
|
6673
|
-
|
6936
|
+
An optional positional parameter can be provided as extended value `Hash`:
|
6674
6937
|
|
6675
|
-
|
|
6676
|
-
|
6677
|
-
|
|
6678
|
-
|
|
6679
|
-
|
|
6680
|
-
|
|
6681
|
-
| certificate.chain | `String` | nil | Path to certificate chain |
|
6682
|
-
| processing | `Hash` | nil | Behavior of post processing |
|
6683
|
-
| processing.script_folder | `String` | . | Prefix added to script path |
|
6684
|
-
| processing.fail_on_error | `Bool` | false | Fail if true and process exits with non zero code |
|
6685
|
-
| processing.timeout_seconds | `Integer`| 60 | Max. execution time before script is killed |
|
6938
|
+
| Parameter | Type | Default | Description |
|
6939
|
+
|-------------------|----------|-------|-----------------------------------------------------|
|
6940
|
+
| **server info** | - | - | See [Web service](#web-service). |
|
6941
|
+
| `script_folder` | `String` | `.` | Prefix added to script path (Default: CWD) |
|
6942
|
+
| `fail_on_error` | `Bool` | false | Fail if true and process exits with non-zero code |
|
6943
|
+
| `timeout_seconds` | `Integer`| 60 | Time out before script is killed |
|
6686
6944
|
|
6687
|
-
|
6688
|
-
|
6689
|
-
- If `http` or `https` is used
|
6690
|
-
- The local port number (default 443 for HTTPS, 80 for HTTP)
|
6691
|
-
- The **base path**, i.e. the path under which requests are received, if a reverse proxy is used this can be used to route.
|
6692
|
-
|
6693
|
-
When a request is received the following happens:
|
6945
|
+
When a request on `ascli` is received the following happens:
|
6694
6946
|
|
6695
6947
|
- `ascli` gets the path of the URL called
|
6696
6948
|
- It removes the **base path** of base URL.
|
6697
6949
|
- It prepends it with the value of `script_folder`
|
6698
|
-
- It executes the script
|
6950
|
+
- It executes the script at that path
|
6699
6951
|
- Upon success, a success code is returned
|
6700
6952
|
|
6701
6953
|
For example:
|
6702
6954
|
|
6703
|
-
|
6704
|
-
|
6955
|
+
```bash
|
6956
|
+
ascli faspex5 postprocessing @json:'{"url":"http://localhost:8080/processing","script_folder":"/opt/scripts"}'
|
6957
|
+
```
|
6705
6958
|
|
6706
6959
|
In Faspex 5, the URL of the webhook endpoint shall be reachable from within Faspex containers.
|
6707
6960
|
For example, if `ascli` in running in the base host, the URL hostname shall not be localhost, as this refers to the local address inside Faspex container.
|
6708
|
-
Instead, one can specify the IP address of the host or `host.containers.internal
|
6961
|
+
Instead, one can specify the **IP address of the host** or `host.containers.internal` (Check `podman` manual).
|
6962
|
+
|
6963
|
+
Let's define the web hook:
|
6709
6964
|
|
6710
6965
|
**Webhook endpoint URI** : `http://host.containers.internal:8080/processing/script1.sh`
|
6711
6966
|
|
6712
|
-
Then the
|
6967
|
+
Then the post-processing script executed will be `/opt/scripts/script1.sh`.
|
6713
6968
|
|
6714
6969
|
### Faspex 5: Faspex 4 Gateway
|
6715
6970
|
|
6716
6971
|
> **Note:** This is not a feature for production. It's provided for testing only.
|
6717
6972
|
|
6718
|
-
For legacy
|
6973
|
+
For legacy Faspex client applications that use the `send` API (only) of Faspex v4, the command `gateway` provides the capability to present an API compatible with Faspex 4, and it will call the Faspex 5 API.
|
6719
6974
|
|
6720
6975
|
It takes a single argument which is the URL at which the gateway will be located (locally):
|
6721
6976
|
|
6722
6977
|
```bash
|
6723
|
-
ascli faspex5 gateway https://localhost:12345/aspera/faspex
|
6978
|
+
ascli faspex5 gateway @json:'{"url":"https://localhost:12345/aspera/faspex"}'
|
6724
6979
|
```
|
6725
6980
|
|
6726
6981
|
There are many limitations:
|
6727
6982
|
|
6728
|
-
- It's only to emulate the Faspex 4 `send` API (send package)
|
6729
|
-
- No support for remote sources, only for an actual file transfer by the client
|
6730
|
-
- The client must use the transfer spec returned by the API (not faspe
|
6731
|
-
-
|
6732
|
-
-
|
6733
|
-
-
|
6983
|
+
- It's only to emulate the Faspex 4 `send` API (send package).
|
6984
|
+
- No support for remote sources, only for an actual file transfer by the client.
|
6985
|
+
- The client must use the transfer spec returned by the API (not `faspe:` URL).
|
6986
|
+
- Tags returned in transfer spec must be used in transfer.
|
6987
|
+
- Only a single authentication is possible (per gateway) on Faspex5.
|
6988
|
+
- No authentication of F4 side (ignored).
|
6734
6989
|
|
6735
6990
|
Behavior: The API client calls the Faspex 4 API on the gateway, then the gateway transforms this into a Faspex5 API call, which returns a transfer spec, which is returned to the calling client. The calling client uses this to start a transfer to HSTS which is actually managed by Faspex 5.
|
6736
6991
|
|
6992
|
+
For other parameters, see [Web service](#web-service).
|
6993
|
+
|
6737
6994
|
### Faspex 5: Get Bearer token to use API
|
6738
6995
|
|
6739
6996
|
If a command is missing, then it is still possible to execute command by calling directly the API on the command line using `curl`:
|
@@ -6744,7 +7001,7 @@ curl -H "Authorization: $(ascli ascli bearer)" https://faspex5.example.com/asper
|
|
6744
7001
|
|
6745
7002
|
## Plugin: `faspex`: IBM Aspera Faspex v4
|
6746
7003
|
|
6747
|
-
> **Note:** Faspex v4 is end of support since Sept. 30th, 2024. So this plugin for
|
7004
|
+
> **Note:** Faspex v4 is end of support since Sept. 30th, 2024. So this plugin for Faspex v4 is deprecated. If you still need to use Faspex4, then use `ascli` version 4.19.0 or earlier.
|
6748
7005
|
>
|
6749
7006
|
> **Note:** For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
6750
7007
|
|
@@ -6757,7 +7014,7 @@ Command: `faspex package list`
|
|
6757
7014
|
|
6758
7015
|
#### Option `box`
|
6759
7016
|
|
6760
|
-
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
7017
|
+
By default, it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
6761
7018
|
|
6762
7019
|
#### Option `recipient`
|
6763
7020
|
|
@@ -6770,18 +7027,20 @@ A user can receive a package because the recipient is:
|
|
6770
7027
|
|
6771
7028
|
As inboxes may be large, it is possible to use the following query parameters:
|
6772
7029
|
|
6773
|
-
|
6774
|
-
|
6775
|
-
|
6776
|
-
|
6777
|
-
|
7030
|
+
| Parameter | Evaluation | Description |
|
7031
|
+
|--------------|------------|-------------|
|
7032
|
+
| `count` | API | Number of items in one API call result (default=0, equivalent to 10) |
|
7033
|
+
| `page` | API | ID of page in call (default=0) |
|
7034
|
+
| `startIndex` | API | Index of item to start (default=0) |
|
7035
|
+
| `max` | `ascli` | Maximum number of items |
|
7036
|
+
| `pmax` | `ascli` | Maximum number of pages |
|
6778
7037
|
|
6779
7038
|
(SQL query is `LIMIT <startIndex>, <count>`)
|
6780
7039
|
|
6781
7040
|
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under **Services (API v.3)**.
|
6782
7041
|
|
6783
7042
|
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`).
|
6784
|
-
By default `count` is `0` (`10`), it can be increased to issue less HTTP calls.
|
7043
|
+
By default, `count` is `0` (`10`), it can be increased to issue less HTTP calls.
|
6785
7044
|
|
6786
7045
|
#### Example: List packages in dropbox
|
6787
7046
|
|
@@ -6795,7 +7054,7 @@ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in recei
|
|
6795
7054
|
|
6796
7055
|
The command is `package recv`, possible methods are:
|
6797
7056
|
|
6798
|
-
- Provide a package
|
7057
|
+
- Provide a package ID with option `id`
|
6799
7058
|
- Provide a public link with option `link`
|
6800
7059
|
- Provide a `faspe:` URI with option `link`
|
6801
7060
|
|
@@ -6811,13 +7070,13 @@ ascli faspex package list --recipient='*dropbox_name'
|
|
6811
7070
|
ascli faspex package recv 125 --recipient='*dropbox_name'
|
6812
7071
|
```
|
6813
7072
|
|
6814
|
-
|
7073
|
+
If `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
6815
7074
|
|
6816
7075
|
### Sending a Package
|
6817
7076
|
|
6818
7077
|
The command is `faspex package send`.
|
6819
7078
|
Package information (title, note, metadata, options) is provided in option `delivery_info`.
|
6820
|
-
The
|
7079
|
+
The content of `delivery_info` is directly the contents of the `send` v3 [API of Faspex 4](https://developer.ibm.com/apis/catalog/aspera--aspera-faspex-client-sdk/API%20v.3:%20Send%20Packages).
|
6821
7080
|
|
6822
7081
|
Example:
|
6823
7082
|
|
@@ -6833,7 +7092,7 @@ Additional optional parameters in mandatory option `delivery_info`:
|
|
6833
7092
|
- Package Note: : `"note":"note this and that"`
|
6834
7093
|
- Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
6835
7094
|
|
6836
|
-
It is possible to send from a remote source using option `remote_source`, providing either the numerical
|
7095
|
+
It is possible to send from a remote source using option `remote_source`, providing either the numerical ID, or the name of the remote source using percent selector: `%name:<name>`.
|
6837
7096
|
|
6838
7097
|
Remote source can be browsed if option `storage` is provided.
|
6839
7098
|
`storage` is a `Hash` extended value.
|
@@ -6841,11 +7100,11 @@ The key is the storage name, as listed in `source list` command.
|
|
6841
7100
|
The value is a `Hash` with the following keys:
|
6842
7101
|
|
6843
7102
|
- `node` is a `Hash` with keys: `url`, `username`, `password`
|
6844
|
-
- `path` is the
|
7103
|
+
- `path` is the sub-path inside the node, as configured in Faspex
|
6845
7104
|
|
6846
7105
|
### Email notification on transfer
|
6847
7106
|
|
6848
|
-
Like for any transfer, a notification can be sent by email using options: `notify_to` and `notify_template
|
7107
|
+
Like for any transfer, a notification can be sent by email using options: `notify_to` and `notify_template`.
|
6849
7108
|
|
6850
7109
|
Example:
|
6851
7110
|
|
@@ -6855,7 +7114,7 @@ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipien
|
|
6855
7114
|
|
6856
7115
|
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
6857
7116
|
|
6858
|
-
###
|
7117
|
+
### Operations on dropbox
|
6859
7118
|
|
6860
7119
|
Example:
|
6861
7120
|
|
@@ -6868,7 +7127,7 @@ ascli faspex v4 dropbox delete 36
|
|
6868
7127
|
### Remote sources
|
6869
7128
|
|
6870
7129
|
Faspex lacks an API to list the contents of a remote source (available in web UI).
|
6871
|
-
To
|
7130
|
+
To work around this, the Node API is used, for this it is required to set option: `storage` that links a storage name to a node configuration and sub path.
|
6872
7131
|
|
6873
7132
|
Example:
|
6874
7133
|
|
@@ -6887,7 +7146,7 @@ my_faspex_node:
|
|
6887
7146
|
password: MyNodePassword
|
6888
7147
|
```
|
6889
7148
|
|
6890
|
-
In this example, a
|
7149
|
+
In this example, a Faspex storage named `my_storage` exists in Faspex, and is located
|
6891
7150
|
under the docroot in `/mydir` (this must be the same as configured in Faspex).
|
6892
7151
|
The node configuration name is `my_faspex_node` here.
|
6893
7152
|
|
@@ -6942,7 +7201,7 @@ v4 workgroup list
|
|
6942
7201
|
|
6943
7202
|
## Plugin: `shares`: IBM Aspera Shares v1
|
6944
7203
|
|
6945
|
-
Aspera Shares supports the **
|
7204
|
+
Aspera Shares supports the **Node API** for the file transfer part.
|
6946
7205
|
|
6947
7206
|
Supported commands are listed in Share's API documentation:
|
6948
7207
|
|
@@ -6960,7 +7219,7 @@ user_id=$(ascli shares admin user list --select=@json:'{"username":"entity1"}' -
|
|
6960
7219
|
ascli shares admin share user_permissions $share_id create @json:'{"user_id":'$user_id',"browse_permission":true, "download_permission":true, "mkdir_permission":true,"delete_permission":true,"rename_permission":true,"content_availability_permission":true,"manage_permission":true}'
|
6961
7220
|
```
|
6962
7221
|
|
6963
|
-
To figure out
|
7222
|
+
To figure out an entity's payload, for example for creation, refer to the API documentation above.
|
6964
7223
|
|
6965
7224
|
### Shares sample commands
|
6966
7225
|
|
@@ -6985,6 +7244,8 @@ files delete my_share1/test_file.bin
|
|
6985
7244
|
files download --to-folder=. my_share1/test_file.bin
|
6986
7245
|
files download --to-folder=. my_share1/test_file.bin my_share1/test_file.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://tst.example.com/path@"}'
|
6987
7246
|
files mkdir my_share1/new_folder
|
7247
|
+
files sync start push /data/local_syncsynctst /london-sh1/synctst --sync-info=@json:'{"quiet":false}'
|
7248
|
+
files sync start push /data/local_syncsynctst /london-sh1/synctst --sync-info=@json:'{"reset":true}'
|
6988
7249
|
files upload --to-folder=https://shares.share1 'faux:///testfile?1m' --transfer=httpgw --transfer-info=@json:'{"url":"my_example.com/path@","synchronous":true,"api_version":"v1","upload_chunk_size":100000}'
|
6989
7250
|
files upload --to-folder=https://shares.share1 sendfolder --transfer=httpgw --transfer-info=@json:'{"url":"my_example.com/path@","synchronous":true,"api_version":"v1","upload_chunk_size":100000}'
|
6990
7251
|
files upload --to-folder=my_share1 test_file.bin
|
@@ -7043,7 +7304,7 @@ If you have those parameters already, then following options shall be provided:
|
|
7043
7304
|
- `bucket` bucket name
|
7044
7305
|
- `endpoint` storage endpoint URL, e.g. `https://s3.hkg02.cloud-object-storage.appdomain.cloud`
|
7045
7306
|
- `apikey` API Key
|
7046
|
-
- `crn` resource instance
|
7307
|
+
- `crn` resource instance ID
|
7047
7308
|
|
7048
7309
|
For example, let us create a default configuration:
|
7049
7310
|
|
@@ -7120,14 +7381,14 @@ ascli config preset set default cos mycos
|
|
7120
7381
|
|
7121
7382
|
Let's assume you created a default configuration from one of the two previous steps (else specify the access options on command lines).
|
7122
7383
|
|
7123
|
-
A subset of `node` plugin operations are supported, basically
|
7384
|
+
A subset of `node` plugin operations are supported, basically Node API:
|
7124
7385
|
|
7125
7386
|
```bash
|
7126
7387
|
ascli cos node info
|
7127
7388
|
ascli cos node upload 'faux:///sample1G?1g'
|
7128
7389
|
```
|
7129
7390
|
|
7130
|
-
> **Note:** A dummy file `sample1G` of size
|
7391
|
+
> **Note:** A dummy file `sample1G` of size 2 GB is generated using the `faux` PVCL scheme (see previous section and `man ascp`), but you can, of course, send a real file by specifying a real file path instead.
|
7131
7392
|
|
7132
7393
|
### Cos sample commands
|
7133
7394
|
|
@@ -7181,7 +7442,7 @@ health -N
|
|
7181
7442
|
## Plugin: `preview`: Preview generator for AoC
|
7182
7443
|
|
7183
7444
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
7184
|
-
It uses the **
|
7445
|
+
It uses the **Node API** of Aspera HSTS and requires use of Access Keys and its **storage root**.
|
7185
7446
|
Several options can be used to tune several aspects:
|
7186
7447
|
|
7187
7448
|
- Methods for detection of new files needing generation
|
@@ -7192,7 +7453,7 @@ See also <https://github.com/IBM/aspera-on-cloud-file-previews>
|
|
7192
7453
|
|
7193
7454
|
### Aspera Server configuration
|
7194
7455
|
|
7195
|
-
Specify the
|
7456
|
+
Specify the preview's folder as shown in:
|
7196
7457
|
|
7197
7458
|
<https://ibmaspera.com/help/admin/organization/installing_the_preview_maker>
|
7198
7459
|
|
@@ -7209,36 +7470,40 @@ asnodeadmin --reload
|
|
7209
7470
|
|
7210
7471
|
If another folder is configured on the HSTS, then specify it to `ascli` using the option `previews_folder`.
|
7211
7472
|
|
7212
|
-
The HSTS
|
7213
|
-
This size is internally capped to `1<<24` Bytes (16777216)
|
7473
|
+
The HSTS Node API limits any preview file to a parameter: `max_request_file_create_size_kb` (1 KB is 1024 Bytes).
|
7474
|
+
This size is internally capped to `1<<24` Bytes (16777216), i.e. 16384 KB, i.e. 16 MB.
|
7214
7475
|
|
7215
7476
|
To change this parameter in `aspera.conf`, use `asconfigurator`.
|
7216
7477
|
To display the value, use `asuserdata`:
|
7217
7478
|
|
7218
7479
|
```bash
|
7219
7480
|
asuserdata -a | grep max_request_file_create_size_kb
|
7481
|
+
```
|
7220
7482
|
|
7483
|
+
```console
|
7221
7484
|
max_request_file_create_size_kb: "1024"
|
7485
|
+
```
|
7222
7486
|
|
7487
|
+
```bash
|
7223
7488
|
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
7224
7489
|
```
|
7225
7490
|
|
7226
|
-
If you use a value different
|
7491
|
+
If you use a value different from `16777216`, then specify it using option `max_size`.
|
7227
7492
|
|
7228
|
-
> **Note:** The HSTS parameter (`max_request_file_create_size_kb`) is in **
|
7493
|
+
> **Note:** The HSTS parameter (`max_request_file_create_size_kb`) is in **kilo Bytes** while the generator parameter is in **Bytes** (factor of 1024).
|
7229
7494
|
|
7230
7495
|
### External tools: Linux
|
7231
7496
|
|
7232
7497
|
`ascli` requires the following external tools available in the `PATH`:
|
7233
7498
|
|
7234
|
-
- **ImageMagick**
|
7499
|
+
- **ImageMagick** v7+: `magick` `composite`
|
7235
7500
|
- **OptiPNG** : `optipng`
|
7236
7501
|
- **FFmpeg** : `ffmpeg` `ffprobe`
|
7237
|
-
- **
|
7502
|
+
- **LibreOffice** : `unoconv`
|
7238
7503
|
|
7239
|
-
Here shown on
|
7504
|
+
Here shown on Red Hat/Rocky Linux.
|
7240
7505
|
|
7241
|
-
Other OSes should work as well, but are
|
7506
|
+
Other OSes should work as well, but are not tested.
|
7242
7507
|
|
7243
7508
|
To check if all tools are found properly, execute:
|
7244
7509
|
|
@@ -7253,26 +7518,26 @@ dnf install -y ImageMagick optipng
|
|
7253
7518
|
```
|
7254
7519
|
|
7255
7520
|
You may also install `ghostscript` which adds fonts to ImageMagick.
|
7256
|
-
Available fonts, used to generate
|
7521
|
+
Available fonts, used to generate PNG for text, can be listed with `magick identify -list font`.
|
7257
7522
|
Prefer ImageMagick version >=7.
|
7258
7523
|
|
7259
7524
|
More info on ImageMagick at <https://imagemagick.org/>
|
7260
7525
|
|
7261
7526
|
#### Video: FFmpeg
|
7262
7527
|
|
7263
|
-
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
7528
|
+
The easiest method is to download and install the latest released version of `ffmpeg` with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
7264
7529
|
|
7265
7530
|
```bash
|
7266
7531
|
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
7267
7532
|
```
|
7268
7533
|
|
7269
|
-
#### Office: `unoconv` and
|
7534
|
+
#### Office: `unoconv` and LibreOffice
|
7270
7535
|
|
7271
7536
|
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
7272
7537
|
|
7273
|
-
The generation of preview in based on the use of `unoconv
|
7538
|
+
The generation of preview in based on the use of LibreOffice's `unoconv`.
|
7274
7539
|
|
7275
|
-
-
|
7540
|
+
- RHEL 8/Rocky Linux 8+
|
7276
7541
|
|
7277
7542
|
```bash
|
7278
7543
|
dnf install unoconv
|
@@ -7327,13 +7592,14 @@ For video preview, the whole set of options can be overridden with option `reenc
|
|
7327
7592
|
|
7328
7593
|
### Execution
|
7329
7594
|
|
7330
|
-
`ascli` intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST
|
7595
|
+
`ascli` intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST API too quickly during the scan or event method).
|
7331
7596
|
It needs to be run on a regular basis to create or update preview files.
|
7332
7597
|
For that use your best reliable scheduler, see [Scheduler](#scheduler).
|
7333
7598
|
|
7334
|
-
Typically, for **Access key** access, the system/transfer is `xfer`.
|
7599
|
+
Typically, for **Access key** access, the system/transfer is `xfer`.
|
7600
|
+
So, in order to be consistent, and generate the appropriate access rights, the generation process should be run as user `xfer`.
|
7335
7601
|
|
7336
|
-
|
7602
|
+
Let's do a one shot test, using the configuration previously created:
|
7337
7603
|
|
7338
7604
|
```bash
|
7339
7605
|
su -s /bin/bash - xfer
|
@@ -7342,7 +7608,7 @@ ascli preview scan --overwrite=always
|
|
7342
7608
|
```
|
7343
7609
|
|
7344
7610
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
7345
|
-
in the
|
7611
|
+
in the preview's folder which contains the access key used.
|
7346
7612
|
On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
|
7347
7613
|
|
7348
7614
|
### Configuration for Execution in scheduler
|
@@ -7410,15 +7676,15 @@ The preview generator supports rendering of those file categories:
|
|
7410
7676
|
|
7411
7677
|
To avoid generation for some categories, specify a list using option `skip_types`.
|
7412
7678
|
|
7413
|
-
Each category has a specific rendering method to produce the
|
7679
|
+
Each category has a specific rendering method to produce the PNG thumbnail.
|
7414
7680
|
|
7415
7681
|
The mp4 video preview file is only for category `video`
|
7416
7682
|
|
7417
|
-
File type is primarily based on file extension detected by the
|
7683
|
+
File type is primarily based on file extension detected by the Node API and translated info a mime type returned by the Node API.
|
7418
7684
|
|
7419
7685
|
### `mimemagic`
|
7420
7686
|
|
7421
|
-
By default, the Mime type used for conversion is the one returned by the
|
7687
|
+
By default, the Mime type used for conversion is the one returned by the Node API, based on file name extension.
|
7422
7688
|
|
7423
7689
|
It is also possible to detect the mime type using option `mimemagic`.
|
7424
7690
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`.
|
@@ -7501,29 +7767,30 @@ An interface for the `async` utility is provided in the following plugins:
|
|
7501
7767
|
- `aoc files sync` (uses node)
|
7502
7768
|
- `shares files sync` (uses node)
|
7503
7769
|
|
7504
|
-
One advantage over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`.
|
7505
|
-
Moreover `ascli` supports sync with application requiring token-based authorization.
|
7770
|
+
One advantage of using `ascli` over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`.
|
7771
|
+
Moreover, `ascli` supports sync with application requiring token-based authorization.
|
7506
7772
|
|
7507
|
-
Some
|
7773
|
+
Some `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
|
7508
7774
|
|
7509
|
-
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). The Transfer SDK 1.3 includes this.
|
7775
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). The Aspera Transfer SDK 1.3+ includes this.
|
7510
7776
|
|
7511
|
-
`sync` supports 0 or 3 arguments
|
7777
|
+
`sync` supports 0 or 3 arguments.
|
7778
|
+
If 3 arguments are provided, they are applied to the first (and only) session and mapped, in that order, to:
|
7512
7779
|
|
7513
|
-
- direction
|
7514
|
-
- local
|
7515
|
-
- remote
|
7780
|
+
- `direction`
|
7781
|
+
- `local.path`
|
7782
|
+
- `remote.path`
|
7516
7783
|
|
7517
7784
|
Additional options can be provided with option `sync_info`, for which two syntax are possible, as follows.
|
7518
7785
|
|
7519
|
-
### `async` API and conf format
|
7786
|
+
### `async` API and `conf` format
|
7520
7787
|
|
7521
|
-
It is the same payload as specified on the option `--conf` of `async` or in
|
7522
|
-
This is the preferred syntax and allows a single session definition.
|
7788
|
+
It is the same payload as specified on the option `--conf` of `async` or in Node API `/asyncs`.
|
7789
|
+
This is the **preferred** syntax and allows a single session definition.
|
7523
7790
|
|
7524
|
-
> **Note:**
|
7791
|
+
> **Note:** By default, no progress, nor error messages is provided on terminal. To activate messages, set option `sync_info` parameter `quiet` to `false`.
|
7525
7792
|
|
7526
|
-
Documentation on Async
|
7793
|
+
Documentation on Async Node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
7527
7794
|
|
7528
7795
|
If 3 arguments are provided they are mapped to:
|
7529
7796
|
|
@@ -7537,16 +7804,10 @@ If 3 arguments are provided they are mapped to:
|
|
7537
7804
|
It is based on a JSON representation of `async` command line options.
|
7538
7805
|
It allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
7539
7806
|
|
7540
|
-
This is the mode selection if there
|
7807
|
+
This is the mode selection if there are either keys `sessions` or `instance` in option `sync_info`.
|
7541
7808
|
|
7542
7809
|
> **Note:** Progress and error messages are provided on terminal like regular command line invocation of `async`.
|
7543
7810
|
|
7544
|
-
If 3 arguments are provided, they are applied to the first (and only) session and mapped to:
|
7545
|
-
|
7546
|
-
- `direction`
|
7547
|
-
- `local_dir`
|
7548
|
-
- `remote_dir`
|
7549
|
-
|
7550
7811
|
## Hot folder
|
7551
7812
|
|
7552
7813
|
### Requirements
|
@@ -7583,7 +7844,7 @@ Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
|
7583
7844
|
>
|
7584
7845
|
> **Note:** Usual native `ascp` arguments are available as standard [**transfer-spec**](#transfer-specification) parameters, but not special or advanced options.
|
7585
7846
|
>
|
7586
|
-
> **Note:** Only for the [`direct`](#agent-direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info
|
7847
|
+
> **Note:** Only for the [`direct`](#agent-direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info`.
|
7587
7848
|
|
7588
7849
|
#### Server side and configuration
|
7589
7850
|
|
@@ -7591,11 +7852,11 @@ Virtually any transfer on a **repository** on a regular basis might emulate a ho
|
|
7591
7852
|
|
7592
7853
|
> **Note:** File detection is not based on events (`inotify`, etc...), but on a simple folder scan on source side.
|
7593
7854
|
>
|
7594
|
-
> **Note:** Options may be saved in
|
7855
|
+
> **Note:** Options may be saved in an [Option Preset](#option-preset) and used with `-P`.
|
7595
7856
|
|
7596
7857
|
#### Scheduling
|
7597
7858
|
|
7598
|
-
Once `ascli` command line arguments are defined, run the command using the OS native scheduler, e.g. every
|
7859
|
+
Once `ascli` command line arguments are defined, run the command using the OS native scheduler, e.g. every minute, or 5 minutes, etc...
|
7599
7860
|
Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
7600
7861
|
|
7601
7862
|
### Example: Upload hot folder
|
@@ -7604,7 +7865,7 @@ Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
|
7604
7865
|
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
7605
7866
|
```
|
7606
7867
|
|
7607
|
-
The local folder (here, relative path: `source_hot`) is sent (upload) to an
|
7868
|
+
The local folder (here, relative path: `source_hot`) is sent (upload) to an Aspera server.
|
7608
7869
|
Source files are deleted after transfer.
|
7609
7870
|
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
7610
7871
|
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
@@ -7666,13 +7927,13 @@ OK - [transfer:ok]
|
|
7666
7927
|
The `smtp` option is a `Hash` (extended value) with the following fields:
|
7667
7928
|
|
7668
7929
|
<!-- markdownlint-disable MD034 -->
|
7669
|
-
| field |
|
7930
|
+
| field | Default | example | Description |
|
7670
7931
|
|--------------|---------------------|----------------------------|----------------------------------|
|
7671
|
-
| `server` | - | smtp.gmail.com
|
7672
|
-
| `tls` | true
|
7673
|
-
| `ssl` | false
|
7674
|
-
| `port` | 587 or 465 or 25
|
7675
|
-
| `domain` | domain of server
|
7932
|
+
| `server` | - | `smtp.gmail.com` | SMTP server address |
|
7933
|
+
| `tls` | `true` | `true` | Enable `STARTTLS` (port 587) |
|
7934
|
+
| `ssl` | `false` | `false` | Enable `TLS` (port 465) |
|
7935
|
+
| `port` | `587` or `465` or `25` | `587` | Port for service |
|
7936
|
+
| `domain` | *domain of server* | gmail.com | Email domain of user |
|
7676
7937
|
| `username` | - | john@example.com | User to authenticate on SMTP server, leave empty for open auth. |
|
7677
7938
|
| `password` | - | my_password_here | Password for above username |
|
7678
7939
|
| `from_email` | username if defined | johnny@example.com | Address used if receiver replies |
|
@@ -7708,7 +7969,7 @@ ascli config preset set default config cli_default
|
|
7708
7969
|
|
7709
7970
|
### Email templates
|
7710
7971
|
|
7711
|
-
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
7972
|
+
Sent emails are built using a template that uses the [`ERB`](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
7712
7973
|
|
7713
7974
|
The template is the full SMTP message, including headers.
|
7714
7975
|
|
@@ -7760,10 +8021,10 @@ Transfer is: <%=status%>
|
|
7760
8021
|
|
7761
8022
|
This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: `asession`.
|
7762
8023
|
|
7763
|
-
It aims at simplifying the startup of a FASP session from a programmatic
|
8024
|
+
It aims at simplifying the startup of a FASP session from a programmatic standpoint as formatting a [**transfer-spec**](#transfer-specification) is:
|
7764
8025
|
|
7765
8026
|
- Common to Aspera Node API (HTTP POST /ops/transfer)
|
7766
|
-
- Common to Aspera Connect API (browser
|
8027
|
+
- Common to Aspera Connect API (browser JavaScript `startTransfer`)
|
7767
8028
|
- Easy to generate by using any third party language specific JSON library
|
7768
8029
|
|
7769
8030
|
Hopefully, IBM integrates this directly in `ascp`, and this tool is made redundant.
|
@@ -7780,16 +8041,16 @@ During execution, it generates all low level events, one per line, in JSON forma
|
|
7780
8041
|
|
7781
8042
|
Top level parameters supported by `asession`:
|
7782
8043
|
|
7783
|
-
|
|
8044
|
+
| Parameter | Description |
|
7784
8045
|
|-----------|-------------|
|
7785
8046
|
| `spec` | the [**transfer-spec**](#transfer-specification) |
|
7786
8047
|
| `agent` | same parameters as transfer-info for agent `direct` |
|
7787
8048
|
| `loglevel` | log level of `asession` |
|
7788
|
-
| `file_list_folder` | the folder used to store (for garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists` |
|
8049
|
+
| `file_list_folder` | the folder used to store (for garbage collection) generated file lists. By default, it is `[system tmp folder]/[username]_asession_filelists` |
|
7789
8050
|
|
7790
8051
|
### Comparison of interfaces
|
7791
8052
|
|
7792
|
-
| feature/tool | Transfer SDK |
|
8053
|
+
| feature/tool | Transfer SDK | FASPManager | `ascp` | `asession` |
|
7793
8054
|
|-----------------------|--------------|---------------------------------|--------|------------|
|
7794
8055
|
| language integration | Many | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | Any | Any |
|
7795
8056
|
| required additional components to `ascp` | Daemon | Library<br/>(+headers) | - | Ruby<br/>Aspera gem |
|
@@ -7813,9 +8074,9 @@ asession < session.json
|
|
7813
8074
|
|
7814
8075
|
### Asynchronous commands and Persistent session
|
7815
8076
|
|
7816
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter`
|
8077
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter` → `long_parameter`
|
7817
8078
|
|
7818
|
-
This is particularly useful for a persistent session (
|
8079
|
+
This is particularly useful for a persistent session (with the [**transfer-spec**](#transfer-specification) parameter: `"keepalive":true`)
|
7819
8080
|
|
7820
8081
|
```json
|
7821
8082
|
asession
|
@@ -7828,7 +8089,7 @@ asession
|
|
7828
8089
|
|
7829
8090
|
### Example of language wrapper
|
7830
8091
|
|
7831
|
-
|
8092
|
+
NodeJS: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/aspera)
|
7832
8093
|
|
7833
8094
|
### Help
|
7834
8095
|
|
@@ -7888,7 +8149,7 @@ So, it evolved into `ascli`:
|
|
7888
8149
|
|
7889
8150
|
- Portable: works on platforms supporting `ruby` (and `ascp`)
|
7890
8151
|
- Easy to install with the `gem` utility
|
7891
|
-
- Supports transfers with multiple [Transfer Agents](#transfer-clients-agents), that's why transfer parameters moved from `ascp` command line to [**transfer-spec**](#transfer-specification) (more reliable
|
8152
|
+
- Supports transfers with multiple [Transfer Agents](#transfer-clients-agents), that's why transfer parameters moved from `ascp` command line to [**transfer-spec**](#transfer-specification) (more reliable, more standard)
|
7892
8153
|
- `ruby` is consistent with other Aspera products
|
7893
8154
|
|
7894
8155
|
Over the time, a supported command line tool `aspera` was developed in C++, it was later on deprecated.
|
@@ -7908,7 +8169,7 @@ ascli config coffee
|
|
7908
8169
|
|
7909
8170
|
### Error: "Remote host is not who we expected"
|
7910
8171
|
|
7911
|
-
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
8172
|
+
Cause: `ascp` >= 4.x checks fingerprint of the highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
7912
8173
|
|
7913
8174
|
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the configuration file):
|
7914
8175
|
|
@@ -7943,5 +8204,5 @@ OpenSSH keys only supported if ED25519 is available
|
|
7943
8204
|
Which meant that you do not have Ruby support for ED25519 SSH keys.
|
7944
8205
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
7945
8206
|
|
7946
|
-
In addition, host keys of type: `ecdsa-sha2` and `ecdh-sha2` are also
|
8207
|
+
In addition, host keys of type: `ecdsa-sha2` and `ecdh-sha2` are also deactivated by default.
|
7947
8208
|
To re-activate, set env var `ASCLI_ENABLE_ECDSHA2` to `true`.
|