aspera-cli 4.21.2 → 4.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +1 -1
- data/CHANGELOG.md +34 -16
- data/CONTRIBUTING.md +6 -10
- data/README.md +805 -574
- data/examples/get_proto_file.rb +1 -1
- data/lib/aspera/agent/base.rb +9 -5
- data/lib/aspera/agent/connect.rb +30 -28
- data/lib/aspera/agent/desktop.rb +29 -25
- data/lib/aspera/agent/direct.rb +137 -125
- data/lib/aspera/agent/httpgw.rb +22 -26
- data/lib/aspera/agent/node.rb +14 -11
- data/lib/aspera/agent/transferd.rb +6 -2
- data/lib/aspera/api/aoc.rb +6 -6
- data/lib/aspera/api/cos_node.rb +1 -1
- data/lib/aspera/api/httpgw.rb +7 -3
- data/lib/aspera/api/node.rb +6 -4
- data/lib/aspera/ascmd.rb +3 -3
- data/lib/aspera/ascp/installation.rb +15 -16
- data/lib/aspera/ascp/management.rb +1 -1
- data/lib/aspera/assert.rb +11 -2
- data/lib/aspera/cli/error.rb +2 -2
- data/lib/aspera/cli/extended_value.rb +38 -19
- data/lib/aspera/cli/formatter.rb +48 -48
- data/lib/aspera/cli/hints.rb +1 -1
- data/lib/aspera/cli/main.rb +190 -168
- data/lib/aspera/cli/manager.rb +15 -15
- data/lib/aspera/cli/plugin.rb +23 -20
- data/lib/aspera/cli/plugin_factory.rb +1 -1
- data/lib/aspera/cli/plugins/alee.rb +1 -1
- data/lib/aspera/cli/plugins/aoc.rb +144 -107
- data/lib/aspera/cli/plugins/ats.rb +19 -17
- data/lib/aspera/cli/plugins/config.rb +67 -83
- data/lib/aspera/cli/plugins/console.rb +5 -3
- data/lib/aspera/cli/plugins/faspex.rb +39 -35
- data/lib/aspera/cli/plugins/faspex5.rb +104 -80
- data/lib/aspera/cli/plugins/faspio.rb +13 -1
- data/lib/aspera/cli/plugins/httpgw.rb +13 -1
- data/lib/aspera/cli/plugins/node.rb +306 -179
- data/lib/aspera/cli/plugins/orchestrator.rb +34 -40
- data/lib/aspera/cli/plugins/preview.rb +3 -3
- data/lib/aspera/cli/plugins/server.rb +6 -6
- data/lib/aspera/cli/plugins/shares.rb +5 -5
- data/lib/aspera/cli/sync_actions.rb +19 -18
- data/lib/aspera/cli/transfer_agent.rb +5 -5
- data/lib/aspera/cli/transfer_progress.rb +2 -2
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +116 -95
- data/lib/aspera/coverage.rb +4 -3
- data/lib/aspera/environment.rb +6 -6
- data/lib/aspera/faspex_gw.rb +14 -14
- data/lib/aspera/faspex_postproc.rb +7 -6
- data/lib/aspera/hash_ext.rb +2 -2
- data/lib/aspera/json_rpc.rb +1 -1
- data/lib/aspera/keychain/encrypted_hash.rb +47 -34
- data/lib/aspera/keychain/factory.rb +41 -0
- data/lib/aspera/keychain/hashicorp_vault.rb +71 -0
- data/lib/aspera/keychain/macos_security.rb +19 -11
- data/lib/aspera/log.rb +28 -34
- data/lib/aspera/nagios.rb +6 -6
- data/lib/aspera/node_simulator.rb +8 -8
- data/lib/aspera/oauth/base.rb +8 -6
- data/lib/aspera/oauth/factory.rb +5 -6
- data/lib/aspera/oauth/url_json.rb +6 -6
- data/lib/aspera/persistency_action_once.rb +6 -4
- data/lib/aspera/persistency_folder.rb +2 -2
- data/lib/aspera/preview/generator.rb +1 -1
- data/lib/aspera/preview/options.rb +16 -16
- data/lib/aspera/preview/terminal.rb +3 -3
- data/lib/aspera/preview/utils.rb +11 -13
- data/lib/aspera/products/connect.rb +1 -1
- data/lib/aspera/products/desktop.rb +1 -1
- data/lib/aspera/products/transferd.rb +1 -1
- data/lib/aspera/proxy_auto_config.rb +2 -2
- data/lib/aspera/rest.rb +52 -43
- data/lib/aspera/rest_errors_aspera.rb +1 -1
- data/lib/aspera/secret_hider.rb +5 -5
- data/lib/aspera/ssh.rb +4 -4
- data/lib/aspera/transfer/convert.rb +29 -0
- data/lib/aspera/transfer/error_info.rb +66 -66
- data/lib/aspera/transfer/parameters.rb +13 -68
- data/lib/aspera/transfer/spec.rb +5 -6
- data/lib/aspera/transfer/spec.schema.yaml +753 -0
- data/lib/aspera/transfer/spec_doc.rb +62 -0
- data/lib/aspera/transfer/sync.rb +23 -72
- data/lib/aspera/transfer/sync_instance.schema.yaml +13 -0
- data/lib/aspera/transfer/sync_session.schema.yaml +79 -0
- data/lib/aspera/transfer/uri.rb +6 -6
- data/lib/aspera/uri_reader.rb +1 -1
- data/lib/aspera/web_auth.rb +1 -1
- data/lib/aspera/web_server_simple.rb +53 -44
- data.tar.gz.sig +1 -2
- metadata +37 -4
- metadata.gz.sig +0 -0
- data/examples/build_package.sh +0 -28
- data/lib/aspera/transfer/spec.yaml +0 -718
data/README.md
CHANGED
@@ -1,11 +1,12 @@
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
2
2
|
<!--
|
3
|
-
|
3
|
+
DO NOT EDIT: THIS FILE IS GENERATED, edit docs/README.erb.md, for details, read docs/README.md
|
4
4
|
|
5
5
|
markdownlint-disable MD033 MD003 MD053
|
6
6
|
cspell:ignore Serban Antipolis
|
7
7
|
PANDOC_META_BEGIN
|
8
8
|
subtitle: "ascli 4.22.0.pre"
|
9
|
+
author: "Laurent MARTIN"
|
9
10
|
PANDOC_META_END
|
10
11
|
-->
|
11
12
|
|
@@ -21,7 +22,7 @@ Laurent/2016-2025
|
|
21
22
|
|
22
23
|
This gem provides the `ascli` CLI (Command Line Interface) to IBM Aspera software.
|
23
24
|
|
24
|
-
`ascli` is
|
25
|
+
`ascli` is also great tool to learn Aspera APIs.
|
25
26
|
|
26
27
|
Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
|
27
28
|
|
@@ -66,7 +67,7 @@ It is designed for:
|
|
66
67
|
- `curl` (for REST calls)
|
67
68
|
- Aspera transfer (`ascp`)
|
68
69
|
|
69
|
-
If the need is to perform operations programmatically in languages such as: C, Go, Python,
|
70
|
+
If the need is to perform operations programmatically in languages such as: C, Go, Python, NodeJS, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
70
71
|
|
71
72
|
- Product APIs (REST) : e.g. AoC, Faspex, node
|
72
73
|
- Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, Go, Ruby, Rust, etc...)
|
@@ -76,15 +77,18 @@ Code examples here: <https://github.com/laurent-martin/aspera-api-examples>
|
|
76
77
|
|
77
78
|
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
78
79
|
|
80
|
+
`ascli` is developer-friendly, designed for quickly testing APIs and learning how to work with Aspera APIs.
|
81
|
+
Refer to section: [Logging, Debugging](#logging-debugging).
|
82
|
+
|
79
83
|
So, which is Aspera's CLI ? `ascli` or `ascp`
|
80
84
|
|
81
85
|
`ascp` is the low level Aspera command line for **transfers**.
|
82
86
|
It is in fact the implementation of the FASP protocol.
|
83
|
-
So, ANY Aspera transfer leads to one ascp process running on client side and another on server side.
|
84
|
-
`ascp` can be used as a command line, but it is very low level, and practically it can be used on command line only if there is no Aspera web
|
87
|
+
So, ANY Aspera transfer leads to one `ascp` process running on client side and another on server side.
|
88
|
+
`ascp` can be used as a command line, but it is very low level, and practically it can be used on command line only if there is no Aspera web app (AoC, Faspex, etc...) and ONLY to do a transfer (send/receive), not for any operation on Aspera apps (e.g. listing remote files).
|
85
89
|
`ascp` does not provide a configuration file to store credentials or options, it does not resume automatically on transfer error.
|
86
90
|
|
87
|
-
In fact, `ascli` can do everything that `ascp` does, and much more,
|
91
|
+
In fact, `ascli` can do everything that `ascp` does, and much more, in an easier way.
|
88
92
|
|
89
93
|
### Notations, Shell, Examples
|
90
94
|
|
@@ -94,12 +98,12 @@ Command line arguments beginning with `my_` in examples, e.g. `my_param_value`,
|
|
94
98
|
|
95
99
|
`ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, etc...)
|
96
100
|
|
97
|
-
Some commands will start an Aspera
|
101
|
+
Some commands will start an Aspera transfer (e.g. `upload`).
|
98
102
|
The transfer is not directly implemented in `ascli`, rather `ascli` uses one of the external Aspera Transfer Clients called **[Transfer Agents](#transfer-clients-agents)**.
|
99
103
|
|
100
|
-
> **Note:** A **[Transfer
|
101
|
-
|
102
|
-
For example a remote Aspera Server may be used as a transfer agent (using
|
104
|
+
> **Note:** A **[Transfer Agent](#transfer-clients-agents)** is a client for the remote Transfer Server (HSTS).
|
105
|
+
A **[Transfer Agent](#transfer-clients-agents)** can be local or remote...
|
106
|
+
For example a remote Aspera Server may be used as a transfer agent (using Node API).
|
103
107
|
i.e. using option `--transfer=node`
|
104
108
|
|
105
109
|
## Quick Start
|
@@ -121,7 +125,7 @@ Once installation is completed, you can proceed to the first use with a demo ser
|
|
121
125
|
|
122
126
|
If you want to test with Aspera on Cloud, jump to section: [Wizard](#configuration-using-wizard).
|
123
127
|
|
124
|
-
To test with Aspera demo transfer server,
|
128
|
+
To test with Aspera demo transfer server, set up the environment and then test:
|
125
129
|
|
126
130
|
```bash
|
127
131
|
ascli config initdemo
|
@@ -142,10 +146,10 @@ ascli server browse /
|
|
142
146
|
╰────────────┴──────┴───────────┴───────┴───────────────────────────┴───────────────────────╯
|
143
147
|
```
|
144
148
|
|
145
|
-
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define
|
149
|
+
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define an [Option Preset](#option-preset) for the server's authentication options.
|
146
150
|
The following example will:
|
147
151
|
|
148
|
-
- Create
|
152
|
+
- Create an [Option Preset](#option-preset)
|
149
153
|
- Define it as default for the `server` plugin
|
150
154
|
- List files in a folder
|
151
155
|
- Download a file
|
@@ -198,7 +202,7 @@ complete
|
|
198
202
|
|
199
203
|
Get familiar with configuration, options, commands : [Command Line Interface](#command-line-interface).
|
200
204
|
|
201
|
-
Then, follow the section relative to the product you want to interact with (
|
205
|
+
Then, follow the section relative to the product you want to interact with (Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
202
206
|
|
203
207
|
## Installation
|
204
208
|
|
@@ -224,27 +228,33 @@ A package with pre-installed Ruby, gem and `ascp` may also be provided.
|
|
224
228
|
### `ascli` executable
|
225
229
|
|
226
230
|
It is planned to provide `ascli` as a single platform-dependent executable.
|
227
|
-
[
|
231
|
+
[Beta releases can be found here](https://ibm.biz/aspera-cli-exe).
|
228
232
|
|
229
|
-
**Note:** This is
|
233
|
+
**Note:** This is a Beta feature. On Linux, the executable requires a minimum GLIBC version. Installation of `ascp` is still required separately. Refer to [Install `ascp`](#installation-of-ascp-through-transferd).
|
230
234
|
|
231
|
-
On Linux, check your GLIBC version with `ldd`:
|
235
|
+
On Linux, check the minimum required GLIBC on this site: [repology.org](https://repology.org/project/glibc/versions), or check your GLIBC version with `ldd`:
|
236
|
+
|
237
|
+
```bash
|
238
|
+
ldd --version | head -n1
|
239
|
+
```
|
232
240
|
|
233
241
|
```console
|
234
|
-
$ ldd --version | head -n1
|
235
242
|
ldd (GNU libc) 2.34
|
236
243
|
```
|
237
244
|
|
238
|
-
Check an executable's
|
245
|
+
Check an executable's (`ascli`, `ascp`) minimum required GLIBC version:
|
246
|
+
|
247
|
+
```bash
|
248
|
+
objdump -p /bin/bash | sed -n 's/^.*GLIBC_//p' | sort -V | tail -n1
|
249
|
+
```
|
239
250
|
|
240
251
|
```console
|
241
|
-
$ objdump -p /bin/bash | sed -n 's/^.*GLIBC_//p' | sort -V | tail -n1
|
242
252
|
2.34
|
243
253
|
```
|
244
254
|
|
245
|
-
> **Note:** if `objdump` is not available use `strings` or `grep -z 'GLIBC_'|tr \\0 \\n`
|
255
|
+
> **Note:** if `objdump` is not available, then use `strings` or `grep -z 'GLIBC_'|tr \\0 \\n`
|
246
256
|
|
247
|
-
The `
|
257
|
+
The required GLIBC version for `ascp` can be found in the [Release Notes of HSTS](https://www.ibm.com/docs/en/ahts) or [here](https://eudemo.asperademo.com/download/sdk.html).
|
248
258
|
|
249
259
|
### Ruby
|
250
260
|
|
@@ -254,7 +264,7 @@ Required Ruby version: >= 3.1.
|
|
254
264
|
|
255
265
|
> **Deprecation notice**: the minimum Ruby version will be 3.2 in a future version.
|
256
266
|
|
257
|
-
**Ruby can be installed using any method** : rpm
|
267
|
+
**Ruby can be installed using any method** : `rpm`, `yum`, `dnf`, `rvm`, `brew`, Windows installer, ...
|
258
268
|
|
259
269
|
**In priority**, refer to the official Ruby documentation:
|
260
270
|
|
@@ -263,7 +273,7 @@ Required Ruby version: >= 3.1.
|
|
263
273
|
|
264
274
|
For convenience, you may refer to the following sections for a proposed method for specific operating systems.
|
265
275
|
|
266
|
-
Latest version of `ascli` requires a
|
276
|
+
Latest version of `ascli` requires a Ruby version [at least under maintenance support](https://www.ruby-lang.org/en/downloads/branches/).
|
267
277
|
If an older Ruby version is needed, then use an older version of `ascli` that supports it.
|
268
278
|
|
269
279
|
#### Windows: Installer
|
@@ -272,9 +282,9 @@ Manual installation:
|
|
272
282
|
|
273
283
|
- Navigate to [https://rubyinstaller.org/](https://rubyinstaller.org/) → **Downloads**.
|
274
284
|
- Download the latest Ruby installer **"with devkit"**. (`Msys2` is needed to install some native extensions, such as `grpc`)
|
275
|
-
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
276
|
-
- At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and mingw)
|
277
|
-
- Then install the aspera-cli gem and Aspera SDK (see next sections)
|
285
|
+
- Execute the installer which installs by default in: `C:\RubyVV-x64` (`VV` is the version number)
|
286
|
+
- At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and `mingw`)
|
287
|
+
- Then install the `aspera-cli` gem and Aspera SDK (see next sections)
|
278
288
|
|
279
289
|
Automated installation, with internet access:
|
280
290
|
|
@@ -289,18 +299,41 @@ rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\asper
|
|
289
299
|
#### macOS: `brew`
|
290
300
|
|
291
301
|
**macOS** comes with Ruby 2.6.
|
292
|
-
It is an old unsupported version and [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes)
|
302
|
+
It is an old unsupported version and [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes).
|
303
|
+
It will be removed from macOS in the future.
|
293
304
|
Do not use it.
|
294
305
|
|
295
|
-
The recommended way is to
|
306
|
+
The recommended way is to use [Homebrew](https://brew.sh/).
|
296
307
|
|
297
308
|
```bash
|
298
309
|
brew install ruby
|
299
310
|
```
|
300
311
|
|
301
|
-
This installs a recent
|
312
|
+
This installs a recent Ruby suitable for `ascli`.
|
302
313
|
|
303
|
-
|
314
|
+
To add PATH to Ruby on Apple Silicon, add this in your shell configuration file (e.g. `~/.bash_profile` or `~/.zshrc`):
|
315
|
+
|
316
|
+
```bash
|
317
|
+
use_ruby(){
|
318
|
+
version=$1
|
319
|
+
local prefix=$(brew --prefix ruby${version:+@}$version)
|
320
|
+
if ! test -d $prefix;then
|
321
|
+
echo "No such ruby version: $version"
|
322
|
+
brew list|grep ruby
|
323
|
+
return 1
|
324
|
+
fi
|
325
|
+
export PATH="$prefix/bin:$PATH"
|
326
|
+
export PATH="$(gem env gemdir)/bin:$PATH"
|
327
|
+
export LDFLAGS="-L$prefix/lib"
|
328
|
+
export CPPFLAGS="-I$prefix/include"
|
329
|
+
export PKG_CONFIG_PATH="$prefix/lib/pkgconfig"
|
330
|
+
echo "Using: $prefix"
|
331
|
+
ruby -v
|
332
|
+
}
|
333
|
+
use_ruby
|
334
|
+
```
|
335
|
+
|
336
|
+
If using [RVM](https://rvm.io/), one way to force use of OpenSSL 3.0 is:
|
304
337
|
|
305
338
|
```bash
|
306
339
|
RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@3.0)" rvm install 3.4.0
|
@@ -365,7 +398,7 @@ Install `rvm`.
|
|
365
398
|
Follow [https://rvm.io/](https://rvm.io/).
|
366
399
|
|
367
400
|
Execute the shell/curl command.
|
368
|
-
As regular user, it installs in the user's home: `~/.rvm
|
401
|
+
As regular user, it installs in the user's home: `~/.rvm`.
|
369
402
|
|
370
403
|
```bash
|
371
404
|
\curl -sSL https://get.rvm.io | bash -s stable
|
@@ -420,7 +453,7 @@ rvm version
|
|
420
453
|
|
421
454
|
#### Linux as non-root
|
422
455
|
|
423
|
-
If you don't have root access, you can install Ruby in your home directory using `rbenv` see [rbenv-installer](https://github.com/rbenv/rbenv-installer#rbenv-installer):
|
456
|
+
If you don't have root access, you can install Ruby in your home directory using `rbenv` see [`rbenv-installer`](https://github.com/rbenv/rbenv-installer#rbenv-installer):
|
424
457
|
|
425
458
|
```bash
|
426
459
|
curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash
|
@@ -473,7 +506,7 @@ make install
|
|
473
506
|
All what is needed is a JVM (Java Virtual Machine) on your system (`java`).
|
474
507
|
The JRuby package comes pre-complied and does not require compilation of native extensions.
|
475
508
|
Use a version of JRuby compatible with Ruby version supported by `ascli`.
|
476
|
-
Refer to [the
|
509
|
+
Refer to [the Wikipedia page](https://en.wikipedia.org/wiki/JRuby) to match JRuby and Ruby versions.
|
477
510
|
Choose the latest version from:
|
478
511
|
|
479
512
|
<https://www.jruby.org/download>
|
@@ -502,18 +535,18 @@ Some additional gems are required for some specific features, see [Gemfile](Gemf
|
|
502
535
|
|
503
536
|
| name | version | comment |
|
504
537
|
| ---- | ------- | ------- |
|
505
|
-
| grpc | ~> 1.
|
538
|
+
| grpc | ~> 1.71 | for Aspera Transfer Daemon |
|
506
539
|
| mimemagic | ~> 0.4 | for preview |
|
507
|
-
| rmagick | ~>
|
508
|
-
| symmetric-encryption | ~> 4.6 | for file
|
540
|
+
| rmagick | ~> 6.1 | for terminal view |
|
541
|
+
| symmetric-encryption | ~> 4.6 | for encrypted hash file secrets |
|
509
542
|
| bigdecimal | ~> 3.1.9 | if RUBY_VERSION >= '3.4' for symmetric-encryption ? |
|
510
543
|
|
511
544
|
Install like this:
|
512
545
|
|
513
546
|
```bash
|
514
|
-
gem install grpc -v '~> 1.
|
547
|
+
gem install grpc -v '~> 1.71'
|
515
548
|
gem install mimemagic -v '~> 0.4'
|
516
|
-
gem install rmagick -v '~>
|
549
|
+
gem install rmagick -v '~> 6.1'
|
517
550
|
gem install symmetric-encryption -v '~> 4.6'
|
518
551
|
gem install bigdecimal -v '~> 3.1.9'
|
519
552
|
```
|
@@ -535,7 +568,7 @@ gem update aspera-cli
|
|
535
568
|
```
|
536
569
|
|
537
570
|
During its execution, `ascli` checks every week if a new version is available and notifies the user in a WARN log.
|
538
|
-
To
|
571
|
+
To deactivate this feature, globally set the option `version_check_days` to `0`, or specify a different period in days.
|
539
572
|
|
540
573
|
To check if a new version is available (independently of `version_check_days`):
|
541
574
|
|
@@ -545,7 +578,7 @@ ascli config check_update
|
|
545
578
|
|
546
579
|
#### Gem installation with signature verification
|
547
580
|
|
548
|
-
The gem is signed with a private key, and the public certificate is available in the
|
581
|
+
The gem is signed with a private key, and the public certificate is available in the GitHub repository (`certs/aspera-cli-public-cert.pem`).
|
549
582
|
When installing the gem, the signature can be optionally verified.
|
550
583
|
|
551
584
|
For [secure installation](https://guides.rubygems.org/security/#using-gems), one can install the gem with the public key:
|
@@ -568,7 +601,7 @@ Most file transfers will be executed using the **FASP** protocol, using `ascp`.
|
|
568
601
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
569
602
|
|
570
603
|
- `ascp`
|
571
|
-
- `aspera-license` (in same folder, or
|
604
|
+
- `aspera-license` (in same folder, or `../etc`)
|
572
605
|
|
573
606
|
This can be installed either be installing an Aspera transfer software or using an `ascli` command.
|
574
607
|
|
@@ -593,7 +626,7 @@ ascli config transferd install
|
|
593
626
|
The installation of the transfer binary follows those steps:
|
594
627
|
|
595
628
|
- Check the value of option `sdk_url`: if the value is the default value `DEF`, then the procedure follows, else it specified a URL where to take the archive from.
|
596
|
-
- The location of archives is retrieved from the
|
629
|
+
- The location of archives is retrieved from the URL specified by option `locations_url` whose default value is <https://ibm.biz/sdk_location>
|
597
630
|
- The archive for the current system architecture (CPU and OS) is selected and downloaded.
|
598
631
|
|
599
632
|
The option `locations_url` can be set to override the URL where the list of versions is located, in case of air-gap environment or for testing.
|
@@ -634,7 +667,7 @@ If the embedded method is not used, the following packages are also suitable:
|
|
634
667
|
- IBM Aspera Connect Client (Free)
|
635
668
|
- IBM Aspera Desktop Client (Free)
|
636
669
|
- IBM Aspera High Speed Transfer Server (Licensed)
|
637
|
-
- IBM Aspera High Speed Transfer
|
670
|
+
- IBM Aspera High Speed Transfer Endpoint (Licensed)
|
638
671
|
|
639
672
|
For instance, Aspera Connect Client can be installed by visiting the page:
|
640
673
|
[https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
@@ -652,7 +685,7 @@ Refer to section: [Transfer Agents](#transfer-clients-agents)
|
|
652
685
|
|
653
686
|
#### Gem files and dependencies
|
654
687
|
|
655
|
-
The sample script: [examples/build_package.sh](examples/build_package.sh) can be used to download all necessary gems and dependencies in a tar
|
688
|
+
The sample script: [examples/build_package.sh](examples/build_package.sh) can be used to download all necessary gems and dependencies in a `tar.gz`.
|
656
689
|
|
657
690
|
```console
|
658
691
|
$ ./build_package.sh aspera-cli 4.18.0
|
@@ -690,7 +723,7 @@ source ~/.rvm/scripts/rvm
|
|
690
723
|
ascli config ascp install --sdk-url=file:///[SDK archive file path]
|
691
724
|
```
|
692
725
|
|
693
|
-
- Add those lines to shell
|
726
|
+
- Add those lines to shell environment (`.profile`)
|
694
727
|
|
695
728
|
```bash
|
696
729
|
source ~/.rvm/scripts/rvm
|
@@ -711,7 +744,7 @@ It is essentially the same procedure as installation for Windows with internet,
|
|
711
744
|
|
712
745
|
- Create an archive with necessary gems like in previous section
|
713
746
|
|
714
|
-
- Download the SDK following [Install `ascp`](#installation-of-ascp-through-transferd)
|
747
|
+
- Download the SDK following: [Install `ascp`](#installation-of-ascp-through-transferd)
|
715
748
|
|
716
749
|
- Create a Zip with all those files and transfer to the target system.
|
717
750
|
|
@@ -751,13 +784,14 @@ podman --version
|
|
751
784
|
#### Container: Quick start
|
752
785
|
|
753
786
|
**Wanna start quickly ?** With an interactive shell ?
|
787
|
+
|
754
788
|
Execute this:
|
755
789
|
|
756
790
|
```bash
|
757
791
|
podman run --rm --tty --interactive --entrypoint bash docker.io/martinlaurent/ascli:latest
|
758
792
|
```
|
759
793
|
|
760
|
-
> **Note:** This command changes the
|
794
|
+
> **Note:** This command changes the entry point to an interactive shell instead of direct execution of `ascli`.
|
761
795
|
|
762
796
|
Then, execute individual `ascli` commands such as:
|
763
797
|
|
@@ -809,7 +843,7 @@ Add options:
|
|
809
843
|
--user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli
|
810
844
|
```
|
811
845
|
|
812
|
-
> **Note:** If you are using a `podman machine`, e.g. on macOS
|
846
|
+
> **Note:** If you are using a `podman machine`, e.g. on macOS, make sure that the folder is also shared between the VM and the host, so that sharing is: container → VM → Host: `podman machine init ... --volume="/Users:/Users"`
|
813
847
|
|
814
848
|
As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available,
|
815
849
|
you can change the entry point, add option:
|
@@ -853,7 +887,7 @@ Some environment variables can be set for this script to adapt its behavior:
|
|
853
887
|
|----------------|------------------------------------|--------------------------|--------------------------|
|
854
888
|
| `ASCLI_HOME` | Configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascli_config` |
|
855
889
|
| `docker_args` | Additional options to `podman` | <empty> | `--volume /Users:/Users` |
|
856
|
-
| `image` | Container image name | `docker.io/martinlaurent/ascli`
|
890
|
+
| `image` | Container image name | `docker.io/martinlaurent/ascli` | n/a |
|
857
891
|
| `version` | Container image version | Latest | `4.8.0.pre` |
|
858
892
|
|
859
893
|
The wrapping script maps the folder `$ASCLI_HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
@@ -900,7 +934,7 @@ podman load -i ascli_image_latest.tar.gz
|
|
900
934
|
#### Container: `aspera.conf`
|
901
935
|
|
902
936
|
`ascp`'s configuration file `aspera.conf` is located in the container at: `/ibm_aspera/aspera.conf` (see Dockerfile).
|
903
|
-
As the container is immutable, it is not recommended
|
937
|
+
As the container is immutable, it is not recommended modifying this file.
|
904
938
|
If one wants to change the content, it is possible to tell `ascp` to use another file using `ascp` option `-f`, e.g. by locating it on the host folder `$HOME/.aspera/ascli` mapped to the container folder `/home/cliuser/.aspera/ascli`:
|
905
939
|
|
906
940
|
```bash
|
@@ -945,7 +979,7 @@ singularity shell ascli.sif
|
|
945
979
|
|
946
980
|
`ascli` uses the Ruby `openssl` gem which uses by default the system's `openssl` library and its CA certificate bundle.
|
947
981
|
|
948
|
-
To display the version of **
|
982
|
+
To display the version of **OpenSSL** used in `ascli`:
|
949
983
|
|
950
984
|
```bash
|
951
985
|
ascli config echo @ruby:OpenSSL::OPENSSL_VERSION --format=text
|
@@ -977,7 +1011,7 @@ To display trusted certificate store locations:
|
|
977
1011
|
ascli --show-config --fields=cert_stores
|
978
1012
|
```
|
979
1013
|
|
980
|
-
Certificates are checked against the [Ruby default certificate store](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html) `OpenSSL::X509::DEFAULT_CERT_FILE` and `OpenSSL::X509::DEFAULT_CERT_DIR`, which are typically the ones of `openssl` on Unix-like systems (Linux, macOS, etc
|
1014
|
+
Certificates are checked against the [Ruby default certificate store](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html) `OpenSSL::X509::DEFAULT_CERT_FILE` and `OpenSSL::X509::DEFAULT_CERT_DIR`, which are typically the ones of `openssl` on Unix-like systems (Linux, macOS, etc...).
|
981
1015
|
Ruby's default values can be overridden using env vars: `SSL_CERT_FILE` and `SSL_CERT_DIR`.
|
982
1016
|
|
983
1017
|
One can display those default values:
|
@@ -998,13 +1032,13 @@ update-ca-trust extract
|
|
998
1032
|
```
|
999
1033
|
|
1000
1034
|
The SSL CA certificate bundle can be specified using option `cert_stores`, which is a list of files or folders.
|
1001
|
-
By default it uses Ruby's default certificate store.
|
1035
|
+
By default, it uses Ruby's default certificate store.
|
1002
1036
|
|
1003
1037
|
If you use this option, then default locations are not used.
|
1004
1038
|
Default locations can be added using special value `DEF`.
|
1005
1039
|
The value can be either an `Array` or `String` (path).
|
1006
1040
|
Successive options add paths incrementally.
|
1007
|
-
All files of a
|
1041
|
+
All files of a folder are added.
|
1008
1042
|
|
1009
1043
|
JRuby uses its own implementation and CA bundles.
|
1010
1044
|
|
@@ -1016,8 +1050,8 @@ For example, on Linux to force the use the system's certificate store:
|
|
1016
1050
|
|
1017
1051
|
`ascp` also needs to validate certificates when using **WSS** for transfer TCP part (instead of **SSH**).
|
1018
1052
|
|
1019
|
-
By default,`ascp` uses
|
1020
|
-
Original `ascp`'s
|
1053
|
+
By default,`ascp` uses a hard coded root location `OPENSSLDIR`.
|
1054
|
+
Original `ascp`'s hard coded locations can be found using:
|
1021
1055
|
|
1022
1056
|
```bash
|
1023
1057
|
ascli config ascp info --fields=openssldir
|
@@ -1025,7 +1059,7 @@ ascli config ascp info --fields=openssldir
|
|
1025
1059
|
|
1026
1060
|
E.g. on macOS: `/Library/Aspera/ssl`.
|
1027
1061
|
Then trusted certificates are taken from `[OPENSSLDIR]/cert.pem` and files in `[OPENSSLDIR]/certs`.
|
1028
|
-
`ascli` overrides the default
|
1062
|
+
`ascli` overrides the default hard coded location used by `ascp` for WSS and uses the same locations as specified in `cert_stores` (using the `-i` option of `ascp`).
|
1029
1063
|
|
1030
1064
|
To update trusted root certificates for `ascli`:
|
1031
1065
|
Display the trusted certificate store locations used by `ascli`.
|
@@ -1075,17 +1109,17 @@ The `aspera-cli` gem provides a command line interface (CLI) which interacts wit
|
|
1075
1109
|
|
1076
1110
|
`ascli` provides the following features:
|
1077
1111
|
|
1078
|
-
-
|
1112
|
+
- Commands to Aspera server products (on-premise and SaaS)
|
1079
1113
|
- Any command line **options** (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files, ...
|
1080
|
-
-
|
1114
|
+
- Commands, Options, and Option values shortcuts
|
1081
1115
|
- FASP [Transfer Agents](#transfer-clients-agents) can be: local `ascp`, or Connect Client, or any transfer node
|
1082
1116
|
- Transfer parameters can be altered by modification of [**transfer-spec**](#transfer-specification), this includes requiring multi-session
|
1083
1117
|
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
1084
|
-
-
|
1085
|
-
-
|
1118
|
+
- faspstream creation (using Node API)
|
1119
|
+
- **Watchfolder** creation (using Node API)
|
1086
1120
|
- Additional command plugins can be written by the user
|
1087
|
-
-
|
1088
|
-
-
|
1121
|
+
- Download of Faspex and Aspera on Cloud "external" links
|
1122
|
+
- **Legacy** SSH based FASP transfers and remote commands (`ascmd`)
|
1089
1123
|
|
1090
1124
|
Basic usage is displayed by executing:
|
1091
1125
|
|
@@ -1108,7 +1142,7 @@ Using `ascli` with plugin `server` for command line gives advantages over `ascp`
|
|
1108
1142
|
- Choice of transfer agents
|
1109
1143
|
- Integrated support of multi-session
|
1110
1144
|
|
1111
|
-
|
1145
|
+
All `ascp` options are supported either through transfer spec parameters (listed with `conf ascp spec`) and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args` in `transfer_info`).
|
1112
1146
|
|
1113
1147
|
### Command line parsing, Special Characters
|
1114
1148
|
|
@@ -1118,10 +1152,10 @@ The way arguments are parsed and provided to `ascli` depend on the Operating Sys
|
|
1118
1152
|
|
1119
1153
|
#### Shell parsing for Unix-like systems: Linux, macOS, AIX
|
1120
1154
|
|
1121
|
-
Linux command line parsing is well
|
1155
|
+
Linux command line parsing is well-defined:
|
1122
1156
|
It is fully documented in the shell's documentation.
|
1123
1157
|
|
1124
|
-
On Unix-like environments, this is typically a POSIX-like shell (bash
|
1158
|
+
On Unix-like environments, this is typically a POSIX-like shell (`bash`, `zsh`, `ksh`, `sh`).
|
1125
1159
|
A c-shell (`csh`, `tcsh`) or other shell can also be used.
|
1126
1160
|
In this environment the shell parses the command line, possibly replacing variables, etc...
|
1127
1161
|
See [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation).
|
@@ -1135,7 +1169,7 @@ Command line parsing first depends on the shell used.
|
|
1135
1169
|
MS Windows command line parsing is not like Unix-like systems simply because Windows does not provide a list of arguments to the executable (Ruby): it provides the whole command line as a single string, but the shell may interpret some special characters.
|
1136
1170
|
|
1137
1171
|
So command line parsing is not handled by the shell (`cmd.exe`), not handled by the operating system, but it is handled by the executable.
|
1138
|
-
Typically, Windows executables use the [
|
1172
|
+
Typically, Windows executables use the [Microsoft library for this parsing](https://learn.microsoft.com/en-us/cpp/cpp/main-function-command-line-args).
|
1139
1173
|
|
1140
1174
|
As far as `ascli` is concerned: the executable is Ruby.
|
1141
1175
|
It has its own parsing algorithm, close to a Linux shell parsing.
|
@@ -1152,7 +1186,7 @@ Hello World
|
|
1152
1186
|
2
|
1153
1187
|
```
|
1154
1188
|
|
1155
|
-
> **Note:** Use `pp` instead of `puts` to display as
|
1189
|
+
> **Note:** Use `pp` instead of `puts` to display as Ruby `Array`.
|
1156
1190
|
|
1157
1191
|
Once the shell has dealt with the command line "special" characters for it, the shell calls Windows' `CreateProcess` with just the whole command line as a single string.
|
1158
1192
|
(Unlike Unix-like systems where the command line is split into arguments by the shell.)
|
@@ -1225,11 +1259,11 @@ PS C:\> ascli config echo --% @json:'{"k":"v","x":"y"}'
|
|
1225
1259
|
PS C:\> ascli config echo @json:'{"""k""":"""v""","""x""":"""y"""}'
|
1226
1260
|
```
|
1227
1261
|
|
1228
|
-
> **Note:** The special
|
1262
|
+
> **Note:** The special Powershell argument `--%` places Powershell in legacy parsing mode.
|
1229
1263
|
|
1230
1264
|
#### Extended Values (JSON, Ruby, ...)
|
1231
1265
|
|
1232
|
-
Some
|
1266
|
+
Some values provided to `ascli` (options, **Command Parameters**) are expected to be [Extended Values](#extended-value-syntax), i.e. not a simple `String`, but a composite structure (`Hash`, `Array`).
|
1233
1267
|
Typically, the `@json:` modifier is used, it expects a [JSON](https://www.json.org/) string.
|
1234
1268
|
JSON itself has some special syntax: for example `"` is used to enclose a `String`.
|
1235
1269
|
|
@@ -1256,8 +1290,8 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
1256
1290
|
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
1257
1291
|
|
1258
1292
|
In the following examples (using a POSIX shell, such as `bash`), several equivalent commands are provided.
|
1259
|
-
For all example, most
|
1260
|
-
It depends on the underlying syntax: shell
|
1293
|
+
For all example, most special character handling is not specific to `ascli`:
|
1294
|
+
It depends on the underlying syntax: shell, JSON, etc...
|
1261
1295
|
Depending on the case, a different `format` option is used to display the actual value.
|
1262
1296
|
|
1263
1297
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
@@ -1307,7 +1341,7 @@ ascli config echo '"'
|
|
1307
1341
|
```
|
1308
1342
|
|
1309
1343
|
Double quote in JSON is a little tricky because `"` is special both for the shell and JSON.
|
1310
|
-
Both shell and JSON syntax allow
|
1344
|
+
Both shell and JSON syntax allow protecting `"`, but only the shell allows protection using single quote.
|
1311
1345
|
|
1312
1346
|
```bash
|
1313
1347
|
ascli config echo @json:'"\""' --format=text
|
@@ -1479,7 +1513,7 @@ The advantages of using a **Command Parameter** instead of an option for the sam
|
|
1479
1513
|
The disadvantage is that it is not possible to define a default value in a configuration file or environment variable using an option value.
|
1480
1514
|
Nevertheless, [Extended Values](#extended-value-syntax) syntax is supported, so it is possible to retrieve a value from the configuration file (using `@preset:`) or environment variable (using `@env:`).
|
1481
1515
|
|
1482
|
-
If a **Command
|
1516
|
+
If a **Command Parameter** begins with `-`, then either use the `@val:` syntax (see [Extended Values](#extended-value-syntax)), or use the `--` separator (see below).
|
1483
1517
|
|
1484
1518
|
A few **Command Parameters** are optional, they are always located at the end of the command line.
|
1485
1519
|
|
@@ -1488,7 +1522,7 @@ A few **Command Parameters** are optional, they are always located at the end of
|
|
1488
1522
|
All options, e.g. `--log-level=debug`, are command line arguments that:
|
1489
1523
|
|
1490
1524
|
- Start with `--`
|
1491
|
-
- Have a name, in lowercase, using `-` as word separator in name
|
1525
|
+
- Have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
|
1492
1526
|
- Have a value, separated from name with a `=`
|
1493
1527
|
- Can be used by prefix (not recommended), provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
|
1494
1528
|
- Is optional on command line (it has a default value or no value)
|
@@ -1512,7 +1546,7 @@ ascli config echo -- --sample
|
|
1512
1546
|
|
1513
1547
|
> **Note:** Here, `--sample` is taken as an argument, and not as an option, due to `--`.
|
1514
1548
|
|
1515
|
-
Options may have a (
|
1549
|
+
Options may have a (hard coded) default value.
|
1516
1550
|
|
1517
1551
|
Options can be placed anywhere on command line and are evaluated in order.
|
1518
1552
|
Usually the last value evaluated overrides previous values, but some options are cumulative, e.g. `ts`.
|
@@ -1591,7 +1625,7 @@ By default, result of type `single_object` and `object_list` are displayed using
|
|
1591
1625
|
|
1592
1626
|
The table style can be customized with option: `table_style` which expects a `Hash`, options are the ones described in gem [`terminal-table`](https://github.com/tj/terminal-table).
|
1593
1627
|
|
1594
|
-
For example, to display a table with thick
|
1628
|
+
For example, to display a table with thick Unicode borders:
|
1595
1629
|
|
1596
1630
|
```bash
|
1597
1631
|
ascli config preset over --table-style=@ruby:'{border: :unicode_thick_edge}'
|
@@ -1601,13 +1635,13 @@ ascli config preset over --table-style=@ruby:'{border: :unicode_thick_edge}'
|
|
1601
1635
|
|
1602
1636
|
#### Option: `flat_hash`: `.`-join keys
|
1603
1637
|
|
1604
|
-
This
|
1638
|
+
This option controls how object fields are displayed for complex objects.
|
1605
1639
|
|
1606
1640
|
Effective only when `format` is `table` to display `single_object` or `object_list`.
|
1607
1641
|
|
1608
1642
|
If value is `no`, then object's `field` names are only the first level keys of the `Hash` result and values that are `Hash` are displayed as such in Ruby syntax.
|
1609
1643
|
|
1610
|
-
If value is `yes` (default), then object are flattened, i.e. deep `Hash` are transformed into 1-level Hash
|
1644
|
+
If value is `yes` (default), then object are flattened, i.e. deep `Hash` are transformed into 1-level `Hash`, where keys are `.`-junction of deep keys.
|
1611
1645
|
In this case, it is possible to filter fields using the option `fields` using the compound field name using `.` (dot) as separator.
|
1612
1646
|
|
1613
1647
|
Example: Result of command is a list of objects with a single object:
|
@@ -1635,7 +1669,7 @@ This option controls if object fields are displayed as columns or lines.
|
|
1635
1669
|
If value is `no` (default), `object_list` are displayed with one object per line, with fields as columns (see above).
|
1636
1670
|
`single_object` are displayed with one field per line (and columns are: `field`, `value`).
|
1637
1671
|
|
1638
|
-
If
|
1672
|
+
If a `object_list` has a single element, it is possible to have `ascli` display the object as a single object (one field per line instead of columns) with option: `multi_single` set to `single`.
|
1639
1673
|
|
1640
1674
|
This parameter can be set as a global default with:
|
1641
1675
|
|
@@ -1672,11 +1706,9 @@ The option `display` controls the level of output:
|
|
1672
1706
|
|
1673
1707
|
#### Option: `show_secrets`: Hide or show secrets in results
|
1674
1708
|
|
1675
|
-
If value is `no` (default), then secrets are redacted from command results.
|
1676
|
-
|
1677
|
-
If
|
1678
|
-
|
1679
|
-
If `display` is `data`, secrets are included to allows piping results.
|
1709
|
+
- If value is `no` (default), then secrets are redacted from command results.
|
1710
|
+
- If value is `yes`, then secrets shown in clear in results.
|
1711
|
+
- If `display` is `data`, secrets are included to allow piping results.
|
1680
1712
|
|
1681
1713
|
#### Option: `fields`: Selection of output object fields
|
1682
1714
|
|
@@ -1704,7 +1736,7 @@ Examples:
|
|
1704
1736
|
|
1705
1737
|
Table output (`object_list`) can be filtered using option `select`.
|
1706
1738
|
This option is either a `Hash` or `Proc`.
|
1707
|
-
The `Proc` takes as argument a line (`Hash`) in the table and is a Ruby lambda expression that shall
|
1739
|
+
The `Proc` takes as argument a line (`Hash`) in the table and is a Ruby lambda expression that shall return `true` to select or `false` to remove an entry.
|
1708
1740
|
|
1709
1741
|
Example:
|
1710
1742
|
|
@@ -1753,7 +1785,7 @@ ascli aoc admin user show %name:john
|
|
1753
1785
|
### Extended Value Syntax
|
1754
1786
|
|
1755
1787
|
Most options and arguments are specified by a simple string (e.g. `username` or `url`).
|
1756
|
-
|
1788
|
+
Sometimes it is convenient to read a value from a file: for example read the PEM value of a private key, or a list of files.
|
1757
1789
|
Some options expect a more complex value such as `Hash` or `Array`.
|
1758
1790
|
|
1759
1791
|
The **Extended Value** Syntax allows to specify such values and even read values from other sources than the command line itself.
|
@@ -1764,7 +1796,7 @@ The syntax is:
|
|
1764
1796
|
<0 or more decoders><some text value or nothing>
|
1765
1797
|
```
|
1766
1798
|
|
1767
|
-
Decoders act like a function with its parameter on right
|
1799
|
+
Decoders act like a function with its parameter on right-hand side and are recognized by the prefix: `@` and suffix `:`
|
1768
1800
|
|
1769
1801
|
The following decoders are supported:
|
1770
1802
|
|
@@ -1779,7 +1811,7 @@ The following decoders are supported:
|
|
1779
1811
|
| `list` | `String` | `Array` | Split a string in multiple items taking first character as separator and return an array |
|
1780
1812
|
| `none` | None | Nil | A null value |
|
1781
1813
|
| `path` | `String` | `String` | Performs path expansion on specified path (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml` |
|
1782
|
-
| `preset` | `String` | `Hash` | Get whole option preset value by name. Sub-values can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]` |
|
1814
|
+
| `preset` | `String` | `Hash` | Get whole option preset value by name. Sub-values can also be used, using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]` |
|
1783
1815
|
| `extend` | `String` | `String` | Evaluates embedded extended value syntax in string |
|
1784
1816
|
| `re` | `String` | `Regexp` | Ruby Regular Expression (short for `@ruby:/.../`) |
|
1785
1817
|
| `ruby` | `String` | Any | Execute specified Ruby code |
|
@@ -1789,7 +1821,7 @@ The following decoders are supported:
|
|
1789
1821
|
| `uri` | `String` | `String` | Read value from specified URL, e.g. `--fpac=@uri:http://serv/f.pac` |
|
1790
1822
|
| `val` | `String` | `String` | Prevent decoders on the right to be decoded. e.g. `--key=@val:@file:foo` sets the option `key` to value `@file:foo`. |
|
1791
1823
|
| `yaml` | `String` | Any | Decode YAML |
|
1792
|
-
| `zlib` | `String` | `String` |
|
1824
|
+
| `zlib` | `String` | `String` | Decompress data using zlib |
|
1793
1825
|
|
1794
1826
|
> **Note:** A few commands support a value of type `Proc` (lambda expression).
|
1795
1827
|
For example, the **Extended Value** `@ruby:'->(i){i["attr"]}'` is a lambda expression that returns the value for key `attr` of the `Hash` `i`.
|
@@ -1817,7 +1849,7 @@ Example: Create a `Hash` value with one key and the value is read from a file:
|
|
1817
1849
|
ascli config echo @ruby:'{"token_verification_key"=>File.read("mykey.txt")}'
|
1818
1850
|
```
|
1819
1851
|
|
1820
|
-
Example: read a
|
1852
|
+
Example: read a CSV file and create an `Array` of `Hash` for bulk provisioning:
|
1821
1853
|
|
1822
1854
|
```bash
|
1823
1855
|
cat test.csv
|
@@ -1857,7 +1889,7 @@ ascli config echo @json:@extend:'{"hello":true,"version":"@preset:config.version
|
|
1857
1889
|
╰─────────┴────────╯
|
1858
1890
|
```
|
1859
1891
|
|
1860
|
-
Example: Create a `Hash` from YAML provided as shell **
|
1892
|
+
Example: Create a `Hash` from YAML provided as shell **Here document**:
|
1861
1893
|
|
1862
1894
|
```bash
|
1863
1895
|
ascli config echo @yaml:@stdin: --format=json<<EOF
|
@@ -1907,7 +1939,7 @@ C:\Users\Kenji\.aspera\ascli
|
|
1907
1939
|
```
|
1908
1940
|
|
1909
1941
|
When OAuth is used (AoC, Faspex5) `ascli` keeps a cache of generated bearer tokens in folder `persist_store` in configuration folder by default.
|
1910
|
-
Option `cache_tokens` (**yes**/no) allows
|
1942
|
+
Option `cache_tokens` (**yes**/no) allows controlling if OAuth tokens are cached on file system, or generated for each request.
|
1911
1943
|
The command `config tokens flush` clears that cache.
|
1912
1944
|
Tokens are kept on disk for a maximum of 30 minutes (`TOKEN_CACHE_EXPIRY_SEC`) and garbage collected after that.
|
1913
1945
|
When a token has expired, then a new token is generated, either using a refresh_token if it is available, or by the default method.
|
@@ -1933,11 +1965,11 @@ A configuration file provides a way to define default values, especially for aut
|
|
1933
1965
|
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overridden with option `--config-file=path` or its env var).
|
1934
1966
|
|
1935
1967
|
The configuration file is a catalog of named lists of options, called: [Option Preset](#option-preset).
|
1936
|
-
Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of
|
1968
|
+
Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of an [Option Preset](#option-preset) (e.g. `mypreset`) using the option `preset`: `--preset=mypreset` or its shortcut: `-Pmypreset`.
|
1937
1969
|
|
1938
1970
|
#### Option Preset
|
1939
1971
|
|
1940
|
-
|
1972
|
+
An [Option Preset](#option-preset) is a collection of options and their associated values in a named section in the configuration file.
|
1941
1973
|
|
1942
1974
|
A named [Option Preset](#option-preset) can be modified directly using `ascli`, which will update the configuration file :
|
1943
1975
|
|
@@ -1951,9 +1983,9 @@ The command `update` allows the easy creation of [Option Preset](#option-preset)
|
|
1951
1983
|
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here --ts=@json:'{"precalculate_job_size":true}'
|
1952
1984
|
```
|
1953
1985
|
|
1954
|
-
- This creates
|
1986
|
+
- This creates an [Option Preset](#option-preset) `demo_server` with all provided options.
|
1955
1987
|
|
1956
|
-
The command `set` allows setting individual options in
|
1988
|
+
The command `set` allows setting individual options in an [Option Preset](#option-preset).
|
1957
1989
|
|
1958
1990
|
```bash
|
1959
1991
|
ascli config preset set demo_server password my_password_here
|
@@ -1978,7 +2010,7 @@ ascli config preset list
|
|
1978
2010
|
```
|
1979
2011
|
|
1980
2012
|
A good practice is to not manually edit the configuration file and use modification commands instead.
|
1981
|
-
If necessary, the configuration file can opened in a text editor with:
|
2013
|
+
If necessary, the configuration file can be opened in a text editor with:
|
1982
2014
|
|
1983
2015
|
```bash
|
1984
2016
|
ascli config open
|
@@ -1986,7 +2018,7 @@ ascli config open
|
|
1986
2018
|
|
1987
2019
|
> **Note:** This starts the editor specified by env var `EDITOR` if defined.
|
1988
2020
|
|
1989
|
-
|
2021
|
+
The former format for commands is still supported:
|
1990
2022
|
|
1991
2023
|
```bash
|
1992
2024
|
ascli config preset set|delete|show|initialize|update <name>
|
@@ -1994,14 +2026,14 @@ ascli config preset over
|
|
1994
2026
|
ascli config preset list
|
1995
2027
|
```
|
1996
2028
|
|
1997
|
-
It is possible to load
|
2029
|
+
It is possible to load an [Option Preset](#option-preset) from within another [Option Preset](#option-preset) using the `preset` option.
|
1998
2030
|
For example if `pcommon` is a preset with common options, and `pspecific` is a preset with specific options, then `pspecific` can load `pcommon` using:
|
1999
2031
|
|
2000
2032
|
```bash
|
2001
2033
|
ascli config preset set pspecific preset pcommon
|
2002
2034
|
```
|
2003
2035
|
|
2004
|
-
When `pspecific` is loaded, then cumulative option `preset` will be set and it will also load `pcommon`.
|
2036
|
+
When `pspecific` is loaded, then cumulative option `preset` will be set, and it will also load `pcommon`.
|
2005
2037
|
|
2006
2038
|
#### Special Option Preset: `config`
|
2007
2039
|
|
@@ -2009,7 +2041,7 @@ This preset name is reserved and contains a single key: `version`. This is the v
|
|
2009
2041
|
|
2010
2042
|
#### Special Option Preset: `default`
|
2011
2043
|
|
2012
|
-
This preset name is reserved and contains an array of key-value
|
2044
|
+
This preset name is reserved and contains an array of key-value, where the key is the name of a plugin, and the value is the name of another preset.
|
2013
2045
|
|
2014
2046
|
When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
|
2015
2047
|
|
@@ -2081,6 +2113,7 @@ ascp install --sdk-folder=sdk_test_dir
|
|
2081
2113
|
ascp install 1.1.3
|
2082
2114
|
ascp products list
|
2083
2115
|
ascp products use 'IBM Aspera Connect'
|
2116
|
+
ascp schema --format=jsonpp
|
2084
2117
|
ascp show
|
2085
2118
|
ascp spec
|
2086
2119
|
check_update
|
@@ -2186,7 +2219,7 @@ We can see here:
|
|
2186
2219
|
- The [Option Preset](#option-preset) `demo_server` defines some options: the URL and credentials
|
2187
2220
|
- The default [Option Preset](#option-preset) to load in any case is : `cli_default`
|
2188
2221
|
|
2189
|
-
Two [Option
|
2222
|
+
Two [Option Presets](#option-preset) are reserved:
|
2190
2223
|
|
2191
2224
|
- `config` contains a single value: `version` showing the version used to create the configuration file.
|
2192
2225
|
It is used to check compatibility.
|
@@ -2241,7 +2274,7 @@ ascli config preset set cli_default interactive no
|
|
2241
2274
|
ascli config preset set default config cli_default
|
2242
2275
|
```
|
2243
2276
|
|
2244
|
-
|
2277
|
+
An [Option Preset](#option-preset) value can be removed with `unset`:
|
2245
2278
|
|
2246
2279
|
```bash
|
2247
2280
|
ascli config preset unset cli_default interactive
|
@@ -2261,9 +2294,25 @@ ascli -N --preset=@json:'{"url":"_url_here_","password":"my_password_here","user
|
|
2261
2294
|
|
2262
2295
|
#### Wizard
|
2263
2296
|
|
2264
|
-
The wizard is a command that asks the user for information and creates
|
2297
|
+
The wizard is a command that asks the user for information and creates an [Option Preset](#option-preset) with the provided information for a given application.
|
2298
|
+
|
2299
|
+
It takes three optional arguments:
|
2300
|
+
|
2301
|
+
- the URL of the application, else it will ask for it
|
2302
|
+
- the plugin name: it limits detection to a given plugin, else it will try to detect known plugins from the URL
|
2303
|
+
- the preset name: it will create a new [Option Preset](#option-preset) with this name, else it will use specific information to generate a unique preset name.
|
2265
2304
|
|
2266
|
-
|
2305
|
+
Special options are also available to the wizard:
|
2306
|
+
|
2307
|
+
| Option | Value | Description |
|
2308
|
+
|-------------|----------|-------------|
|
2309
|
+
| `default` | [yes]/no | Set as default configuration for specified plugin. |
|
2310
|
+
| `override` | yes/[no] | Override existing default preset name for the plugin, if it exists. |
|
2311
|
+
| `key-path` | path | Path to private key for JWT. |
|
2312
|
+
| `test-mode` | yes/[no] | Skip private key check step. |
|
2313
|
+
|
2314
|
+
Other options can be provided to the wizard, such as `--username`, etc...
|
2315
|
+
They will be added to the [Option Preset](#option-preset) created by the wizard.
|
2267
2316
|
|
2268
2317
|
The simplest invocation is:
|
2269
2318
|
|
@@ -2275,7 +2324,7 @@ ascli config wizard
|
|
2275
2324
|
|
2276
2325
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
2277
2326
|
only username/password and URL are required (either on command line, or from configuration file).
|
2278
|
-
Those can usually
|
2327
|
+
Those can be usually provided on the command line:
|
2279
2328
|
|
2280
2329
|
```bash
|
2281
2330
|
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=my_password_here
|
@@ -2315,7 +2364,7 @@ ascli config preset set default shares shares06
|
|
2315
2364
|
ascli config preset overview
|
2316
2365
|
```
|
2317
2366
|
|
2318
|
-
- Execute a command on the
|
2367
|
+
- Execute a command on the **Shares** application using default options
|
2319
2368
|
|
2320
2369
|
```bash
|
2321
2370
|
ascli shares repo browse /
|
@@ -2323,8 +2372,8 @@ ascli shares repo browse /
|
|
2323
2372
|
|
2324
2373
|
### Secret Vault
|
2325
2374
|
|
2326
|
-
Secrets
|
2327
|
-
|
2375
|
+
Secrets, e.g. passwords, keys, are needed when connecting to applications.
|
2376
|
+
Those secrets are usually provided as command options, on command line, env vars, files etc.
|
2328
2377
|
|
2329
2378
|
For security reasons, those secrets shall not be exposed in clear, either:
|
2330
2379
|
|
@@ -2332,13 +2381,13 @@ For security reasons, those secrets shall not be exposed in clear, either:
|
|
2332
2381
|
- In logs
|
2333
2382
|
- In command output
|
2334
2383
|
|
2335
|
-
Instead, they shall be hidden or encrypted.
|
2384
|
+
Instead, they shall be hidden (logs) or encrypted (configuration).
|
2336
2385
|
|
2337
2386
|
Terminal output secret removal is controlled by option `show_secrets` (default: `no`).
|
2338
2387
|
Log secret removal is controlled by option `log_secrets` (default: `no`).
|
2339
2388
|
Mandatory command line options can be requested interactively (e.g. password) using option `interactive`.
|
2340
2389
|
Or it is possible to use extended value `@secret:[name]` to ask for a secret interactively.
|
2341
|
-
It is also possible to enter an option as an environment variable, e.g. `ASCLI_PASSWORD` for option `password` and read the env var
|
2390
|
+
It is also possible to enter an option as an environment variable, e.g. `ASCLI_PASSWORD` for option `password` and read the env var like this:
|
2342
2391
|
|
2343
2392
|
```bash
|
2344
2393
|
read -s ASCLI_PASSWORD
|
@@ -2358,18 +2407,31 @@ The vault is used with options `vault` and `vault_password`.
|
|
2358
2407
|
`vault_password` specifies the password for the vault.
|
2359
2408
|
|
2360
2409
|
Although it can be specified on command line, for security reason you should avoid exposing the secret.
|
2361
|
-
For example it can be securely specified on command line like this:
|
2410
|
+
For example, it can be securely specified on command line like this:
|
2362
2411
|
|
2363
2412
|
```bash
|
2364
2413
|
read -s ASCLI_VAULT_PASSWORD
|
2365
2414
|
export ASCLI_VAULT_PASSWORD
|
2366
2415
|
```
|
2367
2416
|
|
2368
|
-
#### Vault:
|
2417
|
+
#### Vault: IBM HashiCorp Vault
|
2418
|
+
|
2419
|
+
<https://developer.hashicorp.com/vault>
|
2420
|
+
|
2421
|
+
Quick start macOS:
|
2422
|
+
|
2423
|
+
```bash
|
2424
|
+
gem install vault
|
2425
|
+
brew tap hashicorp/tap
|
2426
|
+
brew install hashicorp/tap/vault
|
2427
|
+
vault server -dev -dev-root-token-id=dev-only-token
|
2428
|
+
```
|
2429
|
+
|
2430
|
+
#### Vault: System keychain
|
2369
2431
|
|
2370
2432
|
> **Note:** **macOS only**
|
2371
2433
|
|
2372
|
-
It is possible to manage secrets in macOS
|
2434
|
+
It is possible to manage secrets in macOS keychain (only read supported currently).
|
2373
2435
|
|
2374
2436
|
```bash
|
2375
2437
|
--vault=@json:'{"type":"system","name":"ascli"}'
|
@@ -2408,7 +2470,7 @@ The lookup is done by comparing the service URL and username (or access key).
|
|
2408
2470
|
|
2409
2471
|
#### Securing passwords and secrets
|
2410
2472
|
|
2411
|
-
A
|
2473
|
+
A password can be saved in clear in an [Option Preset](#option-preset) together with other account information (URL, username, etc...).
|
2412
2474
|
Example:
|
2413
2475
|
|
2414
2476
|
```bash
|
@@ -2512,15 +2574,35 @@ For example: <https://cryptotools.net/rsagen>
|
|
2512
2574
|
|
2513
2575
|
> **Note:** Be careful that private keys are sensitive information, and shall be kept secret (like a password), so using online tools is risky.
|
2514
2576
|
|
2577
|
+
### Web service
|
2578
|
+
|
2579
|
+
Some plugins start a local web server.
|
2580
|
+
This server can server HTTP or HTTPS (with certificate):
|
2581
|
+
|
2582
|
+
The following parameters are supported:
|
2583
|
+
|
2584
|
+
| Parameter | Type | Default | Description |
|
2585
|
+
|-------------------|----------|---------|-----------------------------------------------------|
|
2586
|
+
| `url` | `String` | `http://localhost:8080` | Base URL on which requests are listened, a path can be provided. | <!-- markdownlint-disable-line -->
|
2587
|
+
| `cert` | `String` | - | (HTTPS) Path to certificate file (with ext. `.pfx` or `.p12` for PKCS12) |
|
2588
|
+
| `key` | `String` | - | (HTTPS) Path to private key file (PEM), or passphrase for PKCS12 |
|
2589
|
+
| `chain` | `String` | - | (HTTPS) Path to certificate chain (PEM only) |
|
2590
|
+
|
2591
|
+
Parameter `url` (base URL) defines:
|
2592
|
+
|
2593
|
+
- If `http` or `https` is used
|
2594
|
+
- The local port number (default 443 for HTTPS, 80 for HTTP)
|
2595
|
+
- The **base path**, i.e. the path under which requests are received, if a reverse proxy is used this can be used to route.
|
2596
|
+
|
2515
2597
|
### Image and video thumbnails
|
2516
2598
|
|
2517
2599
|
`ascli` can display thumbnails for images and videos in the terminal.
|
2518
2600
|
This is available:
|
2519
2601
|
|
2520
|
-
-
|
2521
|
-
-
|
2602
|
+
- In the `thumbnail` command of `node` when using **gen4/access key** API.
|
2603
|
+
- When using the `show` command of `preview` plugin.
|
2522
2604
|
- `coffee` and `image` commands of `config` plugin.
|
2523
|
-
-
|
2605
|
+
- Any displayed value which is a URL to image can be displayed with option `format` set to `image`
|
2524
2606
|
|
2525
2607
|
The following options can be specified in the option `image`:
|
2526
2608
|
|
@@ -2550,8 +2632,8 @@ Some actions may require the use of a graphical tool:
|
|
2550
2632
|
- A browser for Aspera on Cloud authentication (web auth method)
|
2551
2633
|
- A text editor for configuration file edition
|
2552
2634
|
|
2553
|
-
By default `ascli` assumes that a graphical environment is available on Windows, and on other systems, rely on the presence of the `DISPLAY` environment variable.
|
2554
|
-
It is also possible to force the graphical mode with option
|
2635
|
+
By default, `ascli` assumes that a graphical environment is available on Windows, and on other systems, rely on the presence of the `DISPLAY` environment variable.
|
2636
|
+
It is also possible to force the graphical mode with option `--ui` :
|
2555
2637
|
|
2556
2638
|
- `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
2557
2639
|
- `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
@@ -2559,10 +2641,10 @@ It is also possible to force the graphical mode with option --ui :
|
|
2559
2641
|
### Logging, Debugging
|
2560
2642
|
|
2561
2643
|
The gem is equipped with traces, mainly for debugging and learning APIs.
|
2562
|
-
By default logging level is `warn` and the output channel is `stderr`.
|
2644
|
+
By default, logging level is `warn` and the output channel is `stderr`.
|
2563
2645
|
To increase debug level, use option `log_level` (e.g. using command line `--log-level=xx`, env var `ASCLI_LOG_LEVEL`, or an [Option Preset](#option-preset)).
|
2564
2646
|
|
2565
|
-
By default passwords and secrets are redacted from logs.
|
2647
|
+
By default, passwords and secrets are redacted from logs.
|
2566
2648
|
Set option `log_secrets` to `yes` to include secrets in logs.
|
2567
2649
|
|
2568
2650
|
Option `logger`: `stdout`, `stderr`, `syslog`.
|
@@ -2609,16 +2691,19 @@ To disable the warning, use option `silent_insecure` set to `no`.
|
|
2609
2691
|
|
2610
2692
|
HTTP connection parameters (not `ascp` WSS) can be adjusted using option `http_options`:
|
2611
2693
|
|
2612
|
-
| Parameter | Default
|
2613
|
-
|
2614
|
-
| `read_timeout` | 60 |
|
2615
|
-
| `write_timeout` | 60 |
|
2616
|
-
| `open_timeout` | 60 |
|
2617
|
-
| `keep_alive_timeout` | 2 |
|
2618
|
-
| `user_agent` | `ascli`
|
2619
|
-
| `download_partial_suffix` |
|
2620
|
-
| `retry_on_error` |
|
2621
|
-
| `
|
2694
|
+
| Parameter | Type | Default | Handler |
|
2695
|
+
|---------------------------|------|-----------------|-----------|
|
2696
|
+
| `read_timeout` | int | `60` | Ruby |
|
2697
|
+
| `write_timeout` | int | `60` | Ruby |
|
2698
|
+
| `open_timeout` | int | `60` | Ruby |
|
2699
|
+
| `keep_alive_timeout` | int | `2` | Ruby |
|
2700
|
+
| `user_agent` | int | `ascli` | `ascli` |
|
2701
|
+
| `download_partial_suffix` | int | `.http_partial` | `ascli` |
|
2702
|
+
| `retry_on_error` | bool | `false` | `ascli` |
|
2703
|
+
| `retry_on_timeout` | bool | `true` | `ascli` |
|
2704
|
+
| `retry_on_unavailable` | bool | `true` | `ascli` |
|
2705
|
+
| `retry_max` | int | `1` | `ascli` |
|
2706
|
+
| `retry_sleep` | int | `4` | `ascli` |
|
2622
2707
|
|
2623
2708
|
Time values are in set **seconds** and can be of type either `Integer` or `Float`.
|
2624
2709
|
Default values are the ones of Ruby:
|
@@ -2647,7 +2732,7 @@ Refer to the following sections.
|
|
2647
2732
|
REST API calls and transfers based on HTTP Gateway both use Ruby's `Net::HTTP` class.
|
2648
2733
|
Refer to [Ruby find proxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
2649
2734
|
|
2650
|
-
When Ruby HTTP is used, there are two possibilities to define an HTTP proxy to be used
|
2735
|
+
When Ruby HTTP is used, there are two possibilities to define an HTTP proxy to be used.
|
2651
2736
|
|
2652
2737
|
The `http_proxy` environment variable (**lower case**) can be set to the **URL** of the proxy (with optional credentials).
|
2653
2738
|
Syntax is: `(http|https)://[user:password@]host:port`.
|
@@ -2667,7 +2752,7 @@ Option `http_proxy` does the same (set env var) but on command line:
|
|
2667
2752
|
ascli --http-proxy=http://username:password@host:port ...
|
2668
2753
|
```
|
2669
2754
|
|
2670
|
-
Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config)
|
2755
|
+
Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) JavaScript value.
|
2671
2756
|
|
2672
2757
|
Note that proxy credentials are not supported in PAC files.
|
2673
2758
|
|
@@ -2739,7 +2824,7 @@ Or, alternatively, (prefer transfer spec like above, generally):
|
|
2739
2824
|
|
2740
2825
|
`ascli` uses one of the transfer agents to execute transfers.
|
2741
2826
|
|
2742
|
-
By default it uses the `direct` agent, which is basically a local `ascp`.
|
2827
|
+
By default, it uses the `direct` agent, which is basically a local `ascp`.
|
2743
2828
|
Nevertheless, `ascli` does not come with `ascp` installed.
|
2744
2829
|
This is the reason why it is advised to install the Aspera Transfer SDK during installation (`ascli config ascp install`).
|
2745
2830
|
|
@@ -2750,11 +2835,11 @@ The way to specify the location of `ascp` is to use either options:
|
|
2750
2835
|
- `ascp_path`
|
2751
2836
|
- `use_product`
|
2752
2837
|
|
2753
|
-
The `config` plugin allows
|
2754
|
-
It provides the following commands for `ascp`
|
2838
|
+
The `config` plugin allows finding and specifying the location of `ascp`.
|
2839
|
+
It provides the following commands for `ascp` sub-command:
|
2755
2840
|
|
2756
2841
|
- `show` : shows the path of `ascp` used
|
2757
|
-
- `use` : specify the ascp path to use
|
2842
|
+
- `use` : specify the `ascp` path to use
|
2758
2843
|
- `products` : list Aspera transfer products available locally
|
2759
2844
|
- `connect` : list and download connect client versions available on internet
|
2760
2845
|
|
@@ -2891,7 +2976,7 @@ Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
|
2891
2976
|
|
2892
2977
|
### Transfer Clients: Agents
|
2893
2978
|
|
2894
|
-
Some
|
2979
|
+
Some actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
2895
2980
|
Transfers will be executed by a transfer client, here called **Transfer Agent**.
|
2896
2981
|
|
2897
2982
|
The following agents are supported and selected with option `transfer`:
|
@@ -2908,7 +2993,7 @@ The following agents are supported and selected with option `transfer`:
|
|
2908
2993
|
> **Note:** All transfer operations are seen from the point of view of the agent.
|
2909
2994
|
For example, an agent executing an **upload**, or **package send** operation will effectively push files to the related server from the system where the agent runs.
|
2910
2995
|
|
2911
|
-
All
|
2996
|
+
All above agents (including `direct`) receive transfer parameters as a [**transfer-spec**](#transfer-specification).
|
2912
2997
|
Parameters in transfer-spec can be modified with option `ts`.
|
2913
2998
|
|
2914
2999
|
**Specific** options for agents are provided with option `transfer_info`.
|
@@ -2930,7 +3015,7 @@ The `transfer_info` option accepts the following optional parameters to control
|
|
2930
3015
|
| `quiet` | `Bool` | If `true`, then `ascp` progress bar is not shown.<br/>Default: `false` |
|
2931
3016
|
| `trusted_certs` | `Array` | List of repositories for trusted certificates. |
|
2932
3017
|
| `client_ssh_key` | `String` | SSH Keys to use for token-based transfers. One of: `dsa_rsa`, `rsa`, `per_client`. Default: `rsa` |
|
2933
|
-
| `ascp_args` | `Array` | Array of strings with native `ascp` arguments.<br/>Default: `[]` |
|
3018
|
+
| `ascp_args` | `Array` | `Array` of strings with native `ascp` arguments.<br/>Default: `[]` |
|
2934
3019
|
| `spawn_timeout_sec` | `Float` | Multi session<br/>Verification time that `ascp` is running<br/>Default: `3` |
|
2935
3020
|
| `spawn_delay_sec` | `Float` | Multi session<br/>Delay between startup of sessions<br/>Default: `2` |
|
2936
3021
|
| `multi_incr_udp` | `Bool` | Multi Session<br/>Increment UDP port on multi-session<br/>If `true`, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: `true` on Windows, else `false` |
|
@@ -2939,14 +3024,20 @@ The `transfer_info` option accepts the following optional parameters to control
|
|
2939
3024
|
| `resume.sleep_initial` | `Integer` | First Sleep before retry<br/>Default: `2` |
|
2940
3025
|
| `resume.sleep_factor` | `Integer` | Multiplier of sleep period between attempts<br/>Default: `2` |
|
2941
3026
|
| `resume.sleep_max` | `Integer` | Default: `60` |
|
3027
|
+
| `monitor` | `Bool` | Use management port.<br/>Default: `true` |
|
2942
3028
|
|
2943
3029
|
In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
|
2944
3030
|
Sleep between iterations is given by the following formula where `iter_index` is the current iteration index, starting at 0:
|
2945
3031
|
|
2946
3032
|
```bash
|
2947
|
-
max( sleep_max
|
3033
|
+
max( sleep_max, sleep_initial * sleep_factor ^ iter_index )
|
2948
3034
|
```
|
2949
3035
|
|
3036
|
+
To display the native progress bar of `ascp`, use `--progress-bar=no --transfer-info=@json:'{"quiet":false}'`.
|
3037
|
+
|
3038
|
+
To skip usage of management port (which disables custom progress bar), set option `monitor` to `false`.
|
3039
|
+
In that, use the native progress bar: `--transfer-info=@json:'{"monitor":false,"quiet":false}'`
|
3040
|
+
|
2950
3041
|
By default, Ruby's root CA store is used to validate any HTTPS endpoint used by `ascp` (e.g. WSS).
|
2951
3042
|
In order to use a custom certificate store, use the `trusted_certs` option of direct agent's option `transfer_info`.
|
2952
3043
|
To use `ascp`'s default, use option: `--transfer-info=@json:'{"trusted_certs":null}'`.
|
@@ -3070,12 +3161,12 @@ This is especially useful for direct node-to-node transfers.
|
|
3070
3161
|
|
3071
3162
|
Parameters provided in option `transfer_info` are:
|
3072
3163
|
|
3073
|
-
|
|
3074
|
-
|
3075
|
-
| `url` | `String` | URL of the
|
3076
|
-
| `username` | `String` | Node
|
3164
|
+
| Parameter | Type | Description |
|
3165
|
+
|------------|----------|----------------------------------------------------|
|
3166
|
+
| `url` | `String` | URL of the Node API<br/>Mandatory |
|
3167
|
+
| `username` | `String` | Node API user or access key<br/>Mandatory |
|
3077
3168
|
| `password` | `String` | Password, secret or bearer token<br/>Mandatory |
|
3078
|
-
| `root_id` | `String` | Root file
|
3169
|
+
| `root_id` | `String` | Root file ID<br/>Mandatory only for bearer token |
|
3079
3170
|
|
3080
3171
|
Like any other option, `transfer_info` can get its value from a pre-configured [Option Preset](#option-preset) :
|
3081
3172
|
|
@@ -3091,8 +3182,8 @@ or be specified using the extended value syntax :
|
|
3091
3182
|
|
3092
3183
|
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
3093
3184
|
|
3094
|
-
If the `password` value begins with `Bearer` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the file
|
3095
|
-
It can be either the access key's root file
|
3185
|
+
If the `password` value begins with `Bearer` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the file ID of the top folder to use on the node using this access key.
|
3186
|
+
It can be either the access key's root file ID, or any authorized file ID underneath it.
|
3096
3187
|
|
3097
3188
|
#### Agent: HTTP Gateway
|
3098
3189
|
|
@@ -3147,11 +3238,11 @@ On Windows the compilation may fail for various reasons (3.1.1):
|
|
3147
3238
|
|
3148
3239
|
- `cannot find -lx64-ucrt-ruby310`
|
3149
3240
|
|
3150
|
-
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
3241
|
+
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the `dll` extension)
|
3151
3242
|
|
3152
3243
|
- `conflicting types for 'gettimeofday'`
|
3153
3244
|
|
3154
|
-
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)
|
3245
|
+
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)`, i.e. change `struct timezone` to `void`
|
3155
3246
|
|
3156
3247
|
<!-- spellchecker: enable -->
|
3157
3248
|
|
@@ -3161,13 +3252,13 @@ Some commands lead to file transfer (upload/download).
|
|
3161
3252
|
All parameters necessary for this transfer are described in a [**transfer-spec**](#transfer-specification) (Transfer Specification), such as:
|
3162
3253
|
|
3163
3254
|
- Server address
|
3164
|
-
- Transfer
|
3255
|
+
- Transfer username
|
3165
3256
|
- Credentials
|
3166
3257
|
- File list
|
3167
3258
|
- Etc...
|
3168
3259
|
|
3169
3260
|
`ascli` builds the [**transfer-spec**](#transfer-specification) internally as a `Hash`.
|
3170
|
-
It is not necessary to provide additional parameters on the command line for
|
3261
|
+
It is not necessary to provide additional parameters on the command line for a transfer.
|
3171
3262
|
|
3172
3263
|
It is possible to modify or add any of the supported [**transfer-spec**](#transfer-specification) parameter using the `ts` option.
|
3173
3264
|
The `ts` option accepts a [`Hash` Extended Value](#extended-value-syntax) containing one or several [**transfer-spec**](#transfer-specification) parameters.
|
@@ -3188,7 +3279,7 @@ The use of a [**transfer-spec**](#transfer-specification) instead of `ascp` comm
|
|
3188
3279
|
### Transfer Parameters
|
3189
3280
|
|
3190
3281
|
All standard [**transfer-spec**](#transfer-specification) parameters can be specified.
|
3191
|
-
[**transfer-spec**](#transfer-specification) can also be saved/overridden in the configuration file.
|
3282
|
+
A [**transfer-spec**](#transfer-specification) can also be saved/overridden in the configuration file.
|
3192
3283
|
|
3193
3284
|
References:
|
3194
3285
|
|
@@ -3203,102 +3294,126 @@ ascli config ascp spec
|
|
3203
3294
|
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
3204
3295
|
```
|
3205
3296
|
|
3206
|
-
|
3297
|
+
A JSON Schema can be generated with command:
|
3298
|
+
|
3299
|
+
```bash
|
3300
|
+
ascli config ascp schema --format=jsonpp
|
3301
|
+
```
|
3207
3302
|
|
3208
|
-
|
3209
|
-
|
3210
|
-
|
3211
|
-
|
3212
|
-
|
3303
|
+
An optional parameter can be specified to display the schema for a specific transfer agent:
|
3304
|
+
|
3305
|
+
```bash
|
3306
|
+
ascli config ascp schema transferd --format=jsonpp
|
3307
|
+
```
|
3213
3308
|
|
3214
3309
|
`ascp` argument or environment variable is provided in description.
|
3215
3310
|
|
3216
|
-
|
|
3311
|
+
| ID | Name |
|
3312
|
+
| -- | ---- |
|
3313
|
+
| A | Direct |
|
3314
|
+
| C | Connect |
|
3315
|
+
| D | Desktop |
|
3316
|
+
| H | Httpgw |
|
3317
|
+
| N | Node |
|
3318
|
+
| T | Transferd |
|
3319
|
+
|
3320
|
+
| Field | Type | A | C | D | H | N | T | Description |
|
3217
3321
|
| ----- | ---- | - | - | - | - | - | - | ----------- |
|
3218
|
-
| apply_local_docroot |
|
3219
|
-
| authentication | string | | | | | |
|
3220
|
-
| cipher | string | Y | Y | Y | Y | Y | Y | In transit encryption
|
3221
|
-
| cipher_allowed | string |
|
3222
|
-
| compression |
|
3223
|
-
| content_protection | string | Y | Y | Y | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt
|
3322
|
+
| apply_local_docroot | boolean | Y | | | | | Y | Apply local docroot to source paths.<br/>(--apply-local-docroot) |
|
3323
|
+
| authentication | string | | Y | | | | | value=token for SSH bypass keys, else password asked if not provided. |
|
3324
|
+
| cipher | string | Y | Y | Y | Y | Y | Y | In transit encryption algorithms.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum}) |
|
3325
|
+
| cipher_allowed | string | | Y | | | | | returned by node API. Valid literals include "aes-128" and "none". |
|
3326
|
+
| compression | integer | | | | | | | ascp4 only, 0 / 1? |
|
3327
|
+
| content_protection | string | Y | Y | Y | Y | Y | Y | Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt={enum}) |
|
3224
3328
|
| content_protection_password | string | Y | Y | Y | Y | Y | Y | Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS) |
|
3225
3329
|
| cookie | string | Y | Y | Y | Y | Y | Y | Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE) |
|
3226
|
-
| create_dir |
|
3227
|
-
| delete_before_transfer |
|
3228
|
-
| delete_source |
|
3330
|
+
| create_dir | boolean | Y | Y | Y | Y | Y | Y | Specifies whether to create new directories.<br/>(-d) |
|
3331
|
+
| delete_before_transfer | boolean | Y | Y | Y | Y | Y | Y | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects on the source are not deleted.<br/>(--delete-before-transfer) |
|
3332
|
+
| delete_source | boolean | Y | | | | Y | Y | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
3229
3333
|
| destination_root | string | Y | Y | Y | Y | Y | Y | Destination root directory.<br/>(<special>) |
|
3230
|
-
| destination_root_id | string | | | | | |
|
3231
|
-
| dgram_size |
|
3232
|
-
| direction | string | Y | Y | Y | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode
|
3233
|
-
| exclude_newer_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are newer than the specified number of seconds added to the source computer's epoch
|
3234
|
-
| exclude_older_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are older than the specified number of seconds added to the source computer's epoch
|
3235
|
-
|
|
3236
|
-
|
|
3237
|
-
|
|
3238
|
-
|
|
3239
|
-
|
|
3240
|
-
|
|
3241
|
-
|
|
3242
|
-
|
|
3243
|
-
|
|
3244
|
-
|
|
3245
|
-
|
|
3246
|
-
|
|
3247
|
-
|
|
3248
|
-
|
|
3249
|
-
|
|
3250
|
-
|
|
3251
|
-
|
|
3252
|
-
|
|
3253
|
-
|
|
3254
|
-
|
|
3334
|
+
| destination_root_id | string | | | | | | Y | The file ID of the destination root directory.<br/>Required when using Bearer token auth for the destination node. |
|
3335
|
+
| dgram_size | integer | Y | Y | Y | Y | Y | Y | UDP datagram size in bytes<br/>(-Z {integer}) |
|
3336
|
+
| direction | string | Y | Y | Y | Y | Y | Y | Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode=(conversion){enum}) |
|
3337
|
+
| exclude_newer_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are newer than the specified number of seconds added to the source computer's epoch.<br/>e.g. "-86400" for newer than a day back.<br/>(--exclude-newer-than={string}) |
|
3338
|
+
| exclude_older_than | string | Y | Y | Y | Y | Y | Y | Exclude files, but not directories, from the transfer if they are older than the specified number of seconds added to the source computer's epoch.<br/>e.g. "-86400" for older than a day back.<br/>(--exclude-older-than={string}) |
|
3339
|
+
| fail_bad_filepass | boolean | Y | | | | | Y | Fail on bad file decryption passphrase.<br/>(--fail-bad-filepass) |
|
3340
|
+
| fasp_port | integer | Y | Y | Y | Y | Y | Y | Specifies fasp (UDP) port.<br/>(-O {integer}) |
|
3341
|
+
| fasp_proxy | object | | | | | | Y | Proxy for communications between the remote server and the (local) client. |
|
3342
|
+
| fasp_url | string | | | | | | | Only used in Faspex. |
|
3343
|
+
| file_checksum | string | Y | | | | Y | | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none<br/>(--file-checksum={enum}) |
|
3344
|
+
| http_fallback | boolean<br/>string | Y | Y | Y | Y | Y | Y | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){boolean\|string}) |
|
3345
|
+
| http_fallback_port | integer | Y | Y | Y | Y | Y | Y | Specifies http port when no cipher is used<br/>(-t {integer}) |
|
3346
|
+
| https_fallback_port | integer | Y | Y | Y | Y | Y | Y | Specifies https port when cipher is used<br/>(-t {integer}) |
|
3347
|
+
| icos | object | | | | | | Y | Configuration parameters for IBM Cloud Object Storage (ICOS). |
|
3348
|
+
| keepalive | boolean | Y | | | | | Y | The session is running in persistent session mode.<br/>(--keepalive) |
|
3349
|
+
| lock_min_rate | boolean | | Y | | | | | TODO: remove ? |
|
3350
|
+
| lock_min_rate_kbps | boolean | | Y | | | | Y | If true, lock the minimum transfer rate to the value set for min_rate_kbps.<br/>If false, users can adjust the transfer rate up to the value set for target_rate_cap_kbps. |
|
3351
|
+
| lock_rate_policy | boolean | | Y | | | | Y | If true, lock the rate policy to the default value. |
|
3352
|
+
| lock_target_rate | boolean | | Y | | | | | TODO: remove ? |
|
3353
|
+
| lock_target_rate_kbps | boolean | | Y | | | | Y | If true, lock the target transfer rate to the default value set for target_rate_kbps.<br/>If false, users can adjust the transfer rate up to the value set for target_rate_cap_kbps. |
|
3354
|
+
| min_rate_cap_kbps | integer | | Y | | | | Y | The highest minimum rate that an incoming transfer can request, in kilobits per second.<br/>Client minimum rate requests that exceed the minimum rate cap are ignored.<br/>The default value of unlimited applies no cap to the minimum rate. (Default: 0) |
|
3355
|
+
| min_rate_kbps | integer | Y | Y | Y | Y | Y | Y | Set the minimum transfer rate in kilobits per second.<br/>(-m {integer}) |
|
3356
|
+
| move_after_transfer | string | Y | | | | Y | Y | The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer={string}) |
|
3357
|
+
| multi_session | integer | Y | Y | Y | Y | Y | Y | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/>(<special>) |
|
3358
|
+
| multi_session_threshold | integer | Y | | | | Y | Y | Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold={integer}) |
|
3359
|
+
| obfuscate_file_names | boolean | | | | Y | | | HTTP Gateway obfuscates file names when set to true. |
|
3360
|
+
| overwrite | string | Y | Y | Y | Y | Y | Y | Overwrite files at the destination with source files of the same name based on the policy:<br/>- always – Always overwrite the file.<br/>- never – Never overwrite the file. If the destination contains partial files that are older or the same as the source files and resume is enabled, the partial files resume transfer. Partial files with checksums or sizes that differ from the source files are not overwritten.<br/>- diff – Overwrite the file if it is different from the source, depending on the compare method (default is size). If the destination is object storage, diff has the same effect as always. If resume is not enabled, partial files are overwritten if they are different from the source, otherwise they are skipped. If resume is enabled, only partial files with different sizes or checksums from the source are overwritten; otherwise, files resume. <br/>- diff+older – Overwrite the file if it is older and different from the source, depending on the compare method (default is size). If resume is not enabled, partial files are overwritten if they are older and different from the source, otherwise they are skipped. If resume is enabled, only partial files that are different and older than the source are overwritten, otherwise they are resumed. <br/>- older – Overwrite the file if its timestamp is older than the source timestamp. <br/>If you set an overwrite policy of diff or diff+older, difference is determined by the value set for resume_policy:<br/>"none" - the source and destination files are always considered different and the destination file is always overwritten<br/>"attributes" - the source and destination files are compared based on file attributes <br/>"sparse_checksum" - the source and destination files are compared based on sparse checksums, (currently file size)<br/>"full_checksum" - the source and destination files are compared based on full checksums <br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite={enum}) |
|
3361
|
+
| password | string | | | | | Y | | Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only. |
|
3255
3362
|
| paths | array | Y | Y | Y | Y | Y | Y | Array of path to the source (required) and a path to the destination (optional).<br/>(<special>) |
|
3256
|
-
| precalculate_job_size |
|
3257
|
-
| preserve_access_time |
|
3258
|
-
| preserve_acls | string | Y | |
|
3259
|
-
| preserve_creation_time |
|
3260
|
-
| preserve_extended_attrs | string |
|
3261
|
-
| preserve_file_owner_gid |
|
3262
|
-
| preserve_file_owner_uid |
|
3263
|
-
| preserve_modification_time |
|
3264
|
-
| preserve_remote_acls | string | Y | |
|
3265
|
-
|
|
3266
|
-
|
|
3267
|
-
|
|
3268
|
-
|
|
3269
|
-
|
|
3270
|
-
|
|
3271
|
-
|
|
3272
|
-
|
|
3363
|
+
| precalculate_job_size | boolean | Y | Y | Y | Y | Y | Y | Specifies whether to precalculate the job size.<br/>(--precalculate-job-size) |
|
3364
|
+
| preserve_access_time | boolean | Y | Y | Y | Y | Y | Y | Preserve the source-file access timestamps at the destination.<br/>Because source access times are updated by the transfer operation, the timestamp that is preserved is the one just before to the transfer.<br/>(--preserve-access-time) |
|
3365
|
+
| preserve_acls | string | Y | | | | | Y | Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls={enum}) |
|
3366
|
+
| preserve_creation_time | boolean | Y | Y | Y | Y | Y | Y | (Windows only) Preserve source-file creation timestamps at the destination.<br/>Only Windows systems retain information about creation time.<br/>If the destination is not a Windows computer, this option is ignored.<br/>(--preserve-creation-time) |
|
3367
|
+
| preserve_extended_attrs | string | Y | | | | | Y | Preserve the extended attributes.<br/>Allowed values: none, native, metafile<br/>(--preserve-xattrs={enum}) |
|
3368
|
+
| preserve_file_owner_gid | boolean | Y | | | | | Y | Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid) |
|
3369
|
+
| preserve_file_owner_uid | boolean | Y | | | | | Y | Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid) |
|
3370
|
+
| preserve_modification_time | boolean | Y | Y | Y | Y | Y | Y | Set the modification time, the last time a file or directory was modified (written), of a transferred file to the modification of the source file or directory.<br/>Preserve source-file modification timestamps at the destination.<br/>(--preserve-modification-time) |
|
3371
|
+
| preserve_remote_acls | string | Y | | | | | Y | Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls={enum}) |
|
3372
|
+
| preserve_remote_extended_attrs | string | Y | | | | | Y | Preserve remote extended attributes.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-xattrs={enum}) |
|
3373
|
+
| preserve_source_access_time | boolean | Y | | | | | Y | Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time) |
|
3374
|
+
| preserve_times | boolean | Y | | | | Y | Y | Preserve file timestamps.<br/>(-p {boolean}) |
|
3375
|
+
| proxy | string | Y | | | | | | Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy={string}) |
|
3376
|
+
| rate_policy | string | Y | Y | Y | Y | Y | Y | The transfer rate policy to use when sharing bandwidth. Allowable values:<br/>- high : When sharing bandwidth, transfer at twice the rate of a transfer using a fair policy.<br/>- fair : (Default) Share bandwidth equally with other traffic.<br/>- low : Use only unused bandwidth.<br/>- fixed : Transfer at the target rate, regardless of the actual network capacity. Do not share bandwidth. Aspera recommends that you do not use this setting except under special circumstances, otherwise the destination storage can be damaged. <br/>Allowed values: low, fair, high, fixed<br/>(--policy={enum}) |
|
3377
|
+
| rate_policy_allowed | string | | Y | | | | | Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed |
|
3378
|
+
| read_threads | integer | | | | | | | ascp4 only |
|
3379
|
+
| remote_access_key | string | | | | | | Y | The access key ID of the access key that was used to construct the bearer token that is used to authenticate to the remote node. |
|
3380
|
+
| remote_host | string | Y | Y | Y | Y | Y | Y | IP or fully qualified domain name of the remote server<br/>(--host={string}) |
|
3273
3381
|
| remote_password | string | Y | Y | Y | Y | Y | Y | SSH session password<br/>(env:ASPERA_SCP_PASS) |
|
3274
|
-
| remote_user | string | Y | Y | Y | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user
|
3275
|
-
| remove_after_transfer |
|
3276
|
-
| remove_empty_directories |
|
3277
|
-
|
|
3278
|
-
|
|
3279
|
-
|
|
3280
|
-
|
|
3281
|
-
|
|
3282
|
-
|
|
3283
|
-
|
|
3284
|
-
|
|
3285
|
-
|
|
3286
|
-
|
|
3287
|
-
|
|
3288
|
-
|
|
3289
|
-
|
|
3290
|
-
|
|
3291
|
-
|
|
3292
|
-
|
|
3293
|
-
|
|
3294
|
-
|
|
3382
|
+
| remote_user | string | Y | Y | Y | Y | Y | Y | Remote user. Default value is "xfer" on node or connect.<br/>(--user={string}) |
|
3383
|
+
| remove_after_transfer | boolean | Y | | | | Y | Y | Remove SRC files after transfer success<br/>(--remove-after-transfer) |
|
3384
|
+
| remove_empty_directories | boolean | Y | | | | Y | Y | Specifies whether to remove empty directories.<br/>(--remove-empty-directories) |
|
3385
|
+
| remove_empty_source_dir | boolean | | | | | | Y | Remove empty source subdirectories and remove the source directory itself, if empty. |
|
3386
|
+
| remove_empty_source_directory | boolean | Y | | | | | | Remove empty source subdirectories and remove the source directory itself, if empty.<br/>(--remove-empty-source-directory) |
|
3387
|
+
| remove_skipped | boolean | Y | Y | | | Y | | Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped) |
|
3388
|
+
| resume_policy | string | Y | Y | Y | Y | Y | Y | If a transfer is interrupted or fails to finish, this policy directs the transfer to resume without retransferring the files. Allowable values:<br/>- none : Always re-transfer the entire file<br/>- attrs : Compare file attributes and resume if they match, and re-transfer if they do not<br/>- sparse_csum : Compare file attributes and the sparse file checksums; resume if they match, and re-transfer if they do not<br/>- full_csum : Compare file attributes and the full file checksums; resume if they match, and re-transfer if they do not. <br/>Note: transferd uses values: attributes, sparse_checksum, full_checksum.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum}) |
|
3389
|
+
| retry_duration | integer<br/>string | | | | | | Y | Specifies how long to wait before retrying transfer. (e.g. "5min") |
|
3390
|
+
| save_before_overwrite | boolean | Y | | | | Y | Y | If a transfer would result in an existing file <filename>.<ext> being overwritten, move that file to <filename>.yyyy.mm.dd.hh.mm.ss.index.<ext> (where index is set to 1 at the beginning of each new second and incremented for each file saved in this manner during the same second) in the same directory before writing the new file.<br/>File attributes are maintained in the renamed file.<br/>(--save-before-overwrite) |
|
3391
|
+
| skip_duplicate_check | boolean | Y | | | | | Y | Don't check for duplicate files at the destination.<br/>(--skip-dir-traversal-dupes) |
|
3392
|
+
| skip_special_files | boolean | Y | | | | | Y | All assets other than files, directories and symbolic links are considered special. A transfer will fail if the user attempts to transfer special assets. If true, ascp skips special assets and proceeds with the transfer of all other assets. <br/>(--skip-special-files) |
|
3393
|
+
| source_root | string | Y | Y | Y | Y | Y | Y | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64=(conversion){string}) |
|
3394
|
+
| source_root_id | string | | | | | Y | Y | The file ID of the source root directory. Required when using Bearer token auth for the source node. |
|
3395
|
+
| src_base | string | Y | | | | Y | Y | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64=(conversion){string}) |
|
3396
|
+
| src_base64 | string | Y | | | | | Y | The folder name below which the directory structure is preserved (base64 encoded).<br/>(--src-base64={string}) |
|
3397
|
+
| ssh_args | array | Y | | | | | Y | Array of arguments to pass to SSH. Use with caution.<br/>(-i {array}) |
|
3398
|
+
| ssh_port | integer | Y | Y | Y | Y | Y | Y | Specifies SSH (TCP) port.<br/>(-P {integer}) |
|
3399
|
+
| ssh_private_key | string | Y | | | | | Y | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----\nMII...<br/>Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY) |
|
3400
|
+
| ssh_private_key_passphrase | string | Y | | | | | Y | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS) |
|
3401
|
+
| ssh_private_key_path | string | Y | | | | | Y | Path to private key for SSH.<br/>(-i {string}) |
|
3402
|
+
| sshfp | string | Y | Y | Y | Y | Y | Y | Check it against server SSH host key fingerprint.<br/>(--check-sshfp={string}) |
|
3403
|
+
| symlink_policy | string | Y | Y | Y | Y | Y | Y | Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links={enum}) |
|
3404
|
+
| tags | object | Y | Y | Y | Y | Y | Y | Metadata for transfer as JSON. Key `aspera` is reserved. Key `aspera.xfer_retry` specifies a retry timeout for node api initiated transfers.<br/>(--tags64=(conversion){object}) |
|
3405
|
+
| tags64 | string | Y | | | | | Y | Metadata for transfer as JSON. Key `aspera` is reserved. Key `aspera.xfer_retry` specifies a retry timeout for node api initiated transfers.<br/>(--tags64={string}) |
|
3406
|
+
| target_rate_cap_kbps | integer | | Y | | | | Y | Maximum target rate for incoming transfers, in kilobits per second. Returned by upload/download_setup node API. |
|
3407
|
+
| target_rate_kbps | integer | Y | Y | Y | Y | Y | Y | Specifies desired speed for the transfer.<br/>(-l {integer}) |
|
3408
|
+
| target_rate_percentage | string | | | | | | | TODO: remove ? |
|
3409
|
+
| title | string | | Y | | | Y | Y | Title of the transfer. |
|
3295
3410
|
| token | string | Y | Y | Y | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
3296
|
-
| use_ascp4 |
|
3297
|
-
| use_system_ssh | string |
|
3298
|
-
| write_threads |
|
3299
|
-
| wss_enabled |
|
3300
|
-
| wss_port |
|
3301
|
-
| xfer_max_retries |
|
3411
|
+
| use_ascp4 | boolean | Y | | | | Y | Y | Specify version of protocol. Do not use ascp4.<br/>(<special>) |
|
3412
|
+
| use_system_ssh | string | Y | | | | | Y | TODO, comment...<br/>(-SSH {string}) |
|
3413
|
+
| write_threads | integer | | | | | | | ascp4 only |
|
3414
|
+
| wss_enabled | boolean | Y | Y | Y | Y | Y | Y | Server has Web Socket service enabled<br/>(<special>) |
|
3415
|
+
| wss_port | integer | Y | Y | Y | Y | Y | Y | TCP port used for websocket service feed<br/>(<special>) |
|
3416
|
+
| xfer_max_retries | integer | | | | | Y | | Maximum number of retries, for node API initiated transfers. Shall not exceed aspera.conf `transfer_manager_max_retries` (default 5). |
|
3302
3417
|
|
3303
3418
|
#### Destination folder for transfers
|
3304
3419
|
|
@@ -3310,18 +3425,18 @@ The destination folder is set by `ascli` by default to:
|
|
3310
3425
|
It is specified by the [**transfer-spec**](#transfer-specification) parameter `destination_root`.
|
3311
3426
|
As such, it can be modified with option: `--ts=@json:'{"destination_root":"<path>"}'`.
|
3312
3427
|
The option `to_folder` provides an equivalent and convenient way to change this parameter:
|
3313
|
-
`--to-folder=<path
|
3428
|
+
`--to-folder=<path>`.
|
3314
3429
|
|
3315
3430
|
#### List of files for transfers
|
3316
3431
|
|
3317
3432
|
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
3318
3433
|
|
3319
|
-
By default the list of files to transfer is simply provided on the command line.
|
3434
|
+
By default, the list of files to transfer is simply provided on the command line.
|
3320
3435
|
|
3321
3436
|
The list of (source) files to transfer is specified by (extended value) option `sources` (default: `@args`).
|
3322
3437
|
The list is either simply the list of source files, or a combined source/destination list (see below) depending on value of option `src_type` (default: `list`).
|
3323
3438
|
|
3324
|
-
In `ascli`, all transfer parameters, including file list, are provided to the transfer agent in a [**transfer-spec**](#transfer-specification) so that execution of a transfer is independent of the transfer agent (direct, connect, node, transfer
|
3439
|
+
In `ascli`, all transfer parameters, including file list, are provided to the transfer agent in a [**transfer-spec**](#transfer-specification) so that execution of a transfer is independent of the transfer agent (direct, connect, node, transfer daemon...).
|
3325
3440
|
So, eventually, the list of files to transfer is provided to the transfer agent using the [**transfer-spec**](#transfer-specification) field: `"paths"` which is a list (array) of pairs of `"source"` (mandatory) and `"destination"` (optional).
|
3326
3441
|
The `sources` and `src_type` options provide convenient ways to populate the transfer spec with the source file list.
|
3327
3442
|
|
@@ -3430,16 +3545,16 @@ In the following table source folder `d3` contains 2 files: `f1` and `d4/f2`.
|
|
3430
3545
|
|
3431
3546
|
| Source files | Destination | Folders on Dest. |`create_dir`| Destination Files |
|
3432
3547
|
|--------------|-------------|-------------------|------------|-----------------------------|
|
3433
|
-
| f1
|
3434
|
-
| f1
|
3435
|
-
| f1
|
3436
|
-
| f1
|
3437
|
-
| f1 f2
|
3438
|
-
| d3
|
3439
|
-
| f1
|
3440
|
-
| f1 f2
|
3441
|
-
| d1/f1 d2/f2
|
3442
|
-
| d3
|
3548
|
+
| `f1` | `d/f` | - | false | Error: `d` does not exist. |
|
3549
|
+
| `f1` | `d/f` | `d` | false | `d/f` (renamed) |
|
3550
|
+
| `f1` | `d/f/.` | `d` | false | `d/f` (renamed) |
|
3551
|
+
| `f1` | `d/f` | `d/f` | false | `d/f/f1` |
|
3552
|
+
| `f1` `f2` | `d` | `d` | false | `d/f1` `d/f2` |
|
3553
|
+
| `d3` | `d` | - | false | `d/f1` `d/f2` (renamed) |
|
3554
|
+
| `f1` | `d` | - | true | `d/f1` |
|
3555
|
+
| `f1` `f2` | `d` | - | true | `d/f1` `d/f2` |
|
3556
|
+
| `d1/f1` `d2/f2` | `d` | - | true | `d/f1` `d/f2` |
|
3557
|
+
| `d3` | `d` | - | true | `d/d3/f1` `d/d3/d4/f2` |
|
3443
3558
|
|
3444
3559
|
If a file par list is provided then it is possible to rename or specify a different destination folder for each source (relative to the destination).
|
3445
3560
|
|
@@ -3447,7 +3562,7 @@ If transfer spec has a `src_base`, it has the side effect that the simple source
|
|
3447
3562
|
|
3448
3563
|
| Source files | Destination |`src_base`| Destination Files |
|
3449
3564
|
|-------------------|-------------|----------|-----------------------------|
|
3450
|
-
| d1/d2/f2 d1/d3/f3 | d
|
3565
|
+
| `d1/d2/f2` `d1/d3/f3` | `d` | `d1` | `d/d2/f2` `d/d3/f3` |
|
3451
3566
|
|
3452
3567
|
Advanced Example: Send files `./file1` and `./folder2/files2` to server (e.g. `/Upload`) and keep the original file names and folders, i.e. send `file1` to `/Upload/file1` and `files2` to `/Upload/folder2/files2`.
|
3453
3568
|
|
@@ -3497,7 +3612,7 @@ Activating CSEAR consists in using transfer spec parameters:
|
|
3497
3612
|
- `content_protection` : activate encryption (`encrypt` for upload) or decryption (`decrypt` for download)
|
3498
3613
|
- `content_protection_password` : the passphrase to be used.
|
3499
3614
|
|
3500
|
-
Example: parameter to download a
|
3615
|
+
Example: parameter to download a Faspex package and decrypt on the fly
|
3501
3616
|
|
3502
3617
|
```bash
|
3503
3618
|
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"my_password_here"}'
|
@@ -3517,7 +3632,7 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
3517
3632
|
--ts=@json:'{"ssh_port":33002}'
|
3518
3633
|
```
|
3519
3634
|
|
3520
|
-
- Force
|
3635
|
+
- Force HTTP fallback mode:
|
3521
3636
|
|
3522
3637
|
```bash
|
3523
3638
|
--ts=@json:'{"http_fallback":"force"}'
|
@@ -3531,12 +3646,10 @@ Example: parameter to download a faspex package and decrypt on the fly
|
|
3531
3646
|
|
3532
3647
|
### Transfer progress bar
|
3533
3648
|
|
3534
|
-
File transfer operations are monitored and a progress bar is displayed on the terminal if option `progress_bar` (`Bool`) is set to `yes` (default if the output is a terminal).
|
3649
|
+
File transfer operations are monitored, and a progress bar is displayed on the terminal if option `progress_bar` (`Bool`) is set to `yes` (default if the output is a terminal).
|
3535
3650
|
|
3536
3651
|
The same progress bar is used for any type of transfer, using `ascp`, server to server, using HTTPS, etc...
|
3537
3652
|
|
3538
|
-
To display the native progress bar of `ascp`, use `--progress-bar=no --transfer-info=@json:'{"quiet":false}'`.
|
3539
|
-
|
3540
3653
|
### Scheduler
|
3541
3654
|
|
3542
3655
|
It is useful to configure automated scheduled execution.
|
@@ -3550,17 +3663,17 @@ It can be configured:
|
|
3550
3663
|
|
3551
3664
|
- Using utility [`schtasks.exe`](https://learn.microsoft.com/fr-fr/windows-server/administration/windows-commands/schtasks-create)
|
3552
3665
|
|
3553
|
-
- Using
|
3666
|
+
- Using Powershell function [`scheduletasks`](https://learn.microsoft.com/en-us/powershell/module/scheduledtasks)
|
3554
3667
|
|
3555
3668
|
- Using `taskschd.msc` (UI)
|
3556
3669
|
|
3557
3670
|
#### Unix-like Scheduler
|
3558
3671
|
|
3559
|
-
Unix-like systems (Linux, ...) provide cron
|
3672
|
+
Unix-like systems (Linux, ...) provide `cron`, configured using a [`crontab`](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
3560
3673
|
|
3561
3674
|
Linux also provides `anacron`, if tasks are hourly or daily.
|
3562
3675
|
|
3563
|
-
For example, on Linux it is convenient to create a wrapping script, e.g. `cron_ascli` that will
|
3676
|
+
For example, on Linux it is convenient to create a wrapping script, e.g. `cron_ascli` that will set up the environment (e.g. Ruby) to properly start `ascli`:
|
3564
3677
|
|
3565
3678
|
```bash
|
3566
3679
|
#!/bin/bash
|
@@ -3582,13 +3695,70 @@ crontab<<EOF
|
|
3582
3695
|
EOF
|
3583
3696
|
```
|
3584
3697
|
|
3585
|
-
> **Note:** Logging options are kept here in the cron file instead of configuration file to allow execution on command line with output on command line.
|
3698
|
+
> **Note:** Logging options are kept here in the `cron` file instead of configuration file to allow execution on command line with output on command line.
|
3699
|
+
|
3700
|
+
### Running as service
|
3701
|
+
|
3702
|
+
Some commands result in `ascli` running as a server, listening on a port.
|
3703
|
+
In this case, it is usually desirable to run `ascli` as a service.
|
3704
|
+
On Linux, typically, [SystemD](https://systemd.io/) is used.
|
3705
|
+
|
3706
|
+
A convenient way is to write a startup script, and run this script as a service.
|
3707
|
+
|
3708
|
+
Let's give a base name for our service: `my_ascli_svc`
|
3709
|
+
|
3710
|
+
The startup script can be simply the `ascli` command line, for example: `/usr/local/bin/start_my_ascli_svc.sh`:
|
3711
|
+
|
3712
|
+
```bash
|
3713
|
+
#!/bin/bash
|
3714
|
+
set -e
|
3715
|
+
echo "Starting my_ascli_svc at $(date)"
|
3716
|
+
# set PATH to find ascli, and other environment setup
|
3717
|
+
exec ascli .....
|
3718
|
+
```
|
3719
|
+
|
3720
|
+
And make this script executable:
|
3721
|
+
|
3722
|
+
```bash
|
3723
|
+
chmod a+x /usr/local/bin/start_my_ascli_svc.sh
|
3724
|
+
```
|
3725
|
+
|
3726
|
+
Create a startup file: `/etc/systemd/system/my_ascli_svc.service` :
|
3727
|
+
|
3728
|
+
```ini
|
3729
|
+
[Unit]
|
3730
|
+
Description=My ascli daemon
|
3731
|
+
After=network.target
|
3732
|
+
|
3733
|
+
[Service]
|
3734
|
+
ExecStart=/usr/local/bin/start_my_ascli_svc.sh
|
3735
|
+
Restart=on-failure
|
3736
|
+
RestartSec=15
|
3737
|
+
User=xfer
|
3738
|
+
# Optional, specify a working directory
|
3739
|
+
# WorkingDirectory=/path/to/working/dir
|
3740
|
+
# Optional, redirect output
|
3741
|
+
StandardOutput=journal
|
3742
|
+
StandardError=journal
|
3743
|
+
|
3744
|
+
[Install]
|
3745
|
+
WantedBy=multi-user.target
|
3746
|
+
```
|
3747
|
+
|
3748
|
+
Then enable and start with:
|
3749
|
+
|
3750
|
+
```bash
|
3751
|
+
sudo systemctl daemon-reload
|
3752
|
+
sudo systemctl enable --now my_ascli_svc.service
|
3753
|
+
systemctl status my_ascli_svc.service
|
3754
|
+
journalctl -u my_ascli_svc.service
|
3755
|
+
```
|
3586
3756
|
|
3587
3757
|
### Locking for exclusive execution
|
3588
3758
|
|
3589
3759
|
In some cases one needs to ensure that `ascli` is not executed several times in parallel.
|
3590
3760
|
|
3591
|
-
When `ascli` is executed automatically on a schedule basis, one generally desires that a new execution is not started if a previous execution is still running because an
|
3761
|
+
When `ascli` is executed automatically on a schedule basis, one generally desires that a new execution is not started if a previous execution is still running because an ongoing operation may last longer than the scheduling period:
|
3592
3762
|
|
3593
3763
|
- Executing instances may pile-up and kill the system
|
3594
3764
|
- The same file may be transferred by multiple instances at the same time.
|
@@ -3597,7 +3767,7 @@ When `ascli` is executed automatically on a schedule basis, one generally desire
|
|
3597
3767
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
3598
3768
|
|
3599
3769
|
- The Windows scheduler does this by default
|
3600
|
-
- Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
3770
|
+
- Linux `cron` can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
3601
3771
|
|
3602
3772
|
```bash
|
3603
3773
|
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
@@ -3658,10 +3828,9 @@ It is a pseudo file system allowing generation of file data without actual stora
|
|
3658
3828
|
|
3659
3829
|
### `faux:` for testing
|
3660
3830
|
|
3661
|
-
This is an extract of the man page of `ascp`.
|
3662
|
-
This feature is a feature of `ascp`, not `ascli`.
|
3663
|
-
|
3664
3831
|
This adapter can be used to simulate a file or a directory.
|
3832
|
+
This is a feature of `ascp`, not `ascli`.
|
3833
|
+
The following is an extract of the man page of `ascp`.
|
3665
3834
|
|
3666
3835
|
To discard data at the destination, the destination argument is set to `faux://`.
|
3667
3836
|
|
@@ -3678,7 +3847,7 @@ where:
|
|
3678
3847
|
|
3679
3848
|
> **Note:** Characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
3680
3849
|
|
3681
|
-
For all sizes, a suffix can be added (case
|
3850
|
+
For all sizes, a suffix can be added (case-insensitive) to the size: k, m, g, t, p, e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8\*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
3682
3851
|
|
3683
3852
|
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
3684
3853
|
|
@@ -3704,17 +3873,18 @@ The sequence parameter is applied as follows:
|
|
3704
3873
|
|
3705
3874
|
- If `seq` is `random` then each file size is:
|
3706
3875
|
|
3707
|
-
- size +/- (inc * rand())
|
3876
|
+
- `size +/- (inc * rand())`
|
3708
3877
|
- Where rand is a random number between 0 and 1
|
3709
3878
|
- Note that file size must not be negative, inc will be set to size if it is greater than size
|
3710
|
-
- Similarly, overall file size must be less than 8
|
3879
|
+
- Similarly, overall file size must be less than 8\*2<sup>60</sup>.
|
3880
|
+
If size + inc is greater, inc will be reduced to limit size + inc to 7\*2<sup>60</sup>.
|
3711
3881
|
|
3712
3882
|
- If `seq` is `sequential` then each file size is:
|
3713
3883
|
|
3714
3884
|
- `size + ((file_index - 1) * inc)`
|
3715
3885
|
- Where first file is index 1
|
3716
3886
|
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
3717
|
-
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less
|
3887
|
+
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less than 8\*2<sup>60</sup>.
|
3718
3888
|
|
3719
3889
|
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
3720
3890
|
|
@@ -3732,12 +3902,18 @@ ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
|
3732
3902
|
ascli server upload /tmp/sample --to-folder=faux://
|
3733
3903
|
```
|
3734
3904
|
|
3735
|
-
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the
|
3905
|
+
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the base name of each file being `testfile` to /Upload
|
3736
3906
|
|
3737
3907
|
```bash
|
3738
3908
|
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
3739
3909
|
```
|
3740
3910
|
|
3911
|
+
- Upload a faux directory `mydir` containing 1000 files, of size 1 byte, with the base name of each file being `testfile` to /Upload
|
3912
|
+
|
3913
|
+
```bash
|
3914
|
+
ascli server upload "faux:///mydir?file=testfile&count=1000&size=1" --to-folder=/Upload
|
3915
|
+
```
|
3916
|
+
|
3741
3917
|
### Usage
|
3742
3918
|
|
3743
3919
|
```text
|
@@ -3852,7 +4028,7 @@ OPTIONS:
|
|
3852
4028
|
|
3853
4029
|
|
3854
4030
|
COMMAND: node
|
3855
|
-
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse central delete download events health
|
4031
|
+
SUBCOMMANDS: access_keys api_details asperabrowser async basic_token bearer_token browse cat central delete download events health info license mkdir mkfile mklink rename search service simulator slash space ssync stream sync telemetry transfer transport upload watch_folder
|
3856
4032
|
OPTIONS:
|
3857
4033
|
--url=VALUE URL of application, e.g. https://faspex.example.com/aspera/faspex
|
3858
4034
|
--username=VALUE User's name to log in
|
@@ -4043,16 +4219,16 @@ OPTIONS:
|
|
4043
4219
|
|
4044
4220
|
Bulk creation and deletion of resources are possible using option `bulk` (`yes`,`no`(default)).
|
4045
4221
|
In that case, the operation expects an `Array` of `Hash` instead of a simple `Hash` using the [Extended Value Syntax](#extended-value-syntax).
|
4046
|
-
This option is available only for some
|
4222
|
+
This option is available only for some resources: if you need it: try and see if the entities you try to create or delete support this option.
|
4047
4223
|
|
4048
4224
|
### Option: `query`
|
4049
4225
|
|
4050
|
-
The `query` option can generally be used to add URL parameters to commands that list
|
4226
|
+
The `query` option can generally be used to add URL parameters to commands that list resources.
|
4051
4227
|
It takes either a `Hash` or an `Array`, corresponding to key/value pairs that appear in the query part of request.
|
4052
4228
|
|
4053
4229
|
For example: `--query=@json:'{"p1":"v1","p2":"v2"}'` leads to query: `?p1=v1&p2=v2`.
|
4054
4230
|
|
4055
|
-
If the same parameter needs to be provided several times, then it's possible as well to provide an Array or 2-element Array
|
4231
|
+
If the same parameter needs to be provided several times, then it's possible as well to provide an `Array` or 2-element `Array`: `--query=@json:'[["p1":,"v1"],["p2":"v2"]]'` leads to the same result as previously.
|
4056
4232
|
|
4057
4233
|
If PHP's style array is used, then one can use either:
|
4058
4234
|
|
@@ -4088,9 +4264,9 @@ Most plugins will take the URL option: `url` to identify their location.
|
|
4088
4264
|
|
4089
4265
|
REST APIs of Aspera legacy applications (Aspera Node, Faspex 4, Shares, Console, Orchestrator) use simple username/password authentication: HTTP Basic Authentication using options: `username` and `password`.
|
4090
4266
|
|
4091
|
-
Aspera on Cloud and Faspex 5 rely on
|
4267
|
+
Aspera on Cloud and Faspex 5 rely on OAuth.
|
4092
4268
|
|
4093
|
-
By default plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
4269
|
+
By default, plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
4094
4270
|
|
4095
4271
|
```bash
|
4096
4272
|
ascli --show-config --fields=plugin_folder
|
@@ -4112,17 +4288,18 @@ ascli --plugin-folder=. foo
|
|
4112
4288
|
|
4113
4289
|
## Plugin: `aoc`: IBM Aspera on Cloud
|
4114
4290
|
|
4115
|
-
Aspera on Cloud API requires the use of
|
4291
|
+
Aspera on Cloud API requires the use of OAuth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
4116
4292
|
|
4117
4293
|
It is recommended to use the wizard to set it up, although manual configuration is also possible.
|
4118
4294
|
|
4119
4295
|
### Configuration: Using Wizard
|
4120
4296
|
|
4121
|
-
`ascli` provides a configuration wizard.
|
4297
|
+
`ascli` provides a configuration [wizard](#wizard).
|
4122
4298
|
|
4123
4299
|
The wizard guides you through the steps to create a new configuration preset for Aspera on Cloud.
|
4124
4300
|
|
4125
|
-
The first
|
4301
|
+
The first optional argument is the URL of your Aspera on Cloud instance, e.g. `https://_your_instance_.ibmaspera.com` or simply the organization name, and a second optional argument can also be provided to specify the plugin name, e.g. `aoc` for Aspera on Cloud.
|
4302
|
+
If optional arguments are not provided, the wizard will ask interactively and try to detect the application.
|
4126
4303
|
|
4127
4304
|
Here is a sample invocation :
|
4128
4305
|
|
@@ -4159,6 +4336,13 @@ ascli config wizard _your_instance_ aoc
|
|
4159
4336
|
|
4160
4337
|
> **Note:** In above example, replace `_your_instance_` with the first part of your actual AoC URL: `https://_your_instance_.ibmaspera.com`.
|
4161
4338
|
|
4339
|
+
After successful completion of the wizard, a new configuration preset is created, and set as default for the `aoc` plugin.
|
4340
|
+
This can be verified with command:
|
4341
|
+
|
4342
|
+
```bash
|
4343
|
+
ascli config preset over
|
4344
|
+
```
|
4345
|
+
|
4162
4346
|
### Configuration: Using manual setup
|
4163
4347
|
|
4164
4348
|
> **Note:** If you used the wizard (recommended): skip this section.
|
@@ -4167,17 +4351,17 @@ ascli config wizard _your_instance_ aoc
|
|
4167
4351
|
|
4168
4352
|
Several types of OAuth authentication are supported:
|
4169
4353
|
|
4170
|
-
- JSON Web Token (JWT) : authentication is secured by a private key (recommended
|
4354
|
+
- JSON Web Token (JWT) : authentication is secured by a private key (recommended)
|
4171
4355
|
- Web based authentication : authentication is made by user using a browser
|
4172
|
-
- URL Token : external
|
4356
|
+
- URL Token : external user's authentication with URL tokens (public links)
|
4173
4357
|
|
4174
4358
|
The authentication method is controlled by option `auth`.
|
4175
4359
|
|
4176
4360
|
For a **quick start**, follow the mandatory and sufficient section: [API Client Registration](#api-client-registration) (auth=web) as well as [[Option Preset](#option-preset) for Aspera on Cloud](#configuration-for-aspera-on-cloud).
|
4177
4361
|
|
4178
|
-
For a more convenient, browser-less, experience follow the [JWT](#authentication-with-private-key) section (auth=jwt) in addition to Client Registration.
|
4362
|
+
For a more convenient, browser-less, experience follow the [JWT](#authentication-with-private-key) section (`auth=jwt`) in addition to Client Registration.
|
4179
4363
|
|
4180
|
-
In
|
4364
|
+
In OAuth, a **Bearer** token is generated to authenticate REST calls.
|
4181
4365
|
Bearer tokens are valid for a period of time defined (by the AoC app, configurable by admin) at its creation.
|
4182
4366
|
`ascli` saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
4183
4367
|
|
@@ -4189,7 +4373,7 @@ If you use the built-in client_id and client_secret, skip this and do not set th
|
|
4189
4373
|
|
4190
4374
|
Else you can use a specific OAuth API client_id, the first step is to declare `ascli` in Aspera on Cloud using the admin interface.
|
4191
4375
|
|
4192
|
-
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client)
|
4376
|
+
([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client)).
|
4193
4377
|
|
4194
4378
|
Let's start by a registration with web based authentication (auth=web):
|
4195
4379
|
|
@@ -4210,9 +4394,9 @@ Once the client is registered, a **Client ID** and **Secret** are created, these
|
|
4210
4394
|
|
4211
4395
|
#### Configuration for Aspera on Cloud
|
4212
4396
|
|
4213
|
-
If you did not use the wizard, you can also manually create
|
4397
|
+
If you did not use the wizard, you can also manually create an [Option Preset](#option-preset) for `ascli` in its configuration file.
|
4214
4398
|
|
4215
|
-
|
4399
|
+
Let's create an [Option Preset](#option-preset) called: `my_aoc_org` using `ask` for interactive input (client info from previous step):
|
4216
4400
|
|
4217
4401
|
```bash
|
4218
4402
|
ascli config preset ask my_aoc_org url client_id client_secret
|
@@ -4262,11 +4446,15 @@ ascli aoc admin client list
|
|
4262
4446
|
```
|
4263
4447
|
|
4264
4448
|
```output
|
4265
|
-
|
4266
|
-
|
4267
|
-
|
4268
|
-
|
4269
|
-
|
4449
|
+
Items: 4/4
|
4450
|
+
╭────────────┬─────────────────────╮
|
4451
|
+
│ id │ name │
|
4452
|
+
╞════════════╪═════════════════════╡
|
4453
|
+
│ oXPUyJ7JpQ │ PRI Sydney │
|
4454
|
+
│ TaoAmAG8Rg │ ascli_test_web │
|
4455
|
+
│ TDNl2bLZqw │ ascli_web │
|
4456
|
+
│ VTh92i5OfQ │ shannon │
|
4457
|
+
╰────────────┴─────────────────────╯
|
4270
4458
|
```
|
4271
4459
|
|
4272
4460
|
```bash
|
@@ -4333,7 +4521,7 @@ ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.asp
|
|
4333
4521
|
|
4334
4522
|
> **Note:** The private key argument represents the actual PEM string.
|
4335
4523
|
In order to read the content from a file, use the `@file:` prefix.
|
4336
|
-
But if the
|
4524
|
+
But if the `@file:` argument is used as is, it will read the file and set in the configuration file.
|
4337
4525
|
So, to keep the `@file:` tag in the configuration file, the `@val:` prefix is added.
|
4338
4526
|
|
4339
4527
|
After this last step, commands do not require web login anymore.
|
@@ -4402,12 +4590,12 @@ Values are directly sent to the API call and used as a filter on server side.
|
|
4402
4590
|
|
4403
4591
|
The following parameters are supported:
|
4404
4592
|
|
4405
|
-
- `q` : a filter on name of resource (case
|
4593
|
+
- `q` : a filter on name of resource (case-insensitive, matches if value is contained in name)
|
4406
4594
|
- `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
4407
4595
|
- `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
4408
4596
|
- `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
4409
|
-
- `page` : native
|
4410
|
-
- `per_page` : native
|
4597
|
+
- `page` : native API parameter, in general do not use (added by `ascli`)
|
4598
|
+
- `per_page` : native API parameter, number of items par API call, in general do not use
|
4411
4599
|
- Other specific parameters depending on resource type.
|
4412
4600
|
|
4413
4601
|
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
@@ -4446,24 +4634,25 @@ Refer to the AoC API for full list of query parameters, or use the browser in de
|
|
4446
4634
|
|
4447
4635
|
#### Selecting a resource
|
4448
4636
|
|
4449
|
-
Resources are identified by a unique `id`, as well as a unique `name` (case
|
4637
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case-insensitive).
|
4450
4638
|
|
4451
4639
|
To execute an action on a specific resource, select it using one of those methods:
|
4452
4640
|
|
4453
|
-
- **recommended**: give
|
4641
|
+
- **recommended**: give ID directly on command line **after the action**: `aoc admin node show 123`
|
4454
4642
|
- Give name on command line **after the action**: `aoc admin node show name abc`
|
4455
4643
|
- Provide option `id` : `aoc admin node show 123`
|
4456
4644
|
- Provide option `name` : `aoc admin node show --name=abc`
|
4457
4645
|
|
4458
4646
|
#### Creating a resource
|
4459
4647
|
|
4460
|
-
New resources (users, groups, workspaces, etc
|
4648
|
+
New resources (users, groups, workspaces, etc...) can be created using a command like:
|
4461
4649
|
|
4462
4650
|
```bash
|
4463
4651
|
ascli aoc admin create <resource type> @json:'{<...parameters...>}'
|
4464
4652
|
```
|
4465
4653
|
|
4466
|
-
Some
|
4654
|
+
Some API endpoints are described in [IBM API Hub](https://developer.ibm.com/apis/catalog?search=%22aspera%20on%20cloud%20api%22).
|
4655
|
+
Sadly, not all.
|
4467
4656
|
|
4468
4657
|
Nevertheless, it is possible to guess the structure of the creation value by simply dumping an existing resource, and use the same parameters for the creation.
|
4469
4658
|
|
@@ -4520,7 +4709,8 @@ The activity app can be queried with:
|
|
4520
4709
|
ascli aoc admin analytics transfers
|
4521
4710
|
```
|
4522
4711
|
|
4523
|
-
It can also support filters and send notification using option `notify_to`.
|
4712
|
+
It can also support filters and send notification using option `notify_to`.
|
4713
|
+
A template is defined using option `notify_template` :
|
4524
4714
|
|
4525
4715
|
`mytemplate.erb`:
|
4526
4716
|
|
@@ -4552,11 +4742,11 @@ Options:
|
|
4552
4742
|
- `query` filter (on API call)
|
4553
4743
|
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
4554
4744
|
|
4555
|
-
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now]
|
4745
|
+
> **Note:** This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is `[date of previous execution]..[now]`.
|
4556
4746
|
|
4557
4747
|
#### Transfer: Using specific transfer ports
|
4558
4748
|
|
4559
|
-
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
4749
|
+
By default, transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
4560
4750
|
The option `default_ports` ([yes]/no) allows `ascli` to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
4561
4751
|
|
4562
4752
|
#### Using ATS
|
@@ -4569,7 +4759,7 @@ Aspera on Cloud Shared folders are implemented through a special type of file: `
|
|
4569
4759
|
A `link` is the equivalent of a symbolic link on a file system: it points to another folder (not file).
|
4570
4760
|
|
4571
4761
|
Listing a link (in terminal position of path) will information on the link itself, not the content of the folder it points to.
|
4572
|
-
To list the target folder content, add a `/`
|
4762
|
+
To list the target folder content, add a `/` at the end of the path.
|
4573
4763
|
|
4574
4764
|
Example:
|
4575
4765
|
|
@@ -4662,7 +4852,7 @@ ascli aoc user workspaces list
|
|
4662
4852
|
|
4663
4853
|
#### Example: Create a sub access key in a `node`
|
4664
4854
|
|
4665
|
-
Creation of a sub-access key is like creation of access key with the following difference: authentication to
|
4855
|
+
Creation of a sub-access key is like creation of access key with the following difference: authentication to Node API is made with access key (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
4666
4856
|
|
4667
4857
|
```bash
|
4668
4858
|
ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create @json:'{"storage":{"path":"/folder1"}}'
|
@@ -4717,14 +4907,14 @@ Other query parameters:
|
|
4717
4907
|
|
4718
4908
|
#### Example: Add all members of a workspace to another workspace
|
4719
4909
|
|
4720
|
-
a- Get
|
4910
|
+
a- Get ID of first workspace
|
4721
4911
|
|
4722
4912
|
```bash
|
4723
4913
|
WS1='First Workspace'
|
4724
4914
|
WS1ID=$(ascli aoc admin workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
4725
4915
|
```
|
4726
4916
|
|
4727
|
-
b- Get
|
4917
|
+
b- Get ID of second workspace
|
4728
4918
|
|
4729
4919
|
```bash
|
4730
4920
|
WS2='Second Workspace'
|
@@ -4743,7 +4933,7 @@ d- Convert to creation data for second workspace:
|
|
4743
4933
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
4744
4934
|
```
|
4745
4935
|
|
4746
|
-
or, using jq
|
4936
|
+
or, using `jq`:
|
4747
4937
|
|
4748
4938
|
```bash
|
4749
4939
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
@@ -4786,7 +4976,7 @@ ascli aoc admin group create @json:'{"name":"group 1","description":"my super gr
|
|
4786
4976
|
|
4787
4977
|
Group: `11111`
|
4788
4978
|
|
4789
|
-
- Get the workspace
|
4979
|
+
- Get the workspace ID
|
4790
4980
|
|
4791
4981
|
```bash
|
4792
4982
|
ascli aoc admin workspace list --query=@json:'{"q":"myworkspace"}' --fields=id --format=csv --display=data
|
@@ -4800,7 +4990,7 @@ Workspace: 22222
|
|
4800
4990
|
ascli aoc admin workspace_membership create @json:'{"workspace_id":22222,"member_type":"user","member_id":11111}'
|
4801
4991
|
```
|
4802
4992
|
|
4803
|
-
- Get a user's
|
4993
|
+
- Get a user's ID
|
4804
4994
|
|
4805
4995
|
```bash
|
4806
4996
|
ascli aoc admin user list --query=@json:'{"q":"manu.macron@example.com"}' --fields=id --format=csv --display=data
|
@@ -4818,7 +5008,7 @@ ascli aoc admin group_membership create @json:'{"group_id":11111,"member_type":"
|
|
4818
5008
|
|
4819
5009
|
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
4820
5010
|
|
4821
|
-
First,
|
5011
|
+
First, set up the environment (skip if already done)
|
4822
5012
|
|
4823
5013
|
```bash
|
4824
5014
|
ascli config wizard --url=https://sedemo.ibmaspera.com --username=someuser@example.com
|
@@ -4846,7 +5036,7 @@ You can test with:
|
|
4846
5036
|
ascli aoc user profile show
|
4847
5037
|
```
|
4848
5038
|
|
4849
|
-
This creates the option preset `aoc_[org name]` to allow seamless command line access and sets it as default for
|
5039
|
+
This creates the option preset `aoc_[org name]` to allow seamless command line access and sets it as default for Aspera on Cloud.
|
4850
5040
|
|
4851
5041
|
Then, create two shared folders located in two regions, in your files home, in a workspace.
|
4852
5042
|
|
@@ -4856,20 +5046,18 @@ Then, transfer between those:
|
|
4856
5046
|
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
4857
5047
|
```
|
4858
5048
|
|
4859
|
-
#### Example: Create registration key to register a node
|
5049
|
+
#### Example: Create registration key to register a tethered node
|
4860
5050
|
|
4861
|
-
|
4862
|
-
ascli aoc admin client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
4863
|
-
```
|
5051
|
+
The following command will create and display a secret token to register a self-managed Aspera Transfer Server:
|
4864
5052
|
|
4865
|
-
```
|
4866
|
-
|
5053
|
+
```bash
|
5054
|
+
ascli aoc admin client_registration_token create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --show-secrets=yes
|
4867
5055
|
```
|
4868
5056
|
|
4869
5057
|
#### Example: Delete all registration keys
|
4870
5058
|
|
4871
5059
|
```bash
|
4872
|
-
ascli aoc admin
|
5060
|
+
ascli aoc admin client_registration_token list --fields=id --format=csv|ascli aoc admin client_registration_token delete @lines:@stdin: --bulk=yes
|
4873
5061
|
```
|
4874
5062
|
|
4875
5063
|
```output
|
@@ -4905,9 +5093,9 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
4905
5093
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
4906
5094
|
```
|
4907
5095
|
|
4908
|
-
Once executed, the access key `id` and `secret`, randomly generated by the
|
5096
|
+
Once executed, the access key `id` and `secret`, randomly generated by the Node API, is displayed.
|
4909
5097
|
|
4910
|
-
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking
|
5098
|
+
> **Note:** Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking IBM support.
|
4911
5099
|
|
4912
5100
|
- Create the AoC node entity
|
4913
5101
|
|
@@ -4930,12 +5118,12 @@ Creation of a node with a self-managed node is similar, but the command `aoc adm
|
|
4930
5118
|
Source files are provided as a list with the `sources` option.
|
4931
5119
|
Refer to section [File list](#list-of-files-for-transfers)
|
4932
5120
|
|
4933
|
-
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file
|
5121
|
+
> **Note:** A special case is when the source files are located on **Aspera on Cloud** (i.e. using access keys and the `file ID` API).
|
4934
5122
|
|
4935
5123
|
Source files are located on **Aspera on cloud**, when :
|
4936
5124
|
|
4937
|
-
- The server is Aspera on Cloud, and executing a download or recv
|
4938
|
-
- The agent is Aspera on Cloud, and executing an upload or send
|
5125
|
+
- The server is Aspera on Cloud, and executing a `download` or `recv`
|
5126
|
+
- The agent is Aspera on Cloud, and executing an `upload` or `send`
|
4939
5127
|
|
4940
5128
|
In this case:
|
4941
5129
|
|
@@ -4968,10 +5156,10 @@ ascli aoc packages shared_inboxes list
|
|
4968
5156
|
|
4969
5157
|
Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: **user** or **shared inbox**:
|
4970
5158
|
|
4971
|
-
- Provide either
|
5159
|
+
- Provide either IDs as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
4972
5160
|
- or just names: `"recipients":[{"The Dest"}]`.
|
4973
5161
|
|
4974
|
-
ascli will resolve the list of email addresses and dropbox names to the expected type/
|
5162
|
+
ascli will resolve the list of email addresses and dropbox names to the expected type/ID list, based on case-insensitive partial match.
|
4975
5163
|
|
4976
5164
|
If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and:
|
4977
5165
|
|
@@ -4980,17 +5168,20 @@ If a user recipient (email) is not already registered and the workspace allows e
|
|
4980
5168
|
|
4981
5169
|
#### List packages
|
4982
5170
|
|
4983
|
-
By default, when using `aoc packages list`, the following `query` is performed:
|
5171
|
+
By default, when using `aoc packages list` or `aoc packages receive ALL`, the following `query` is performed:
|
4984
5172
|
|
4985
|
-
|
|
4986
|
-
|
5173
|
+
| Query parameter | Value |
|
5174
|
+
|----------------------------|---------|
|
4987
5175
|
| `archived` | `false` |
|
4988
5176
|
| `has_content` | `true` |
|
4989
5177
|
| `received` | `true` |
|
4990
5178
|
| `completed` | `true` |
|
4991
|
-
| `workspace_id` | based on current workspace |
|
4992
|
-
| `
|
4993
|
-
| `
|
5179
|
+
| `workspace_id` | Set based on current workspace |
|
5180
|
+
| `dropbox_id` | Set according to `dropbox_name`, if provided |
|
5181
|
+
| `exclude_dropbox_packages` | `true` unless `dropbox_id` is provided |
|
5182
|
+
|
5183
|
+
Parameters provided using option `query` override this query.
|
5184
|
+
To remove a parameter, set it to `null`.
|
4994
5185
|
|
4995
5186
|
#### Example: Send a package with one file to two users, using their email
|
4996
5187
|
|
@@ -5018,7 +5209,7 @@ When user packages are listed, the following query is used:
|
|
5018
5209
|
{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}
|
5019
5210
|
```
|
5020
5211
|
|
5021
|
-
To list packages in a shared inbox, the query has to be specified with the
|
5212
|
+
To list packages in a shared inbox, the query has to be specified with the shared inbox by name or its identifier.
|
5022
5213
|
Additional parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode).
|
5023
5214
|
The current workspace is added unless specified in the query.
|
5024
5215
|
|
@@ -5030,7 +5221,7 @@ Using shared inbox name:
|
|
5030
5221
|
ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
5031
5222
|
```
|
5032
5223
|
|
5033
|
-
Using shared inbox identifier: first retrieve the
|
5224
|
+
Using shared inbox identifier: first retrieve the ID of the shared inbox, and then list packages with the appropriate filter.
|
5034
5225
|
|
5035
5226
|
```bash
|
5036
5227
|
shared_box_id=$(ascli aoc packages shared_inboxes show --name='My Shared Inbox' --format=csv --display=data --fields=id)
|
@@ -5046,6 +5237,8 @@ ascli aoc packages list --query=@json:'{"dropbox_id":"'$shared_box_id'","archive
|
|
5046
5237
|
ascli aoc packages recv ALL --workspace=_workspace_ --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"_shared_inbox_name_","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
5047
5238
|
```
|
5048
5239
|
|
5240
|
+
To list packages that would be downloaded, without actually downloading them, replace `recv ALL` with `list` (keep options `once_only` and `query`)
|
5241
|
+
|
5049
5242
|
#### Example: Send a package with files from the Files app
|
5050
5243
|
|
5051
5244
|
Find files in Files app:
|
@@ -5079,7 +5272,7 @@ It is possible to automatically download new packages, like using Aspera Cargo:
|
|
5079
5272
|
ascli aoc packages recv ALL --once-only=yes --lock-port=12345
|
5080
5273
|
```
|
5081
5274
|
|
5082
|
-
- `ALL` (case
|
5275
|
+
- `ALL` (case-sensitive) will download all packages
|
5083
5276
|
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
5084
5277
|
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
5085
5278
|
|
@@ -5097,12 +5290,12 @@ ascli aoc package browse my5CnbeWng /
|
|
5097
5290
|
|
5098
5291
|
To list recursively, use command `find`.
|
5099
5292
|
|
5100
|
-
To download only some
|
5293
|
+
To download only some files listed in the package, just add the path of the files on the command line.
|
5101
5294
|
|
5102
5295
|
For advanced users, it's also possible to pipe node information for the package and use node operations:
|
5103
5296
|
|
5104
5297
|
```bash
|
5105
|
-
ascli aoc package node_info <package
|
5298
|
+
ascli aoc package node_info <package ID here> / --format=json --show-secrets=yes --display=data | ascli node -N --preset=@json:@stdin: access_key do self browse /
|
5106
5299
|
```
|
5107
5300
|
|
5108
5301
|
### Files
|
@@ -5131,13 +5324,13 @@ ascli aoc files download <single file path>
|
|
5131
5324
|
#### Shared folders
|
5132
5325
|
|
5133
5326
|
Like in AoC web UI, "Shared Folders" can be created and shared with either **Private** or **Public** links.
|
5134
|
-
**Private** links require the collaborator to log
|
5327
|
+
**Private** links require the collaborator to log in to access the shared folder.
|
5135
5328
|
**Public** links include a passcode that enables the user to access the shared folder without login-in.
|
5136
5329
|
|
5137
5330
|
Shared folders can be created either:
|
5138
5331
|
|
5139
5332
|
- by users in a workspace: they can share personal folders with other users in the same workspace: `aoc files perm`
|
5140
|
-
- by administrators: they can share a folder with users in any workspace: `aoc admin node do [node
|
5333
|
+
- by administrators: they can share a folder with users in any workspace: `aoc admin node do [node ID] perm`
|
5141
5334
|
|
5142
5335
|
Technically (API), shared folder are managed through [permissions](https://developer.ibm.com/apis/catalog/aspera--aspera-node-api/Introduction) on node and an event is sent to AoC to create a **link** in the user's home folder to the shared folder.
|
5143
5336
|
In both cases, it is necessary to specify a workspace.
|
@@ -5157,18 +5350,18 @@ The basic payload (last argument at creation usually specified with `@json:`) is
|
|
5157
5350
|
`ascli` expects the same payload for creation.
|
5158
5351
|
`ascli` automatically populates this payload like this:
|
5159
5352
|
|
5160
|
-
- `file_id` : the
|
5353
|
+
- `file_id` : the ID of the folder to share whose path is specified in the command line
|
5161
5354
|
- `access_levels` : are set by default to full access.
|
5162
|
-
- `tags` : are set with expected values for AoC:
|
5355
|
+
- `tags` : are set with expected values for AoC: username who creates, and workspace in which the shared folder is created.
|
5163
5356
|
- `access_type` and `access_id` : need to be set by the user, or using special key as follows.
|
5164
5357
|
|
5165
5358
|
To change `access_levels`, just provide the list of levels in the `@json:` payload.
|
5166
5359
|
|
5167
|
-
In order to declare/create the shared folder in the workspace, a special value for `access_id` is used: `ASPERA_ACCESS_KEY_ADMIN_WS_[workspace
|
5360
|
+
In order to declare/create the shared folder in the workspace, a special value for `access_id` is used: `ASPERA_ACCESS_KEY_ADMIN_WS_[workspace ID]]`. This is conveniently set by `ascli` using an empty string for the pseudo key `with`. In order to share a folder with a different, special tags are set, but this is conveniently done by `ascli` using the `as` key.
|
5168
5361
|
|
5169
5362
|
The following optional additional helper keys are supported by `ascli`:
|
5170
5363
|
|
5171
|
-
- `with` : Recipient of shared folder. Can be a
|
5364
|
+
- `with` : Recipient of shared folder. Can be a username, a group name, or a workspace name. `ascli` will resolve the name to the proper type and ID in fields `access_type` and `access_id`. If the value is the empty string, then it declares the shared folder in the workspace (first action to do, see below).
|
5172
5365
|
- `link_name` : The name of the link file created in the user's home folder for private links.
|
5173
5366
|
- `as` : The name of the link file created in the user's home folder for admin shared folders.
|
5174
5367
|
|
@@ -5209,7 +5402,7 @@ ascli aoc admin node do 1234 perm folder_on_node create @json:'{"with":"group 1"
|
|
5209
5402
|
ascli aoc admin node do 1234 perm folder_on_node create @json:'{"with":"my ws","as":"folder_for_users"}' --workspace="my ws"
|
5210
5403
|
```
|
5211
5404
|
|
5212
|
-
> **Note:** The node can be conveniently identified using the **percent selector** instead of numerical
|
5405
|
+
> **Note:** The node can be conveniently identified using the **percent selector** instead of numerical ID.
|
5213
5406
|
|
5214
5407
|
#### Cross Organization transfers
|
5215
5408
|
|
@@ -5219,9 +5412,9 @@ Although optional, the creation of [Option Preset](#option-preset) is recommende
|
|
5219
5412
|
|
5220
5413
|
Procedure to send a file from org1 to org2:
|
5221
5414
|
|
5222
|
-
- Get access to Organization 1 and create
|
5415
|
+
- Get access to Organization 1 and create an [Option Preset](#option-preset): e.g. `org1`, for instance, use the [Wizard](#configuration-using-wizard)
|
5223
5416
|
- Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
5224
|
-
- Get access to Organization 2 and create
|
5417
|
+
- Get access to Organization 2 and create an [Option Preset](#option-preset): e.g. `org2`
|
5225
5418
|
- Check that access works and locate the destination folder `mydestfolder`
|
5226
5419
|
- Execute the following:
|
5227
5420
|
|
@@ -5232,18 +5425,18 @@ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | as
|
|
5232
5425
|
Explanation:
|
5233
5426
|
|
5234
5427
|
- `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
5235
|
-
- `files node_info /mydestfolder` generate transfer information including
|
5428
|
+
- `files node_info /mydestfolder` generate transfer information including Node API credential and root ID, suitable for the next command
|
5236
5429
|
- `--format=json` format the output in JSON (instead of default text table)
|
5237
5430
|
- `--display=data` display only the result, and remove other information, such as workspace name
|
5238
5431
|
- `|` the standard output of the first command is fed into the second one
|
5239
5432
|
- `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
5240
5433
|
- `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
5241
5434
|
- `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agent-direct)
|
5242
|
-
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e.
|
5435
|
+
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. Node API credentials, those are expected in JSON format and read from standard input
|
5243
5436
|
|
5244
5437
|
#### Find Files
|
5245
5438
|
|
5246
|
-
The command `aoc files find` allows
|
5439
|
+
The command `aoc files find` allows searching for files in a given workspace.
|
5247
5440
|
|
5248
5441
|
It works also on `node` resource using the `v4` command:
|
5249
5442
|
|
@@ -5283,6 +5476,7 @@ admin dropbox_membership list
|
|
5283
5476
|
admin group list
|
5284
5477
|
admin kms_profile list
|
5285
5478
|
admin node do %name:my_node_name --secret=my_ak_secret browse /
|
5479
|
+
admin node do %name:my_node_name --secret=my_ak_secret browse /folder_sub --node-cache=no
|
5286
5480
|
admin node do %name:my_node_name --secret=my_ak_secret delete /folder1
|
5287
5481
|
admin node do %name:my_node_name --secret=my_ak_secret delete /folder_sub
|
5288
5482
|
admin node do %name:my_node_name --secret=my_ak_secret mkdir /folder1
|
@@ -5326,13 +5520,13 @@ files browse / --url=my_public_link_folder_pass --password=my_public_link_passwo
|
|
5326
5520
|
files browse my_remote_file
|
5327
5521
|
files browse my_remote_folder
|
5328
5522
|
files browse my_remote_folder/
|
5523
|
+
files cat --to-folder=. testdst/test_file.bin
|
5329
5524
|
files delete /testsrc
|
5330
5525
|
files download --transfer=connect testdst/test_file.bin
|
5331
5526
|
files download --transfer=desktop testdst/test_file.bin
|
5332
5527
|
files find /
|
5333
5528
|
files find / '\.partial$'
|
5334
5529
|
files find / @ruby:'->(f){f["type"].eql?("file")}'
|
5335
|
-
files http_node_download --to-folder=. testdst/test_file.bin
|
5336
5530
|
files mkdir /testsrc
|
5337
5531
|
files modify my_test_folder
|
5338
5532
|
files permission my_test_folder list
|
@@ -5348,8 +5542,8 @@ files sync admin status --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","
|
|
5348
5542
|
files sync start --sync-info=@json:'{"name":"my_aoc_sync1","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
|
5349
5543
|
files sync start --sync-info=@json:'{"name":"my_aoc_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
|
5350
5544
|
files sync start --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/testdst","reset":true}]}'
|
5351
|
-
files sync start pull /
|
5352
|
-
files sync start pull /
|
5545
|
+
files sync start pull /data/local_sync /testdst --sync-info=@json:'{"reset":true,"quiet":false}'
|
5546
|
+
files sync start pull /data/local_sync /testdst --sync-info=@json:'{"sessions":[{"reset":true}]}'
|
5353
5547
|
files thumbnail my_test_folder/video_file.mpg
|
5354
5548
|
files thumbnail my_test_folder/video_file.mpg --query=@json:'{"text":true,"double":true}'
|
5355
5549
|
files transfer push /testsrc --to-folder=/testdst test_file.bin
|
@@ -5357,7 +5551,7 @@ files upload --to-folder=/ test_file.bin --url=my_public_link_folder_no_pass
|
|
5357
5551
|
files upload --to-folder=/testsrc test_file.bin
|
5358
5552
|
files upload --workspace=my_other_workspace --to-folder=my_other_folder test_file.bin --transfer=node --transfer-info=@json:@stdin:
|
5359
5553
|
files v3 info
|
5360
|
-
gateway --pid-file=pid_aocfxgw https://localhost:12345/aspera/faspex &
|
5554
|
+
gateway --pid-file=pid_aocfxgw @json:'{"url":"https://localhost:12345/aspera/faspex"}' &
|
5361
5555
|
org --url=my_public_link_recv_from_aoc_user
|
5362
5556
|
organization
|
5363
5557
|
packages browse package_id3 /
|
@@ -5397,12 +5591,12 @@ ATS is usable either :
|
|
5397
5591
|
|
5398
5592
|
- Or from an IBM Cloud subscription : `ascli ats` : use IBM Cloud API key authentication
|
5399
5593
|
|
5400
|
-
### IBM Cloud ATS : Creation of
|
5594
|
+
### IBM Cloud ATS : Creation of API key
|
5401
5595
|
|
5402
5596
|
This section is about using ATS with an IBM cloud subscription.
|
5403
5597
|
If you are using ATS as part of AoC, then authentication is through AoC, not IBM Cloud.
|
5404
5598
|
|
5405
|
-
First get your IBM Cloud
|
5599
|
+
First get your IBM Cloud API key. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
5406
5600
|
|
5407
5601
|
```bash
|
5408
5602
|
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
@@ -5430,7 +5624,7 @@ References:
|
|
5430
5624
|
|
5431
5625
|
<!-- spellchecker: enable -->
|
5432
5626
|
|
5433
|
-
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
5627
|
+
Then, to register the key by default for the `ats` plugin, create a preset. Execute:
|
5434
5628
|
|
5435
5629
|
```bash
|
5436
5630
|
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here
|
@@ -5477,7 +5671,7 @@ Those are directly the parameters expected by the [ATS API](https://developer.ib
|
|
5477
5671
|
|
5478
5672
|
### Misc. Examples
|
5479
5673
|
|
5480
|
-
Example: create access key on IBM Cloud (
|
5674
|
+
Example: create access key on IBM Cloud (Softlayer):
|
5481
5675
|
|
5482
5676
|
```bash
|
5483
5677
|
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"my_secret_here","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
@@ -5495,7 +5689,7 @@ Example: create access key on Azure SAS:
|
|
5495
5689
|
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
5496
5690
|
```
|
5497
5691
|
|
5498
|
-
|
5692
|
+
> **Note:** The blob name is mandatory after server address and before parameters, and that parameter `sr=c` is mandatory.
|
5499
5693
|
|
5500
5694
|
Example: create access key on Azure:
|
5501
5695
|
|
@@ -5623,13 +5817,13 @@ For file operation command (browse, delete), the Ruby SSH client library `Net::S
|
|
5623
5817
|
|
5624
5818
|
For a list of SSH client options, refer to the Ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html#method-c-start).
|
5625
5819
|
|
5626
|
-
|
5820
|
+
Among the 50 available SSH options:
|
5627
5821
|
|
5628
5822
|
- `verbose`
|
5629
5823
|
- `use_agent`
|
5630
5824
|
- `passphrase`
|
5631
5825
|
|
5632
|
-
By default the SSH library will check if a local `ssh-agent` is running.
|
5826
|
+
By default, the SSH library will check if a local `ssh-agent` is running.
|
5633
5827
|
|
5634
5828
|
On Linux, if you get an error message such as:
|
5635
5829
|
|
@@ -5643,7 +5837,7 @@ or on Windows:
|
|
5643
5837
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
5644
5838
|
```
|
5645
5839
|
|
5646
|
-
This means that your environment suggests
|
5840
|
+
This means that your environment suggests using an agent, but you don't have such an SSH agent running, then:
|
5647
5841
|
|
5648
5842
|
- Check env var: `SSH_AGENT_SOCK`
|
5649
5843
|
- Check your file: `$HOME/.ssh/config`
|
@@ -5663,12 +5857,12 @@ It is equivalent to setting both options `ssh_options.passphrase` and `ts.ssh_pr
|
|
5663
5857
|
### Other session channels for `server`
|
5664
5858
|
|
5665
5859
|
URL schemes `local` and `https` are also supported (mainly for testing purpose).
|
5666
|
-
(`--url=local
|
5860
|
+
(`--url=local:`, `--url=https://...`)
|
5667
5861
|
|
5668
5862
|
- `local` will execute `ascmd` locally, instead of using an SSH connection.
|
5669
5863
|
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
5670
5864
|
|
5671
|
-
As, most of the time, SSH is used, if
|
5865
|
+
As, most of the time, SSH is used, if a `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
5672
5866
|
|
5673
5867
|
### Examples: `server`
|
5674
5868
|
|
@@ -5680,9 +5874,9 @@ ascli server browse /aspera-test-dir-large
|
|
5680
5874
|
ascli server download /aspera-test-dir-large/200MB
|
5681
5875
|
```
|
5682
5876
|
|
5683
|
-
`initdemo` creates
|
5877
|
+
`initdemo` creates an [Option Preset](#option-preset) `demoserver` and set it as default for plugin `server`.
|
5684
5878
|
|
5685
|
-
If an SSH private key is used for authentication with a passphrase, the passphrase needs to be provided to both options: `ssh_options
|
5879
|
+
If an SSH private key is used for authentication with a passphrase, the passphrase needs to be provided to both options: `ssh_options` (for browsing) and `ts` (for transfers):
|
5686
5880
|
|
5687
5881
|
```bash
|
5688
5882
|
ascli server --url=ssh://_server_address_here_:33001 --username=_user_here_ --ssh_keys=_private_key_path_here_ --passphrase=_passphrase_here_
|
@@ -5690,13 +5884,13 @@ ascli server --url=ssh://_server_address_here_:33001 --username=_user_here_ --ss
|
|
5690
5884
|
|
5691
5885
|
## Plugin: `node`: IBM Aspera High Speed Transfer Server Node
|
5692
5886
|
|
5693
|
-
This plugin gives access to capabilities provided by the HSTS
|
5887
|
+
This plugin gives access to capabilities provided by the HSTS Node API.
|
5694
5888
|
|
5695
5889
|
The authentication is `username` and `password` or `access_key` and `secret` through options: `username` and `password`.
|
5696
5890
|
|
5697
|
-
> **Note:** Capabilities of this plugin are used in other plugins which access to the
|
5891
|
+
> **Note:** Capabilities of this plugin are used in other plugins which access to the Node API, such as `aoc`, `ats`, `shares`.
|
5698
5892
|
>
|
5699
|
-
> **Note:** This plugin can be used with any type of **Aspera Node**, either on
|
5893
|
+
> **Note:** This plugin can be used with any type of **Aspera Node**, either on premise or ATS, provided that you have Node API credentials. Those credentials can be either Node API user or Access Key (e.g. on ATS).
|
5700
5894
|
|
5701
5895
|
### File Operations
|
5702
5896
|
|
@@ -5735,7 +5929,7 @@ The command `find <folder> [filter_expr]` is available for **gen4/access key**,
|
|
5735
5929
|
The argument `<folder>` is mandatory and is the root from which search is performed.
|
5736
5930
|
The argument `[filter_expr]` is optional and represent the matching criteria.
|
5737
5931
|
|
5738
|
-
It recursively scans storage to find files/folders matching
|
5932
|
+
It recursively scans storage to find files/folders matching criteria and then returns a list of matching entries.
|
5739
5933
|
|
5740
5934
|
`[filter_expr]` is either:
|
5741
5935
|
|
@@ -5777,7 +5971,7 @@ The following are examples of Ruby lambda code to be provided in the following t
|
|
5777
5971
|
->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}
|
5778
5972
|
```
|
5779
5973
|
|
5780
|
-
- Find files larger than
|
5974
|
+
- Find files larger than 1 MB
|
5781
5975
|
|
5782
5976
|
```ruby
|
5783
5977
|
->(f){f["type"].eql?("file") and f["size"].to_i>1000000}
|
@@ -5805,7 +5999,7 @@ ascli node access_keys do self find / @ruby:'->(f){f["type"].eql?("file") and (D
|
|
5805
5999
|
|
5806
6000
|
### Listing transfer events
|
5807
6001
|
|
5808
|
-
When a transfer is run, its information is stored (
|
6002
|
+
When a transfer is run, its information is stored (typically, 1 day) in the HSTS database (Redis).
|
5809
6003
|
This information can be retrieved with command: `transfer list`.
|
5810
6004
|
|
5811
6005
|
If the number of transfers is too large, then the list will be retrieved using several API calls.
|
@@ -5816,14 +6010,14 @@ In addition, it is possible to list "only new information" using option `once_on
|
|
5816
6010
|
ascli node transfer list --once-only=yes
|
5817
6011
|
```
|
5818
6012
|
|
5819
|
-
The `iteration_token` that keeps memory of latest event is stored in the
|
6013
|
+
The `iteration_token` that keeps memory of the latest event is stored in the persistence repository of `ascli`.
|
5820
6014
|
To reset it, add option: `--query=@json:'{"reset": true}'`.
|
5821
6015
|
To list only a number of events, use the `max` parameter in query.
|
5822
6016
|
Other parameters are directly transmitted to the underlying API (`GET /ops/transfers`).
|
5823
6017
|
|
5824
6018
|
### Central
|
5825
6019
|
|
5826
|
-
The central
|
6020
|
+
The central sub-command uses the **reliable query** API (session and file).
|
5827
6021
|
It allows listing transfer sessions and transferred files.
|
5828
6022
|
|
5829
6023
|
Filtering can be applied:
|
@@ -5841,12 +6035,12 @@ By providing the `validator` option, offline transfer validation can be done.
|
|
5841
6035
|
There are three sync types of operations:
|
5842
6036
|
|
5843
6037
|
- `sync`: perform a local sync, by executing `async` locally
|
5844
|
-
- `async`: calls legacy async API on node : `/async`
|
5845
|
-
- `ssync` : calls newer async API on node : `/asyncs`
|
6038
|
+
- `async`: calls legacy `async` API on node : `/async`
|
6039
|
+
- `ssync` : calls newer `async` API on node : `/asyncs`
|
5846
6040
|
|
5847
6041
|
### FASP Stream
|
5848
6042
|
|
5849
|
-
It is possible to start a
|
6043
|
+
It is possible to start a faspstream session using the Node API:
|
5850
6044
|
|
5851
6045
|
Use the command `ascli node stream create --ts=@json:<value>`, with [**transfer-spec**](#transfer-specification):
|
5852
6046
|
|
@@ -5858,10 +6052,10 @@ Use the command `ascli node stream create --ts=@json:<value>`, with [**transfer-
|
|
5858
6052
|
|
5859
6053
|
Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/entsrv/3.7.4/es_admin_linux/webhelp/index.html#watchfolder_external/dita/json_conf.html) for watch folder creation.
|
5860
6054
|
|
5861
|
-
`ascli` supports remote operations through the
|
6055
|
+
`ascli` supports remote operations through the Node API. Operations are:
|
5862
6056
|
|
5863
6057
|
- Start `watchd` and `watchfolderd` services running as a system user having access to files
|
5864
|
-
- Configure a **
|
6058
|
+
- Configure a **Watchfolder** to define automated transfers
|
5865
6059
|
|
5866
6060
|
```bash
|
5867
6061
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -5901,7 +6095,7 @@ updated
|
|
5901
6095
|
|
5902
6096
|
Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner.
|
5903
6097
|
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
|
5904
|
-
Simply create an **Aspera Transfer Service** instance, which provides access to the
|
6098
|
+
Simply create an **Aspera Transfer Service** instance, which provides access to the Node API.
|
5905
6099
|
Then create a configuration for the **SHOD** instance in the configuration file: in section **shares**, a configuration named: `aws_shod`.
|
5906
6100
|
Create another configuration for the Azure ATS instance: in section **node**, named `azure_ats`.
|
5907
6101
|
Then execute the following command:
|
@@ -5914,7 +6108,7 @@ This will get transfer information from the SHOD instance and tell the Azure ATS
|
|
5914
6108
|
|
5915
6109
|
### Node file information
|
5916
6110
|
|
5917
|
-
When
|
6111
|
+
When Node API is used with an **Access key**, extra information can be retrieved, such as preview.
|
5918
6112
|
|
5919
6113
|
> **Note:** Display of preview on terminal requires installation of extra gem: `rmagick`
|
5920
6114
|
|
@@ -5935,7 +6129,7 @@ Previews are mainly used in AoC, this also works with AoC:
|
|
5935
6129
|
ascli aoc files thumbnail /preview_samples/Aspera.mpg
|
5936
6130
|
```
|
5937
6131
|
|
5938
|
-
> **Note:** To specify the file by its file
|
6132
|
+
> **Note:** To specify the file by its file ID, use the selector syntax: `%id:_file_id_here_`
|
5939
6133
|
>
|
5940
6134
|
> **Note:** To force textual display of the preview on **iTerm**, prefix command with: `env -u TERM_PROGRAM -u LC_TERMINAL` or use option: ``
|
5941
6135
|
|
@@ -5946,7 +6140,7 @@ ascli node access_key create @json:'{"id":"myaccesskey","secret":"my_secret_here
|
|
5946
6140
|
```
|
5947
6141
|
|
5948
6142
|
> **Note:** The `id` and `secret` are optional.
|
5949
|
-
> If not provided, they will be generated and returned
|
6143
|
+
> If not provided, they will be generated and returned into the result.
|
5950
6144
|
|
5951
6145
|
Access keys support extra overriding parameters using parameter: `configuration` and sub keys `transfer` and `server`. For example, an access key can be modified or created with the following options:
|
5952
6146
|
|
@@ -5966,27 +6160,27 @@ Bearer tokens are part of the **gen4/access key** API.
|
|
5966
6160
|
It follows the model of OAuth 2.
|
5967
6161
|
For example, they are used in Aspera on Cloud.
|
5968
6162
|
This is also available for developers for any application integrating Aspera.
|
5969
|
-
In this API, files, users and groups are identified by an
|
6163
|
+
In this API, files, users and groups are identified by an ID (a `String`, e.g. `"125"`, not necessarily numerical).
|
5970
6164
|
|
5971
|
-
Bearer tokens are typically generated by the
|
6165
|
+
Bearer tokens are typically generated by the authenticating application and then recognized by the Node API.
|
5972
6166
|
A bearer token is authorized on the node by creating `permissions` on a **folder**.
|
5973
6167
|
|
5974
|
-
Bearer tokens can be generated using command `bearer_token`: it takes two arguments:
|
6168
|
+
Bearer tokens can be generated using `ascli` command `bearer_token`: it takes two arguments:
|
5975
6169
|
|
5976
|
-
- The private key used to sign the token
|
6170
|
+
- The private key used to sign the token.
|
5977
6171
|
- The token information, which is a `Hash` containing the following elements:
|
5978
6172
|
|
5979
|
-
|
|
6173
|
+
| Parameter | Default | Type | Description |
|
5980
6174
|
| ------------------------ |-----------------------------|-----------|----------------------------------------------------------|
|
5981
|
-
| _scope
|
5982
|
-
| _validity
|
5983
|
-
| user_id
|
5984
|
-
| scope
|
5985
|
-
| expires_at
|
5986
|
-
| auth_type
|
5987
|
-
| group_ids
|
5988
|
-
| organization_id
|
5989
|
-
| watermarking_json_base64
|
6175
|
+
| `_scope` | `user:all` | Special | Either `user:all` or `admin:all` |
|
6176
|
+
| `_validity` | 86400 | Special | Validity in seconds from now. |
|
6177
|
+
| `user_id` | - | Mandatory | Identifier of user |
|
6178
|
+
| `scope` | `node.<access_key>:<_scope>`| Mandatory | API scope, e.g. `node.<access_key>:<node scope>` |
|
6179
|
+
| `expires_at` | `now+<_validity>` | Mandatory | Format: `%Y-%m-%dT%H:%M:%SZ` e.g. `2021-12-31T23:59:59Z` |
|
6180
|
+
| `auth_type` | `access_key` | Optional | `access_key`, `node_user` |
|
6181
|
+
| `group_ids` | - | Optional | List of group IDs |
|
6182
|
+
| `organization_id` | - | Optional | Organization ID |
|
6183
|
+
| `watermarking_json_base64` | - | Optional | Watermarking information (not used) |
|
5990
6184
|
|
5991
6185
|
> **Note:** For convenience, `ascli` provides additional parameters `_scope` and `_validity`.
|
5992
6186
|
> They are not part of the API and are removed from the final payload.
|
@@ -5995,7 +6189,7 @@ Bearer tokens can be generated using command `bearer_token`: it takes two argume
|
|
5995
6189
|
#### Bearer token: Environment
|
5996
6190
|
|
5997
6191
|
An access key shall be created to grant access for transfers to its storage.
|
5998
|
-
The
|
6192
|
+
The access key and its secret represent administrative access to the storage as it has access rights to the whole storage of the access key.
|
5999
6193
|
|
6000
6194
|
They way to create access keys depend slightly on the type of HSTS:
|
6001
6195
|
|
@@ -6007,36 +6201,40 @@ They way to create access keys depend slightly on the type of HSTS:
|
|
6007
6201
|
|
6008
6202
|
- If Aspera on Cloud or ATS is used, then the SaaS API for access key creation is used.
|
6009
6203
|
|
6204
|
+
> **Note:** Refer to [HSTS manual](https://www.ibm.com/docs/en/ahts): `Access key authentication` section for more details on access key creation.
|
6205
|
+
|
6206
|
+
In the next sections, we will assume that an access key has been created and that `ascli` is configured to use this access key by default using `node`.
|
6207
|
+
|
6010
6208
|
#### Bearer token: Preparation
|
6011
6209
|
|
6012
6210
|
Let's assume that the access key was created, and a default configuration is set to use this **access key**.
|
6013
6211
|
Using `ascli`, an access key can be created using the `access_key create` on the node (using main node credentials) or ATS.
|
6014
6212
|
|
6015
|
-
|
6213
|
+
Create a private key (organization key) that will be used to sign bearer tokens:
|
6016
6214
|
|
6017
|
-
|
6018
|
-
|
6019
|
-
|
6020
|
-
|
6215
|
+
```bash
|
6216
|
+
my_private_pem=./myorgkey.pem
|
6217
|
+
ascli config genkey $my_private_pem
|
6218
|
+
```
|
6021
6219
|
|
6022
|
-
|
6023
|
-
|
6220
|
+
> **Note:** This key is not used for authentication, it is used to sign bearer tokens.
|
6221
|
+
> Refer to section [private key](#private-key) for more details on generation.
|
6024
6222
|
|
6025
|
-
|
6223
|
+
The corresponding public key shall be placed as an attribute of the **access key** (done with `PUT /access_keys/<id>`):
|
6026
6224
|
|
6027
|
-
|
6028
|
-
|
6029
|
-
|
6225
|
+
```bash
|
6226
|
+
ascli node access_key set_bearer_key self @file:$my_private_pem
|
6227
|
+
```
|
6030
6228
|
|
6031
|
-
|
6032
|
-
|
6033
|
-
|
6229
|
+
> **Note:** Either the public or private key can be provided, and only the public key is used.
|
6230
|
+
> This will enable to check the signature of the bearer token.
|
6231
|
+
> Above command is executed with access key credentials.
|
6034
6232
|
|
6035
|
-
|
6233
|
+
Alternatively, use the following equivalent command, as `ascli` kindly extracts the public key with extension `.pub`:
|
6036
6234
|
|
6037
|
-
|
6038
|
-
|
6039
|
-
|
6235
|
+
```bash
|
6236
|
+
ascli node access_key modify %id:self @ruby:'{token_verification_key: File.read("'$my_private_pem'.pub")}'
|
6237
|
+
```
|
6040
6238
|
|
6041
6239
|
#### Bearer token: Configuration for user
|
6042
6240
|
|
@@ -6048,14 +6246,14 @@ Using `ascli`, an access key can be created using the `access_key create` on the
|
|
6048
6246
|
|
6049
6247
|
> **Note:** Here we simply select `/`, but any folder can be selected in the access key storage.
|
6050
6248
|
|
6051
|
-
- Let's designate a user by its
|
6249
|
+
- Let's designate a user by its ID:
|
6052
6250
|
|
6053
6251
|
```bash
|
6054
6252
|
my_user_id=777
|
6055
6253
|
```
|
6056
6254
|
|
6057
6255
|
> **Note:** This is an arbitrary identifier, typically managed by the web application.
|
6058
|
-
> Not related to Linux user
|
6256
|
+
> Not related to Linux user IDs or anything else.
|
6059
6257
|
|
6060
6258
|
- Grant this user access to the selected folder:
|
6061
6259
|
|
@@ -6069,18 +6267,20 @@ Using `ascli`, an access key can be created using the `access_key create` on the
|
|
6069
6267
|
ascli node bearer_token @file:./myorgkey.pem @json:'{"user_id":"'$my_user_id'","_validity":3600}' --output=bearer.txt
|
6070
6268
|
```
|
6071
6269
|
|
6270
|
+
> **Note:** The Bearer token can also be created using command `asnodeadmin` on HSTS. Refer to the [HSTS manual](https://www.ibm.com/docs/en/ahts): `Bearer tokens` section. Code for token generation is provided in [lib/aspera/api/node.rb](lib/aspera/api/node.rb)
|
6271
|
+
|
6072
6272
|
#### Bearer token: User side
|
6073
6273
|
|
6074
6274
|
Now, let's assume we are the user, the only information received are:
|
6075
6275
|
|
6076
|
-
- The URL of the
|
6276
|
+
- The URL of the Node API
|
6077
6277
|
- A Bearer token
|
6078
|
-
- A file
|
6278
|
+
- A file ID for which we have access
|
6079
6279
|
|
6080
6280
|
Let's use it:
|
6081
6281
|
|
6082
6282
|
```bash
|
6083
|
-
ascli node -N --url
|
6283
|
+
ascli node -N --url=https://... --password="Bearer $(cat bearer.txt)" --root-id=$my_folder_id access_key do self br /
|
6084
6284
|
```
|
6085
6285
|
|
6086
6286
|
### Node sample commands
|
@@ -6120,6 +6320,7 @@ async show ALL
|
|
6120
6320
|
basic_token
|
6121
6321
|
bearer_token @file:my_private_key @json:'{"user_id":"666"}' --output=bearer_666
|
6122
6322
|
browse / --log-level=trace2
|
6323
|
+
cat my_upload_folder/test_file.bin --to-folder=.
|
6123
6324
|
central file list
|
6124
6325
|
central file modify --validator=1 @json:'{"files":[]}'
|
6125
6326
|
central session list
|
@@ -6127,7 +6328,6 @@ delete @list:,my_upload_folder/a_folder,my_upload_folder/tdlink,my_upload_folder
|
|
6127
6328
|
delete my_upload_folder/test_file.bin
|
6128
6329
|
download my_upload_folder/test_file.bin --to-folder=.
|
6129
6330
|
health
|
6130
|
-
http_node_download my_upload_folder/test_file.bin --to-folder=.
|
6131
6331
|
info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
|
6132
6332
|
license
|
6133
6333
|
mkdir my_upload_folder/a_folder
|
@@ -6167,6 +6367,32 @@ upload test_file.bin --to-folder=my_upload_folder --ts=@json:'{"target_rate_cap_
|
|
6167
6367
|
watch_folder list
|
6168
6368
|
```
|
6169
6369
|
|
6370
|
+
### Open Telemetry
|
6371
|
+
|
6372
|
+
The Node plugin supports Open Telemetry (OTel) for monitoring and tracing.
|
6373
|
+
|
6374
|
+
`ascli` can poll the Node API for transfer events and send them to an OTel collector.
|
6375
|
+
|
6376
|
+
The command expects the following parameters provided as a `Hash` positional parameter:
|
6377
|
+
|
6378
|
+
| Parameter | Type | Default | Description |
|
6379
|
+
|-------------|----------|---------|---------------------------------|
|
6380
|
+
| `url` | `String` | - | URL of the Instana backend. |
|
6381
|
+
| `apikey` | `String` | - | Token for the OTel collector. |
|
6382
|
+
| `interval` | `Float` | 10 | Polling interval in seconds. |
|
6383
|
+
|
6384
|
+
For convenience, those parameters can be provided in a preset, e.g. `otel_default`.
|
6385
|
+
|
6386
|
+
```bash
|
6387
|
+
ascli config preset init otel_default @json:'{"url":"https://otlp-orange-saas.instana.io:4318","apikey":"*********","interval":1.1}'
|
6388
|
+
```
|
6389
|
+
|
6390
|
+
Then it is invoked like this (assuming a default node is configured):
|
6391
|
+
|
6392
|
+
```bash
|
6393
|
+
ascli node telemetry @preset:otel_default
|
6394
|
+
```
|
6395
|
+
|
6170
6396
|
## Plugin: `faspex5`: IBM Aspera Faspex v5
|
6171
6397
|
|
6172
6398
|
IBM Aspera's newer self-managed application.
|
@@ -6182,7 +6408,7 @@ IBM Aspera's newer self-managed application.
|
|
6182
6408
|
|
6183
6409
|
> **Note:** If you have a Faspex 5 public link, provide it, unchanged, through the option `url`
|
6184
6410
|
|
6185
|
-
For a quick start, one can use the wizard, which will help
|
6411
|
+
For a quick start, one can use the wizard, which will help to create an [Option Preset](#option-preset):
|
6186
6412
|
|
6187
6413
|
```bash
|
6188
6414
|
ascli config wizard
|
@@ -6204,7 +6430,7 @@ option: key_path>
|
|
6204
6430
|
Using existing key:
|
6205
6431
|
/Users/someuser/.aspera/ascli/my_key
|
6206
6432
|
option: username> someuser@example.com
|
6207
|
-
Ask the ascli client
|
6433
|
+
Ask the ascli client ID and secret to your Administrator.
|
6208
6434
|
Admin should login to: https://faspex5.example.com/aspera/faspex
|
6209
6435
|
Navigate to: :: → Admin → Configurations → API clients
|
6210
6436
|
Create an API client with:
|
@@ -6243,11 +6469,11 @@ Activation is in two steps:
|
|
6243
6469
|
- As Admin, Navigate to the web UI: Admin → Configurations → API Clients → Create
|
6244
6470
|
- Give a name, like `ascli`
|
6245
6471
|
- Activate JWT
|
6246
|
-
- There is an option to set a
|
6472
|
+
- There is an option to set a global public key allowing the owner of the private key to impersonate any user. Unless you want to do this, leave this field empty.
|
6247
6473
|
- Click on `Create` Button
|
6248
|
-
- Take note of Client
|
6474
|
+
- Take note of Client ID (and Client Secret, but not used in current version)
|
6249
6475
|
|
6250
|
-
- The user will authenticate with a private key and set the public key in his
|
6476
|
+
- The user will authenticate with a private key and set the public key in his Faspex 5 profile.
|
6251
6477
|
|
6252
6478
|
**Note:** If you don't have a private key refer to section [Private Key](#private-key) to generate one.
|
6253
6479
|
|
@@ -6289,7 +6515,7 @@ The administrator must create an API client in Faspex for an external web app su
|
|
6289
6515
|
- Do not Activate JWT
|
6290
6516
|
- Set **Redirect URI** to `https://127.0.0.1:8888`
|
6291
6517
|
- Click on `Create` Button
|
6292
|
-
- Take note of Client Id (and Client Secret, but not used in current version)
|
6518
|
+
- Take note of the Client Id (and Client Secret, but not used in current version)
|
6293
6519
|
|
6294
6520
|
The user will use the following options:
|
6295
6521
|
|
@@ -6351,7 +6577,7 @@ admin smtp show
|
|
6351
6577
|
admin smtp test my_email_external
|
6352
6578
|
admin workgroups list
|
6353
6579
|
bearer_token
|
6354
|
-
gateway --pid-file=pid_f5_fxgw https://localhost:12346/aspera/faspex &
|
6580
|
+
gateway --pid-file=pid_f5_fxgw @json:'{"url":"https://localhost:12346/aspera/faspex"}' &
|
6355
6581
|
health
|
6356
6582
|
invitation list
|
6357
6583
|
invitations create @json:'{"email_address":"aspera.user1+u@gmail.com"}'
|
@@ -6365,7 +6591,7 @@ packages receive --box=my_workgroup --group-type=workgroups workgroup_package_id
|
|
6365
6591
|
packages receive ALL --once-only=yes --to-folder=.
|
6366
6592
|
packages receive INIT --once-only=yes
|
6367
6593
|
packages receive f5_pack_id --to-folder=. --ts=@json:'{"content_protection_password":"my_secret_here"}'
|
6368
|
-
packages send --shared-folder=%name:my_shared_folder_name @json:'{"title":"test title","recipients":["my_email_internal"]}' my_shared_folder_file --fields=id --display=data --output=
|
6594
|
+
packages send --shared-folder=%name:my_shared_folder_name @json:'{"title":"test title","recipients":["my_email_internal"]}' my_shared_folder_file --fields=id --display=data --output=f5_pack_id
|
6369
6595
|
packages send --url=my_public_link_send_f5_user @json:'{"title":"test title"}' test_file.bin
|
6370
6596
|
packages send --url=my_public_link_send_shared_box @json:'{"title":"test title"}' test_file.bin
|
6371
6597
|
packages send @json:'{"title":"test title","recipients":["my_shared_box_name"],"metadata":{"Options":"Opt1","TextInput":"example text"}}' test_file.bin
|
@@ -6375,7 +6601,7 @@ packages show --box=my_shared_box_name package_box_id1
|
|
6375
6601
|
packages show --box=my_workgroup --group-type=workgroups workgroup_package_id1
|
6376
6602
|
packages show f5_pack_id
|
6377
6603
|
packages status f5_pack_id
|
6378
|
-
postprocessing --pid-file=pid_f5_postproc @json:'{"url":"
|
6604
|
+
postprocessing --pid-file=pid_f5_postproc @json:'{"url":"https://localhost:8553/asclihook","script_folder":"","cert":"localhost.p12","key":"changeit"}' &
|
6379
6605
|
shared browse %name:my_src
|
6380
6606
|
shared list
|
6381
6607
|
shared_folders browse %name:my_shared_folder_name
|
@@ -6428,7 +6654,7 @@ Example (assuming a default preset is created for the connection information):
|
|
6428
6654
|
ascli faspex5 packages send @json:'{"title":"some title","recipients":["user@example.com"]}' mybygfile1
|
6429
6655
|
```
|
6430
6656
|
|
6431
|
-
Longer example for the
|
6657
|
+
Longer example for the payload of `@json:`:
|
6432
6658
|
|
6433
6659
|
```json
|
6434
6660
|
{"title":"some title","recipients":[{"recipient_type":"user","name":"user@example.com"}]}
|
@@ -6445,10 +6671,10 @@ Longer example for the ayload of `@json:`:
|
|
6445
6671
|
`ascli` adds some convenience:
|
6446
6672
|
The API expects the field `recipients` to be an `Array` of `Hash`, each with field `name` and optionally `recipient_type`.
|
6447
6673
|
`ascli` also accepts an `Array` of `String`, with simply a recipient name.
|
6448
|
-
Then, `ascli` will
|
6674
|
+
Then, `ascli` will look up existing contacts among all possible types, use it if a single match is found, and set the `name` and `recipient_type` accordingly.
|
6449
6675
|
Else an exception is sent.
|
6450
6676
|
|
6451
|
-
> **Note:** The lookup is case
|
6677
|
+
> **Note:** The lookup is case-insensitive and on partial matches.
|
6452
6678
|
|
6453
6679
|
```json
|
6454
6680
|
{"title":"some title","recipients":["user@example.com"]}
|
@@ -6477,11 +6703,11 @@ Option `box` can be used to list packages from a specific box (see [Inbox Select
|
|
6477
6703
|
|
6478
6704
|
Option `query` can be used to filter the list of packages, based on native API parameters, directly sent to [Faspex 5 API `GET /packages`](https://developer.ibm.com/apis/catalog/aspera--ibm-aspera-faspex-5-0-api/api/API--aspera--ibm-aspera-faspex-api#getAllPackages).
|
6479
6705
|
|
6480
|
-
|
|
6706
|
+
| Parameter | Type | Description |
|
6481
6707
|
|-----------|---------|-------------|
|
6482
6708
|
| `offset` | Native | Managed by `ascli`: Offset of first package. Default: 0 |
|
6483
6709
|
| `limit` | Native | Managed by `ascli`: # of packages per API call. Default: 100 |
|
6484
|
-
| `q` | Native | General search string (case
|
6710
|
+
| `q` | Native | General search string (case-insensitive, matches if value is contained in several fields) |
|
6485
6711
|
| ... | Native | Other native parameters are supported (Refer to API documentation) |
|
6486
6712
|
| `max` | Special | Maximum number of items to retrieve (stop pages when the maximum is passed) |
|
6487
6713
|
| `pmax` | Special | Maximum number of pages to request (stop pages when the maximum is passed) |
|
@@ -6504,32 +6730,43 @@ Examples:
|
|
6504
6730
|
ascli faspex5 packages list @ruby:'->(p){p["state"].eql?("released")}'
|
6505
6731
|
```
|
6506
6732
|
|
6507
|
-
### Faspex 5:
|
6733
|
+
### Faspex 5: Browsing folder content
|
6508
6734
|
|
6509
|
-
|
6735
|
+
Several entities support folder browsing: Packages, Nodes, Shared Folders.
|
6736
|
+
All support two modes: paging and legacy API.
|
6737
|
+
By default, paging is used.
|
6510
6738
|
|
6511
|
-
|
6739
|
+
Option `query` is available with parameters supported by the API and `ascli` :
|
6512
6740
|
|
6513
|
-
|
6741
|
+
| Parameter | Evaluation | Description |
|
6742
|
+
|--------------|--------------|-------------|
|
6743
|
+
| `paging` | `ascli` | Use paging API. Default: `true` |
|
6744
|
+
| `recursive` | `ascli` | List inside folders. Default: `false` |
|
6745
|
+
| `max` | `ascli` | Maximum number of items. |
|
6746
|
+
| `filter` | API | Refer to API doc. Default: `{"basenames":[]}` |
|
6747
|
+
| `offset` | API (legacy) | Index of first item. Default: `0` |
|
6748
|
+
| `limit` | API (legacy) | Number of items in one API call result. Default: `500` |
|
6749
|
+
| `per_page` | API (paging) | Number of items in one API call result. Default: `500` |
|
6514
6750
|
|
6515
|
-
|
6751
|
+
### Faspex 5: Content of a received Package
|
6516
6752
|
|
6517
|
-
> **Note:**
|
6753
|
+
> **Note:** Listing content also applies to sent packages using `--box=outbox`.
|
6518
6754
|
|
6519
|
-
To
|
6755
|
+
To list the content of a received package, use command `faspex5 packages browse <package id>`.
|
6756
|
+
Optionally, provide a folder path.
|
6520
6757
|
|
6521
6758
|
### Faspex 5: Receive a package
|
6522
6759
|
|
6523
6760
|
To receive one, or several packages at once, use command `faspex5 packages receive`.
|
6524
|
-
Provide either a single package
|
6761
|
+
Provide either a single package ID, or an extended value `Array` of package IDs, e.g. `@list:,1,2,3` as argument.
|
6525
6762
|
|
6526
6763
|
The same options as for `faspex5 packages list` can be used to select the box and filter the packages to download.
|
6527
6764
|
I.e. options `box` and `query`, as well as last **Command Parameter** `Proc` (filter).
|
6528
6765
|
|
6529
6766
|
Option `--once-only=yes` can be used, for "cargo-like" behavior.
|
6530
|
-
Special package
|
6767
|
+
Special package ID `INIT` initializes the persistency of already received packages when option `--once-only=yes` is used.
|
6531
6768
|
|
6532
|
-
Special package
|
6769
|
+
Special package ID `ALL` selects all packages (of the selected box).
|
6533
6770
|
In this case, typically, only `completed` packages should be downloaded, so use option `--query=@json:'{"status":"completed"}'`.
|
6534
6771
|
|
6535
6772
|
If a package is password protected, then the content protection password is asked interactively.
|
@@ -6654,7 +6891,7 @@ To start package purge, i.e. permanently remove packages marked for deletion old
|
|
6654
6891
|
ascli faspex5 admin clean_deleted
|
6655
6892
|
```
|
6656
6893
|
|
6657
|
-
> **Note:** The expiration
|
6894
|
+
> **Note:** The expiration period taken by default is the one from `admin configuration show`. To use a different period than the default, specify it on command line with: `@json:'{"days_before_deleting_package_records":15}'`
|
6658
6895
|
|
6659
6896
|
To delete all packages, one can use the following command:
|
6660
6897
|
|
@@ -6667,7 +6904,7 @@ ascli faspex5 packages list --box=ALL --format=yaml --fields=id | ascli faspex5
|
|
6667
6904
|
|
6668
6905
|
### Faspex 5: Admin: Unlock user
|
6669
6906
|
|
6670
|
-
To unlock a user, you can
|
6907
|
+
To unlock a user, you can deactivate and then re-activate the user:
|
6671
6908
|
|
6672
6909
|
```bash
|
6673
6910
|
ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":false}'
|
@@ -6677,89 +6914,83 @@ ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account
|
|
6677
6914
|
ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":true}'
|
6678
6915
|
```
|
6679
6916
|
|
6680
|
-
> **Note:**
|
6917
|
+
> **Note:** Here we use the convenient percent selector, but the numerical ID can be used as well.
|
6681
6918
|
|
6682
6919
|
To send a password reset link to a user, use command `reset_password` on the `account`.
|
6683
6920
|
|
6684
|
-
### Faspex 5: Faspex 4-style
|
6921
|
+
### Faspex 5: Faspex 4-style post-processing
|
6685
6922
|
|
6686
|
-
The command
|
6687
|
-
It implements Faspex 5
|
6688
|
-
Environment variables at set to the values provided by the web hook which are the same as Faspex 4
|
6923
|
+
The command `ascli faspex5 postprocessing` emulates Faspex 4 post-processing script execution in Faspex 5.
|
6924
|
+
It implements a web hook for Faspex 5 and calls a script with the same environment variables as set by Faspex 4.
|
6925
|
+
Environment variables at set to the values provided by the web hook which are the same as Faspex 4 post-processing.
|
6689
6926
|
|
6690
|
-
It allows to quickly migrate workflows to Faspex 5 while preserving scripts.
|
6691
|
-
Nevertheless, on long term, a native approach shall be considered, such as using Aspera Orchestrator or other workflow engine.
|
6927
|
+
It allows to quickly migrate workflows from Faspex 4 to Faspex 5 while preserving scripts.
|
6928
|
+
Nevertheless, on long term, a native approach shall be considered, such as using Aspera Orchestrator or other workflow engine, using Faspex 5 native web hooks or File Processing.
|
6692
6929
|
|
6693
6930
|
It is invoked like this:
|
6694
6931
|
|
6695
6932
|
```bash
|
6696
|
-
ascli faspex5 postprocessing
|
6933
|
+
ascli faspex5 postprocessing
|
6697
6934
|
```
|
6698
6935
|
|
6699
|
-
|
6936
|
+
An optional positional parameter can be provided as extended value `Hash`:
|
6700
6937
|
|
6701
|
-
|
|
6702
|
-
|
6703
|
-
|
|
6704
|
-
|
|
6705
|
-
|
|
6706
|
-
|
|
6707
|
-
| certificate.chain | `String` | nil | Path to certificate chain |
|
6708
|
-
| processing | `Hash` | nil | Behavior of post processing |
|
6709
|
-
| processing.script_folder | `String` | . | Prefix added to script path |
|
6710
|
-
| processing.fail_on_error | `Bool` | false | Fail if true and process exits with non zero code |
|
6711
|
-
| processing.timeout_seconds | `Integer`| 60 | Max. execution time before script is killed |
|
6938
|
+
| Parameter | Type | Default | Description |
|
6939
|
+
|-------------------|----------|-------|-----------------------------------------------------|
|
6940
|
+
| **server info** | - | - | See [Web service](#web-service). |
|
6941
|
+
| `script_folder` | `String` | `.` | Prefix added to script path (Default: CWD) |
|
6942
|
+
| `fail_on_error` | `Bool` | false | Fail if true and process exits with non-zero code |
|
6943
|
+
| `timeout_seconds` | `Integer`| 60 | Time out before script is killed |
|
6712
6944
|
|
6713
|
-
|
6714
|
-
|
6715
|
-
- If `http` or `https` is used
|
6716
|
-
- The local port number (default 443 for HTTPS, 80 for HTTP)
|
6717
|
-
- The **base path**, i.e. the path under which requests are received, if a reverse proxy is used this can be used to route.
|
6718
|
-
|
6719
|
-
When a request is received the following happens:
|
6945
|
+
When a request on `ascli` is received the following happens:
|
6720
6946
|
|
6721
6947
|
- `ascli` gets the path of the URL called
|
6722
6948
|
- It removes the **base path** of base URL.
|
6723
6949
|
- It prepends it with the value of `script_folder`
|
6724
|
-
- It executes the script
|
6950
|
+
- It executes the script at that path
|
6725
6951
|
- Upon success, a success code is returned
|
6726
6952
|
|
6727
6953
|
For example:
|
6728
6954
|
|
6729
|
-
|
6730
|
-
|
6955
|
+
```bash
|
6956
|
+
ascli faspex5 postprocessing @json:'{"url":"http://localhost:8080/processing","script_folder":"/opt/scripts"}'
|
6957
|
+
```
|
6731
6958
|
|
6732
6959
|
In Faspex 5, the URL of the webhook endpoint shall be reachable from within Faspex containers.
|
6733
6960
|
For example, if `ascli` in running in the base host, the URL hostname shall not be localhost, as this refers to the local address inside Faspex container.
|
6734
|
-
Instead, one can specify the IP address of the host or `host.containers.internal
|
6961
|
+
Instead, one can specify the **IP address of the host** or `host.containers.internal` (Check `podman` manual).
|
6962
|
+
|
6963
|
+
Let's define the web hook:
|
6735
6964
|
|
6736
6965
|
**Webhook endpoint URI** : `http://host.containers.internal:8080/processing/script1.sh`
|
6737
6966
|
|
6738
|
-
Then the
|
6967
|
+
Then the post-processing script executed will be `/opt/scripts/script1.sh`.
|
6739
6968
|
|
6740
6969
|
### Faspex 5: Faspex 4 Gateway
|
6741
6970
|
|
6742
6971
|
> **Note:** This is not a feature for production. It's provided for testing only.
|
6743
6972
|
|
6744
|
-
For legacy
|
6973
|
+
For legacy Faspex client applications that use the `send` API (only) of Faspex v4, the command `gateway` provides the capability to present an API compatible with Faspex 4, and it will call the Faspex 5 API.
|
6745
6974
|
|
6746
6975
|
It takes a single argument which is the URL at which the gateway will be located (locally):
|
6747
6976
|
|
6748
6977
|
```bash
|
6749
|
-
ascli faspex5 gateway https://localhost:12345/aspera/faspex
|
6978
|
+
ascli faspex5 gateway @json:'{"url":"https://localhost:12345/aspera/faspex"}'
|
6750
6979
|
```
|
6751
6980
|
|
6752
6981
|
There are many limitations:
|
6753
6982
|
|
6754
|
-
- It's only to emulate the Faspex 4 `send` API (send package)
|
6755
|
-
- No support for remote sources, only for an actual file transfer by the client
|
6756
|
-
- The client must use the transfer spec returned by the API (not faspe
|
6757
|
-
-
|
6758
|
-
-
|
6759
|
-
-
|
6983
|
+
- It's only to emulate the Faspex 4 `send` API (send package).
|
6984
|
+
- No support for remote sources, only for an actual file transfer by the client.
|
6985
|
+
- The client must use the transfer spec returned by the API (not `faspe:` URL).
|
6986
|
+
- Tags returned in transfer spec must be used in transfer.
|
6987
|
+
- Only a single authentication is possible (per gateway) on Faspex5.
|
6988
|
+
- No authentication of F4 side (ignored).
|
6760
6989
|
|
6761
6990
|
Behavior: The API client calls the Faspex 4 API on the gateway, then the gateway transforms this into a Faspex5 API call, which returns a transfer spec, which is returned to the calling client. The calling client uses this to start a transfer to HSTS which is actually managed by Faspex 5.
|
6762
6991
|
|
6992
|
+
For other parameters, see [Web service](#web-service).
|
6993
|
+
|
6763
6994
|
### Faspex 5: Get Bearer token to use API
|
6764
6995
|
|
6765
6996
|
If a command is missing, then it is still possible to execute command by calling directly the API on the command line using `curl`:
|
@@ -6770,7 +7001,7 @@ curl -H "Authorization: $(ascli ascli bearer)" https://faspex5.example.com/asper
|
|
6770
7001
|
|
6771
7002
|
## Plugin: `faspex`: IBM Aspera Faspex v4
|
6772
7003
|
|
6773
|
-
> **Note:** Faspex v4 is end of support since Sept. 30th, 2024. So this plugin for
|
7004
|
+
> **Note:** Faspex v4 is end of support since Sept. 30th, 2024. So this plugin for Faspex v4 is deprecated. If you still need to use Faspex4, then use `ascli` version 4.19.0 or earlier.
|
6774
7005
|
>
|
6775
7006
|
> **Note:** For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
6776
7007
|
|
@@ -6783,7 +7014,7 @@ Command: `faspex package list`
|
|
6783
7014
|
|
6784
7015
|
#### Option `box`
|
6785
7016
|
|
6786
|
-
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
7017
|
+
By default, it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
6787
7018
|
|
6788
7019
|
#### Option `recipient`
|
6789
7020
|
|
@@ -6796,18 +7027,20 @@ A user can receive a package because the recipient is:
|
|
6796
7027
|
|
6797
7028
|
As inboxes may be large, it is possible to use the following query parameters:
|
6798
7029
|
|
6799
|
-
|
6800
|
-
|
6801
|
-
|
6802
|
-
|
6803
|
-
|
7030
|
+
| Parameter | Evaluation | Description |
|
7031
|
+
|--------------|------------|-------------|
|
7032
|
+
| `count` | API | Number of items in one API call result (default=0, equivalent to 10) |
|
7033
|
+
| `page` | API | ID of page in call (default=0) |
|
7034
|
+
| `startIndex` | API | Index of item to start (default=0) |
|
7035
|
+
| `max` | `ascli` | Maximum number of items |
|
7036
|
+
| `pmax` | `ascli` | Maximum number of pages |
|
6804
7037
|
|
6805
7038
|
(SQL query is `LIMIT <startIndex>, <count>`)
|
6806
7039
|
|
6807
7040
|
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under **Services (API v.3)**.
|
6808
7041
|
|
6809
7042
|
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`).
|
6810
|
-
By default `count` is `0` (`10`), it can be increased to issue less HTTP calls.
|
7043
|
+
By default, `count` is `0` (`10`), it can be increased to issue less HTTP calls.
|
6811
7044
|
|
6812
7045
|
#### Example: List packages in dropbox
|
6813
7046
|
|
@@ -6821,7 +7054,7 @@ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in recei
|
|
6821
7054
|
|
6822
7055
|
The command is `package recv`, possible methods are:
|
6823
7056
|
|
6824
|
-
- Provide a package
|
7057
|
+
- Provide a package ID with option `id`
|
6825
7058
|
- Provide a public link with option `link`
|
6826
7059
|
- Provide a `faspe:` URI with option `link`
|
6827
7060
|
|
@@ -6837,13 +7070,13 @@ ascli faspex package list --recipient='*dropbox_name'
|
|
6837
7070
|
ascli faspex package recv 125 --recipient='*dropbox_name'
|
6838
7071
|
```
|
6839
7072
|
|
6840
|
-
|
7073
|
+
If `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
6841
7074
|
|
6842
7075
|
### Sending a Package
|
6843
7076
|
|
6844
7077
|
The command is `faspex package send`.
|
6845
7078
|
Package information (title, note, metadata, options) is provided in option `delivery_info`.
|
6846
|
-
The
|
7079
|
+
The content of `delivery_info` is directly the contents of the `send` v3 [API of Faspex 4](https://developer.ibm.com/apis/catalog/aspera--aspera-faspex-client-sdk/API%20v.3:%20Send%20Packages).
|
6847
7080
|
|
6848
7081
|
Example:
|
6849
7082
|
|
@@ -6859,7 +7092,7 @@ Additional optional parameters in mandatory option `delivery_info`:
|
|
6859
7092
|
- Package Note: : `"note":"note this and that"`
|
6860
7093
|
- Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
6861
7094
|
|
6862
|
-
It is possible to send from a remote source using option `remote_source`, providing either the numerical
|
7095
|
+
It is possible to send from a remote source using option `remote_source`, providing either the numerical ID, or the name of the remote source using percent selector: `%name:<name>`.
|
6863
7096
|
|
6864
7097
|
Remote source can be browsed if option `storage` is provided.
|
6865
7098
|
`storage` is a `Hash` extended value.
|
@@ -6867,11 +7100,11 @@ The key is the storage name, as listed in `source list` command.
|
|
6867
7100
|
The value is a `Hash` with the following keys:
|
6868
7101
|
|
6869
7102
|
- `node` is a `Hash` with keys: `url`, `username`, `password`
|
6870
|
-
- `path` is the
|
7103
|
+
- `path` is the sub-path inside the node, as configured in Faspex
|
6871
7104
|
|
6872
7105
|
### Email notification on transfer
|
6873
7106
|
|
6874
|
-
Like for any transfer, a notification can be sent by email using options: `notify_to` and `notify_template
|
7107
|
+
Like for any transfer, a notification can be sent by email using options: `notify_to` and `notify_template`.
|
6875
7108
|
|
6876
7109
|
Example:
|
6877
7110
|
|
@@ -6881,7 +7114,7 @@ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipien
|
|
6881
7114
|
|
6882
7115
|
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
6883
7116
|
|
6884
|
-
###
|
7117
|
+
### Operations on dropbox
|
6885
7118
|
|
6886
7119
|
Example:
|
6887
7120
|
|
@@ -6894,7 +7127,7 @@ ascli faspex v4 dropbox delete 36
|
|
6894
7127
|
### Remote sources
|
6895
7128
|
|
6896
7129
|
Faspex lacks an API to list the contents of a remote source (available in web UI).
|
6897
|
-
To
|
7130
|
+
To work around this, the Node API is used, for this it is required to set option: `storage` that links a storage name to a node configuration and sub path.
|
6898
7131
|
|
6899
7132
|
Example:
|
6900
7133
|
|
@@ -6913,7 +7146,7 @@ my_faspex_node:
|
|
6913
7146
|
password: MyNodePassword
|
6914
7147
|
```
|
6915
7148
|
|
6916
|
-
In this example, a
|
7149
|
+
In this example, a Faspex storage named `my_storage` exists in Faspex, and is located
|
6917
7150
|
under the docroot in `/mydir` (this must be the same as configured in Faspex).
|
6918
7151
|
The node configuration name is `my_faspex_node` here.
|
6919
7152
|
|
@@ -6968,7 +7201,7 @@ v4 workgroup list
|
|
6968
7201
|
|
6969
7202
|
## Plugin: `shares`: IBM Aspera Shares v1
|
6970
7203
|
|
6971
|
-
Aspera Shares supports the **
|
7204
|
+
Aspera Shares supports the **Node API** for the file transfer part.
|
6972
7205
|
|
6973
7206
|
Supported commands are listed in Share's API documentation:
|
6974
7207
|
|
@@ -6986,7 +7219,7 @@ user_id=$(ascli shares admin user list --select=@json:'{"username":"entity1"}' -
|
|
6986
7219
|
ascli shares admin share user_permissions $share_id create @json:'{"user_id":'$user_id',"browse_permission":true, "download_permission":true, "mkdir_permission":true,"delete_permission":true,"rename_permission":true,"content_availability_permission":true,"manage_permission":true}'
|
6987
7220
|
```
|
6988
7221
|
|
6989
|
-
To figure out
|
7222
|
+
To figure out an entity's payload, for example for creation, refer to the API documentation above.
|
6990
7223
|
|
6991
7224
|
### Shares sample commands
|
6992
7225
|
|
@@ -7011,6 +7244,8 @@ files delete my_share1/test_file.bin
|
|
7011
7244
|
files download --to-folder=. my_share1/test_file.bin
|
7012
7245
|
files download --to-folder=. my_share1/test_file.bin my_share1/test_file.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://tst.example.com/path@"}'
|
7013
7246
|
files mkdir my_share1/new_folder
|
7247
|
+
files sync start push /data/local_syncsynctst /london-sh1/synctst --sync-info=@json:'{"quiet":false}'
|
7248
|
+
files sync start push /data/local_syncsynctst /london-sh1/synctst --sync-info=@json:'{"reset":true}'
|
7014
7249
|
files upload --to-folder=https://shares.share1 'faux:///testfile?1m' --transfer=httpgw --transfer-info=@json:'{"url":"my_example.com/path@","synchronous":true,"api_version":"v1","upload_chunk_size":100000}'
|
7015
7250
|
files upload --to-folder=https://shares.share1 sendfolder --transfer=httpgw --transfer-info=@json:'{"url":"my_example.com/path@","synchronous":true,"api_version":"v1","upload_chunk_size":100000}'
|
7016
7251
|
files upload --to-folder=my_share1 test_file.bin
|
@@ -7069,7 +7304,7 @@ If you have those parameters already, then following options shall be provided:
|
|
7069
7304
|
- `bucket` bucket name
|
7070
7305
|
- `endpoint` storage endpoint URL, e.g. `https://s3.hkg02.cloud-object-storage.appdomain.cloud`
|
7071
7306
|
- `apikey` API Key
|
7072
|
-
- `crn` resource instance
|
7307
|
+
- `crn` resource instance ID
|
7073
7308
|
|
7074
7309
|
For example, let us create a default configuration:
|
7075
7310
|
|
@@ -7146,14 +7381,14 @@ ascli config preset set default cos mycos
|
|
7146
7381
|
|
7147
7382
|
Let's assume you created a default configuration from one of the two previous steps (else specify the access options on command lines).
|
7148
7383
|
|
7149
|
-
A subset of `node` plugin operations are supported, basically
|
7384
|
+
A subset of `node` plugin operations are supported, basically Node API:
|
7150
7385
|
|
7151
7386
|
```bash
|
7152
7387
|
ascli cos node info
|
7153
7388
|
ascli cos node upload 'faux:///sample1G?1g'
|
7154
7389
|
```
|
7155
7390
|
|
7156
|
-
> **Note:** A dummy file `sample1G` of size
|
7391
|
+
> **Note:** A dummy file `sample1G` of size 2 GB is generated using the `faux` PVCL scheme (see previous section and `man ascp`), but you can, of course, send a real file by specifying a real file path instead.
|
7157
7392
|
|
7158
7393
|
### Cos sample commands
|
7159
7394
|
|
@@ -7207,7 +7442,7 @@ health -N
|
|
7207
7442
|
## Plugin: `preview`: Preview generator for AoC
|
7208
7443
|
|
7209
7444
|
The `preview` generates thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
7210
|
-
It uses the **
|
7445
|
+
It uses the **Node API** of Aspera HSTS and requires use of Access Keys and its **storage root**.
|
7211
7446
|
Several options can be used to tune several aspects:
|
7212
7447
|
|
7213
7448
|
- Methods for detection of new files needing generation
|
@@ -7218,7 +7453,7 @@ See also <https://github.com/IBM/aspera-on-cloud-file-previews>
|
|
7218
7453
|
|
7219
7454
|
### Aspera Server configuration
|
7220
7455
|
|
7221
|
-
Specify the
|
7456
|
+
Specify the preview's folder as shown in:
|
7222
7457
|
|
7223
7458
|
<https://ibmaspera.com/help/admin/organization/installing_the_preview_maker>
|
7224
7459
|
|
@@ -7235,8 +7470,8 @@ asnodeadmin --reload
|
|
7235
7470
|
|
7236
7471
|
If another folder is configured on the HSTS, then specify it to `ascli` using the option `previews_folder`.
|
7237
7472
|
|
7238
|
-
The HSTS
|
7239
|
-
This size is internally capped to `1<<24` Bytes (16777216)
|
7473
|
+
The HSTS Node API limits any preview file to a parameter: `max_request_file_create_size_kb` (1 KB is 1024 Bytes).
|
7474
|
+
This size is internally capped to `1<<24` Bytes (16777216), i.e. 16384 KB, i.e. 16 MB.
|
7240
7475
|
|
7241
7476
|
To change this parameter in `aspera.conf`, use `asconfigurator`.
|
7242
7477
|
To display the value, use `asuserdata`:
|
@@ -7253,9 +7488,9 @@ asuserdata -a | grep max_request_file_create_size_kb
|
|
7253
7488
|
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
7254
7489
|
```
|
7255
7490
|
|
7256
|
-
If you use a value different
|
7491
|
+
If you use a value different from `16777216`, then specify it using option `max_size`.
|
7257
7492
|
|
7258
|
-
> **Note:** The HSTS parameter (`max_request_file_create_size_kb`) is in **
|
7493
|
+
> **Note:** The HSTS parameter (`max_request_file_create_size_kb`) is in **kilo Bytes** while the generator parameter is in **Bytes** (factor of 1024).
|
7259
7494
|
|
7260
7495
|
### External tools: Linux
|
7261
7496
|
|
@@ -7264,11 +7499,11 @@ If you use a value different than `16777216`, then specify it using option `max_
|
|
7264
7499
|
- **ImageMagick** v7+: `magick` `composite`
|
7265
7500
|
- **OptiPNG** : `optipng`
|
7266
7501
|
- **FFmpeg** : `ffmpeg` `ffprobe`
|
7267
|
-
- **
|
7502
|
+
- **LibreOffice** : `unoconv`
|
7268
7503
|
|
7269
|
-
Here shown on
|
7504
|
+
Here shown on Red Hat/Rocky Linux.
|
7270
7505
|
|
7271
|
-
Other OSes should work as well, but are
|
7506
|
+
Other OSes should work as well, but are not tested.
|
7272
7507
|
|
7273
7508
|
To check if all tools are found properly, execute:
|
7274
7509
|
|
@@ -7283,24 +7518,24 @@ dnf install -y ImageMagick optipng
|
|
7283
7518
|
```
|
7284
7519
|
|
7285
7520
|
You may also install `ghostscript` which adds fonts to ImageMagick.
|
7286
|
-
Available fonts, used to generate
|
7521
|
+
Available fonts, used to generate PNG for text, can be listed with `magick identify -list font`.
|
7287
7522
|
Prefer ImageMagick version >=7.
|
7288
7523
|
|
7289
7524
|
More info on ImageMagick at <https://imagemagick.org/>
|
7290
7525
|
|
7291
7526
|
#### Video: FFmpeg
|
7292
7527
|
|
7293
|
-
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
7528
|
+
The easiest method is to download and install the latest released version of `ffmpeg` with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
7294
7529
|
|
7295
7530
|
```bash
|
7296
7531
|
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
7297
7532
|
```
|
7298
7533
|
|
7299
|
-
#### Office: `unoconv` and
|
7534
|
+
#### Office: `unoconv` and LibreOffice
|
7300
7535
|
|
7301
7536
|
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
7302
7537
|
|
7303
|
-
The generation of preview in based on the use of
|
7538
|
+
The generation of preview in based on the use of LibreOffice's `unoconv`.
|
7304
7539
|
|
7305
7540
|
- RHEL 8/Rocky Linux 8+
|
7306
7541
|
|
@@ -7357,13 +7592,14 @@ For video preview, the whole set of options can be overridden with option `reenc
|
|
7357
7592
|
|
7358
7593
|
### Execution
|
7359
7594
|
|
7360
|
-
`ascli` intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST
|
7595
|
+
`ascli` intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST API too quickly during the scan or event method).
|
7361
7596
|
It needs to be run on a regular basis to create or update preview files.
|
7362
7597
|
For that use your best reliable scheduler, see [Scheduler](#scheduler).
|
7363
7598
|
|
7364
|
-
Typically, for **Access key** access, the system/transfer is `xfer`.
|
7599
|
+
Typically, for **Access key** access, the system/transfer is `xfer`.
|
7600
|
+
So, in order to be consistent, and generate the appropriate access rights, the generation process should be run as user `xfer`.
|
7365
7601
|
|
7366
|
-
|
7602
|
+
Let's do a one shot test, using the configuration previously created:
|
7367
7603
|
|
7368
7604
|
```bash
|
7369
7605
|
su -s /bin/bash - xfer
|
@@ -7372,7 +7608,7 @@ ascli preview scan --overwrite=always
|
|
7372
7608
|
```
|
7373
7609
|
|
7374
7610
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
7375
|
-
in the
|
7611
|
+
in the preview's folder which contains the access key used.
|
7376
7612
|
On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
|
7377
7613
|
|
7378
7614
|
### Configuration for Execution in scheduler
|
@@ -7440,15 +7676,15 @@ The preview generator supports rendering of those file categories:
|
|
7440
7676
|
|
7441
7677
|
To avoid generation for some categories, specify a list using option `skip_types`.
|
7442
7678
|
|
7443
|
-
Each category has a specific rendering method to produce the
|
7679
|
+
Each category has a specific rendering method to produce the PNG thumbnail.
|
7444
7680
|
|
7445
7681
|
The mp4 video preview file is only for category `video`
|
7446
7682
|
|
7447
|
-
File type is primarily based on file extension detected by the
|
7683
|
+
File type is primarily based on file extension detected by the Node API and translated info a mime type returned by the Node API.
|
7448
7684
|
|
7449
7685
|
### `mimemagic`
|
7450
7686
|
|
7451
|
-
By default, the Mime type used for conversion is the one returned by the
|
7687
|
+
By default, the Mime type used for conversion is the one returned by the Node API, based on file name extension.
|
7452
7688
|
|
7453
7689
|
It is also possible to detect the mime type using option `mimemagic`.
|
7454
7690
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`.
|
@@ -7531,29 +7767,30 @@ An interface for the `async` utility is provided in the following plugins:
|
|
7531
7767
|
- `aoc files sync` (uses node)
|
7532
7768
|
- `shares files sync` (uses node)
|
7533
7769
|
|
7534
|
-
One advantage over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`.
|
7535
|
-
Moreover `ascli` supports sync with application requiring token-based authorization.
|
7770
|
+
One advantage of using `ascli` over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`.
|
7771
|
+
Moreover, `ascli` supports sync with application requiring token-based authorization.
|
7536
7772
|
|
7537
|
-
Some
|
7773
|
+
Some `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
|
7538
7774
|
|
7539
|
-
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). The Transfer SDK 1.3 includes this.
|
7775
|
+
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). The Aspera Transfer SDK 1.3+ includes this.
|
7540
7776
|
|
7541
|
-
`sync` supports 0 or 3 arguments
|
7777
|
+
`sync` supports 0 or 3 arguments.
|
7778
|
+
If 3 arguments are provided, they are applied to the first (and only) session and mapped, in that order, to:
|
7542
7779
|
|
7543
|
-
- direction
|
7544
|
-
- local
|
7545
|
-
- remote
|
7780
|
+
- `direction`
|
7781
|
+
- `local.path`
|
7782
|
+
- `remote.path`
|
7546
7783
|
|
7547
7784
|
Additional options can be provided with option `sync_info`, for which two syntax are possible, as follows.
|
7548
7785
|
|
7549
|
-
### `async` API and conf format
|
7786
|
+
### `async` API and `conf` format
|
7550
7787
|
|
7551
|
-
It is the same payload as specified on the option `--conf` of `async` or in
|
7552
|
-
This is the preferred syntax and allows a single session definition.
|
7788
|
+
It is the same payload as specified on the option `--conf` of `async` or in Node API `/asyncs`.
|
7789
|
+
This is the **preferred** syntax and allows a single session definition.
|
7553
7790
|
|
7554
|
-
> **Note:**
|
7791
|
+
> **Note:** By default, no progress, nor error messages is provided on terminal. To activate messages, set option `sync_info` parameter `quiet` to `false`.
|
7555
7792
|
|
7556
|
-
Documentation on Async
|
7793
|
+
Documentation on Async Node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
7557
7794
|
|
7558
7795
|
If 3 arguments are provided they are mapped to:
|
7559
7796
|
|
@@ -7567,16 +7804,10 @@ If 3 arguments are provided they are mapped to:
|
|
7567
7804
|
It is based on a JSON representation of `async` command line options.
|
7568
7805
|
It allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
7569
7806
|
|
7570
|
-
This is the mode selection if there
|
7807
|
+
This is the mode selection if there are either keys `sessions` or `instance` in option `sync_info`.
|
7571
7808
|
|
7572
7809
|
> **Note:** Progress and error messages are provided on terminal like regular command line invocation of `async`.
|
7573
7810
|
|
7574
|
-
If 3 arguments are provided, they are applied to the first (and only) session and mapped to:
|
7575
|
-
|
7576
|
-
- `direction`
|
7577
|
-
- `local_dir`
|
7578
|
-
- `remote_dir`
|
7579
|
-
|
7580
7811
|
## Hot folder
|
7581
7812
|
|
7582
7813
|
### Requirements
|
@@ -7613,7 +7844,7 @@ Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
|
7613
7844
|
>
|
7614
7845
|
> **Note:** Usual native `ascp` arguments are available as standard [**transfer-spec**](#transfer-specification) parameters, but not special or advanced options.
|
7615
7846
|
>
|
7616
|
-
> **Note:** Only for the [`direct`](#agent-direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info
|
7847
|
+
> **Note:** Only for the [`direct`](#agent-direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer_info`.
|
7617
7848
|
|
7618
7849
|
#### Server side and configuration
|
7619
7850
|
|
@@ -7621,11 +7852,11 @@ Virtually any transfer on a **repository** on a regular basis might emulate a ho
|
|
7621
7852
|
|
7622
7853
|
> **Note:** File detection is not based on events (`inotify`, etc...), but on a simple folder scan on source side.
|
7623
7854
|
>
|
7624
|
-
> **Note:** Options may be saved in
|
7855
|
+
> **Note:** Options may be saved in an [Option Preset](#option-preset) and used with `-P`.
|
7625
7856
|
|
7626
7857
|
#### Scheduling
|
7627
7858
|
|
7628
|
-
Once `ascli` command line arguments are defined, run the command using the OS native scheduler, e.g. every
|
7859
|
+
Once `ascli` command line arguments are defined, run the command using the OS native scheduler, e.g. every minute, or 5 minutes, etc...
|
7629
7860
|
Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
7630
7861
|
|
7631
7862
|
### Example: Upload hot folder
|
@@ -7634,7 +7865,7 @@ Refer to section [Scheduler](#scheduler). (on use of option `lock_port`)
|
|
7634
7865
|
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
7635
7866
|
```
|
7636
7867
|
|
7637
|
-
The local folder (here, relative path: `source_hot`) is sent (upload) to an
|
7868
|
+
The local folder (here, relative path: `source_hot`) is sent (upload) to an Aspera server.
|
7638
7869
|
Source files are deleted after transfer.
|
7639
7870
|
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
7640
7871
|
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
@@ -7696,13 +7927,13 @@ OK - [transfer:ok]
|
|
7696
7927
|
The `smtp` option is a `Hash` (extended value) with the following fields:
|
7697
7928
|
|
7698
7929
|
<!-- markdownlint-disable MD034 -->
|
7699
|
-
| field |
|
7930
|
+
| field | Default | example | Description |
|
7700
7931
|
|--------------|---------------------|----------------------------|----------------------------------|
|
7701
|
-
| `server` | - | smtp.gmail.com
|
7702
|
-
| `tls` | true
|
7703
|
-
| `ssl` | false
|
7704
|
-
| `port` | 587 or 465 or 25
|
7705
|
-
| `domain` | domain of server
|
7932
|
+
| `server` | - | `smtp.gmail.com` | SMTP server address |
|
7933
|
+
| `tls` | `true` | `true` | Enable `STARTTLS` (port 587) |
|
7934
|
+
| `ssl` | `false` | `false` | Enable `TLS` (port 465) |
|
7935
|
+
| `port` | `587` or `465` or `25` | `587` | Port for service |
|
7936
|
+
| `domain` | *domain of server* | gmail.com | Email domain of user |
|
7706
7937
|
| `username` | - | john@example.com | User to authenticate on SMTP server, leave empty for open auth. |
|
7707
7938
|
| `password` | - | my_password_here | Password for above username |
|
7708
7939
|
| `from_email` | username if defined | johnny@example.com | Address used if receiver replies |
|
@@ -7738,7 +7969,7 @@ ascli config preset set default config cli_default
|
|
7738
7969
|
|
7739
7970
|
### Email templates
|
7740
7971
|
|
7741
|
-
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
7972
|
+
Sent emails are built using a template that uses the [`ERB`](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
7742
7973
|
|
7743
7974
|
The template is the full SMTP message, including headers.
|
7744
7975
|
|
@@ -7790,10 +8021,10 @@ Transfer is: <%=status%>
|
|
7790
8021
|
|
7791
8022
|
This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: `asession`.
|
7792
8023
|
|
7793
|
-
It aims at simplifying the startup of a FASP session from a programmatic
|
8024
|
+
It aims at simplifying the startup of a FASP session from a programmatic standpoint as formatting a [**transfer-spec**](#transfer-specification) is:
|
7794
8025
|
|
7795
8026
|
- Common to Aspera Node API (HTTP POST /ops/transfer)
|
7796
|
-
- Common to Aspera Connect API (browser
|
8027
|
+
- Common to Aspera Connect API (browser JavaScript `startTransfer`)
|
7797
8028
|
- Easy to generate by using any third party language specific JSON library
|
7798
8029
|
|
7799
8030
|
Hopefully, IBM integrates this directly in `ascp`, and this tool is made redundant.
|
@@ -7810,16 +8041,16 @@ During execution, it generates all low level events, one per line, in JSON forma
|
|
7810
8041
|
|
7811
8042
|
Top level parameters supported by `asession`:
|
7812
8043
|
|
7813
|
-
|
|
8044
|
+
| Parameter | Description |
|
7814
8045
|
|-----------|-------------|
|
7815
8046
|
| `spec` | the [**transfer-spec**](#transfer-specification) |
|
7816
8047
|
| `agent` | same parameters as transfer-info for agent `direct` |
|
7817
8048
|
| `loglevel` | log level of `asession` |
|
7818
|
-
| `file_list_folder` | the folder used to store (for garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists` |
|
8049
|
+
| `file_list_folder` | the folder used to store (for garbage collection) generated file lists. By default, it is `[system tmp folder]/[username]_asession_filelists` |
|
7819
8050
|
|
7820
8051
|
### Comparison of interfaces
|
7821
8052
|
|
7822
|
-
| feature/tool | Transfer SDK |
|
8053
|
+
| feature/tool | Transfer SDK | FASPManager | `ascp` | `asession` |
|
7823
8054
|
|-----------------------|--------------|---------------------------------|--------|------------|
|
7824
8055
|
| language integration | Many | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | Any | Any |
|
7825
8056
|
| required additional components to `ascp` | Daemon | Library<br/>(+headers) | - | Ruby<br/>Aspera gem |
|
@@ -7843,9 +8074,9 @@ asession < session.json
|
|
7843
8074
|
|
7844
8075
|
### Asynchronous commands and Persistent session
|
7845
8076
|
|
7846
|
-
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter`
|
8077
|
+
`asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter` → `long_parameter`
|
7847
8078
|
|
7848
|
-
This is particularly useful for a persistent session (
|
8079
|
+
This is particularly useful for a persistent session (with the [**transfer-spec**](#transfer-specification) parameter: `"keepalive":true`)
|
7849
8080
|
|
7850
8081
|
```json
|
7851
8082
|
asession
|
@@ -7858,7 +8089,7 @@ asession
|
|
7858
8089
|
|
7859
8090
|
### Example of language wrapper
|
7860
8091
|
|
7861
|
-
|
8092
|
+
NodeJS: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/aspera)
|
7862
8093
|
|
7863
8094
|
### Help
|
7864
8095
|
|
@@ -7918,7 +8149,7 @@ So, it evolved into `ascli`:
|
|
7918
8149
|
|
7919
8150
|
- Portable: works on platforms supporting `ruby` (and `ascp`)
|
7920
8151
|
- Easy to install with the `gem` utility
|
7921
|
-
- Supports transfers with multiple [Transfer Agents](#transfer-clients-agents), that's why transfer parameters moved from `ascp` command line to [**transfer-spec**](#transfer-specification) (more reliable
|
8152
|
+
- Supports transfers with multiple [Transfer Agents](#transfer-clients-agents), that's why transfer parameters moved from `ascp` command line to [**transfer-spec**](#transfer-specification) (more reliable, more standard)
|
7922
8153
|
- `ruby` is consistent with other Aspera products
|
7923
8154
|
|
7924
8155
|
Over the time, a supported command line tool `aspera` was developed in C++, it was later on deprecated.
|
@@ -7938,7 +8169,7 @@ ascli config coffee
|
|
7938
8169
|
|
7939
8170
|
### Error: "Remote host is not who we expected"
|
7940
8171
|
|
7941
|
-
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
8172
|
+
Cause: `ascp` >= 4.x checks fingerprint of the highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
7942
8173
|
|
7943
8174
|
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the configuration file):
|
7944
8175
|
|
@@ -7973,5 +8204,5 @@ OpenSSH keys only supported if ED25519 is available
|
|
7973
8204
|
Which meant that you do not have Ruby support for ED25519 SSH keys.
|
7974
8205
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
7975
8206
|
|
7976
|
-
In addition, host keys of type: `ecdsa-sha2` and `ecdh-sha2` are also
|
8207
|
+
In addition, host keys of type: `ecdsa-sha2` and `ecdh-sha2` are also deactivated by default.
|
7977
8208
|
To re-activate, set env var `ASCLI_ENABLE_ECDSHA2` to `true`.
|