aspera-cli 4.9.0 → 4.10.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/README.md +749 -667
- data/docs/test_env.conf +6 -4
- data/examples/dascli +9 -4
- data/examples/faspex4.rb +1 -1
- data/examples/node.rb +1 -1
- data/examples/server.rb +1 -1
- data/lib/aspera/cli/info.rb +2 -2
- data/lib/aspera/cli/main.rb +1 -1
- data/lib/aspera/cli/manager.rb +6 -4
- data/lib/aspera/cli/plugin.rb +5 -1
- data/lib/aspera/cli/plugins/aoc.rb +1 -1
- data/lib/aspera/cli/plugins/config.rb +100 -84
- data/lib/aspera/cli/plugins/faspex.rb +3 -3
- data/lib/aspera/cli/plugins/faspex5.rb +5 -1
- data/lib/aspera/cli/plugins/node.rb +24 -4
- data/lib/aspera/cli/plugins/preview.rb +1 -1
- data/lib/aspera/cli/transfer_agent.rb +3 -2
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/environment.rb +19 -1
- data/lib/aspera/fasp/agent_direct.rb +4 -0
- data/lib/aspera/fasp/agent_httpgw.rb +147 -88
- data/lib/aspera/fasp/installation.rb +4 -4
- data/lib/aspera/fasp/parameters.rb +13 -1
- data/lib/aspera/fasp/parameters.yaml +5 -2
- data/lib/aspera/fasp/resume_policy.rb +1 -1
- data/lib/aspera/keychain/encrypted_hash.rb +36 -98
- data/lib/aspera/keychain/macos_security.rb +127 -57
- data/lib/aspera/persistency_folder.rb +3 -2
- data/lib/aspera/proxy_auto_config.rb +12 -5
- data/lib/aspera/rest.rb +19 -10
- data.tar.gz.sig +0 -0
- metadata +5 -33
- metadata.gz.sig +0 -0
data/README.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
4
4
|
|
5
|
-
Version : 4.
|
5
|
+
Version : 4.10.0
|
6
6
|
|
7
7
|
Laurent/2016-2022
|
8
8
|
|
@@ -14,7 +14,7 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
14
14
|
|
15
15
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
16
16
|
|
17
|
-
Minimum required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.
|
17
|
+
Minimum required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.7 in a future version.
|
18
18
|
|
19
19
|
[Aspera APIs on IBM developer](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
20
20
|
[Link 2](https://developer.ibm.com/apis/catalog/?search=aspera)
|
@@ -33,26 +33,25 @@ One can also [create one's own plugin](#createownplugin).
|
|
33
33
|
|
34
34
|
`ascli` is designed to be used as a command line tool to:
|
35
35
|
|
36
|
-
|
37
|
-
|
36
|
+
- Execute commands remotely on Aspera products
|
37
|
+
- Transfer to/from Aspera products
|
38
38
|
|
39
39
|
So it is designed for:
|
40
40
|
|
41
|
-
|
42
|
-
|
41
|
+
- Interactive operations on a text terminal (typically, VT100 compatible), e.g. for maintenance
|
42
|
+
- Scripting, e.g. batch operations in (shell) scripts (e.g. cron job)
|
43
43
|
|
44
44
|
`ascli` can be seen as a command line tool integrating:
|
45
45
|
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
46
|
+
- A configuration file (config.yaml)
|
47
|
+
- Advanced command line options
|
48
|
+
- cURL (for REST calls)
|
49
|
+
- Aspera transfer (ascp)
|
50
50
|
|
51
|
-
|
52
|
-
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
51
|
+
If the need is to perform operations programmatically in languages such as: C, Go, Python, nodejs, ... then it is better to directly use [Aspera APIs](https://ibm.biz/aspera_api)
|
53
52
|
|
54
|
-
|
55
|
-
|
53
|
+
- Product APIs (REST) : e.g. AoC, Faspex, node
|
54
|
+
- Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
56
55
|
|
57
56
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
58
57
|
|
@@ -78,7 +77,7 @@ ascli --version
|
|
78
77
|
```
|
79
78
|
|
80
79
|
```bash
|
81
|
-
4.
|
80
|
+
4.10.0
|
82
81
|
```
|
83
82
|
|
84
83
|
### First use
|
@@ -110,10 +109,10 @@ ascli server browse /
|
|
110
109
|
|
111
110
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
112
111
|
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
112
|
+
- create a [option preset](#lprt)
|
113
|
+
- define it as default for `server` plugin
|
114
|
+
- list files in a folder
|
115
|
+
- download a file
|
117
116
|
|
118
117
|
```bash
|
119
118
|
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_
|
@@ -175,9 +174,9 @@ It is possible to install *either* directly on the host operating system (Linux,
|
|
175
174
|
|
176
175
|
The direct installation is recommended and consists in installing:
|
177
176
|
|
178
|
-
|
179
|
-
|
180
|
-
|
177
|
+
- [Ruby](#ruby) (version: >= 2.4. Deprecation notice: the minimum will be 2.7 in a future version)
|
178
|
+
- [aspera-cli](#the_gem)
|
179
|
+
- [Aspera SDK (ascp)](#fasp_prot)
|
181
180
|
|
182
181
|
The following sections provide information on the various installation methods.
|
183
182
|
|
@@ -187,43 +186,56 @@ An internet connection is required for the installation. If you don't have inter
|
|
187
186
|
|
188
187
|
This method installs a docker image that contains: Ruby, `ascli` and the FASP SDK.
|
189
188
|
|
190
|
-
The image is: <https://hub.docker.com/r/martinlaurent/ascli>.
|
189
|
+
The image is: <https://hub.docker.com/r/martinlaurent/ascli>.
|
190
|
+
It is built from this [Dockerfile](Dockerfile).
|
191
191
|
|
192
|
-
Ensure that you have
|
192
|
+
Ensure that you have `docker` (or `podman`) installed.
|
193
193
|
|
194
194
|
```bash
|
195
195
|
docker --version
|
196
196
|
```
|
197
197
|
|
198
|
-
|
198
|
+
Download the script [`dascli`](../examples/dascli) from [the GIT repo](https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli) :
|
199
199
|
|
200
|
-
|
201
|
-
cp $(ascli conf gem path)/../examples/dascli ascli
|
202
|
-
```
|
200
|
+
> If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
203
201
|
|
204
|
-
|
202
|
+
Note that ascli is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default.
|
205
203
|
|
206
|
-
|
207
|
-
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
208
|
-
```
|
204
|
+
Some environment variables can be set to alter the behaviour of the script:
|
209
205
|
|
210
|
-
|
211
|
-
|
212
|
-
|
206
|
+
| env var | description | default | example |
|
207
|
+
|-------------|------------------------------------|--------------------------|--------------------------|
|
208
|
+
| ASCLI__HOME | configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascliconfig` |
|
209
|
+
| docker_args | additional options to `docker` | <empty> | `--volume /Users:/Users` |
|
210
|
+
| image | container image name | martinlaurent/ascli | |
|
211
|
+
| version | container image version | latest | `4.8.0.pre` |
|
212
|
+
|
213
|
+
The wrapping script maps the folder `$ASCLI__HOME` on host to `/home/cliuser/.aspera/ascli` in the container.
|
214
|
+
(value expected in the container).
|
215
|
+
This allows having persistent configuration on the host.
|
216
|
+
|
217
|
+
To add local storage as a volume, you can use the env var `docker_args`:
|
213
218
|
|
214
|
-
|
219
|
+
Example of use:
|
215
220
|
|
216
221
|
```bash
|
217
|
-
|
218
|
-
|
222
|
+
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/examples/dascli
|
223
|
+
chmod a+x ascli
|
224
|
+
export ASCLI__HOME=$HOME/.ascliconf
|
225
|
+
mkdir -p $ASCLI__HOME
|
226
|
+
chmod -R 777 $ASCLI__HOME
|
227
|
+
export xferdir=$HOME/xferdir
|
228
|
+
mkdir -p $xferdir
|
229
|
+
chmod -R 777 $xferdir
|
230
|
+
export docker_args="--volume $xferdir:/xferfolder"
|
219
231
|
|
220
|
-
|
232
|
+
./ascli conf init
|
221
233
|
|
222
|
-
|
223
|
-
|
234
|
+
touch $xferdir/samplefile
|
235
|
+
./ascli server upload /xferfolder/samplefile --to-folder=/Upload
|
236
|
+
```
|
224
237
|
|
225
|
-
|
226
|
-
To add local storage as a volume edit the script: `ascli` and add a `--volume` stanza near the existing one.
|
238
|
+
> The local file (`samplefile`) is specified relative to storage view from container (`/xferfolder`) mapped to the host folder `$HOME/xferdir`
|
227
239
|
|
228
240
|
### <a id="ruby"></a>Ruby
|
229
241
|
|
@@ -231,7 +243,7 @@ Use this method to install on the native host.
|
|
231
243
|
|
232
244
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
233
245
|
|
234
|
-
Required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.
|
246
|
+
Required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.7 in a future version.
|
235
247
|
|
236
248
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
237
249
|
|
@@ -239,7 +251,7 @@ Refer to the following sections for a proposed method for specific operating sys
|
|
239
251
|
|
240
252
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
241
253
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
242
|
-
If you have a simpler better way to install Ruby : use it ! (version: >= 2.4. Deprecation notice: the minimum will be 2.
|
254
|
+
If you have a simpler better way to install Ruby : use it ! (version: >= 2.4. Deprecation notice: the minimum will be 2.7 in a future version)
|
243
255
|
|
244
256
|
#### Generic: RVM: single user installation (not root)
|
245
257
|
|
@@ -247,18 +259,14 @@ Use this method which provides more flexibility.
|
|
247
259
|
|
248
260
|
Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
249
261
|
|
250
|
-
Install the 2 keys
|
251
|
-
|
252
|
-
```bash
|
253
|
-
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
254
|
-
```
|
255
|
-
|
256
262
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
257
263
|
|
258
264
|
```bash
|
259
265
|
\curl -sSL https://get.rvm.io | bash -s stable
|
260
266
|
```
|
261
267
|
|
268
|
+
Follow on-screen instructions to install keys, and then re-execute the command.
|
269
|
+
|
262
270
|
If you keep the same terminal (not needed if re-login):
|
263
271
|
|
264
272
|
```bash
|
@@ -306,12 +314,11 @@ rvm version
|
|
306
314
|
|
307
315
|
Install Latest stable Ruby:
|
308
316
|
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
314
|
-
|
317
|
+
- Navigate to [https://rubyinstaller.org/](https://rubyinstaller.org/) → **Downloads**.
|
318
|
+
- Download the latest Ruby installer **with devkit**. (Msys2 is needed to install some native extensions, such as `grpc`)
|
319
|
+
- Execute the installer which installs by default in: `C:\RubyVV-x64` (VV is the version number)
|
320
|
+
- At the end of the installation procedure, the Msys2 installer is automatically executed, select option 3 (msys and mingw)
|
321
|
+
- for the installation of
|
315
322
|
#### macOS: pre-installed or `brew`
|
316
323
|
|
317
324
|
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
@@ -420,8 +427,8 @@ ascli conf check_update
|
|
420
427
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
421
428
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
422
429
|
|
423
|
-
|
424
|
-
|
430
|
+
- ascp
|
431
|
+
- aspera-license (in same folder, or ../etc)
|
425
432
|
|
426
433
|
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
427
434
|
|
@@ -443,11 +450,11 @@ The format is: `file:///<path>`, where `<path>` can be either a relative path (n
|
|
443
450
|
|
444
451
|
If the embedded method is not used, the following packages are also suitable:
|
445
452
|
|
446
|
-
|
447
|
-
|
448
|
-
|
449
|
-
|
450
|
-
|
453
|
+
- IBM Aspera Connect Client (Free)
|
454
|
+
- IBM Aspera Desktop Client (Free)
|
455
|
+
- IBM Aspera CLI (Free)
|
456
|
+
- IBM Aspera High Speed Transfer Server (Licensed)
|
457
|
+
- IBM Aspera High Speed Transfer EndPoint (Licensed)
|
451
458
|
|
452
459
|
For instance, Aspera Connect Client can be installed
|
453
460
|
by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
|
@@ -465,28 +472,28 @@ A method to build one is provided here:
|
|
465
472
|
|
466
473
|
The procedure:
|
467
474
|
|
468
|
-
|
469
|
-
|
475
|
+
- Follow the non-root installation procedure with RVM, including gem
|
476
|
+
- Archive (zip, tar) the main RVM folder (includes ascli):
|
470
477
|
|
471
478
|
```bash
|
472
479
|
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
473
480
|
```
|
474
481
|
|
475
|
-
|
482
|
+
- Get the Aspera SDK.
|
476
483
|
|
477
484
|
```bash
|
478
485
|
ascli conf --show-config --fields=sdk_url
|
479
486
|
```
|
480
487
|
|
481
|
-
|
488
|
+
- Download the SDK archive from that URL.
|
482
489
|
|
483
490
|
```bash
|
484
491
|
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
485
492
|
```
|
486
493
|
|
487
|
-
|
494
|
+
- Transfer those 2 files to the target system
|
488
495
|
|
489
|
-
|
496
|
+
- On target system
|
490
497
|
|
491
498
|
```bash
|
492
499
|
cd $HOME
|
@@ -498,7 +505,7 @@ source ~/.rvm/scripts/rvm
|
|
498
505
|
ascli conf ascp install --sdk-url=file:///SDK.zip
|
499
506
|
```
|
500
507
|
|
501
|
-
|
508
|
+
- Add those lines to shell init (`.profile`)
|
502
509
|
|
503
510
|
```bash
|
504
511
|
source ~/.rvm/scripts/rvm
|
@@ -508,27 +515,27 @@ source ~/.rvm/scripts/rvm
|
|
508
515
|
|
509
516
|
The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
510
517
|
|
511
|
-
|
512
|
-
|
513
|
-
|
514
|
-
|
515
|
-
|
516
|
-
|
517
|
-
|
518
|
+
- IBM Aspera High Speed Transfer Server (FASP and Node)
|
519
|
+
- IBM Aspera on Cloud (including ATS)
|
520
|
+
- IBM Aspera Faspex
|
521
|
+
- IBM Aspera Shares
|
522
|
+
- IBM Aspera Console
|
523
|
+
- IBM Aspera Orchestrator
|
524
|
+
- and more...
|
518
525
|
|
519
526
|
`ascli` provides the following features:
|
520
527
|
|
521
|
-
|
522
|
-
|
523
|
-
|
524
|
-
|
525
|
-
|
526
|
-
|
527
|
-
|
528
|
-
|
529
|
-
|
530
|
-
|
531
|
-
|
528
|
+
- Supports most Aspera server products (on-premise and SaaS)
|
529
|
+
- Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
530
|
+
- Supports Commands, Option values and Parameters shortcuts
|
531
|
+
- FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
532
|
+
- Transfer parameters can be altered by modification of [*transfer-spec*](#transferspec), this includes requiring multi-session
|
533
|
+
- Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
534
|
+
- Supports FaspStream creation (using Node API)
|
535
|
+
- Supports Watchfolder creation (using Node API)
|
536
|
+
- Additional command plugins can be written by the user
|
537
|
+
- Supports download of faspex and Aspera on Cloud "external" links
|
538
|
+
- Supports "legacy" ssh based FASP transfers and remote commands (ascmd)
|
532
539
|
|
533
540
|
Basic usage is displayed by executing:
|
534
541
|
|
@@ -546,10 +553,10 @@ If you want to use `ascp` directly as a command line, refer to IBM Aspera docume
|
|
546
553
|
|
547
554
|
Using `ascli` with plugin `server` for command line gives advantages over ascp:
|
548
555
|
|
549
|
-
|
550
|
-
|
551
|
-
|
552
|
-
|
556
|
+
- automatic resume on error
|
557
|
+
- configuration file
|
558
|
+
- choice of transfer agents
|
559
|
+
- integrated support of multi-session
|
553
560
|
|
554
561
|
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`EX_ascp_args`).
|
555
562
|
|
@@ -573,8 +580,8 @@ On Windows, `cmd.exe` is typically used.
|
|
573
580
|
Windows process creation does not receive the list of arguments but just the whole line.
|
574
581
|
It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
575
582
|
|
576
|
-
|
577
|
-
|
583
|
+
- [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
584
|
+
- [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
578
585
|
|
579
586
|
#### Extended Values (JSON, Ruby, ...)
|
580
587
|
|
@@ -666,9 +673,9 @@ Here a single quote or a backslash protects the double quote to avoid shell proc
|
|
666
673
|
|
667
674
|
Construction of values with special characters is done like this:
|
668
675
|
|
669
|
-
|
676
|
+
- First select a syntax to represent the extended value, e.g. JSON or Ruby
|
670
677
|
|
671
|
-
|
678
|
+
- Write the expression using this syntax, for example, using JSON:
|
672
679
|
|
673
680
|
```json
|
674
681
|
{"title":"Test \" ' & \\"}
|
@@ -683,7 +690,7 @@ or using Ruby:
|
|
683
690
|
|
684
691
|
Both `"` and `\` are special characters for JSON and Ruby and can be protected with `\` (unless Ruby's extended single quote notation `%q` is used).
|
685
692
|
|
686
|
-
|
693
|
+
- Then, since the value will be evaluated by shell, any shell special characters must be protected, either using preceding `\` for each character to protect, or by enclosing in single quote:
|
687
694
|
|
688
695
|
```bash
|
689
696
|
ascli conf echo @json:{\"title\":\"Test\ \\\"\ \'\ \&\ \\\\\"} --format=json
|
@@ -748,10 +755,10 @@ There are two types of command line arguments: Commands and Options. Example :
|
|
748
755
|
ascli command subcommand --option-name=VAL1 VAL2
|
749
756
|
```
|
750
757
|
|
751
|
-
|
752
|
-
|
753
|
-
|
754
|
-
|
758
|
+
- executes *command*: `command subcommand`
|
759
|
+
- with one *option*: `option_name`
|
760
|
+
- this option is given a *value* of: `VAL1`
|
761
|
+
- the command has one additional *argument*: `VAL2`
|
755
762
|
|
756
763
|
When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example `ascli conf ov` is the same as `ascli config overview`.
|
757
764
|
|
@@ -761,16 +768,16 @@ The value of options and arguments is evaluated with the [Extended Value Syntax]
|
|
761
768
|
|
762
769
|
All options, e.g. `--log-level=debug`, are command line arguments that:
|
763
770
|
|
764
|
-
|
765
|
-
|
766
|
-
|
767
|
-
|
771
|
+
- start with `--`
|
772
|
+
- have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
|
773
|
+
- have a value, separated from name with a `=`
|
774
|
+
- can be used by prefix, provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
|
768
775
|
|
769
776
|
Exceptions:
|
770
777
|
|
771
|
-
|
772
|
-
|
773
|
-
|
778
|
+
- some options accept a short form, e.g. `-Ptoto` is equivalent to `--preset=toto`, refer to the manual or `-h`.
|
779
|
+
- some options (flags) don't take a value, e.g. `-r`
|
780
|
+
- the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
774
781
|
|
775
782
|
```bash
|
776
783
|
ascli config echo -- --sample
|
@@ -786,9 +793,9 @@ Options can be optional or mandatory, with or without (hardcoded) default value.
|
|
786
793
|
|
787
794
|
The value for *any* options can come from the following locations (in this order, last value evaluated overrides previous value):
|
788
795
|
|
789
|
-
|
790
|
-
|
791
|
-
|
796
|
+
- [Configuration file](#configfile).
|
797
|
+
- Environment variable
|
798
|
+
- Command line
|
792
799
|
|
793
800
|
Environment variable starting with prefix: ASCLI_ are taken as option values, e.g. `ASCLI_OPTION_NAME` is for `--option-name`.
|
794
801
|
|
@@ -804,11 +811,11 @@ Some options and parameters are mandatory and other optional. By default, the to
|
|
804
811
|
|
805
812
|
The behavior can be controlled with:
|
806
813
|
|
807
|
-
|
808
|
-
|
809
|
-
|
810
|
-
|
811
|
-
|
814
|
+
- --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
815
|
+
- yes : missing mandatory parameters/options are asked to the user
|
816
|
+
- no : missing mandatory parameters/options raise an error message
|
817
|
+
- --ask-options=<yes|no> (default=no)
|
818
|
+
- optional parameters/options are asked to user
|
812
819
|
|
813
820
|
### Output
|
814
821
|
|
@@ -819,12 +826,12 @@ The information displayed depends on the action.
|
|
819
826
|
|
820
827
|
Depending on action, the output will contain:
|
821
828
|
|
822
|
-
|
823
|
-
|
824
|
-
|
825
|
-
|
826
|
-
|
827
|
-
|
829
|
+
- `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
830
|
+
- `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
831
|
+
- `value_list` : a table with one column.
|
832
|
+
- `empty` : nothing
|
833
|
+
- `status` : a message
|
834
|
+
- `other_struct` : a complex structure that cannot be displayed as an array
|
828
835
|
|
829
836
|
#### Format of output
|
830
837
|
|
@@ -841,13 +848,13 @@ If transposition of single object is not desired, use option: `transpose_single`
|
|
841
848
|
|
842
849
|
The style of output can be set using the `format` parameter, supporting:
|
843
850
|
|
844
|
-
|
845
|
-
|
846
|
-
|
847
|
-
|
848
|
-
|
849
|
-
|
850
|
-
|
851
|
+
- `text` : Value as String
|
852
|
+
- `table` : Text table
|
853
|
+
- `ruby` : Ruby code
|
854
|
+
- `json` : JSON code
|
855
|
+
- `jsonpp` : JSON pretty printed
|
856
|
+
- `yaml` : YAML
|
857
|
+
- `csv` : Comma Separated Values
|
851
858
|
|
852
859
|
#### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
853
860
|
|
@@ -872,15 +879,15 @@ Note that `select` filters selected elements from the result of API calls, while
|
|
872
879
|
|
873
880
|
Output messages are categorized in 3 types:
|
874
881
|
|
875
|
-
|
876
|
-
|
877
|
-
|
882
|
+
- `info` output contain additional information, such as number of elements in a table
|
883
|
+
- `data` output contain the actual output of the command (object, or list of objects)
|
884
|
+
- `error`output contain error messages
|
878
885
|
|
879
886
|
The option `display` controls the level of output:
|
880
887
|
|
881
|
-
|
882
|
-
|
883
|
-
|
888
|
+
- `info` displays all messages: `info`, `data`, and `error`
|
889
|
+
- `data` display `data` and `error` messages
|
890
|
+
- `error` display only error messages.
|
884
891
|
|
885
892
|
By default, secrets are removed from output: option `show_secrets` defaults to `no`, unless `display` is `data`, to allows piping results.
|
886
893
|
To hide secrets from output, set option `show_secrets` to `no`.
|
@@ -889,12 +896,12 @@ To hide secrets from output, set option `show_secrets` to `no`.
|
|
889
896
|
|
890
897
|
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
891
898
|
|
892
|
-
|
893
|
-
|
894
|
-
|
895
|
-
|
896
|
-
|
897
|
-
|
899
|
+
- DEF : default display of columns (that's the default, when not set)
|
900
|
+
- ALL : all columns available
|
901
|
+
- a,b,c : the list of attributes specified by the comma separated list
|
902
|
+
- Array extended value: for instance, @json:'["a","b","c"]' same as above
|
903
|
+
- +a,b,c : add selected properties to the default selection.
|
904
|
+
- -a,b,c : remove selected properties from the default selection.
|
898
905
|
|
899
906
|
### <a id="extended"></a>Extended Value Syntax
|
900
907
|
|
@@ -910,24 +917,24 @@ The difference between reader and decoder is order and ordinality. Both act like
|
|
910
917
|
|
911
918
|
The following "readers" are supported (returns value in []):
|
912
919
|
|
913
|
-
|
914
|
-
|
915
|
-
|
916
|
-
|
917
|
-
|
918
|
-
|
919
|
-
|
920
|
+
- @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
|
921
|
+
- @file:PATH : [String] read value from a URL, e.g. `--fpac=@uri:http://serv/f.pac`
|
922
|
+
- @uri:URL : [String] read value from a file (prefix `~/` is replaced with the users home folder), e.g. `--key=@file:~/.ssh/mykey`
|
923
|
+
- @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
|
924
|
+
- @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
925
|
+
- @stdin: : [String] read from stdin (no value on right)
|
926
|
+
- @preset:NAME : [Hash] get whole option preset value by name. Subvalues can also be used using `.` as separator. e.g. foo.bar is conf[foo][bar]
|
920
927
|
|
921
928
|
In addition it is possible to decode a value, using one or multiple decoders :
|
922
929
|
|
923
|
-
|
924
|
-
|
925
|
-
|
926
|
-
|
927
|
-
|
928
|
-
|
929
|
-
|
930
|
-
|
930
|
+
- @base64: [String] decode a base64 encoded string
|
931
|
+
- @json: [any] decode JSON values (convenient to provide complex structures)
|
932
|
+
- @zlib: [String] uncompress data
|
933
|
+
- @ruby: [any] execute ruby code
|
934
|
+
- @csvt: [Array] decode a titled CSV value
|
935
|
+
- @lines: [Array] split a string in multiple lines and return an array
|
936
|
+
- @list: [Array] split a string in multiple items taking first character as separator and return an array
|
937
|
+
- @incps: [Hash] include values of presets specified by key `incps` in input hash
|
931
938
|
|
932
939
|
To display the result of an extended value, use the `config echo` command.
|
933
940
|
|
@@ -1057,7 +1064,7 @@ The command `update` allows the easy creation of [option preset](#lprt) by simpl
|
|
1057
1064
|
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_ --ts=@json:'{"precalculate_job_size":true}'
|
1058
1065
|
```
|
1059
1066
|
|
1060
|
-
|
1067
|
+
- This creates a [option preset](#lprt) `demo_server` with all provided options.
|
1061
1068
|
|
1062
1069
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
1063
1070
|
|
@@ -1195,16 +1202,16 @@ demo_server:
|
|
1195
1202
|
|
1196
1203
|
We can see here:
|
1197
1204
|
|
1198
|
-
|
1199
|
-
|
1200
|
-
|
1201
|
-
|
1205
|
+
- The configuration was created with CLI version 0.3.7
|
1206
|
+
- the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
1207
|
+
- the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
1208
|
+
- the default [option preset](#lprt) to load in any case is : `cli_default`
|
1202
1209
|
|
1203
1210
|
Two [option presets](#lprt) are reserved:
|
1204
1211
|
|
1205
|
-
|
1212
|
+
- `config` contains a single value: `version` showing the CLI
|
1206
1213
|
version used to create the configuration file. It is used to check compatibility.
|
1207
|
-
|
1214
|
+
- `default` is reserved to define the default [option preset](#lprt) name used for known plugins.
|
1208
1215
|
|
1209
1216
|
The user may create as many [option presets](#lprt) as needed. For instance, a particular [option preset](#lprt) can be created for a particular application instance and contain URL and credentials.
|
1210
1217
|
|
@@ -1233,11 +1240,11 @@ Some options are global, some options are available only for some plugins. (the
|
|
1233
1240
|
|
1234
1241
|
Options are loaded using this algorithm:
|
1235
1242
|
|
1236
|
-
|
1237
|
-
|
1238
|
-
|
1239
|
-
|
1240
|
-
|
1243
|
+
- If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
1244
|
+
- else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
1245
|
+
- If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
1246
|
+
- Environment variables are evaluated
|
1247
|
+
- Command line options are evaluated
|
1241
1248
|
|
1242
1249
|
Parameters are evaluated in the order of command line.
|
1243
1250
|
|
@@ -1286,7 +1293,7 @@ ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=_p
|
|
1286
1293
|
|
1287
1294
|
This can also be provisioned in a config file:
|
1288
1295
|
|
1289
|
-
|
1296
|
+
- Build [option preset](#lprt)
|
1290
1297
|
|
1291
1298
|
```bash
|
1292
1299
|
ascli config preset set shares06 url https://10.25.0.6
|
@@ -1306,19 +1313,19 @@ or
|
|
1306
1313
|
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=_pass_here_
|
1307
1314
|
```
|
1308
1315
|
|
1309
|
-
|
1316
|
+
- Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
1310
1317
|
|
1311
1318
|
```bash
|
1312
1319
|
ascli config preset set default shares shares06
|
1313
1320
|
```
|
1314
1321
|
|
1315
|
-
|
1322
|
+
- Display the content of configuration file in table format
|
1316
1323
|
|
1317
1324
|
```bash
|
1318
1325
|
ascli config overview
|
1319
1326
|
```
|
1320
1327
|
|
1321
|
-
|
1328
|
+
- Execute a command on the shares application using default parameters
|
1322
1329
|
|
1323
1330
|
```bash
|
1324
1331
|
ascli shares repo browse /
|
@@ -1326,66 +1333,67 @@ ascli shares repo browse /
|
|
1326
1333
|
|
1327
1334
|
### <a id="vault"></a>Secret Vault
|
1328
1335
|
|
1329
|
-
|
1330
|
-
|
1331
|
-
|
1332
|
-
|
1333
|
-
#### Using system keychain
|
1336
|
+
Password and secrets are command options.
|
1337
|
+
They can be provided on command line, env vars, files etc.
|
1338
|
+
A more secure option is to retrieve values from a secret vault.
|
1334
1339
|
|
1335
|
-
|
1340
|
+
The vault is used with options `vault` and `vault_password`.
|
1336
1341
|
|
1337
|
-
|
1342
|
+
`vault` defines the vault to be used and shall be a Hash, example:
|
1338
1343
|
|
1339
|
-
|
1340
|
-
|
1341
|
-
|
1344
|
+
```json
|
1345
|
+
{"type":"system","name":"ascli"}
|
1346
|
+
```
|
1342
1347
|
|
1343
|
-
|
1344
|
-
|
1348
|
+
`vault_password` specifies the password for the vault.
|
1349
|
+
Although it can be specified on command line, for security reason you can hide the value.
|
1350
|
+
For example it can be specified on command line like this:
|
1345
1351
|
|
1346
|
-
|
1352
|
+
```bash
|
1353
|
+
export ASCLI_VAULT_PASSWORD
|
1354
|
+
read -s ASCLI_VAULT_PASSWORD
|
1355
|
+
```
|
1347
1356
|
|
1348
|
-
|
1357
|
+
#### Vault: System keychain
|
1349
1358
|
|
1350
|
-
|
1351
|
-
* `get`
|
1352
|
-
* `list`
|
1353
|
-
* `delete`
|
1359
|
+
> **macOS only**
|
1354
1360
|
|
1355
|
-
|
1361
|
+
It is possible to manage secrets in macOS keychain (only read supported currently).
|
1356
1362
|
|
1357
|
-
|
1363
|
+
```json
|
1364
|
+
--vault=@json:'{"type":"system","name":"ascli"}'
|
1365
|
+
```
|
1358
1366
|
|
1359
|
-
|
1367
|
+
#### Vault: Encrypted file
|
1360
1368
|
|
1361
|
-
|
1369
|
+
It is possible to store and use secrets encrypted in a file.
|
1362
1370
|
|
1363
|
-
|
1371
|
+
```json
|
1372
|
+
--vault=@json:'{"type":"file","name":"vault.bin"}'
|
1373
|
+
```
|
1364
1374
|
|
1365
|
-
|
1366
|
-
ascli conf id my_secrets set 'access_key1' 'secret1'
|
1375
|
+
`name` is the file path, absolute or relative to the config folder `ASCLI__HOME`.
|
1367
1376
|
|
1368
|
-
|
1377
|
+
#### Vault: Operations
|
1369
1378
|
|
1370
|
-
|
1379
|
+
For this use the `config vault` command.
|
1371
1380
|
|
1372
|
-
|
1373
|
-
```
|
1381
|
+
Then secrets can be manipulated using commands:
|
1374
1382
|
|
1375
|
-
|
1376
|
-
|
1383
|
+
- `create`
|
1384
|
+
- `show`
|
1385
|
+
- `list`
|
1386
|
+
- `delete`
|
1377
1387
|
|
1378
1388
|
```bash
|
1379
|
-
ascli conf
|
1389
|
+
ascli conf vault create mylabel @json:'{"password":"__value_here__","description":"for this account"}'
|
1380
1390
|
```
|
1381
1391
|
|
1382
|
-
|
1392
|
+
#### <a id="config_finder"></a>Configuration Finder
|
1383
1393
|
|
1384
|
-
|
1394
|
+
When a secret is needed by a sub command, the command can search for existing configurations in the config file.
|
1385
1395
|
|
1386
|
-
|
1387
|
-
ascli conf vault get --username=access_key1
|
1388
|
-
```
|
1396
|
+
The lookup is done by comparing the service URL and username (or access key).
|
1389
1397
|
|
1390
1398
|
### <a id="private_key"></a>Private Key
|
1391
1399
|
|
@@ -1397,15 +1405,17 @@ The private key can be protected by a passphrase or not.
|
|
1397
1405
|
If the key is protected by a passphrase, then it will be prompted.
|
1398
1406
|
(some plugins support option `passphrase`)
|
1399
1407
|
|
1400
|
-
|
1401
|
-
|
1402
|
-
|
1408
|
+
The following commands use the shell variable `PRIVKEYFILE`.
|
1409
|
+
Set it to the desired safe location of the private key.
|
1410
|
+
Typically, in `$HOME/.ssh` or `$HOME/.aspera/ascli`:
|
1403
1411
|
|
1404
1412
|
```bash
|
1405
1413
|
PRIVKEYFILE=~/.aspera/ascli/my_private_key
|
1406
1414
|
```
|
1407
1415
|
|
1408
|
-
|
1416
|
+
Several methods can be used to generate a key pair:
|
1417
|
+
|
1418
|
+
- `ascli`
|
1409
1419
|
|
1410
1420
|
The generated key is of type RSA, by default: 4096 bit.
|
1411
1421
|
For convenience, the public key is also extracted with extension `.pub`.
|
@@ -1415,23 +1425,30 @@ The key is not passphrase protected.
|
|
1415
1425
|
ascli config genkey ${PRIVKEYFILE} 4096
|
1416
1426
|
```
|
1417
1427
|
|
1418
|
-
|
1428
|
+
- `ssh-keygen`
|
1419
1429
|
|
1420
1430
|
Both private and public keys are generated, option `-N` is for passphrase.
|
1421
1431
|
|
1422
1432
|
```bash
|
1423
|
-
ssh-keygen -t rsa -b 4096 -N '' -f ${PRIVKEYFILE}
|
1433
|
+
ssh-keygen -t rsa -b 4096 -m PEM -N '' -f ${PRIVKEYFILE}
|
1424
1434
|
```
|
1425
1435
|
|
1426
|
-
|
1436
|
+
- `openssl`
|
1427
1437
|
|
1428
|
-
|
1429
|
-
To generate a private key pair without passphrase the following can be used on any system:
|
1438
|
+
To generate a private key pair with a passphrase the following can be used on any system:
|
1430
1439
|
|
1431
1440
|
```bash
|
1432
|
-
openssl genrsa -passout pass:
|
1433
|
-
openssl rsa -passin pass:dummypassword -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1441
|
+
openssl genrsa -passout pass:_passphrase_here_ -out ${PRIVKEYFILE}.protected 4096
|
1434
1442
|
openssl rsa -pubout -in ${PRIVKEYFILE} -out ${PRIVKEYFILE}.pub
|
1443
|
+
```
|
1444
|
+
|
1445
|
+
`openssl` is sometimes compiled to support option `-nodes` (no DES, i.e. no passphrase, e.g. on macOS).
|
1446
|
+
In that case, add option `-nodes` instead of `-passout pass:_passphrase_here_` to generate a key without passphrase.
|
1447
|
+
|
1448
|
+
If option `-nodes` is not available, the passphrase can be removed using this method:
|
1449
|
+
|
1450
|
+
```bash
|
1451
|
+
openssl rsa -passin pass:_passphrase_here_ -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1435
1452
|
rm -f ${PRIVKEYFILE}.protected
|
1436
1453
|
```
|
1437
1454
|
|
@@ -1505,9 +1522,9 @@ REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, O
|
|
1505
1522
|
|
1506
1523
|
Those are using options:
|
1507
1524
|
|
1508
|
-
|
1509
|
-
|
1510
|
-
|
1525
|
+
- url
|
1526
|
+
- username
|
1527
|
+
- password
|
1511
1528
|
|
1512
1529
|
Those can be provided using command line, parameter set, env var, see section above.
|
1513
1530
|
|
@@ -1532,13 +1549,13 @@ Note that when using the `direct` agent (`ascp`), additional transfer logs can b
|
|
1532
1549
|
|
1533
1550
|
Examples:
|
1534
1551
|
|
1535
|
-
|
1552
|
+
- display debugging log on `stdout`:
|
1536
1553
|
|
1537
1554
|
```bash
|
1538
1555
|
ascli conf over --log-level=debug --logger=stdout
|
1539
1556
|
```
|
1540
1557
|
|
1541
|
-
|
1558
|
+
- log errors to `syslog`:
|
1542
1559
|
|
1543
1560
|
```bash
|
1544
1561
|
ascli conf over --log-level=error --logger=syslog
|
@@ -1582,76 +1599,100 @@ ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
|
1582
1599
|
|
1583
1600
|
Some actions may require the use of a graphical tool:
|
1584
1601
|
|
1585
|
-
|
1586
|
-
|
1602
|
+
- a browser for Aspera on Cloud authentication (web auth method)
|
1603
|
+
- a text editor for configuration file edition
|
1587
1604
|
|
1588
1605
|
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1589
1606
|
It is also possible to force the graphical mode with option --ui :
|
1590
1607
|
|
1591
|
-
|
1592
|
-
|
1608
|
+
- `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
1609
|
+
- `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
1593
1610
|
|
1594
1611
|
### Proxy
|
1595
1612
|
|
1596
|
-
There are several types of network connections, each of them use a different mechanism to define a
|
1613
|
+
There are several types of network connections, each of them use a different mechanism to define a (forward) **proxy**:
|
1597
1614
|
|
1598
|
-
|
1615
|
+
- Ruby HTTP: REST and HTTPGW client
|
1616
|
+
- Legacy Aspera HTTP/S Fallback
|
1617
|
+
- Aspera FASP
|
1599
1618
|
|
1600
|
-
|
1601
|
-
See [Ruby findproxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1619
|
+
Refer to the following sections.
|
1602
1620
|
|
1603
|
-
|
1604
|
-
export http_proxy=http://myproxy.org.net:3128
|
1605
|
-
```
|
1621
|
+
### Proxy for REST and HTTPGW
|
1606
1622
|
|
1607
|
-
|
1623
|
+
There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is used.
|
1608
1624
|
|
1609
|
-
|
1625
|
+
The `http_proxy` environment variable (**lower case**, preferred) can be set to the URL of the proxy, e.g. `http://myproxy.org.net:3128`.
|
1626
|
+
Refer to [Ruby findproxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1610
1627
|
|
1611
|
-
|
1612
|
-
|
1613
|
-
#### HTTP proxy legacy Aspera HTTP fallback transfers
|
1628
|
+
> Note: Ruby expects a URL and `myproxy.org.net:3128` alone is **not** accepted.
|
1614
1629
|
|
1615
|
-
|
1616
|
-
|
1617
|
-
|
1618
|
-
#### Proxy auto config
|
1630
|
+
```bash
|
1631
|
+
export http_proxy=http://proxy.example.com:3128
|
1632
|
+
```
|
1619
1633
|
|
1620
|
-
The `fpac` option
|
1621
|
-
To read the script from a URL (`http:`, `https:` and `file:`), use: `@uri:`.
|
1622
|
-
A minimal script can be specified
|
1634
|
+
The `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) javascript value.
|
1635
|
+
To read the script from a URL (`http:`, `https:` and `file:`), use prefix: `@uri:`.
|
1636
|
+
A minimal script can be specified to define the use of a local proxy:
|
1623
1637
|
|
1624
1638
|
```bash
|
1625
|
-
|
1639
|
+
ascli --fpac='function FindProxyForURL(url, host){return "PROXY localhost:3128"}' ...
|
1626
1640
|
```
|
1627
1641
|
|
1628
|
-
The PAC file
|
1629
|
-
|
1630
|
-
The PAC file can be tested with command: `config proxy_check`. Example, using command line option:
|
1642
|
+
The result of a PAC file can be tested with command: `config proxy_check`.
|
1643
|
+
Example, using command line option:
|
1631
1644
|
|
1632
1645
|
```bash
|
1633
|
-
ascli conf proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:
|
1646
|
+
ascli conf proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:3128;DIRECT";}' http://example.com
|
1647
|
+
```
|
1648
|
+
|
1649
|
+
```text
|
1634
1650
|
PROXY proxy.example.com:1234;DIRECT
|
1635
1651
|
```
|
1636
1652
|
|
1637
1653
|
```bash
|
1638
1654
|
ascli config proxy_check --fpac=@file:./proxy.pac http://www.example.com
|
1655
|
+
```
|
1656
|
+
|
1657
|
+
```text
|
1639
1658
|
PROXY proxy.example.com:8080
|
1640
1659
|
```
|
1641
1660
|
|
1642
1661
|
```bash
|
1643
1662
|
ascli config proxy_check --fpac=@uri:http://server/proxy.pac http://www.example.com
|
1663
|
+
```
|
1664
|
+
|
1665
|
+
```text
|
1644
1666
|
PROXY proxy.example.com:8080
|
1645
1667
|
```
|
1646
1668
|
|
1669
|
+
If the proxy requires credentials, then use option `proxy_credentials` with username and password provided as an `Array`:
|
1670
|
+
|
1671
|
+
```bash
|
1672
|
+
ascli --proxy-credentials=@json:'["__username_here__","__password_here__"]' ...
|
1673
|
+
```
|
1674
|
+
|
1675
|
+
```bash
|
1676
|
+
ascli --proxy-credentials=@list::__username_here__:__password_here__ ...
|
1677
|
+
```
|
1678
|
+
|
1679
|
+
### Proxy for Legacy Aspera HTTP/S Fallback
|
1680
|
+
|
1681
|
+
To specify a proxy for legacy HTTP fallback, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url` (only supported with the `direct` agent).
|
1682
|
+
(It is also possible to use `EX_ascp_args` and native options in `direct`)
|
1683
|
+
|
1684
|
+
### FASP proxy (forward) for transfers
|
1685
|
+
|
1686
|
+
To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) parameter: `EX_fasp_proxy_url` (only supported with the `direct` agent).
|
1687
|
+
|
1647
1688
|
### <a id="client"></a>FASP configuration
|
1648
1689
|
|
1649
1690
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
1650
1691
|
|
1651
|
-
|
1652
|
-
|
1653
|
-
|
1654
|
-
|
1692
|
+
- `show` : shows the path of ascp used
|
1693
|
+
- `use` : list,download connect client versions available on internet
|
1694
|
+
- `products` : list Aspera transfer products available locally
|
1695
|
+
- `connect` : list,download connect client versions available on internet
|
1655
1696
|
|
1656
1697
|
#### Show path of currently used `ascp`
|
1657
1698
|
|
@@ -1793,11 +1834,11 @@ This [*transfer-spec*](#transferspec) will be executed by a transfer client, her
|
|
1793
1834
|
|
1794
1835
|
There are currently 3 agents:
|
1795
1836
|
|
1796
|
-
|
1797
|
-
|
1798
|
-
|
1799
|
-
|
1800
|
-
|
1837
|
+
- [`direct`](#agt_direct) : a local execution of `ascp`
|
1838
|
+
- [`connect`](#agt_connect) : use of a local Connect Client
|
1839
|
+
- [`node`](#agt_node) : use of an Aspera Transfer Node (potentially *remote*).
|
1840
|
+
- [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1841
|
+
- [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1801
1842
|
|
1802
1843
|
Note that all transfer operation are seen from the point of view of the agent.
|
1803
1844
|
For instance, a node agent making an "upload", or "package send" operation,
|
@@ -1861,13 +1902,13 @@ To store ascp logs in file `aspera-scp-transfer.log` in a folder, use `--ts=@jso
|
|
1861
1902
|
|
1862
1903
|
In addition to standard methods described in section [File List](#file_list), it is possible to specify the list of file using those additional methods:
|
1863
1904
|
|
1864
|
-
|
1905
|
+
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_file_list`
|
1865
1906
|
|
1866
1907
|
```javascript
|
1867
1908
|
--sources=@ts --ts=@json:'{"EX_file_list":"filelist.txt"}'
|
1868
1909
|
```
|
1869
1910
|
|
1870
|
-
|
1911
|
+
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_ascp_args`
|
1871
1912
|
|
1872
1913
|
```javascript
|
1873
1914
|
--sources=@ts --ts=@json:'{"EX_ascp_args":["--file-list","myfilelist"]}'
|
@@ -1907,7 +1948,16 @@ If the `password` value begins with `Bearer` then the `username` is expected to
|
|
1907
1948
|
|
1908
1949
|
#### <a id="agt_httpgw"></a>HTTP Gateway
|
1909
1950
|
|
1910
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
1951
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
1952
|
+
|
1953
|
+
Parameters provided in option `transfer_info` are:
|
1954
|
+
|
1955
|
+
<table>
|
1956
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1957
|
+
<tr><td>url</td><td>string</td><td>URL of the HTTP GW</br>Mandatory</td></tr>
|
1958
|
+
<tr><td>upload_bar_refresh_sec</td><td>float</td><td>Refresh rate for upload progress bar</td></tr>
|
1959
|
+
<tr><td>upload_chunksize</td><td>int</td><td>Size in bytes of chunks for upload</td></tr>
|
1960
|
+
</table>
|
1911
1961
|
|
1912
1962
|
Example:
|
1913
1963
|
|
@@ -1915,7 +1965,7 @@ Example:
|
|
1915
1965
|
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1916
1966
|
```
|
1917
1967
|
|
1918
|
-
|
1968
|
+
> The gateway only supports transfers authorized with a token.
|
1919
1969
|
|
1920
1970
|
#### <a id="agt_trsdk"></a>Transfer SDK
|
1921
1971
|
|
@@ -1923,7 +1973,8 @@ Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
|
1923
1973
|
|
1924
1974
|
By default it will listen on local port `55002` on `127.0.0.1`.
|
1925
1975
|
|
1926
|
-
The gem `grpc` was removed from dependencies, as it requires compilation of a native part.
|
1976
|
+
The gem `grpc` was removed from dependencies, as it requires compilation of a native part.
|
1977
|
+
So, to use the Transfer SDK you should install this gem:
|
1927
1978
|
|
1928
1979
|
```bash
|
1929
1980
|
gem install grpc
|
@@ -1931,9 +1982,9 @@ gem install grpc
|
|
1931
1982
|
|
1932
1983
|
On Windows the compilation may fail for various reasons (3.1.1):
|
1933
1984
|
|
1934
|
-
|
1985
|
+
- `cannot find -lx64-ucrt-ruby310`
|
1935
1986
|
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
1936
|
-
|
1987
|
+
- `conflicting types for 'gettimeofday'`
|
1937
1988
|
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)` ,i.e. change `struct timezone` to `void`
|
1938
1989
|
|
1939
1990
|
### <a id="transferspec"></a>Transfer Specification
|
@@ -1941,11 +1992,11 @@ On Windows the compilation may fail for various reasons (3.1.1):
|
|
1941
1992
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
1942
1993
|
is described in a [*transfer-spec*](#transferspec) (Transfer Specification), such as:
|
1943
1994
|
|
1944
|
-
|
1945
|
-
|
1946
|
-
|
1947
|
-
|
1948
|
-
|
1995
|
+
- server address
|
1996
|
+
- transfer user name
|
1997
|
+
- credentials
|
1998
|
+
- file list
|
1999
|
+
- etc...
|
1949
2000
|
|
1950
2001
|
`ascli` builds a default [*transfer-spec*](#transferspec) internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1951
2002
|
|
@@ -1955,8 +2006,8 @@ It is possible to specify ascp options when the `transfer` option is set to [`di
|
|
1955
2006
|
|
1956
2007
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
1957
2008
|
|
1958
|
-
|
1959
|
-
|
2009
|
+
- common to all [Transfer Agent](#agents)
|
2010
|
+
- not dependent on command line limitations (special characters...)
|
1960
2011
|
|
1961
2012
|
A [*transfer-spec*](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
1962
2013
|
|
@@ -1967,9 +2018,9 @@ All standard [*transfer-spec*](#transferspec) parameters can be specified.
|
|
1967
2018
|
|
1968
2019
|
References:
|
1969
2020
|
|
1970
|
-
|
1971
|
-
|
1972
|
-
|
2021
|
+
- [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
2022
|
+
- [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
2023
|
+
- [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
1973
2024
|
|
1974
2025
|
Parameters can be displayed with commands:
|
1975
2026
|
|
@@ -1980,9 +2031,9 @@ ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
|
1980
2031
|
|
1981
2032
|
Columns:
|
1982
2033
|
|
1983
|
-
|
1984
|
-
|
1985
|
-
|
2034
|
+
- D=Direct (local `ascp` execution)
|
2035
|
+
- N=Node API
|
2036
|
+
- C=Connect Client
|
1986
2037
|
|
1987
2038
|
`ascp` argument or environment variable is provided in description.
|
1988
2039
|
|
@@ -1993,79 +2044,79 @@ Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct).
|
|
1993
2044
|
<tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>DEPRECATED: Prefer to use standard parameter: content_protection_password<br/>(env:ASPERA_SCP_FILEPASS)</td></tr>
|
1994
2045
|
<tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr>
|
1995
2046
|
<tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr>
|
1996
|
-
<tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr>
|
1997
|
-
<tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr>
|
2047
|
+
<tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x {string})</td></tr>
|
2048
|
+
<tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j {int})</td></tr>
|
1998
2049
|
<tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr>
|
1999
2050
|
<tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr>
|
2000
2051
|
<tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr>
|
2001
2052
|
<tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/>May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr>
|
2002
|
-
<tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr>
|
2053
|
+
<tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i {array})</td></tr>
|
2003
2054
|
<tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr>
|
2004
2055
|
<tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr>
|
2005
|
-
<tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c)</td></tr>
|
2056
|
+
<tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm<br/>(-c (conversion){enum})</td></tr>
|
2006
2057
|
<tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr>
|
2007
|
-
<tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt)</td></tr>
|
2058
|
+
<tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side encryption at rest. (CSEAR, content protection)<br/>Allowed values: encrypt, decrypt<br/>(--file-crypt {enum})</td></tr>
|
2008
2059
|
<tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password. (content protection)<br/>(env:ASPERA_SCP_FILEPASS)</td></tr>
|
2009
2060
|
<tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer specified by application<br/>(env:ASPERA_SCP_COOKIE)</td></tr>
|
2010
2061
|
<tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr>
|
2011
2062
|
<tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects<br/>on the source are not deleted.<br/>(--delete-before-transfer)</td></tr>
|
2012
2063
|
<tr><td>delete_source</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr>
|
2013
2064
|
<tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr>
|
2014
|
-
<tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>UDP datagram size in bytes<br/>(-Z)</td></tr>
|
2015
|
-
<tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr>
|
2016
|
-
<tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr>
|
2017
|
-
<tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr>
|
2018
|
-
<tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr>
|
2065
|
+
<tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>UDP datagram size in bytes<br/>(-Z {int})</td></tr>
|
2066
|
+
<tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode (conversion){enum})</td></tr>
|
2067
|
+
<tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than {int})</td></tr>
|
2068
|
+
<tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than {int})</td></tr>
|
2069
|
+
<tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O {int})</td></tr>
|
2019
2070
|
<tr><td>file_checksum</td><td>string</td><td>Y</td><td>Y</td><td> </td><td>Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: sha-512, sha-384, sha-256, sha1, md5, none</td></tr>
|
2020
|
-
<tr><td>http_fallback</td><td>bool<br/>string</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y)</td></tr>
|
2021
|
-
<tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port when no cipher is used<br/>(-t)</td></tr>
|
2022
|
-
<tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port when cipher is used<br/>(-t)</td></tr>
|
2071
|
+
<tr><td>http_fallback</td><td>bool<br/>string</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(-y (conversion){bool}|{string})</td></tr>
|
2072
|
+
<tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port when no cipher is used<br/>(-t {int})</td></tr>
|
2073
|
+
<tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port when cipher is used<br/>(-t {int})</td></tr>
|
2023
2074
|
<tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2024
2075
|
<tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2025
2076
|
<tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2026
2077
|
<tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2027
2078
|
<tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2028
2079
|
<tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2029
|
-
<tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr>
|
2030
|
-
<tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td
|
2080
|
+
<tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m {int})</td></tr>
|
2081
|
+
<tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td> </td><td>The relative path to which the files will be moved after the transfer at the source side. Available as of 3.8.0.<br/>(--move-after-transfer {string})</td></tr>
|
2031
2082
|
<tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with resume_policy=none.<br/></td></tr>
|
2032
|
-
<tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold)</td></tr>
|
2033
|
-
<tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr>
|
2083
|
+
<tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(--multi-session-threshold {int})</td></tr>
|
2084
|
+
<tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite {enum})</td></tr>
|
2034
2085
|
<tr><td>password</td><td>string</td><td> </td><td>Y</td><td> </td><td>Password for local Windows user when transfer user associated with node api user is not the same as the one running asperanoded.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node api only.</td></tr>
|
2035
2086
|
<tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Array of path to the source (required) and a path to the destination (optional).</td></tr>
|
2036
2087
|
<tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr>
|
2037
2088
|
<tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr>
|
2038
|
-
<tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls)</td></tr>
|
2089
|
+
<tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowed values: none, native, metafile<br/>(--preserve-acls {enum})</td></tr>
|
2039
2090
|
<tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr>
|
2040
2091
|
<tr><td>preserve_file_owner_gid</td><td>bool</td><td>Y</td><td> </td><td> </td><td>Preserve the group ID for a file owner<br/>(--preserve-file-owner-gid)</td></tr>
|
2041
2092
|
<tr><td>preserve_file_owner_uid</td><td>bool</td><td>Y</td><td> </td><td> </td><td>Preserve the user ID for a file owner<br/>(--preserve-file-owner-uid)</td></tr>
|
2042
2093
|
<tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr>
|
2043
|
-
<tr><td>preserve_remote_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls)</td></tr>
|
2094
|
+
<tr><td>preserve_remote_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve remote access control lists.<br/>Allowed values: none, native, metafile<br/>(--remote-preserve-acls {enum})</td></tr>
|
2044
2095
|
<tr><td>preserve_source_access_time</td><td>bool</td><td>Y</td><td> </td><td> </td><td>Preserve the time logged for when the source file was accessed<br/>(--preserve-source-access-time)</td></tr>
|
2045
2096
|
<tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr>
|
2046
|
-
<tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr>
|
2047
|
-
<tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr>
|
2097
|
+
<tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/>dnat(s)://[user[:password]@]server:port<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy {string})</td></tr>
|
2098
|
+
<tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy {enum})</td></tr>
|
2048
2099
|
<tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/>Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr>
|
2049
|
-
<tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr>
|
2100
|
+
<tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host {string})</td></tr>
|
2050
2101
|
<tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr>
|
2051
|
-
<tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr>
|
2102
|
+
<tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user {string})</td></tr>
|
2052
2103
|
<tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr>
|
2053
2104
|
<tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr>
|
2054
2105
|
<tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>Remove empty source subdirectories and remove the source directory itself, if empty<br/>(--remove-empty-source-directory)</td></tr>
|
2055
2106
|
<tr><td>remove_skipped</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well.<br/>(--remove-skipped)</td></tr>
|
2056
|
-
<tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr>
|
2107
|
+
<tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k (conversion){enum})</td></tr>
|
2057
2108
|
<tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr>
|
2058
|
-
<tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr>
|
2109
|
+
<tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64 (conversion){string})</td></tr>
|
2059
2110
|
<tr><td>source_root_id</td><td>string</td><td> </td><td>Y</td><td> </td><td>The file ID of the source root directory. Required when using Bearer token auth for the source node.</td></tr>
|
2060
|
-
<tr><td>src_base</td><td>string</td><td>Y</td><td>Y</td><td> </td><td>Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64)</td></tr>
|
2061
|
-
<tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies SSH (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr>
|
2111
|
+
<tr><td>src_base</td><td>string</td><td>Y</td><td>Y</td><td> </td><td>Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(--src-base64 (conversion){string})</td></tr>
|
2112
|
+
<tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies SSH (TCP) port. Default: local:22, other:33001<br/>(-P {int})</td></tr>
|
2062
2113
|
<tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----\nMII...<br/>Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr>
|
2063
2114
|
<tr><td>ssh_private_key_passphrase</td><td>string</td><td>Y</td><td> </td><td> </td><td>The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(env:ASPERA_SCP_PASS)</td></tr>
|
2064
|
-
<tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr>
|
2065
|
-
<tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr>
|
2066
|
-
<tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer as JSON<br/>(--tags64)</td></tr>
|
2115
|
+
<tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp {string})</td></tr>
|
2116
|
+
<tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links {enum})</td></tr>
|
2117
|
+
<tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer as JSON<br/>(--tags64 (conversion){hash})</td></tr>
|
2067
2118
|
<tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node API.</td></tr>
|
2068
|
-
<tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr>
|
2119
|
+
<tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l {int})</td></tr>
|
2069
2120
|
<tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr>
|
2070
2121
|
<tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr>
|
2071
2122
|
<tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr>
|
@@ -2078,8 +2129,8 @@ Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct).
|
|
2078
2129
|
|
2079
2130
|
The destination folder is set by `ascli` by default to:
|
2080
2131
|
|
2081
|
-
|
2082
|
-
|
2132
|
+
- `.` for downloads
|
2133
|
+
- `/` for uploads
|
2083
2134
|
|
2084
2135
|
It is specified by the [*transfer-spec*](#transferspec) parameter `destination_root`.
|
2085
2136
|
As such, it can be modified with option: `--ts=@json:'{"destination_root":"<path>"}'`.
|
@@ -2101,7 +2152,7 @@ The `sources` and `src_type` options provide convenient ways to populate the tra
|
|
2101
2152
|
|
2102
2153
|
Possible values for option `sources` are:
|
2103
2154
|
|
2104
|
-
|
2155
|
+
- `@args` : (default) the list of files (or file pair) is directly provided on the command line (after commands): unused arguments (not starting with `-`) are considered as source files.
|
2105
2156
|
So, by default, the list of files to transfer will be simply specified on the command line. Example:
|
2106
2157
|
|
2107
2158
|
```bash
|
@@ -2114,13 +2165,13 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2114
2165
|
ascli server upload --sources=@args --src-type=list ~/mysample.file secondfile
|
2115
2166
|
```
|
2116
2167
|
|
2117
|
-
|
2168
|
+
- an [Extended Value](#extended) with type **Array of String**
|
2118
2169
|
|
2119
2170
|
> Note: extended values can be tested with the command `conf echo`
|
2120
2171
|
|
2121
2172
|
Examples:
|
2122
2173
|
|
2123
|
-
|
2174
|
+
- Using extended value
|
2124
2175
|
|
2125
2176
|
Create the file list:
|
2126
2177
|
|
@@ -2135,28 +2186,28 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2135
2186
|
--sources=@lines:@file:myfilelist.txt
|
2136
2187
|
```
|
2137
2188
|
|
2138
|
-
|
2189
|
+
- Using JSON array
|
2139
2190
|
|
2140
2191
|
```javascript
|
2141
2192
|
--sources=@json:'["file1","file2"]'
|
2142
2193
|
```
|
2143
2194
|
|
2144
|
-
|
2195
|
+
- Using STDIN, one path per line
|
2145
2196
|
|
2146
2197
|
```bash
|
2147
2198
|
--sources=@lines:@stdin:
|
2148
2199
|
```
|
2149
2200
|
|
2150
|
-
|
2201
|
+
- Using ruby code (one path per line in file)
|
2151
2202
|
|
2152
2203
|
```ruby
|
2153
2204
|
--sources=@ruby:'File.read("myfilelist.txt").split("\n")'
|
2154
2205
|
```
|
2155
2206
|
|
2156
|
-
|
2207
|
+
- `@ts` : the user provides the list of files directly in the `paths` field of transfer spec (option `ts`).
|
2157
2208
|
Examples:
|
2158
2209
|
|
2159
|
-
|
2210
|
+
- Using transfer spec
|
2160
2211
|
|
2161
2212
|
```javascript
|
2162
2213
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
@@ -2168,8 +2219,8 @@ The option `src_type` allows specifying if the list specified in option `sources
|
|
2168
2219
|
|
2169
2220
|
Supported values for `src_type` are:
|
2170
2221
|
|
2171
|
-
|
2172
|
-
|
2222
|
+
- `list` : (default) the path of destination is the same as source and each entry is a source file path
|
2223
|
+
- `pair` : the first element is the first source, the second element is the first destination, and so on.
|
2173
2224
|
|
2174
2225
|
Example: Source file `200KB.1` is renamed `sample1` on destination:
|
2175
2226
|
|
@@ -2183,7 +2234,7 @@ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
|
2183
2234
|
|
2184
2235
|
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
2185
2236
|
|
2186
|
-
|
2237
|
+
- when agent=node :
|
2187
2238
|
|
2188
2239
|
```javascript
|
2189
2240
|
--ts=@json:'{"multi_session":10,"multi_session_threshold":1}'
|
@@ -2191,16 +2242,15 @@ Multi session, i.e. starting a transfer of a file set using multiple sessions (o
|
|
2191
2242
|
|
2192
2243
|
Multi-session is directly supported by the node daemon.
|
2193
2244
|
|
2194
|
-
|
2245
|
+
- when agent=direct :
|
2195
2246
|
|
2196
2247
|
```javascript
|
2197
2248
|
--ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'
|
2198
2249
|
```
|
2199
2250
|
|
2200
|
-
Note:
|
2201
|
-
shall be preferred.
|
2251
|
+
Note: `resume_policy` set to `attr` may cause problems: `none` or `sparse_csum` shall be preferred.
|
2202
2252
|
|
2203
|
-
Multi-session
|
2253
|
+
`ascli` starts multiple `ascp` for Multi-session using `direct` agent.
|
2204
2254
|
|
2205
2255
|
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
2206
2256
|
|
@@ -2212,8 +2262,8 @@ using a passphrase, only known by users sharing files. Files stay encrypted on s
|
|
2212
2262
|
|
2213
2263
|
activating CSEAR consists in using transfer spec parameters:
|
2214
2264
|
|
2215
|
-
|
2216
|
-
|
2265
|
+
- `content_protection` : activate encryption (`encrypt` for upload) or decryption (`decrypt` for download)
|
2266
|
+
- `content_protection_password` : the passphrase to be used.
|
2217
2267
|
|
2218
2268
|
Example: parameter to download a faspex package and decrypt on the fly
|
2219
2269
|
|
@@ -2229,25 +2279,25 @@ Note that up to version 4.6.0, the following parameters should be used for agent
|
|
2229
2279
|
|
2230
2280
|
#### Transfer Spec Examples
|
2231
2281
|
|
2232
|
-
|
2282
|
+
- Change target rate
|
2233
2283
|
|
2234
2284
|
```javascript
|
2235
2285
|
--ts=@json:'{"target_rate_kbps":500000}'
|
2236
2286
|
```
|
2237
2287
|
|
2238
|
-
|
2288
|
+
- Override the FASP SSH port to a specific TCP port:
|
2239
2289
|
|
2240
2290
|
```javascript
|
2241
2291
|
--ts=@json:'{"ssh_port":33002}'
|
2242
2292
|
```
|
2243
2293
|
|
2244
|
-
|
2294
|
+
- Force http fallback mode:
|
2245
2295
|
|
2246
2296
|
```javascript
|
2247
2297
|
--ts=@json:'{"http_fallback":"force"}'
|
2248
2298
|
```
|
2249
2299
|
|
2250
|
-
|
2300
|
+
- Activate progress when not activated by default on server
|
2251
2301
|
|
2252
2302
|
```javascript
|
2253
2303
|
--ts=@json:'{"precalculate_job_size":true}'
|
@@ -2259,14 +2309,14 @@ In some conditions, it may be desirable to ensure that `ascli` is not executed s
|
|
2259
2309
|
|
2260
2310
|
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
2261
2311
|
|
2262
|
-
|
2263
|
-
|
2264
|
-
|
2312
|
+
- Executing instances may pile-up and kill the system
|
2313
|
+
- The same file may be transferred by multiple instances at the same time.
|
2314
|
+
- `preview` may generate the same files in multiple instances.
|
2265
2315
|
|
2266
2316
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
2267
2317
|
|
2268
|
-
|
2269
|
-
|
2318
|
+
- The Windows scheduler does this by default
|
2319
|
+
- Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
2270
2320
|
|
2271
2321
|
```bash
|
2272
2322
|
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
@@ -2343,8 +2393,8 @@ faux:///filename?filesize
|
|
2343
2393
|
|
2344
2394
|
where:
|
2345
2395
|
|
2346
|
-
|
2347
|
-
|
2396
|
+
- `filename` is the name that will be assigned to the file on the destination
|
2397
|
+
- `filesize` is the number of bytes that will be sent (in decimal).
|
2348
2398
|
|
2349
2399
|
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
2350
2400
|
|
@@ -2358,8 +2408,8 @@ faux:///dirname?<arg1>=<val1>&...
|
|
2358
2408
|
|
2359
2409
|
where:
|
2360
2410
|
|
2361
|
-
|
2362
|
-
|
2411
|
+
- `dirname` is the folder name and can contain `/` to specify a subfolder.
|
2412
|
+
- supported arguments are:
|
2363
2413
|
|
2364
2414
|
| Name | Type | Description |
|
2365
2415
|
|--------|------|-------------|
|
@@ -2372,19 +2422,19 @@ where:
|
|
2372
2422
|
|
2373
2423
|
The sequence parameter is applied as follows:
|
2374
2424
|
|
2375
|
-
|
2425
|
+
- If `seq` is `random` then each file size is:
|
2376
2426
|
|
2377
|
-
|
2378
|
-
|
2379
|
-
|
2380
|
-
|
2427
|
+
- size +/- (inc * rand())
|
2428
|
+
- Where rand is a random number between 0 and 1
|
2429
|
+
- Note that file size must not be negative, inc will be set to size if it is greater than size
|
2430
|
+
- Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
2381
2431
|
|
2382
|
-
|
2432
|
+
- If `seq` is `sequential` then each file size is:
|
2383
2433
|
|
2384
|
-
|
2385
|
-
|
2386
|
-
|
2387
|
-
|
2434
|
+
- `size + ((fileindex - 1) * inc)`
|
2435
|
+
- Where first file is index 1
|
2436
|
+
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
2437
|
+
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
2388
2438
|
|
2389
2439
|
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
2390
2440
|
|
@@ -2392,19 +2442,19 @@ To discard data at the destination, the destination argument is set to `faux://`
|
|
2392
2442
|
|
2393
2443
|
Examples:
|
2394
2444
|
|
2395
|
-
|
2445
|
+
- Upload 20 gibibytes of random data to file myfile to directory /Upload
|
2396
2446
|
|
2397
2447
|
```bash
|
2398
2448
|
ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
2399
2449
|
```
|
2400
2450
|
|
2401
|
-
|
2451
|
+
- Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
2402
2452
|
|
2403
2453
|
```bash
|
2404
2454
|
ascli server upload /tmp/sample --to-folder=faux://
|
2405
2455
|
```
|
2406
2456
|
|
2407
|
-
|
2457
|
+
- Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
2408
2458
|
|
2409
2459
|
```bash
|
2410
2460
|
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
@@ -2415,7 +2465,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
2415
2465
|
```text
|
2416
2466
|
ascli -h
|
2417
2467
|
NAME
|
2418
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
2468
|
+
ascli -- a command line tool for Aspera Applications (v4.10.0)
|
2419
2469
|
|
2420
2470
|
SYNOPSIS
|
2421
2471
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -2438,23 +2488,23 @@ COMMANDS
|
|
2438
2488
|
OPTIONS
|
2439
2489
|
Options begin with a '-' (minus), and value is provided on command line.
|
2440
2490
|
Special values are supported beginning with special prefix @pfx:, where pfx is one of:
|
2441
|
-
base64, json, zlib, ruby, csvt, lines, list, incps, val, file, path, env, uri, stdin, preset
|
2491
|
+
base64, json, zlib, ruby, csvt, lines, list, incps, vault, val, file, path, env, uri, stdin, preset
|
2442
2492
|
Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'
|
2443
2493
|
|
2444
2494
|
ARGS
|
2445
2495
|
Some commands require mandatory arguments, e.g. a path.
|
2446
2496
|
|
2447
2497
|
OPTIONS: global
|
2448
|
-
--interactive=ENUM use interactive input of missing params:
|
2449
|
-
--ask-options=ENUM ask even optional options:
|
2498
|
+
--interactive=ENUM use interactive input of missing params: [no], yes
|
2499
|
+
--ask-options=ENUM ask even optional options: [no], yes
|
2450
2500
|
--format=ENUM output format: text, nagios, ruby, json, jsonpp, yaml, [table], csv
|
2451
2501
|
--display=ENUM output only some information: [info], data, error
|
2452
2502
|
--fields=VALUE comma separated list of fields, or ALL, or DEF
|
2453
2503
|
--select=VALUE select only some items in lists, extended value: hash (column, value)
|
2454
2504
|
--table-style=VALUE table display style
|
2455
|
-
--flat-hash=ENUM display hash values as additional keys: [yes]
|
2456
|
-
--transpose-single=ENUM single object fields output vertically: [yes]
|
2457
|
-
--show-secrets=ENUM show secrets on command output:
|
2505
|
+
--flat-hash=ENUM display hash values as additional keys: no, [yes]
|
2506
|
+
--transpose-single=ENUM single object fields output vertically: no, [yes]
|
2507
|
+
--show-secrets=ENUM show secrets on command output: [no], yes
|
2458
2508
|
-h, --help Show this message.
|
2459
2509
|
--bash-comp generate bash completion for command
|
2460
2510
|
--show-config Display parameters used for the provided action.
|
@@ -2467,10 +2517,10 @@ OPTIONS: global
|
|
2467
2517
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
2468
2518
|
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2469
2519
|
--http-options=VALUE options for http socket (extended value)
|
2470
|
-
--insecure=ENUM do not validate HTTPS certificate: [yes]
|
2471
|
-
--once-only=ENUM process only new items (some commands):
|
2472
|
-
--log-secrets=ENUM show passwords in logs:
|
2473
|
-
--cache-tokens=ENUM save and reuse Oauth tokens: [yes]
|
2520
|
+
--insecure=ENUM do not validate HTTPS certificate: no, [yes]
|
2521
|
+
--once-only=ENUM process only new items (some commands): [no], yes
|
2522
|
+
--log-secrets=ENUM show passwords in logs: [no], yes
|
2523
|
+
--cache-tokens=ENUM save and reuse Oauth tokens: no, [yes]
|
2474
2524
|
|
2475
2525
|
COMMAND: config
|
2476
2526
|
SUBCOMMANDS: list overview id preset open documentation genkey gem plugin flush_tokens echo wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update initdemo vault
|
@@ -2478,27 +2528,30 @@ OPTIONS:
|
|
2478
2528
|
--value=VALUE extended value for create, update, list filter
|
2479
2529
|
--property=VALUE name of property to set
|
2480
2530
|
--id=VALUE resource identifier (modify,delete,show)
|
2481
|
-
--bulk=ENUM Bulk operation (only some):
|
2531
|
+
--bulk=ENUM Bulk operation (only some): [no], yes
|
2532
|
+
--bfail=ENUM Bulk operation error handling: [no], yes
|
2482
2533
|
--config-file=VALUE read parameters from file in YAML format, current=/usershome/.aspera/ascli/config.yaml
|
2483
2534
|
-N, --no-default do not load default configuration for plugin
|
2484
|
-
--override=ENUM Wizard: override existing value:
|
2485
|
-
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id:
|
2486
|
-
--default=ENUM Wizard: set as default configuration for specified plugin (also: update):
|
2487
|
-
--test-mode=ENUM Wizard: skip private key check step:
|
2535
|
+
--override=ENUM Wizard: override existing value: [no], yes
|
2536
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: [no], yes
|
2537
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): [no], yes
|
2538
|
+
--test-mode=ENUM Wizard: skip private key check step: [no], yes
|
2488
2539
|
-P, --presetVALUE load the named option preset from current config file
|
2489
2540
|
--pkeypath=VALUE Wizard: path to private key for JWT
|
2490
|
-
--ascp-path=VALUE
|
2491
|
-
--use-product=VALUE
|
2492
|
-
--smtp=VALUE
|
2493
|
-
--fpac=VALUE
|
2494
|
-
--
|
2495
|
-
--
|
2541
|
+
--ascp-path=VALUE Path to ascp
|
2542
|
+
--use-product=VALUE Use ascp from specified product
|
2543
|
+
--smtp=VALUE SMTP configuration (extended value: hash)
|
2544
|
+
--fpac=VALUE Proxy auto configuration script
|
2545
|
+
--proxy-credentials=VALUE HTTP proxy credentials (Array with user and password)
|
2546
|
+
--secret=VALUE Secret for access keys
|
2547
|
+
--vault=VALUE Vault for secrets
|
2548
|
+
--vault-password=VALUE Vault password
|
2496
2549
|
--sdk-url=VALUE URL to get SDK
|
2497
2550
|
--sdk-folder=VALUE SDK folder path
|
2498
|
-
--notif-to=VALUE
|
2499
|
-
--notif-template=VALUE
|
2500
|
-
--version-check-days=VALUE
|
2501
|
-
--plugin-folder=VALUE
|
2551
|
+
--notif-to=VALUE Email recipient for notification of transfers
|
2552
|
+
--notif-template=VALUE Email ERB template for notification of transfers
|
2553
|
+
--version-check-days=VALUE Period in days to check new version (zero to disable)
|
2554
|
+
--plugin-folder=VALUE Folder where to find additional plugins
|
2502
2555
|
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
2503
2556
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
2504
2557
|
--to-folder=VALUE destination folder for transfered files
|
@@ -2518,7 +2571,7 @@ OPTIONS:
|
|
2518
2571
|
|
2519
2572
|
|
2520
2573
|
COMMAND: node
|
2521
|
-
SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details health events space info license mkdir mklink mkfile rename delete search
|
2574
|
+
SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async sync central asperabrowser basic_token browse upload download api_details health events space info license mkdir mklink mkfile rename delete search
|
2522
2575
|
OPTIONS:
|
2523
2576
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2524
2577
|
--username=VALUE username to log in
|
@@ -2537,7 +2590,7 @@ OPTIONS:
|
|
2537
2590
|
--password=VALUE user's password
|
2538
2591
|
--params=VALUE parameters hash table, use @json:{"param":"value"}
|
2539
2592
|
--result=VALUE specify result value as: 'work step:parameter'
|
2540
|
-
--synchronous=ENUM work step:parameter expected as result:
|
2593
|
+
--synchronous=ENUM work step:parameter expected as result: [no], yes
|
2541
2594
|
--ret-style=ENUM how return type is requested in api: header, arg, ext
|
2542
2595
|
--auth-style=ENUM authentication type: arg_pass, head_basic, apikey
|
2543
2596
|
|
@@ -2571,7 +2624,7 @@ OPTIONS:
|
|
2571
2624
|
|
2572
2625
|
|
2573
2626
|
COMMAND: faspex5
|
2574
|
-
SUBCOMMANDS: health package admin
|
2627
|
+
SUBCOMMANDS: health version user bearer_token package admin
|
2575
2628
|
OPTIONS:
|
2576
2629
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2577
2630
|
--username=VALUE username to log in
|
@@ -2625,7 +2678,7 @@ OPTIONS:
|
|
2625
2678
|
--case=VALUE basename of output for for test
|
2626
2679
|
--scan-path=VALUE subpath in folder id to start scan in (default=/)
|
2627
2680
|
--scan-id=VALUE forder id in storage to start scan in, default is access key main folder id
|
2628
|
-
--mimemagic=ENUM use Mime type detection of gem mimemagic:
|
2681
|
+
--mimemagic=ENUM use Mime type detection of gem mimemagic: [no], yes
|
2629
2682
|
--overwrite=ENUM when to overwrite result file: always, never, [mtime]
|
2630
2683
|
--file-access=ENUM how to read and write files in repository: [local], remote
|
2631
2684
|
--max-size=VALUE maximum size (in bytes) of preview file
|
@@ -2672,8 +2725,8 @@ OPTIONS:
|
|
2672
2725
|
--new-user-option=VALUE new user creation option for unknown package recipients
|
2673
2726
|
--from-folder=VALUE share to share source folder
|
2674
2727
|
--scope=VALUE OAuth scope for AoC API calls
|
2675
|
-
--default-ports=ENUM use standard FASP ports or get from node api:
|
2676
|
-
--validate-metadata=ENUM validate shared inbox metadata:
|
2728
|
+
--default-ports=ENUM use standard FASP ports or get from node api: [no], yes
|
2729
|
+
--validate-metadata=ENUM validate shared inbox metadata: [no], yes
|
2677
2730
|
|
2678
2731
|
|
2679
2732
|
COMMAND: server
|
@@ -2755,9 +2808,9 @@ ascli config wizard --value=aoc
|
|
2755
2808
|
|
2756
2809
|
Several types of OAuth authentication are supported:
|
2757
2810
|
|
2758
|
-
|
2759
|
-
|
2760
|
-
|
2811
|
+
- JSON Web Token (JWT) : authentication is secured by a private key (recommended for CLI)
|
2812
|
+
- Web based authentication : authentication is made by user using a browser
|
2813
|
+
- URL Token : external users authentication with url tokens (public links)
|
2761
2814
|
|
2762
2815
|
The authentication method is controlled by option `auth`.
|
2763
2816
|
|
@@ -2777,15 +2830,15 @@ Else you can use a specific OAuth API client_id, the first step is to declare `a
|
|
2777
2830
|
|
2778
2831
|
Let's start by a registration with web based authentication (auth=web):
|
2779
2832
|
|
2780
|
-
|
2781
|
-
|
2782
|
-
|
2783
|
-
|
2784
|
-
|
2785
|
-
|
2786
|
-
|
2787
|
-
|
2788
|
-
|
2833
|
+
- Open a web browser, log to your instance: e.g. `https://myorg.ibmaspera.com/`
|
2834
|
+
- Go to Apps→Admin→Organization→Integrations
|
2835
|
+
- Click "Create New"
|
2836
|
+
- Client Name: `ascli`
|
2837
|
+
- Redirect URIs: `http://localhost:12345`
|
2838
|
+
- Origins: `localhost`
|
2839
|
+
- uncheck "Prompt users to allow client to access"
|
2840
|
+
- leave the JWT part for now
|
2841
|
+
- Save
|
2789
2842
|
|
2790
2843
|
Note: for web based authentication, `ascli` listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For ``ascli`, HTTP is required, and 12345 is the default port.
|
2791
2844
|
|
@@ -2826,16 +2879,16 @@ a [private/public key pair](#private_key) must be used.
|
|
2826
2879
|
|
2827
2880
|
If you are not using the built-in client_id and secret, JWT needs to be authorized in Aspera on Cloud. This can be done in two manners:
|
2828
2881
|
|
2829
|
-
|
2882
|
+
- Graphically
|
2830
2883
|
|
2831
|
-
|
2832
|
-
|
2833
|
-
|
2834
|
-
|
2835
|
-
|
2836
|
-
|
2884
|
+
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
2885
|
+
- Go to Apps→Admin→Organization→Integrations
|
2886
|
+
- Click on the previously created application
|
2887
|
+
- select tab : "JSON Web Token Auth"
|
2888
|
+
- Modify options if necessary, for instance: activate both options in section "Settings"
|
2889
|
+
- Click "Save"
|
2837
2890
|
|
2838
|
-
|
2891
|
+
- Using command line
|
2839
2892
|
|
2840
2893
|
```bash
|
2841
2894
|
ascli aoc admin res client list
|
@@ -2865,11 +2918,11 @@ The public key must be assigned to your user. This can be done in two manners:
|
|
2865
2918
|
|
2866
2919
|
Open the previously generated public key located here: `$HOME/.aspera/ascli/my_private_key.pub`
|
2867
2920
|
|
2868
|
-
|
2869
|
-
|
2870
|
-
|
2871
|
-
|
2872
|
-
|
2921
|
+
- Open a web browser, log to your instance: `https://myorg.ibmaspera.com/`
|
2922
|
+
- Click on the user's icon (top right)
|
2923
|
+
- Select "Account Settings"
|
2924
|
+
- Paste the *Public Key* in the "Public Key" section
|
2925
|
+
- Click on "Submit"
|
2873
2926
|
|
2874
2927
|
##### Using command line
|
2875
2928
|
|
@@ -2900,9 +2953,9 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
2900
2953
|
|
2901
2954
|
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the following:
|
2902
2955
|
|
2903
|
-
|
2904
|
-
|
2905
|
-
|
2956
|
+
- change auth method to JWT
|
2957
|
+
- provide location of private key
|
2958
|
+
- provide username to login as (OAuth "subject")
|
2906
2959
|
|
2907
2960
|
Execute:
|
2908
2961
|
|
@@ -2962,13 +3015,13 @@ The option `query` can be optionally used. It expects a Hash using [Extended Val
|
|
2962
3015
|
|
2963
3016
|
The following parameters are supported:
|
2964
3017
|
|
2965
|
-
|
2966
|
-
|
2967
|
-
|
2968
|
-
|
2969
|
-
|
2970
|
-
|
2971
|
-
|
3018
|
+
- `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
3019
|
+
- `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
3020
|
+
- `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
3021
|
+
- `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
3022
|
+
- `page` : native api parameter, in general do not use (added by
|
3023
|
+
- `per_page` : native api parameter, number of items par api call, in general do not use
|
3024
|
+
- Other specific parameters depending on resource type.
|
2972
3025
|
|
2973
3026
|
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
2974
3027
|
|
@@ -2982,19 +3035,19 @@ Other parameters depend on the type of entity (refer to AoC API).
|
|
2982
3035
|
|
2983
3036
|
Examples:
|
2984
3037
|
|
2985
|
-
|
3038
|
+
- List users with `laurent` in name:
|
2986
3039
|
|
2987
3040
|
```javascript
|
2988
3041
|
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2989
3042
|
```
|
2990
3043
|
|
2991
|
-
|
3044
|
+
- List users who logged-in before a date:
|
2992
3045
|
|
2993
3046
|
```javascript
|
2994
3047
|
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2995
3048
|
```
|
2996
3049
|
|
2997
|
-
|
3050
|
+
- List external users and sort in reverse alphabetical order using name:
|
2998
3051
|
|
2999
3052
|
```javascript
|
3000
3053
|
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
@@ -3010,10 +3063,10 @@ Resources are identified by a unique `id`, as well as a unique `name` (case inse
|
|
3010
3063
|
|
3011
3064
|
To execute an action on a specific resource, select it using one of those methods:
|
3012
3065
|
|
3013
|
-
|
3014
|
-
|
3015
|
-
|
3016
|
-
|
3066
|
+
- *recommended*: give id directly on command line *after the action*: `aoc admin res node show 123`
|
3067
|
+
- give name on command line *after the action*: `aoc admin res node show name abc`
|
3068
|
+
- provide option `id` : `aoc admin res node show --id=123`
|
3069
|
+
- provide option `name` : `aoc admin res node show --name=abc`
|
3017
3070
|
|
3018
3071
|
#### <a id="res_create"></a>Creating a resource
|
3019
3072
|
|
@@ -3054,6 +3107,7 @@ If the command returns an error, example:
|
|
3054
3107
|
| | request_id: b0f45d5b-c00a-4711-acef-72b633f8a6ea |
|
3055
3108
|
| | api.ibmaspera.com 422 Unprocessable Entity |
|
3056
3109
|
+----+-----------------------------------------------------------------------------------+```
|
3110
|
+
```
|
3057
3111
|
|
3058
3112
|
Well, remove the offending parameters and try again.
|
3059
3113
|
|
@@ -3061,15 +3115,15 @@ Note that some properties that are shown in the web UI, such as membership, are
|
|
3061
3115
|
|
3062
3116
|
#### Access Key secrets
|
3063
3117
|
|
3064
|
-
In order to access some administrative actions on
|
3065
|
-
|
3118
|
+
In order to access some administrative actions on **nodes** (in fact, access keys), the associated secret is required.
|
3119
|
+
The secret is provided using the `secret` option.
|
3066
3120
|
For example in a command like:
|
3067
3121
|
|
3068
3122
|
```bash
|
3069
3123
|
ascli aoc admin res node --id=123 --secret="secret1" v3 info
|
3070
3124
|
```
|
3071
3125
|
|
3072
|
-
It is also possible to
|
3126
|
+
It is also possible to store secrets in the [secret vault](#vault) and then automatically find the related secret using the [config finder](#config_finder).
|
3073
3127
|
|
3074
3128
|
#### Activity
|
3075
3129
|
|
@@ -3097,7 +3151,7 @@ Thank you.
|
|
3097
3151
|
|
3098
3152
|
The environment provided contains the following additional variable:
|
3099
3153
|
|
3100
|
-
|
3154
|
+
- ev : all details on the transfer event
|
3101
3155
|
|
3102
3156
|
Example:
|
3103
3157
|
|
@@ -3109,9 +3163,9 @@ ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
|
3109
3163
|
|
3110
3164
|
Options:
|
3111
3165
|
|
3112
|
-
|
3113
|
-
|
3114
|
-
|
3166
|
+
- `once_only` keep track of last date it was called, so next call will get only new events
|
3167
|
+
- `query` filter (on API call)
|
3168
|
+
- `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
3115
3169
|
|
3116
3170
|
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
3117
3171
|
|
@@ -3321,7 +3375,7 @@ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_wor
|
|
3321
3375
|
|
3322
3376
|
#### Example: create a group, add to workspace and add user to group
|
3323
3377
|
|
3324
|
-
|
3378
|
+
- Create the group and take note of `id`
|
3325
3379
|
|
3326
3380
|
```bash
|
3327
3381
|
ascli aoc admin res group create @json:'{"name":"group 1","description":"my super group"}'
|
@@ -3329,7 +3383,7 @@ ascli aoc admin res group create @json:'{"name":"group 1","description":"my supe
|
|
3329
3383
|
|
3330
3384
|
Group: `11111`
|
3331
3385
|
|
3332
|
-
|
3386
|
+
- Get the workspace id
|
3333
3387
|
|
3334
3388
|
```bash
|
3335
3389
|
ascli aoc admin res workspace list --query=@json:'{"q":"myworkspace"}' --fields=id --format=csv --display=data
|
@@ -3337,13 +3391,13 @@ ascli aoc admin res workspace list --query=@json:'{"q":"myworkspace"}' --fields=
|
|
3337
3391
|
|
3338
3392
|
Workspace: 22222
|
3339
3393
|
|
3340
|
-
|
3394
|
+
- Add group to workspace
|
3341
3395
|
|
3342
3396
|
```bash
|
3343
3397
|
ascli aoc admin res workspace_membership create @json:'{"workspace_id":22222,"member_type":"user","member_id":11111}'
|
3344
3398
|
```
|
3345
3399
|
|
3346
|
-
|
3400
|
+
- Get a user's id
|
3347
3401
|
|
3348
3402
|
```bash
|
3349
3403
|
ascli aoc admin res user list --query=@json:'{"q":"manu.macron@example.com"}' --fields=id --format=csv --display=data
|
@@ -3351,7 +3405,7 @@ ascli aoc admin res user list --query=@json:'{"q":"manu.macron@example.com"}' --
|
|
3351
3405
|
|
3352
3406
|
User: 33333
|
3353
3407
|
|
3354
|
-
|
3408
|
+
- Add user to group
|
3355
3409
|
|
3356
3410
|
```bash
|
3357
3411
|
ascli aoc admin res group_membership create @json:'{"group_id":11111,"member_type":"user","member_id":33333}'
|
@@ -3430,8 +3484,8 @@ ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res cli
|
|
3430
3484
|
|
3431
3485
|
AoC nodes as actually composed with two related entities:
|
3432
3486
|
|
3433
|
-
|
3434
|
-
|
3487
|
+
- An access key created on the Transfer Server (HSTS/ATS)
|
3488
|
+
- a `node` resource in the AoC application.
|
3435
3489
|
|
3436
3490
|
The web UI allows creation of both entities in one shot.
|
3437
3491
|
For more flexibility, `ascli` allows this in two separate steps.
|
@@ -3439,7 +3493,7 @@ Note that when selecting "Use existing access key" in the web UI, this actually
|
|
3439
3493
|
|
3440
3494
|
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
3441
3495
|
|
3442
|
-
|
3496
|
+
- Create the access key on ATS
|
3443
3497
|
|
3444
3498
|
The creation options are the ones of ATS API, refer to the [section on ATS](#ats_params) for more details and examples.
|
3445
3499
|
|
@@ -3447,9 +3501,11 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3447
3501
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
3448
3502
|
```
|
3449
3503
|
|
3450
|
-
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed
|
3504
|
+
Once executed, the access key `id` and `secret`, randomly generated by the node api, is displayed.
|
3505
|
+
|
3506
|
+
> Note: Once returned by the API, the secret will not be available anymore, so store this preciously. ATS secrets can only be reset by asking to IBM support.
|
3451
3507
|
|
3452
|
-
|
3508
|
+
- Create the AoC node entity
|
3453
3509
|
|
3454
3510
|
First, Retrieve the ATS node address
|
3455
3511
|
|
@@ -3473,16 +3529,16 @@ Note the special case when the source files are located on "Aspera on Cloud" (i.
|
|
3473
3529
|
|
3474
3530
|
Source files are located on "Aspera on cloud", when :
|
3475
3531
|
|
3476
|
-
|
3477
|
-
|
3532
|
+
- the server is Aspera on Cloud, and executing a download or recv
|
3533
|
+
- the agent is Aspera on Cloud, and executing an upload or send
|
3478
3534
|
|
3479
3535
|
In this case:
|
3480
3536
|
|
3481
|
-
|
3482
|
-
|
3483
|
-
|
3484
|
-
|
3485
|
-
|
3537
|
+
- If there is a single file : specify the full path
|
3538
|
+
- Else, if there are multiple files:
|
3539
|
+
- All source files must be in the same source folder
|
3540
|
+
- Specify the source folder as first item in the list
|
3541
|
+
- followed by the list of file names.
|
3486
3542
|
|
3487
3543
|
### Packages
|
3488
3544
|
|
@@ -3498,14 +3554,14 @@ ascli aoc packages send --value=[package extended value] [other parameters such
|
|
3498
3554
|
|
3499
3555
|
Notes:
|
3500
3556
|
|
3501
|
-
|
3502
|
-
|
3503
|
-
|
3504
|
-
|
3505
|
-
|
3506
|
-
|
3507
|
-
|
3508
|
-
|
3557
|
+
- The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
3558
|
+
- List allowed shared inbox destinations with: `ascli aoc packages shared_inboxes list`
|
3559
|
+
- Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
3560
|
+
- Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
3561
|
+
- or just names: `"recipients":[{"The Dest"}]` . ascli will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
3562
|
+
- If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
3563
|
+
- if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
3564
|
+
- if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
3509
3565
|
|
3510
3566
|
#### Example: Send a package with one file to two users, using their email
|
3511
3567
|
|
@@ -3584,15 +3640,15 @@ It is possible to automatically download new packages, like using Aspera Cargo:
|
|
3584
3640
|
ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
3585
3641
|
```
|
3586
3642
|
|
3587
|
-
|
3588
|
-
|
3589
|
-
|
3643
|
+
- `--id=ALL` (case sensitive) will download all packages
|
3644
|
+
- `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
3645
|
+
- `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
3590
3646
|
|
3591
3647
|
Typically, one would execute this command on a regular basis, using the method of your choice:
|
3592
3648
|
|
3593
|
-
|
3594
|
-
|
3595
|
-
|
3649
|
+
- Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
3650
|
+
- Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
3651
|
+
- etc...
|
3596
3652
|
|
3597
3653
|
### Files
|
3598
3654
|
|
@@ -3610,37 +3666,37 @@ For creation, parameters are the same as for node api [permissions](https://deve
|
|
3610
3666
|
Also, the pseudo key `with` is added: it will lookup the name in the contacts and fill the proper type and id.
|
3611
3667
|
The pseudo parameter `link_name` allows changing default "shared as" name.
|
3612
3668
|
|
3613
|
-
|
3669
|
+
- List permissions on a shared folder as user
|
3614
3670
|
|
3615
3671
|
```bash
|
3616
3672
|
ascli aoc files file --path=/shared_folder_test1 perm list
|
3617
3673
|
```
|
3618
3674
|
|
3619
|
-
|
3675
|
+
- Share a personal folder with other users
|
3620
3676
|
|
3621
3677
|
```bash
|
3622
3678
|
ascli aoc files file --path=/shared_folder_test1 perm create @json:'{"with":"laurent"}'
|
3623
3679
|
```
|
3624
3680
|
|
3625
|
-
|
3681
|
+
- Revoke shared access
|
3626
3682
|
|
3627
3683
|
```bash
|
3628
3684
|
ascli aoc files file --path=/shared_folder_test1 perm delete 6161
|
3629
3685
|
```
|
3630
3686
|
|
3631
|
-
|
3687
|
+
- List shared folders in node
|
3632
3688
|
|
3633
3689
|
```bash
|
3634
3690
|
ascli aoc admin res node --id=8669 shared_folders
|
3635
3691
|
```
|
3636
3692
|
|
3637
|
-
|
3693
|
+
- List shared folders in workspace
|
3638
3694
|
|
3639
3695
|
```bash
|
3640
3696
|
ascli aoc admin res workspace --id=10818 shared_folders
|
3641
3697
|
```
|
3642
3698
|
|
3643
|
-
|
3699
|
+
- List members of shared folder
|
3644
3700
|
|
3645
3701
|
```bash
|
3646
3702
|
ascli aoc admin res node --id=8669 v4 perm 82 show
|
@@ -3654,11 +3710,11 @@ Although optional, the creation of [option preset](#lprt) is recommended to avoi
|
|
3654
3710
|
|
3655
3711
|
Procedure to send a file from org1 to org2:
|
3656
3712
|
|
3657
|
-
|
3658
|
-
|
3659
|
-
|
3660
|
-
|
3661
|
-
|
3713
|
+
- Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
3714
|
+
- Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
3715
|
+
- Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
3716
|
+
- Check that access works and locate the destination folder `mydestfolder`
|
3717
|
+
- execute the following:
|
3662
3718
|
|
3663
3719
|
```bash
|
3664
3720
|
ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
@@ -3666,15 +3722,15 @@ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | as
|
|
3666
3722
|
|
3667
3723
|
Explanation:
|
3668
3724
|
|
3669
|
-
|
3670
|
-
|
3671
|
-
|
3672
|
-
|
3673
|
-
|
3674
|
-
|
3675
|
-
|
3676
|
-
|
3677
|
-
|
3725
|
+
- `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
3726
|
+
- `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
3727
|
+
- `--format=json` format the output in JSON (instead of default text table)
|
3728
|
+
- `--display=data` display only the result, and remove other information, such as workspace name
|
3729
|
+
- `|` the standard output of the first command is fed into the second one
|
3730
|
+
- `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
3731
|
+
- `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
3732
|
+
- `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
3733
|
+
- `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
3678
3734
|
|
3679
3735
|
#### Find Files
|
3680
3736
|
|
@@ -3682,34 +3738,34 @@ The command `aoc files find [--value=expression]` will recursively scan storage
|
|
3682
3738
|
|
3683
3739
|
The expression can be of 3 formats:
|
3684
3740
|
|
3685
|
-
|
3686
|
-
|
3741
|
+
- empty (default) : all files, equivalent to value: `exec:true`
|
3742
|
+
- not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
3687
3743
|
|
3688
3744
|
For instance, to find files with a special extension, use `--value='\.myext$'`
|
3689
3745
|
|
3690
|
-
|
3746
|
+
- starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
3691
3747
|
|
3692
3748
|
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
3693
3749
|
|
3694
|
-
|
3750
|
+
- Find files more recent than 100 days
|
3695
3751
|
|
3696
3752
|
```bash
|
3697
3753
|
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
3698
3754
|
```
|
3699
3755
|
|
3700
|
-
|
3756
|
+
- Find files older than 1 year on a given node and store in file list
|
3701
3757
|
|
3702
3758
|
```bash
|
3703
3759
|
ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
3704
3760
|
```
|
3705
3761
|
|
3706
|
-
|
3762
|
+
- Delete the files, one by one
|
3707
3763
|
|
3708
3764
|
```bash
|
3709
3765
|
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 delete "$path" ;done
|
3710
3766
|
```
|
3711
3767
|
|
3712
|
-
|
3768
|
+
- Delete the files in bulk
|
3713
3769
|
|
3714
3770
|
```bash
|
3715
3771
|
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='_secret_here_' v3 delete @lines:@stdin:
|
@@ -3812,9 +3868,9 @@ aoc user workspaces list
|
|
3812
3868
|
|
3813
3869
|
ATS is usable either :
|
3814
3870
|
|
3815
|
-
|
3871
|
+
- from an AoC subscription : ascli aoc admin ats : use AoC authentication
|
3816
3872
|
|
3817
|
-
|
3873
|
+
- or from an IBM Cloud subscription : ascli ats : use IBM Cloud API key authentication
|
3818
3874
|
|
3819
3875
|
### IBM Cloud ATS : creation of api key
|
3820
3876
|
|
@@ -3843,8 +3899,8 @@ UUID ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f
|
|
3843
3899
|
|
3844
3900
|
References:
|
3845
3901
|
|
3846
|
-
|
3847
|
-
|
3902
|
+
- [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
3903
|
+
- [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
3848
3904
|
|
3849
3905
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
3850
3906
|
|
@@ -3987,8 +4043,8 @@ server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_ses
|
|
3987
4043
|
|
3988
4044
|
If SSH is the session control protocol (i.e. not WSS), then following session authentication methods are supported:
|
3989
4045
|
|
3990
|
-
|
3991
|
-
|
4046
|
+
- SSH password
|
4047
|
+
- SSH keys (Multiple SSH key paths can be provided.)
|
3992
4048
|
|
3993
4049
|
If username is not provided, the default transfer user `xfer` is used.
|
3994
4050
|
|
@@ -4030,10 +4086,10 @@ ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant p
|
|
4030
4086
|
|
4031
4087
|
This means that you don't have such an SSH agent running, then:
|
4032
4088
|
|
4033
|
-
|
4034
|
-
|
4035
|
-
|
4036
|
-
|
4089
|
+
- check env var: `SSH_AGENT_SOCK`
|
4090
|
+
- check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option)
|
4091
|
+
- [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
4092
|
+
- To disable the use of `ssh-agent`, use the option `ssh_options` like this:
|
4037
4093
|
|
4038
4094
|
```bash
|
4039
4095
|
ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
@@ -4061,15 +4117,15 @@ This plugin gives access to capabilities provided by HSTS node API.
|
|
4061
4117
|
|
4062
4118
|
It is possible to:
|
4063
4119
|
|
4064
|
-
|
4065
|
-
|
4066
|
-
|
4120
|
+
- browse
|
4121
|
+
- transfer (upload / download)
|
4122
|
+
- ...
|
4067
4123
|
|
4068
4124
|
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
4069
4125
|
|
4070
|
-
|
4071
|
-
|
4072
|
-
|
4126
|
+
- `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token, used as is.
|
4127
|
+
- `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
4128
|
+
- `basic` : transfer spec is created like this:
|
4073
4129
|
|
4074
4130
|
```javascript
|
4075
4131
|
{
|
@@ -4111,8 +4167,8 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
4111
4167
|
|
4112
4168
|
`ascli` supports remote operations through the node API. Operations are:
|
4113
4169
|
|
4114
|
-
|
4115
|
-
|
4170
|
+
- Start watchd and watchfolderd services running as a system user having access to files
|
4171
|
+
- configure a watchfolder to define automated transfers
|
4116
4172
|
|
4117
4173
|
```javascript
|
4118
4174
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
@@ -4170,8 +4226,11 @@ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"myst
|
|
4170
4226
|
```bash
|
4171
4227
|
node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
4172
4228
|
node -N -Ptst_node_preview access_key delete aoc_1
|
4173
|
-
node access_key do my_aoc_ak_name br /
|
4174
|
-
node access_key list
|
4229
|
+
node -Pnode_srv access_key do my_aoc_ak_name br /
|
4230
|
+
node -Pnode_srv access_key list
|
4231
|
+
node -Pnode_srv service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
4232
|
+
node -Pnode_srv service delete service1
|
4233
|
+
node -Pnode_srv service list
|
4175
4234
|
node api_details
|
4176
4235
|
node async bandwidth 1
|
4177
4236
|
node async counters 1
|
@@ -4195,10 +4254,18 @@ node mkfile /delfile1 "hello world"
|
|
4195
4254
|
node mklink /todelete /tdlink
|
4196
4255
|
node rename / delfile1 delfile
|
4197
4256
|
node search / --value=@json:'{"sort":"mtime"}'
|
4198
|
-
node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
4199
|
-
node service delete service1
|
4200
|
-
node service list
|
4201
4257
|
node space /
|
4258
|
+
node sync bandwidth my_syncid
|
4259
|
+
node sync counters my_syncid
|
4260
|
+
node sync create --value=@json:'{"configuration":{"name":"sync1","local":{"path":"my_local_path"},"remote":{"host":"my_host","port":my_port,"user":"my_username","pass":"my_password","path":"my_remote_path"}}}'
|
4261
|
+
node sync delete my_syncid
|
4262
|
+
node sync files my_syncid
|
4263
|
+
node sync list
|
4264
|
+
node sync show my_syncid
|
4265
|
+
node sync start my_syncid
|
4266
|
+
node sync state my_syncid
|
4267
|
+
node sync stop my_syncid
|
4268
|
+
node sync summary my_syncid
|
4202
4269
|
node transfer list --value=@json:'{"active_only":true}'
|
4203
4270
|
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass"}'
|
4204
4271
|
node upload --username=my_aoc_ak_name --password=my_aoc_ak_secret testfile.bin --token-type=basic
|
@@ -4208,79 +4275,81 @@ node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps
|
|
4208
4275
|
|
4209
4276
|
## Plugin: IBM Aspera Faspex5
|
4210
4277
|
|
4211
|
-
|
4212
|
-
|
4213
|
-
This was tested with version Beta 5.
|
4214
|
-
|
4215
|
-
The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "IBM Aspera Faspex API".
|
4216
|
-
|
4217
|
-
### Faspex 5 authentication
|
4278
|
+
IBM Aspera's newer self-managed application.
|
4218
4279
|
|
4219
4280
|
3 authentication methods are supported:
|
4220
4281
|
|
4221
|
-
|
4222
|
-
|
4223
|
-
|
4282
|
+
- jwt
|
4283
|
+
- web
|
4284
|
+
- boot
|
4224
4285
|
|
4225
|
-
|
4286
|
+
### Faspex 5 JWT authentication
|
4226
4287
|
|
4227
|
-
This is the
|
4288
|
+
This is the **recomended** method to use.
|
4228
4289
|
|
4229
4290
|
For `jwt`, create an API client in Faspex with JWT support:
|
4230
4291
|
|
4231
|
-
|
4232
|
-
|
4233
|
-
|
4234
|
-
|
4235
|
-
|
4236
|
-
|
4292
|
+
- Select a private key file: if you don't have any refer to section [Private Key](#private_key)
|
4293
|
+
- Navigate to the web UI: Admin → Configurations → API Clients → Create
|
4294
|
+
- Activate JWT
|
4295
|
+
- Paste **public** key in the appropriate section
|
4296
|
+
- Click on Create Button
|
4297
|
+
- Take note of Client Id (and Client Secret, but not used in current version)
|
4237
4298
|
|
4238
4299
|
Then use these options:
|
4239
4300
|
|
4240
4301
|
```text
|
4241
4302
|
--auth=jwt
|
4242
|
-
--client-id=
|
4243
|
-
--client-secret=
|
4244
|
-
--username=
|
4303
|
+
--client-id=_client_id_here_
|
4304
|
+
--client-secret=_secret_here_
|
4305
|
+
--username=_username_here_
|
4245
4306
|
--private-key=@file:.../path/to/key.pem
|
4246
4307
|
```
|
4247
4308
|
|
4248
|
-
|
4309
|
+
> The `private_key` option must contain the PEM value of the private key which can be read from a file using the modifier: `@file:`, e.g. `@file:/path/to/key.pem`.
|
4310
|
+
|
4311
|
+
### Faspex 5 web authentication
|
4249
4312
|
|
4250
4313
|
For `web` method, create an API client in Faspex without JWT:
|
4251
4314
|
|
4252
|
-
|
4253
|
-
|
4254
|
-
|
4255
|
-
|
4256
|
-
|
4315
|
+
- Navigate to the web UI: Admin → Configurations → API Clients → Create
|
4316
|
+
- Do not Activate JWT
|
4317
|
+
- Set **Redirect URI** to `https://127.0.0.1:8888`
|
4318
|
+
- Click on Create Button
|
4319
|
+
- Take note of Client Id (and Client Secret, but not used in current version)
|
4257
4320
|
|
4258
4321
|
Then use options:
|
4259
4322
|
|
4260
4323
|
```text
|
4261
4324
|
--auth=web
|
4262
|
-
--client-id=
|
4263
|
-
--client-secret=
|
4325
|
+
--client-id=_client_id_here_
|
4326
|
+
--client-secret=_secret_here_
|
4264
4327
|
--redirect-uri=https://127.0.0.1:8888
|
4265
4328
|
```
|
4266
4329
|
|
4267
|
-
|
4330
|
+
### Faspex 5 bootstrap authentication
|
4268
4331
|
|
4269
4332
|
For `boot` method: (will be removed in future)
|
4270
4333
|
|
4271
|
-
|
4272
|
-
|
4273
|
-
|
4274
|
-
|
4334
|
+
- Open a Web Browser
|
4335
|
+
- Start developer mode
|
4336
|
+
- Login to Faspex 5
|
4337
|
+
- Find the first API call with `Authorization` header, and copy the value of the token (series of base64 values with dots)
|
4275
4338
|
|
4276
|
-
Use
|
4339
|
+
Use this token as password and use `--auth=boot`.
|
4277
4340
|
|
4278
4341
|
```bash
|
4279
|
-
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=
|
4342
|
+
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=_token_here_
|
4280
4343
|
```
|
4281
4344
|
|
4282
4345
|
### Faspex 5 sample commands
|
4283
4346
|
|
4347
|
+
Most commands are directly REST API calls.
|
4348
|
+
Parameters to commandsa are carried through option `value`, as extended value.
|
4349
|
+
Usually using JSON format with prefix `@json:`.
|
4350
|
+
|
4351
|
+
> The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
4352
|
+
|
4284
4353
|
```bash
|
4285
4354
|
faspex5 admin res accounts list
|
4286
4355
|
faspex5 admin res contacts list
|
@@ -4291,27 +4360,28 @@ faspex5 admin res registrations list
|
|
4291
4360
|
faspex5 admin res saml_configs list
|
4292
4361
|
faspex5 admin res shared_inboxes list
|
4293
4362
|
faspex5 admin res workgroups list
|
4363
|
+
faspex5 bearer_token
|
4294
4364
|
faspex5 health
|
4295
4365
|
faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
4296
4366
|
faspex5 package receive "my_package_id" --to-folder=. --ts=@json:'{"content_protection_password":"abc123_yo"}'
|
4297
4367
|
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin --ts=@json:'{"content_protection_password":"_content_prot_here_"}'
|
4298
4368
|
```
|
4299
4369
|
|
4300
|
-
|
4370
|
+
Other examples:
|
4301
4371
|
|
4302
|
-
|
4372
|
+
- List all shared inboxes
|
4303
4373
|
|
4304
4374
|
```javascript
|
4305
4375
|
ascli faspex5 admin res shared list --value=@json:'{"all":true}' --fields=id,name
|
4306
4376
|
```
|
4307
4377
|
|
4308
|
-
|
4378
|
+
- Create Metadata profile
|
4309
4379
|
|
4310
4380
|
```javascript
|
4311
4381
|
ascli faspex5 admin res metadata_profiles create --value=@json:'{"name":"the profile","default":false,"title":{"max_length":200,"illegal_chars":[]},"note":{"max_length":400,"illegal_chars":[],"enabled":false},"fields":[{"ordering":0,"name":"field1","type":"text_area","require":true,"illegal_chars":[],"max_length":100},{"ordering":1,"name":"fff2","type":"option_list","require":false,"choices":["opt1","opt2"]}]}'
|
4312
4382
|
```
|
4313
4383
|
|
4314
|
-
|
4384
|
+
- Create a Shared inbox with specific metadata profile
|
4315
4385
|
|
4316
4386
|
```javascript
|
4317
4387
|
ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox","metadata_profile_id":1}'
|
@@ -4321,8 +4391,8 @@ ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox",
|
|
4321
4391
|
|
4322
4392
|
Notes:
|
4323
4393
|
|
4324
|
-
|
4325
|
-
|
4394
|
+
- The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
4395
|
+
- For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
4326
4396
|
|
4327
4397
|
### Listing Packages
|
4328
4398
|
|
@@ -4336,18 +4406,18 @@ By default it looks in box `inbox`, but the following boxes are also supported:
|
|
4336
4406
|
|
4337
4407
|
A user can receive a package because the recipient is:
|
4338
4408
|
|
4339
|
-
|
4340
|
-
|
4409
|
+
- the user himself (default)
|
4410
|
+
- the user is member of a dropbox/workgroup: filter using option `recipient` set with value `*<name of dropbox/workgroup>`
|
4341
4411
|
|
4342
4412
|
#### Option `query`
|
4343
4413
|
|
4344
4414
|
As inboxes may be large, it is possible to use the following query parameters:
|
4345
4415
|
|
4346
|
-
|
4347
|
-
|
4348
|
-
|
4349
|
-
|
4350
|
-
|
4416
|
+
- `count` : (native) number items in one API call (default=0, equivalent to 10)
|
4417
|
+
- `page` : (native) id of page in call (default=0)
|
4418
|
+
- `startIndex` : (native) index of item to start, default=0, oldest index=0
|
4419
|
+
- `max` : maximum number of items
|
4420
|
+
- `pmax` : maximum number of pages
|
4351
4421
|
|
4352
4422
|
(SQL query is `LIMIT <startIndex>, <count>`)
|
4353
4423
|
|
@@ -4367,9 +4437,9 @@ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in recei
|
|
4367
4437
|
|
4368
4438
|
The command is `package recv`, possible methods are:
|
4369
4439
|
|
4370
|
-
|
4371
|
-
|
4372
|
-
|
4440
|
+
- provide a package id with option `id`
|
4441
|
+
- provide a public link with option `link`
|
4442
|
+
- provide a `faspe:` URI with option `link`
|
4373
4443
|
|
4374
4444
|
```bash
|
4375
4445
|
ascli faspex package recv --id=12345
|
@@ -4401,8 +4471,8 @@ If the recipient is a dropbox or workgroup: provide the name of the dropbox or w
|
|
4401
4471
|
|
4402
4472
|
Additional optional parameters in `delivery_info`:
|
4403
4473
|
|
4404
|
-
|
4405
|
-
|
4474
|
+
- Package Note: : `"note":"note this and that"`
|
4475
|
+
- Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
4406
4476
|
|
4407
4477
|
### Email notification on transfer
|
4408
4478
|
|
@@ -4508,16 +4578,16 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
4508
4578
|
|
4509
4579
|
```bash
|
4510
4580
|
shares admin share list
|
4511
|
-
shares admin share user_permissions
|
4581
|
+
shares admin share user_permissions 1
|
4512
4582
|
shares admin user app_authorizations 1
|
4513
4583
|
shares admin user list
|
4514
4584
|
shares admin user share_permissions 1
|
4515
4585
|
shares repository browse /
|
4516
4586
|
shares repository delete my_shares_upload/testfile.bin
|
4517
4587
|
shares repository download --to-folder=. my_shares_upload/testfile.bin
|
4518
|
-
shares repository download --to-folder=. my_shares_upload/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy
|
4588
|
+
shares repository download --to-folder=. my_shares_upload/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
4519
4589
|
shares repository upload --to-folder=my_shares_upload testfile.bin
|
4520
|
-
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy
|
4590
|
+
shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy"}'
|
4521
4591
|
```
|
4522
4592
|
|
4523
4593
|
## Plugin: Console
|
@@ -4525,8 +4595,9 @@ shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=ht
|
|
4525
4595
|
### Console sample commands
|
4526
4596
|
|
4527
4597
|
```bash
|
4528
|
-
console
|
4529
|
-
console transfer
|
4598
|
+
console health
|
4599
|
+
console transfer current list
|
4600
|
+
console transfer smart list
|
4530
4601
|
console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
4531
4602
|
```
|
4532
4603
|
|
@@ -4560,10 +4631,10 @@ There are two possibilities to provide credentials. If you already have the endp
|
|
4560
4631
|
|
4561
4632
|
If you have those parameters already, then following options shall be provided:
|
4562
4633
|
|
4563
|
-
|
4564
|
-
|
4565
|
-
|
4566
|
-
|
4634
|
+
- `bucket` bucket name
|
4635
|
+
- `endpoint` storage endpoint url, e.g. `https://s3.hkg02.cloud-object-storage.appdomain.cloud`
|
4636
|
+
- `apikey` API Key
|
4637
|
+
- `crn` resource instance id
|
4567
4638
|
|
4568
4639
|
For example, let us create a default configuration:
|
4569
4640
|
|
@@ -4617,9 +4688,9 @@ The field `apikey` is for option `apikey`
|
|
4617
4688
|
|
4618
4689
|
The required options for this method are:
|
4619
4690
|
|
4620
|
-
|
4621
|
-
|
4622
|
-
|
4691
|
+
- `bucket` bucket name
|
4692
|
+
- `region` bucket region, e.g. eu-de
|
4693
|
+
- `service_credentials` see below
|
4623
4694
|
|
4624
4695
|
For example, let us create a default configuration:
|
4625
4696
|
|
@@ -4669,9 +4740,9 @@ The `preview` generates thumbnails (office, images, video) and video previews on
|
|
4669
4740
|
It uses the **node API** of Aspera HSTS and requires use of Access Keys and it's **storage root**.
|
4670
4741
|
Several parameters can be used to tune several aspects:
|
4671
4742
|
|
4672
|
-
|
4673
|
-
|
4674
|
-
|
4743
|
+
- Methods for detection of new files needing generation
|
4744
|
+
- Methods for generation of video preview
|
4745
|
+
- Parameters for video handling
|
4675
4746
|
|
4676
4747
|
### Aspera Server configuration
|
4677
4748
|
|
@@ -4713,10 +4784,10 @@ Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* whi
|
|
4713
4784
|
|
4714
4785
|
The tool requires the following external tools available in the `PATH`:
|
4715
4786
|
|
4716
|
-
|
4717
|
-
|
4718
|
-
|
4719
|
-
|
4787
|
+
- ImageMagick : `convert` `composite`
|
4788
|
+
- OptiPNG : `optipng`
|
4789
|
+
- FFmpeg : `ffmpeg` `ffprobe`
|
4790
|
+
- Libreoffice : `libreoffice`
|
4720
4791
|
|
4721
4792
|
Here shown on Redhat/CentOS.
|
4722
4793
|
|
@@ -4752,13 +4823,13 @@ If you don't want to have preview for office documents or if it is too complex y
|
|
4752
4823
|
|
4753
4824
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
4754
4825
|
|
4755
|
-
|
4826
|
+
- CentOS 8
|
4756
4827
|
|
4757
4828
|
```bash
|
4758
4829
|
dnf install unoconv
|
4759
4830
|
```
|
4760
4831
|
|
4761
|
-
|
4832
|
+
- Amazon Linux
|
4762
4833
|
|
4763
4834
|
```bash
|
4764
4835
|
amazon-linux-extras enable libreoffice
|
@@ -4851,19 +4922,19 @@ Note that the logging options are kept in the cronfile instead of conf file to a
|
|
4851
4922
|
|
4852
4923
|
The tool generates preview files using those commands:
|
4853
4924
|
|
4854
|
-
|
4855
|
-
|
4856
|
-
|
4857
|
-
|
4925
|
+
- `trevents` : only recently uploaded files will be tested (transfer events)
|
4926
|
+
- `events` : only recently uploaded files will be tested (file events: not working)
|
4927
|
+
- `scan` : recursively scan all files under the access key's "storage root"
|
4928
|
+
- `test` : test using a local file
|
4858
4929
|
|
4859
4930
|
Once candidate are selected, once candidates are selected,
|
4860
4931
|
a preview is always generated if it does not exist already,
|
4861
4932
|
else if a preview already exist, it will be generated
|
4862
4933
|
using one of three values for the `overwrite` option:
|
4863
4934
|
|
4864
|
-
|
4865
|
-
|
4866
|
-
|
4935
|
+
- `always` : preview is always generated, even if it already exists and is newer than original
|
4936
|
+
- `never` : preview is generated only if it does not exist already
|
4937
|
+
- `mtime` : preview is generated only if the original file is newer than the existing
|
4867
4938
|
|
4868
4939
|
Deletion of preview for deleted source files: not implemented yet (TODO).
|
4869
4940
|
|
@@ -4887,8 +4958,8 @@ For instance to filter out files beginning with `._` do:
|
|
4887
4958
|
|
4888
4959
|
Two types of preview can be generated:
|
4889
4960
|
|
4890
|
-
|
4891
|
-
|
4961
|
+
- png: thumbnail
|
4962
|
+
- mp4: video preview (only for video)
|
4892
4963
|
|
4893
4964
|
Use option `skip_format` to skip generation of a format.
|
4894
4965
|
|
@@ -4896,11 +4967,11 @@ Use option `skip_format` to skip generation of a format.
|
|
4896
4967
|
|
4897
4968
|
The preview generator supports rendering of those file categories:
|
4898
4969
|
|
4899
|
-
|
4900
|
-
|
4901
|
-
|
4902
|
-
|
4903
|
-
|
4970
|
+
- image
|
4971
|
+
- pdf
|
4972
|
+
- plaintext
|
4973
|
+
- office
|
4974
|
+
- video
|
4904
4975
|
|
4905
4976
|
To avoid generation for some categories, specify a list using option `skip_types`.
|
4906
4977
|
|
@@ -4923,31 +4994,31 @@ In this case the `preview` command will first analyze the file content using mim
|
|
4923
4994
|
|
4924
4995
|
If the `mimemagic` gem complains about missing mime info file:
|
4925
4996
|
|
4926
|
-
|
4997
|
+
- any OS:
|
4927
4998
|
|
4928
|
-
|
4929
|
-
|
4930
|
-
|
4999
|
+
- Examine the error message
|
5000
|
+
- Download the file: <https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in>
|
5001
|
+
- move and rename this file to one of the locations expected by mimemagic as specified in the error message
|
4931
5002
|
|
4932
|
-
|
5003
|
+
- Windows:
|
4933
5004
|
|
4934
|
-
|
4935
|
-
|
4936
|
-
|
5005
|
+
- Download the file: <https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in>
|
5006
|
+
- Place this file in the root of Ruby (or elsewhere): `C:\RubyVV-x64\freedesktop.org.xml.in`
|
5007
|
+
- Set a global variable using `SystemPropertiesAdvanced.exe` or using `cmd` (replace `VV` with version) to the exact path of this file:
|
4937
5008
|
|
4938
5009
|
```cmd
|
4939
5010
|
SETX FREEDESKTOP_MIME_TYPES_PATH C:\RubyVV-x64\freedesktop.org.xml.in
|
4940
5011
|
```
|
4941
5012
|
|
4942
|
-
|
5013
|
+
- Close the `cmd` and restart a new one if needed to get refreshed env vars
|
4943
5014
|
|
4944
|
-
|
5015
|
+
- Linux:
|
4945
5016
|
|
4946
5017
|
```bash
|
4947
5018
|
yum install shared-mime-info
|
4948
5019
|
```
|
4949
5020
|
|
4950
|
-
|
5021
|
+
- macOS:
|
4951
5022
|
|
4952
5023
|
```bash
|
4953
5024
|
brew install shared-mime-info
|
@@ -5033,9 +5104,9 @@ The template is the full SMTP message, including headers.
|
|
5033
5104
|
|
5034
5105
|
The following variables are defined by default:
|
5035
5106
|
|
5036
|
-
|
5037
|
-
|
5038
|
-
|
5107
|
+
- from_name
|
5108
|
+
- from_email
|
5109
|
+
- to
|
5039
5110
|
|
5040
5111
|
Other variables are defined depending on context.
|
5041
5112
|
|
@@ -5058,10 +5129,10 @@ A default e-mail template is used, but it can be overridden with option `notif_t
|
|
5058
5129
|
|
5059
5130
|
The environment provided contains the following additional variables:
|
5060
5131
|
|
5061
|
-
|
5062
|
-
|
5063
|
-
|
5064
|
-
|
5132
|
+
- subject
|
5133
|
+
- body
|
5134
|
+
- global_transfer_status
|
5135
|
+
- ts
|
5065
5136
|
|
5066
5137
|
Example of template:
|
5067
5138
|
|
@@ -5079,9 +5150,9 @@ This gem comes with a second executable tool providing a simplified standardized
|
|
5079
5150
|
|
5080
5151
|
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [*transfer-spec*](#transferspec) is:
|
5081
5152
|
|
5082
|
-
|
5083
|
-
|
5084
|
-
|
5153
|
+
- common to Aspera Node API (HTTP POST /ops/transfer)
|
5154
|
+
- common to Aspera Connect API (browser javascript startTransfer)
|
5155
|
+
- easy to generate by using any third party language specific JSON library
|
5085
5156
|
|
5086
5157
|
Hopefully, IBM integrates this diectly in `ascp`, and this tool is made redundant.
|
5087
5158
|
|
@@ -5097,8 +5168,8 @@ During execution, it generates all low level events, one per line, in JSON forma
|
|
5097
5168
|
|
5098
5169
|
Note that there are special "extended" [*transfer-spec*](#transferspec) parameters supported by `asession`:
|
5099
5170
|
|
5100
|
-
|
5101
|
-
|
5171
|
+
- `EX_loglevel` to change log level of the tool
|
5172
|
+
- `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
5102
5173
|
|
5103
5174
|
Note that in addition, many "EX_" [*transfer-spec*](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [*transfer-spec*](#transferspec).
|
5104
5175
|
|
@@ -5176,13 +5247,14 @@ EXAMPLES
|
|
5176
5247
|
|
5177
5248
|
### Requirements
|
5178
5249
|
|
5179
|
-
`ascli` maybe used as a simple hot folder engine.
|
5250
|
+
`ascli` maybe used as a simple hot folder engine.
|
5251
|
+
A hot folder being defined as a tool that:
|
5180
5252
|
|
5181
|
-
|
5182
|
-
|
5183
|
-
|
5184
|
-
|
5185
|
-
|
5253
|
+
- locally (or remotely) detects new files in a top folder
|
5254
|
+
- send detected files to a remote (respectively, local) repository
|
5255
|
+
- only sends new files, do not re-send already sent files
|
5256
|
+
- optionally: sends only files that are not still "growing"
|
5257
|
+
- optionally: after transfer of files, deletes or moves to an archive
|
5186
5258
|
|
5187
5259
|
In addition: the detection should be made "continuously" or on specific time/date.
|
5188
5260
|
|
@@ -5190,43 +5262,53 @@ In addition: the detection should be made "continuously" or on specific time/dat
|
|
5190
5262
|
|
5191
5263
|
The general idea is to rely on :
|
5192
5264
|
|
5193
|
-
|
5194
|
-
|
5195
|
-
|
5265
|
+
- existing `ascp` features for detection and transfer
|
5266
|
+
- take advantage of `ascli` configuration capabilities and server side knowledge
|
5267
|
+
- the OS scheduler for reliability and continuous operation
|
5196
5268
|
|
5197
5269
|
#### ascp features
|
5198
5270
|
|
5199
5271
|
Interesting ascp features are found in its arguments: (see ascp manual):
|
5200
5272
|
|
5201
|
-
|
5202
|
-
|
5203
|
-
|
5204
|
-
|
5273
|
+
- `ascp` already takes care of sending only "new" files: option `-k 1,2,3` (`resume_policy`)
|
5274
|
+
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
5275
|
+
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
5276
|
+
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
5205
5277
|
|
5206
5278
|
Note that:
|
5207
5279
|
|
5208
|
-
|
5209
|
-
|
5210
|
-
|
5280
|
+
- `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `--ts` parameter.
|
5281
|
+
- most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters
|
5282
|
+
- native ascp arguments can be provided with the [*transfer-spec*](#transferspec) parameter: `EX_ascp_args` (array), only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node)
|
5211
5283
|
|
5212
5284
|
#### server side and configuration
|
5213
5285
|
|
5214
|
-
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
5286
|
+
Virtually any transfer on a "repository" on a regular basis might emulate a hot folder.
|
5287
|
+
> file detection is not based on events (inotify, etc...), but on a simple folder scan on source side.
|
5215
5288
|
|
5216
|
-
|
5289
|
+
> parameters may be saved in a [option preset](#lprt) and used with `-P`.
|
5217
5290
|
|
5218
5291
|
#### Scheduling
|
5219
5292
|
|
5220
|
-
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
5293
|
+
Once `ascli` parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc...
|
5294
|
+
Refer to section [Scheduling](#scheduling). (on use of option `lock_port`)
|
5221
5295
|
|
5222
|
-
### Example: upload folder
|
5296
|
+
### Example: upload hot folder
|
5223
5297
|
|
5224
5298
|
```javascript
|
5225
|
-
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"
|
5299
|
+
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than:-8,"src_base":"source_hot"}'
|
5226
5300
|
```
|
5227
5301
|
|
5228
|
-
The local folder (here, relative path: source_hot) is sent (upload) to
|
5302
|
+
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
5303
|
+
Source files are deleted after transfer.
|
5304
|
+
Growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period).
|
5305
|
+
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
5229
5306
|
|
5307
|
+
### Example: unidirectional synchronization
|
5308
|
+
|
5309
|
+
```javascript
|
5310
|
+
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than:-8,"src_base":"source_sync"}'
|
5311
|
+
```
|
5230
5312
|
## Health check and Nagios
|
5231
5313
|
|
5232
5314
|
Most plugin provide a `health` command that will check the health status of the application. Example:
|
@@ -5245,9 +5327,9 @@ ascli console health
|
|
5245
5327
|
|
5246
5328
|
Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
|
5247
5329
|
|
5248
|
-
|
5249
|
-
|
5250
|
-
|
5330
|
+
- issuing a transfer to the server
|
5331
|
+
- checking web app status with `asctl all:status`
|
5332
|
+
- checking daemons process status
|
5251
5333
|
|
5252
5334
|
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
5253
5335
|
|
@@ -5271,9 +5353,9 @@ OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:runnin
|
|
5271
5353
|
|
5272
5354
|
Main components:
|
5273
5355
|
|
5274
|
-
|
5275
|
-
|
5276
|
-
|
5356
|
+
- `Aspera` generic classes for REST and OAuth
|
5357
|
+
- `Aspera::Fasp`: starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.
|
5358
|
+
- `Aspera::Cli`: `ascli`.
|
5277
5359
|
|
5278
5360
|
A working example can be found in the gem, example:
|
5279
5361
|
|
@@ -5310,15 +5392,15 @@ When I joined Aspera, there was only one CLI: `ascp`, which is the implementatio
|
|
5310
5392
|
|
5311
5393
|
There were a few pitfalls:
|
5312
5394
|
|
5313
|
-
|
5314
|
-
|
5395
|
+
- The tool was written in the aging `perl` language while most Aspera application products (but the Transfer Server) are written in `ruby`.
|
5396
|
+
- The tool was only for transfers, but not able to call other products APIs
|
5315
5397
|
|
5316
5398
|
So, it evolved into `ascli`:
|
5317
5399
|
|
5318
|
-
|
5319
|
-
|
5320
|
-
|
5321
|
-
|
5400
|
+
- portable: works on platforms supporting `ruby` (and `ascp`)
|
5401
|
+
- easy to install with the `gem` utility
|
5402
|
+
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from ascp command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
5403
|
+
- `ruby` is consistent with other Aspera products
|
5322
5404
|
|
5323
5405
|
## Common problems
|
5324
5406
|
|