aspera-cli 4.3.0 → 4.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +445 -194
- data/docs/README.erb.md +449 -239
- data/docs/doc_tools.rb +58 -0
- data/lib/aspera/aoc.rb +4 -3
- data/lib/aspera/cli/plugin.rb +13 -6
- data/lib/aspera/cli/plugins/aoc.rb +87 -51
- data/lib/aspera/cli/plugins/config.rb +79 -50
- data/lib/aspera/cli/plugins/faspex.rb +8 -6
- data/lib/aspera/cli/plugins/node.rb +51 -23
- data/lib/aspera/cli/plugins/preview.rb +9 -7
- data/lib/aspera/cli/plugins/server.rb +16 -5
- data/lib/aspera/cli/transfer_agent.rb +5 -20
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/fasp/http_gw.rb +47 -26
- data/lib/aspera/fasp/local.rb +0 -2
- data/lib/aspera/fasp/parameters.rb +16 -3
- data/lib/aspera/faspex_gw.rb +10 -9
- data/lib/aspera/node.rb +10 -0
- metadata +3 -3
- data/lib/aspera/fasp/aoc.rb +0 -24
data/README.md
CHANGED
@@ -1,11 +1,11 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
2
|
<font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
|
3
3
|
|
4
|
-
Version : 4.
|
4
|
+
Version : 4.4.0
|
5
5
|
|
6
6
|
_Laurent/2016-2021_
|
7
7
|
|
8
|
-
This gem provides `ascli
|
8
|
+
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
9
9
|
|
10
10
|
`ascli` is a also great tool to learn Aspera APIs.
|
11
11
|
|
@@ -57,7 +57,7 @@ Command line parameters in examples beginning with `my_`, like `my_param_value`
|
|
57
57
|
|
58
58
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
59
59
|
|
60
|
-
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
60
|
+
On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
61
61
|
|
62
62
|
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
63
63
|
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
@@ -82,7 +82,7 @@ Once the gem is installed, `ascli` shall be accessible:
|
|
82
82
|
|
83
83
|
```
|
84
84
|
$ ascli --version
|
85
|
-
4.
|
85
|
+
4.4.0
|
86
86
|
```
|
87
87
|
|
88
88
|
## First use
|
@@ -109,7 +109,7 @@ $ ascli server browse /
|
|
109
109
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
110
110
|
|
111
111
|
* create a [option preset](#lprt)
|
112
|
-
* define it as default for
|
112
|
+
* define it as default for `server` plugin
|
113
113
|
* list files in a folder
|
114
114
|
* download a file
|
115
115
|
|
@@ -166,6 +166,8 @@ Use this method only if you know what you do, else use the standard recommended
|
|
166
166
|
|
167
167
|
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
168
168
|
|
169
|
+
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
170
|
+
|
169
171
|
Ensure that you have Docker installed.
|
170
172
|
|
171
173
|
```
|
@@ -234,7 +236,7 @@ If you keep the same terminal (ont needed if re-login):
|
|
234
236
|
$ source ~/.rvm/scripts/rvm
|
235
237
|
```
|
236
238
|
|
237
|
-
It is advised to get one of the pre-compiled ruby version, you can list with:
|
239
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
238
240
|
|
239
241
|
```
|
240
242
|
$ rvm list --remote
|
@@ -430,8 +432,8 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
430
432
|
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
431
433
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
432
434
|
|
433
|
-
Several methods are provided
|
434
|
-
other methods are available. Refer to section: [Transfer Agents](#agents)
|
435
|
+
Several methods are provided to start a transfer.
|
436
|
+
Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
435
437
|
|
436
438
|
## <a name="offline_install"></a>Offline Installation (without internet)
|
437
439
|
|
@@ -477,7 +479,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
477
479
|
* Supports most Aspera server products (on-premise and SaaS)
|
478
480
|
* Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
479
481
|
* Supports Commands, Option values and Parameters shortcuts
|
480
|
-
* FASP [Transfer Agents](#agents) can be:
|
482
|
+
* FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
481
483
|
* Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
|
482
484
|
* Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
483
485
|
* Supports FaspStream creation (using Node API)
|
@@ -601,12 +603,12 @@ The style of output can be set using the `format` parameter, supporting:
|
|
601
603
|
* `yaml` : YAML
|
602
604
|
* `csv` : Comma Separated Values
|
603
605
|
|
604
|
-
###
|
606
|
+
### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
605
607
|
|
606
608
|
Table output can be filtered using the `select` parameter. Example:
|
607
609
|
|
608
610
|
```
|
609
|
-
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"
|
611
|
+
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
610
612
|
:...............................:..................................:...........:
|
611
613
|
: name : email : ats_admin :
|
612
614
|
:...............................:..................................:...........:
|
@@ -753,19 +755,15 @@ C:\Users\Kenji\.aspera\ascli
|
|
753
755
|
On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
|
754
756
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
755
757
|
|
756
|
-
Although the file is a standard YAML file, `ascli` provides commands to read and modify it
|
757
|
-
using the `config` command.
|
758
|
+
Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
|
758
759
|
|
759
|
-
All options for `ascli`
|
760
|
+
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
|
760
761
|
|
761
|
-
A configuration file provides a way to define default values, especially
|
762
|
-
for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
762
|
+
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
763
763
|
|
764
|
-
The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
|
765
|
-
(this can be overriden with option `--config-file=path` or equivalent env var).
|
764
|
+
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
|
766
765
|
|
767
|
-
|
768
|
-
called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
766
|
+
The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
769
767
|
|
770
768
|
### <a name="lprt"></a>Option preset
|
771
769
|
|
@@ -865,7 +863,7 @@ demo_server:
|
|
865
863
|
We can see here:
|
866
864
|
|
867
865
|
* The configuration was created with CLI version 0.3.7
|
868
|
-
* the default [option preset](#lprt) to load for plugin
|
866
|
+
* the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
869
867
|
* the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
870
868
|
* the default [option preset](#lprt) to load in any case is : `cli_default`
|
871
869
|
|
@@ -896,26 +894,26 @@ my_aoc_org:
|
|
896
894
|
|
897
895
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
898
896
|
|
897
|
+
### Options evaluation order
|
898
|
+
|
899
|
+
Some options are global, some options are available only for some plugins. (the plugin is the first level command).
|
900
|
+
|
899
901
|
Options are loaded using this algorithm:
|
900
902
|
|
901
|
-
*
|
902
|
-
|
903
|
-
|
904
|
-
*
|
905
|
-
*
|
903
|
+
* If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
904
|
+
* else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
905
|
+
* If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
906
|
+
* Environment variables are evaluated
|
907
|
+
* Command line options are evaluated
|
906
908
|
|
907
909
|
Parameters are evaluated in the order of command line.
|
908
910
|
|
909
|
-
To avoid loading the default [option preset](#lprt) for a plugin,
|
911
|
+
To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
|
910
912
|
|
911
913
|
On command line, words in parameter names are separated by a dash, in configuration file, separator
|
912
914
|
is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
|
913
915
|
|
914
|
-
|
915
|
-
convert olver versions, remove the leading ":" in front of keys.
|
916
|
-
|
917
|
-
The main plugin name is *config*, so it is possible to define a default [option preset](#lprt) for
|
918
|
-
the main plugin with:
|
916
|
+
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
919
917
|
|
920
918
|
```
|
921
919
|
$ ascli config id cli_default set interactive no
|
@@ -928,6 +926,18 @@ A [option preset](#lprt) value can be removed with `unset`:
|
|
928
926
|
$ ascli config id cli_default unset interactive
|
929
927
|
```
|
930
928
|
|
929
|
+
Example: Define options using command line:
|
930
|
+
|
931
|
+
```
|
932
|
+
$ ascli -N --url=x --password=y --username=y node --show-config
|
933
|
+
```
|
934
|
+
|
935
|
+
Example: Define options using a hash:
|
936
|
+
|
937
|
+
```
|
938
|
+
$ ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
939
|
+
```
|
940
|
+
|
931
941
|
### Examples
|
932
942
|
|
933
943
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
@@ -1056,7 +1066,7 @@ $ ascli config ascp info
|
|
1056
1066
|
...
|
1057
1067
|
```
|
1058
1068
|
|
1059
|
-
### Selection of
|
1069
|
+
### Selection of `ascp` location for [`direct`](#direct) agent
|
1060
1070
|
|
1061
1071
|
By default, `ascli` uses any found local product with ascp, including SDK.
|
1062
1072
|
|
@@ -1100,7 +1110,7 @@ $ ascli config ascp products list
|
|
1100
1110
|
:.........................................:................................................:
|
1101
1111
|
```
|
1102
1112
|
|
1103
|
-
### Selection of local client
|
1113
|
+
### Selection of local client for `ascp` for [`direct`](#direct) agent
|
1104
1114
|
|
1105
1115
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1106
1116
|
|
@@ -1155,7 +1165,7 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
1155
1165
|
|
1156
1166
|
There are currently 3 agents:
|
1157
1167
|
|
1158
|
-
* `direct` : a local execution of `ascp`
|
1168
|
+
* [`direct`](#direct) : a local execution of `ascp`
|
1159
1169
|
* `connect` : use of a local Connect Client
|
1160
1170
|
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
1161
1171
|
* `httpgw` : use of an Aspera HTTP Gateway
|
@@ -1167,7 +1177,7 @@ will effectively push files to the related server from the agent node.
|
|
1167
1177
|
`ascli` standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1168
1178
|
|
1169
1179
|
|
1170
|
-
### <a name="direct"></a>Direct (local ascp
|
1180
|
+
### <a name="direct"></a>Direct (local ascp execution)
|
1171
1181
|
|
1172
1182
|
By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
|
1173
1183
|
`ascli` will detect locally installed Aspera products.
|
@@ -1265,7 +1275,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
1265
1275
|
|
1266
1276
|
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
1267
1277
|
|
1268
|
-
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
1278
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
1269
1279
|
|
1270
1280
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
1271
1281
|
|
@@ -1298,7 +1308,7 @@ Columns:
|
|
1298
1308
|
* C=Connect Client
|
1299
1309
|
* arg=`ascp` argument or environment variable
|
1300
1310
|
|
1301
|
-
Fields with EX_ prefix are extensions to transfer agent `direct
|
1311
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in `ascli`).
|
1302
1312
|
|
1303
1313
|
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
1304
1314
|
|
@@ -1316,8 +1326,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1316
1326
|
|
1317
1327
|
### List of files for transfers
|
1318
1328
|
|
1319
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
1320
|
-
|
1329
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
1330
|
+
So, by default, the list of files to transfer will be simply specified on the command line:
|
1321
1331
|
|
1322
1332
|
```
|
1323
1333
|
$ ascli server upload ~/mysample.file secondfile
|
@@ -1329,7 +1339,7 @@ This is equivalent to:
|
|
1329
1339
|
$ ascli server upload --sources=@args ~/mysample.file secondfile
|
1330
1340
|
```
|
1331
1341
|
|
1332
|
-
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1342
|
+
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1333
1343
|
|
1334
1344
|
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
1335
1345
|
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
@@ -1352,13 +1362,15 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
1352
1362
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
1353
1363
|
```
|
1354
1364
|
|
1355
|
-
*
|
1365
|
+
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
1356
1366
|
|
1357
1367
|
```
|
1358
1368
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
1359
1369
|
```
|
1360
1370
|
|
1361
|
-
|
1371
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
1372
|
+
|
1373
|
+
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
1362
1374
|
|
1363
1375
|
* `list` : (default) the path of destination is the same as source
|
1364
1376
|
* `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
|
@@ -1369,6 +1381,8 @@ Example:
|
|
1369
1381
|
$ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
1370
1382
|
```
|
1371
1383
|
|
1384
|
+
Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
|
1385
|
+
|
1372
1386
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
1373
1387
|
|
1374
1388
|
* All files must be in the same source folder.
|
@@ -1433,15 +1447,154 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
1433
1447
|
|
1434
1448
|
|
1435
1449
|
|
1436
|
-
## <a name="scheduling"></a>
|
1450
|
+
## <a name="scheduling"></a>Lock for exclusive execution
|
1451
|
+
|
1452
|
+
In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
|
1453
|
+
|
1454
|
+
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
1455
|
+
|
1456
|
+
* Executing instances may pile-up and kill the system
|
1457
|
+
* The same file may be transfered by multiple instances at the same time.
|
1458
|
+
* `preview` may generate the same files in multiple instances.
|
1459
|
+
|
1460
|
+
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
1461
|
+
|
1462
|
+
* The Windows scheduler does this by default
|
1463
|
+
* Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
1464
|
+
|
1465
|
+
```
|
1466
|
+
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
1467
|
+
```
|
1468
|
+
|
1469
|
+
`ascli` natively supports a locking mechanism with option `lock_port`.
|
1470
|
+
(Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
|
1471
|
+
|
1472
|
+
Example:
|
1473
|
+
|
1474
|
+
Run this same command in two separate terminals within less than 30 seconds:
|
1475
|
+
|
1476
|
+
```
|
1477
|
+
ascli config echo @ruby:'sleep(30)' --lock-port=12345
|
1478
|
+
```
|
1479
|
+
|
1480
|
+
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
1481
|
+
|
1482
|
+
```
|
1483
|
+
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
1484
|
+
```
|
1485
|
+
|
1486
|
+
## "Provençale"
|
1487
|
+
|
1488
|
+
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
1489
|
+
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
1490
|
+
|
1491
|
+
The list of supported *PVCL* adapters can be retried with command:
|
1492
|
+
|
1493
|
+
```
|
1494
|
+
$ ascli conf ascp info
|
1495
|
+
+--------------------+-----------------------------------------------------------+
|
1496
|
+
| key | value |
|
1497
|
+
+--------------------+-----------------------------------------------------------+
|
1498
|
+
-----8<----snip---------
|
1499
|
+
| product_name | IBM Aspera SDK |
|
1500
|
+
| product_version | 4.0.1.182389 |
|
1501
|
+
| process | pvcl |
|
1502
|
+
| shares | pvcl |
|
1503
|
+
| noded | pvcl |
|
1504
|
+
| faux | pvcl |
|
1505
|
+
| file | pvcl |
|
1506
|
+
| stdio | pvcl |
|
1507
|
+
| stdio-tar | pvcl |
|
1508
|
+
+--------------------+-----------------------------------------------------------+
|
1509
|
+
```
|
1510
|
+
|
1511
|
+
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
1512
|
+
|
1513
|
+
Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
|
1514
|
+
|
1515
|
+
The simplified format is:
|
1516
|
+
|
1517
|
+
```
|
1518
|
+
<adapter>:///<sub file path>?<arg1>=<val1>&...
|
1519
|
+
```
|
1520
|
+
|
1521
|
+
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
1522
|
+
|
1523
|
+
## <a name="faux_testing"></a>`faux:` for testing
|
1524
|
+
|
1525
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`
|
1526
|
+
|
1527
|
+
This adapter can be used to simulate a file or a directory.
|
1528
|
+
|
1529
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
|
1530
|
+
|
1531
|
+
* `fname` is the name that will be assigned to the file on the destination
|
1532
|
+
* `fsize` is the number of bytes that will be sent (in decimal).
|
1533
|
+
|
1534
|
+
Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
|
1535
|
+
|
1536
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
|
1537
|
+
|
1538
|
+
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
1539
|
+
|
1540
|
+
```
|
1541
|
+
faux:///dirname?<arg1>=<val1>&...
|
1542
|
+
```
|
1543
|
+
|
1544
|
+
`dirname` is the folder name and can contain `/` to specify a subfolder.
|
1545
|
+
|
1546
|
+
Supported arguments are:
|
1547
|
+
|
1548
|
+
<table>
|
1549
|
+
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
1550
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
|
1551
|
+
<tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
|
1552
|
+
<tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
|
1553
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
|
1554
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
|
1555
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
|
1556
|
+
</table>
|
1557
|
+
|
1558
|
+
|
1559
|
+
The sequence parameter is applied as follows:
|
1560
|
+
|
1561
|
+
* If `seq` is `random` then each file size is:
|
1562
|
+
|
1563
|
+
* size +/- (inc * rand())
|
1564
|
+
* Where rand is a random number between 0 and 1
|
1565
|
+
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
1566
|
+
* Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
|
1437
1567
|
|
1438
|
-
|
1568
|
+
* If `seq` is `sequential` then each file size is:
|
1439
1569
|
|
1440
|
-
|
1570
|
+
* size + ((fileindex - 1) * inc)
|
1571
|
+
* Where first file is index 1
|
1572
|
+
* So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
|
1573
|
+
* As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
|
1441
1574
|
|
1442
|
-
|
1575
|
+
Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
|
1443
1576
|
|
1444
|
-
|
1577
|
+
To discard data at the destination, the destination argument is set to `faux://` .
|
1578
|
+
|
1579
|
+
Examples:
|
1580
|
+
|
1581
|
+
* Upload 20 gigabytes of random data to file myfile to directory /Upload
|
1582
|
+
|
1583
|
+
```
|
1584
|
+
$ ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
1585
|
+
```
|
1586
|
+
|
1587
|
+
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
1588
|
+
|
1589
|
+
```
|
1590
|
+
$ ascli server upload /tmp/sample --to-folder=faux://
|
1591
|
+
```
|
1592
|
+
|
1593
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
|
1594
|
+
|
1595
|
+
```
|
1596
|
+
$ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
1597
|
+
```
|
1445
1598
|
|
1446
1599
|
## <a name="commands"></a>Sample Commands
|
1447
1600
|
|
@@ -1462,7 +1615,7 @@ ascli aoc admin ats cluster clouds
|
|
1462
1615
|
ascli aoc admin ats cluster list
|
1463
1616
|
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
1464
1617
|
ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
|
1465
|
-
ascli aoc admin res
|
1618
|
+
ascli aoc admin res application list
|
1466
1619
|
ascli aoc admin res client list
|
1467
1620
|
ascli aoc admin res client_access_key list
|
1468
1621
|
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
@@ -1520,7 +1673,7 @@ ascli aoc files v3 info
|
|
1520
1673
|
ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
|
1521
1674
|
ascli aoc organization
|
1522
1675
|
ascli aoc packages list
|
1523
|
-
ascli aoc packages list --
|
1676
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
1524
1677
|
ascli aoc packages recv --id="my_package_id" --to-folder=.
|
1525
1678
|
ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
|
1526
1679
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
@@ -1530,6 +1683,8 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
|
|
1530
1683
|
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
1531
1684
|
ascli aoc user info modify @json:'{"name":"dummy change"}'
|
1532
1685
|
ascli aoc user info show
|
1686
|
+
ascli aoc user shared_inboxes
|
1687
|
+
ascli aoc user workspaces
|
1533
1688
|
ascli aoc workspace
|
1534
1689
|
ascli ats access_key --id=ak_aws delete
|
1535
1690
|
ascli ats access_key --id=akibmcloud --secret=somesecret cluster
|
@@ -1556,6 +1711,7 @@ ascli config ascp info
|
|
1556
1711
|
ascli config ascp install
|
1557
1712
|
ascli config ascp products list
|
1558
1713
|
ascli config ascp show
|
1714
|
+
ascli config ascp spec
|
1559
1715
|
ascli config check_update
|
1560
1716
|
ascli config doc
|
1561
1717
|
ascli config doc transfer-parameters
|
@@ -1680,7 +1836,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
1680
1836
|
```
|
1681
1837
|
$ ascli -h
|
1682
1838
|
NAME
|
1683
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
1839
|
+
ascli -- a command line tool for Aspera Applications (v4.4.0)
|
1684
1840
|
|
1685
1841
|
SYNOPSIS
|
1686
1842
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -1737,31 +1893,31 @@ OPTIONS:
|
|
1737
1893
|
--property=VALUE name of property to set
|
1738
1894
|
--id=VALUE resource identifier (modify,delete,show)
|
1739
1895
|
--config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
|
1740
|
-
--override=ENUM override existing value: yes, no
|
1741
1896
|
-N, --no-default do not load default configuration for plugin
|
1742
|
-
--
|
1743
|
-
--
|
1897
|
+
--override=ENUM Wizard: override existing value: yes, no
|
1898
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
|
1899
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
|
1900
|
+
--test-mode=ENUM Wizard: skip private key check step: yes, no
|
1901
|
+
--pkeypath=VALUE Wizard: path to private key for JWT
|
1744
1902
|
--ascp-path=VALUE path to ascp
|
1745
1903
|
--use-product=VALUE use ascp from specified product
|
1746
1904
|
--smtp=VALUE smtp configuration (extended value: hash)
|
1747
1905
|
--fpac=VALUE proxy auto configuration URL
|
1748
1906
|
-P, --presetVALUE load the named option preset from current config file
|
1749
|
-
--default=VALUE set as default configuration for specified plugin
|
1750
1907
|
--secret=VALUE default secret
|
1751
1908
|
--secrets=VALUE secret repository (Hash)
|
1752
1909
|
--sdk-url=VALUE URL to get SDK
|
1753
|
-
--sdk-folder=VALUE SDK folder
|
1910
|
+
--sdk-folder=VALUE SDK folder path
|
1754
1911
|
--notif-to=VALUE email recipient for notification of transfers
|
1755
1912
|
--notif-template=VALUE email ERB template for notification of transfers
|
1756
|
-
--
|
1757
|
-
--version-check-days=VALUE period to check neew version in days (zero to disable)
|
1913
|
+
--version-check-days=VALUE period in days to check new version (zero to disable)
|
1758
1914
|
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
1759
1915
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
1760
1916
|
--to-folder=VALUE destination folder for downloaded files
|
1761
1917
|
--sources=VALUE list of source files (see doc)
|
1762
|
-
--transfer-info=VALUE
|
1918
|
+
--transfer-info=VALUE parameters for transfer agent
|
1763
1919
|
--src-type=ENUM type of file list: list, pair
|
1764
|
-
--transfer=ENUM type of transfer: direct, httpgw, connect, node
|
1920
|
+
--transfer=ENUM type of transfer agent: direct, httpgw, connect, node
|
1765
1921
|
--progress=ENUM type of progress bar: none, native, multi
|
1766
1922
|
|
1767
1923
|
|
@@ -1782,7 +1938,7 @@ OPTIONS:
|
|
1782
1938
|
--validator=VALUE identifier of validator (optional for central)
|
1783
1939
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
1784
1940
|
--name=VALUE sync name
|
1785
|
-
--token=ENUM
|
1941
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
1786
1942
|
|
1787
1943
|
|
1788
1944
|
COMMAND: orchestrator
|
@@ -2213,19 +2369,55 @@ It allows actions (create, update, delete) on "resources": users, group, nodes,
|
|
2213
2369
|
|
2214
2370
|
Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2215
2371
|
|
2216
|
-
|
2372
|
+
#### Listing resources
|
2373
|
+
|
2374
|
+
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
2375
|
+
|
2376
|
+
The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
|
2377
|
+
|
2378
|
+
The following parameters are supported:
|
2379
|
+
|
2380
|
+
* `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
2381
|
+
* `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
2382
|
+
* `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
2383
|
+
* `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
2384
|
+
* `page` : native api parameter, in general do not use (added by
|
2385
|
+
* `per_page` : native api parameter, number of items par api call, in general do not use
|
2386
|
+
* Other specific parameters depending on resource type.
|
2387
|
+
|
2388
|
+
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
2389
|
+
|
2390
|
+
Other parameters are directly sent as parameters to the GET request on API.
|
2391
|
+
|
2392
|
+
`page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
2393
|
+
|
2394
|
+
`q` and `sort` are available on most resrouce types.
|
2395
|
+
|
2396
|
+
Other parameters depend on the type of entity (refer to AoC API).
|
2397
|
+
|
2398
|
+
Examples:
|
2399
|
+
|
2400
|
+
* List users with `laurent` in name:
|
2217
2401
|
|
2218
2402
|
```
|
2219
|
-
--query=@json:'{"
|
2403
|
+
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2220
2404
|
```
|
2221
2405
|
|
2222
|
-
|
2406
|
+
* List users who logded-in before a date:
|
2407
|
+
|
2223
2408
|
```
|
2224
|
-
--query=@json:'{"
|
2225
|
-
--query=@json:'{"q":"laurent"}'
|
2409
|
+
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2226
2410
|
```
|
2227
2411
|
|
2228
|
-
|
2412
|
+
* List external users and sort in reverse alphabetical order using name:
|
2413
|
+
|
2414
|
+
```
|
2415
|
+
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
2416
|
+
```
|
2417
|
+
|
2418
|
+
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
2419
|
+
|
2420
|
+
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
2229
2421
|
|
2230
2422
|
#### Access Key secrets
|
2231
2423
|
|
@@ -2326,17 +2518,20 @@ $ ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_k
|
|
2326
2518
|
$ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
2327
2519
|
```
|
2328
2520
|
|
2329
|
-
|
2330
|
-
|
2331
|
-
|
2332
|
-
|
2521
|
+
Examples of query (TODO: cleanup):
|
2522
|
+
|
2523
|
+
```
|
2524
|
+
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
2525
|
+
|
2526
|
+
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
2527
|
+
|
2333
2528
|
# filter= 'id', 'short_summary', or 'summary'
|
2334
2529
|
# count=nnn
|
2335
2530
|
# tag=x.y.z%3Dvalue
|
2336
2531
|
# iteration_token=nnn
|
2337
2532
|
# after_time=2016-05-01T23:53:09Z
|
2338
2533
|
# active_only=true|false
|
2339
|
-
|
2534
|
+
```
|
2340
2535
|
|
2341
2536
|
* Display node events (events)
|
2342
2537
|
|
@@ -2347,7 +2542,7 @@ $ ascli aoc admin res node --secret=_secret_ v3 events
|
|
2347
2542
|
* display members of a workspace
|
2348
2543
|
|
2349
2544
|
```
|
2350
|
-
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"
|
2545
|
+
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
2351
2546
|
:.............:.........:..................................:
|
2352
2547
|
: member_type : manager : member.email :
|
2353
2548
|
:.............:.........:..................................:
|
@@ -2368,27 +2563,27 @@ other query parameters:
|
|
2368
2563
|
|
2369
2564
|
* <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
|
2370
2565
|
|
2371
|
-
a-
|
2566
|
+
a- Get id of first workspace
|
2372
2567
|
|
2373
2568
|
```
|
2374
2569
|
WS1='First Workspace'
|
2375
2570
|
WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
2376
2571
|
```
|
2377
2572
|
|
2378
|
-
b-
|
2573
|
+
b- Get id of second workspace
|
2379
2574
|
|
2380
2575
|
```
|
2381
2576
|
WS2='Second Workspace'
|
2382
2577
|
WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
|
2383
2578
|
```
|
2384
2579
|
|
2385
|
-
c-
|
2580
|
+
c- Extract membership information
|
2386
2581
|
|
2387
2582
|
```
|
2388
|
-
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"
|
2583
|
+
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
2389
2584
|
```
|
2390
2585
|
|
2391
|
-
d-
|
2586
|
+
d- Convert to creation data for second workspace:
|
2392
2587
|
|
2393
2588
|
```
|
2394
2589
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
@@ -2400,16 +2595,16 @@ or, using jq:
|
|
2400
2595
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
2401
2596
|
```
|
2402
2597
|
|
2403
|
-
e-
|
2598
|
+
e- Add members to second workspace
|
2404
2599
|
|
2405
2600
|
```
|
2406
2601
|
$ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
2407
2602
|
```
|
2408
2603
|
|
2409
|
-
*
|
2604
|
+
* Get users who did not log since a date
|
2410
2605
|
|
2411
2606
|
```
|
2412
|
-
$ ascli aoc admin res user list --fields=email --query=@json:'{"
|
2607
|
+
$ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2413
2608
|
:...............................:
|
2414
2609
|
: email :
|
2415
2610
|
:...............................:
|
@@ -2418,10 +2613,10 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
|
|
2418
2613
|
:...............................:
|
2419
2614
|
```
|
2420
2615
|
|
2421
|
-
*
|
2616
|
+
* List "Limited" users
|
2422
2617
|
|
2423
2618
|
```
|
2424
|
-
$ ascli aoc admin res user list --fields=email --
|
2619
|
+
$ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
2425
2620
|
```
|
2426
2621
|
|
2427
2622
|
* Perform a multi Gbps transfer between two remote shared folders
|
@@ -2464,6 +2659,7 @@ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='
|
|
2464
2659
|
```
|
2465
2660
|
|
2466
2661
|
* create registration key to register a node
|
2662
|
+
|
2467
2663
|
```
|
2468
2664
|
$ ascli aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
2469
2665
|
jfqslfdjlfdjfhdjklqfhdkl
|
@@ -2483,6 +2679,19 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
|
|
2483
2679
|
+-----+---------+
|
2484
2680
|
```
|
2485
2681
|
|
2682
|
+
* List packages in a given shared inbox
|
2683
|
+
|
2684
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
2685
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2686
|
+
|
2687
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
2688
|
+
|
2689
|
+
```
|
2690
|
+
shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
2691
|
+
|
2692
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
2693
|
+
```
|
2694
|
+
|
2486
2695
|
## Shared folders
|
2487
2696
|
|
2488
2697
|
* list shared folders in node
|
@@ -2515,7 +2724,7 @@ Notes:
|
|
2515
2724
|
|
2516
2725
|
* the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
|
2517
2726
|
* to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". ascli will resolve the list of email addresses to expected user ids.
|
2518
|
-
* a
|
2727
|
+
* a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
|
2519
2728
|
* If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
2520
2729
|
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
|
2521
2730
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
@@ -2524,7 +2733,13 @@ Examples:
|
|
2524
2733
|
|
2525
2734
|
```
|
2526
2735
|
$ ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
|
2736
|
+
```
|
2737
|
+
|
2738
|
+
```
|
2527
2739
|
$ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
|
2740
|
+
```
|
2741
|
+
|
2742
|
+
```
|
2528
2743
|
$ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2529
2744
|
```
|
2530
2745
|
|
@@ -2540,7 +2755,7 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
2540
2755
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
2541
2756
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
2542
2757
|
|
2543
|
-
Typically, one would
|
2758
|
+
Typically, one would execute this command on a regular basis, using the method of your choice:
|
2544
2759
|
|
2545
2760
|
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2546
2761
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
@@ -2548,45 +2763,45 @@ Typically, one would regularly execute this command on a regular basis, using th
|
|
2548
2763
|
|
2549
2764
|
## Download Files
|
2550
2765
|
|
2551
|
-
Download of files is straightforward with a specific syntax for the `
|
2766
|
+
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2552
2767
|
|
2553
2768
|
* if only one source is provided, it is downloaded
|
2554
2769
|
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2555
2770
|
|
2556
2771
|
## Find Files
|
2557
2772
|
|
2558
|
-
The command `
|
2773
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2559
2774
|
|
2560
2775
|
The expression can be of 3 formats:
|
2561
2776
|
|
2562
|
-
* empty (default) : all files, equivalent to: `exec:true`
|
2563
|
-
* not starting with `exec:` : the expression is a regular expression, using ruby
|
2777
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
2778
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2564
2779
|
|
2565
2780
|
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2566
2781
|
|
2567
|
-
* starting with `exec:` : the
|
2782
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2568
2783
|
|
2569
|
-
Examples of expressions: (
|
2784
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
2570
2785
|
|
2571
|
-
*
|
2786
|
+
* Find files more recent than 100 days
|
2572
2787
|
|
2573
2788
|
```
|
2574
2789
|
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
2575
2790
|
```
|
2576
2791
|
|
2577
|
-
*
|
2792
|
+
* Find files older than 1 year on a given node and store in file list
|
2578
2793
|
|
2579
2794
|
```
|
2580
2795
|
$ ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
2581
2796
|
```
|
2582
2797
|
|
2583
|
-
*
|
2798
|
+
* Delete the files, one by one
|
2584
2799
|
|
2585
2800
|
```
|
2586
2801
|
$ cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
2587
2802
|
```
|
2588
2803
|
|
2589
|
-
*
|
2804
|
+
* Delete the files in bulk
|
2590
2805
|
|
2591
2806
|
```
|
2592
2807
|
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
@@ -2721,7 +2936,7 @@ Explanation:
|
|
2721
2936
|
* `|` the standard output of the first command is fed into the second one
|
2722
2937
|
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2723
2938
|
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2724
|
-
* `--transfer=node` use transfer agent type `node` instead of default `direct`
|
2939
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
|
2725
2940
|
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2726
2941
|
|
2727
2942
|
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
@@ -2770,6 +2985,14 @@ This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
|
|
2770
2985
|
|
2771
2986
|
Both password and SSH keys auth are supported.
|
2772
2987
|
|
2988
|
+
If not username is provided, the default transfer user `xfer` is used.
|
2989
|
+
|
2990
|
+
If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
|
2991
|
+
|
2992
|
+
```
|
2993
|
+
$ ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
2994
|
+
```
|
2995
|
+
|
2773
2996
|
Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
|
2774
2997
|
|
2775
2998
|
Examples:
|
@@ -2807,7 +3030,7 @@ This means that you dont have such an ssh agent running:
|
|
2807
3030
|
$ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
2808
3031
|
```
|
2809
3032
|
|
2810
|
-
This can also be set as default using a preset
|
3033
|
+
This can also be set as default using a preset.
|
2811
3034
|
|
2812
3035
|
## Example
|
2813
3036
|
|
@@ -2827,18 +3050,36 @@ This creates a [option preset](#lprt) "aspera_demo_server" and set it as default
|
|
2827
3050
|
|
2828
3051
|
This plugin gives access to capabilities provided by HSTS node API.
|
2829
3052
|
|
2830
|
-
##
|
3053
|
+
## File Operations
|
2831
3054
|
|
2832
3055
|
It is possible to:
|
2833
3056
|
* browse
|
2834
3057
|
* transfer (upload / download)
|
2835
3058
|
* ...
|
2836
3059
|
|
3060
|
+
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
3061
|
+
|
3062
|
+
* `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
|
3063
|
+
* `basic` : transfer spec is created like this:
|
3064
|
+
|
3065
|
+
```
|
3066
|
+
{
|
3067
|
+
"remote_host": address of node url,
|
3068
|
+
"remote_user": "xfer",
|
3069
|
+
"ssh_port": 33001,
|
3070
|
+
"token": "Basic <base 64 encoded user/pass>",
|
3071
|
+
"direction": send/recv
|
3072
|
+
}
|
3073
|
+
```
|
3074
|
+
|
3075
|
+
* `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
3076
|
+
|
2837
3077
|
## Central
|
2838
3078
|
|
2839
3079
|
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
|
2840
3080
|
|
2841
3081
|
Filtering can be applied:
|
3082
|
+
|
2842
3083
|
```
|
2843
3084
|
$ ascli node central file list
|
2844
3085
|
```
|
@@ -2864,7 +3105,6 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
2864
3105
|
* Start watchd and watchfolderd services running as a system user having access to files
|
2865
3106
|
* configure a watchfolder to define automated transfers
|
2866
3107
|
|
2867
|
-
|
2868
3108
|
```
|
2869
3109
|
$ ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
2870
3110
|
$ ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
@@ -3182,7 +3422,7 @@ $ ascli cos node info
|
|
3182
3422
|
$ ascli cos node upload 'faux:///sample1G?1g'
|
3183
3423
|
```
|
3184
3424
|
|
3185
|
-
Note: we generate a dummy file `sample1G`
|
3425
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
3186
3426
|
|
3187
3427
|
# Plugin: IBM Aspera Sync
|
3188
3428
|
|
@@ -3279,8 +3519,10 @@ yum install -y ImageMagick optipng
|
|
3279
3519
|
|
3280
3520
|
### Video: FFmpeg
|
3281
3521
|
|
3522
|
+
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
3523
|
+
|
3282
3524
|
```
|
3283
|
-
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt &&
|
3525
|
+
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
3284
3526
|
```
|
3285
3527
|
|
3286
3528
|
### Office: Unoconv and Libreoffice
|
@@ -3578,7 +3820,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
3578
3820
|
* `EX_loglevel` to change log level of the tool
|
3579
3821
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
3580
3822
|
|
3581
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the
|
3823
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
3582
3824
|
|
3583
3825
|
## Comparison of interfaces
|
3584
3826
|
|
@@ -3679,7 +3921,7 @@ Note that:
|
|
3679
3921
|
|
3680
3922
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
3681
3923
|
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3682
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the
|
3924
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
|
3683
3925
|
|
3684
3926
|
### server side and configuration
|
3685
3927
|
|
@@ -3698,7 +3940,7 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
|
|
3698
3940
|
|
3699
3941
|
```
|
3700
3942
|
|
3701
|
-
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3943
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3702
3944
|
|
3703
3945
|
# Aspera Health check and Nagios
|
3704
3946
|
|
@@ -3775,9 +4017,19 @@ So, it evolved into `ascli`:
|
|
3775
4017
|
|
3776
4018
|
# Changes (Release notes)
|
3777
4019
|
|
4020
|
+
* 4.4.0
|
4021
|
+
|
4022
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
4023
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
4024
|
+
* new: `preset` option can specify name or hash value
|
4025
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
4026
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
4027
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
4028
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
4029
|
+
|
3778
4030
|
* 4.3.0
|
3779
4031
|
|
3780
|
-
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on `direct` transfer agent.
|
4032
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
|
3781
4033
|
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
3782
4034
|
|
3783
4035
|
* 4.2.2
|
@@ -3802,20 +4054,19 @@ So, it evolved into `ascli`:
|
|
3802
4054
|
* new: command `aoc remind` to receive organization membership by email
|
3803
4055
|
* new: in `preview` option `value` to filter out on file name
|
3804
4056
|
* new: `initdemo` to initialize for demo server
|
3805
|
-
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
4057
|
+
* new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3806
4058
|
* fix: on Windows `conf ascp use` expects ascp.exe
|
3807
4059
|
* fix: (break) multi_session_threshold is Integer, not String
|
3808
4060
|
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3809
4061
|
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3810
4062
|
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3811
|
-
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
4063
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
|
3812
4064
|
|
3813
4065
|
* 4.1.0
|
3814
4066
|
|
3815
|
-
|
3816
|
-
|
3817
|
-
|
3818
|
-
* new: renamed command `nagios_check` to `health`
|
4067
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
4068
|
+
* new: update documentation with regard to offline and docker installation
|
4069
|
+
* new: renamed command `nagios_check` to `health`
|
3819
4070
|
* new: agent `http_gw` now supports upload
|
3820
4071
|
* new: added option `sdk_url` to install SDK from local file for offline install
|
3821
4072
|
* new: check new gem version periodically
|
@@ -3866,8 +4117,8 @@ So, it evolved into `ascli`:
|
|
3866
4117
|
* allow bulk delete in `aspera files` with option `bulk=yes`
|
3867
4118
|
* fix getting connect versions
|
3868
4119
|
* added section for Aix
|
3869
|
-
* support all ciphers for `
|
3870
|
-
* added transfer spec param `apply_local_docroot` for `
|
4120
|
+
* support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
|
4121
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#direct)
|
3871
4122
|
|
3872
4123
|
* 0.11.4
|
3873
4124
|
|
@@ -3922,7 +4173,7 @@ So, it evolved into `ascli`:
|
|
3922
4173
|
* 0.10.12
|
3923
4174
|
|
3924
4175
|
* added support for AoC node registration keys
|
3925
|
-
* replaced option : `local_resume` with `transfer_info` for agent `direct`
|
4176
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
|
3926
4177
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
3927
4178
|
* `@incps` : new extended value modifier
|
3928
4179
|
* ATS: no more provides access keys secrets: now user must provide it
|
@@ -3972,9 +4223,9 @@ So, it evolved into `ascli`:
|
|
3972
4223
|
|
3973
4224
|
* 0.10.2
|
3974
4225
|
|
3975
|
-
|
3976
|
-
|
3977
|
-
|
4226
|
+
* updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
|
4227
|
+
* added doc for "cargo" like actions
|
4228
|
+
* added doc for multi-session
|
3978
4229
|
|
3979
4230
|
* 0.10.1
|
3980
4231
|
|
@@ -4002,22 +4253,22 @@ So, it evolved into `ascli`:
|
|
4002
4253
|
|
4003
4254
|
* 0.9.33
|
4004
4255
|
|
4005
|
-
|
4006
|
-
|
4007
|
-
|
4008
|
-
|
4256
|
+
* new command to display basic token of node
|
4257
|
+
* new command to display bearer token of node in AoC
|
4258
|
+
* the --fields= option, support +_fieldname_ to add a field to default fields
|
4259
|
+
* many small changes
|
4009
4260
|
|
4010
4261
|
* 0.9.32
|
4011
4262
|
|
4012
|
-
|
4013
|
-
|
4014
|
-
|
4263
|
+
* all Faspex public links are now supported
|
4264
|
+
* removed faspex operation recv_publink
|
4265
|
+
* replaced with option `link` (consistent with AoC)
|
4015
4266
|
|
4016
4267
|
* 0.9.31
|
4017
4268
|
|
4018
|
-
|
4019
|
-
|
4020
|
-
|
4269
|
+
* added more support for public link: receive and send package, to user or dropbox and files view.
|
4270
|
+
* delete expired file lists
|
4271
|
+
* changed text table gem from text-table to terminal-table because it supports multiline values
|
4021
4272
|
|
4022
4273
|
* 0.9.27
|
4023
4274
|
|
@@ -4037,149 +4288,149 @@ So, it evolved into `ascli`:
|
|
4037
4288
|
|
4038
4289
|
* 0.9.24
|
4039
4290
|
|
4040
|
-
|
4041
|
-
|
4291
|
+
* fix bug where AoC node to node transfer did not work
|
4292
|
+
* fix bug on error if ED25519 private key is defined in .ssh
|
4042
4293
|
|
4043
4294
|
* 0.9.23
|
4044
4295
|
|
4045
|
-
|
4046
|
-
|
4296
|
+
* defined REST error handlers, more error conditions detected
|
4297
|
+
* commands to select specific ascp location
|
4047
4298
|
|
4048
4299
|
* 0.9.21
|
4049
4300
|
|
4050
|
-
|
4051
|
-
|
4301
|
+
* supports simplified wizard using global client
|
4302
|
+
* only ascp binary is required, other SDK (keys) files are now generated
|
4052
4303
|
|
4053
4304
|
* 0.9.20
|
4054
4305
|
|
4055
|
-
|
4056
|
-
|
4057
|
-
|
4306
|
+
* improved wizard (prepare for AoC global client id)
|
4307
|
+
* preview generator: addedoption : --skip-format=<png,mp4>
|
4308
|
+
* removed outdated pictures from this doc
|
4058
4309
|
|
4059
4310
|
* 0.9.19
|
4060
4311
|
|
4061
|
-
|
4312
|
+
* added command aspera bearer --scope=xx
|
4062
4313
|
|
4063
4314
|
* 0.9.18
|
4064
4315
|
|
4065
|
-
|
4316
|
+
* enhanced aspera admin events to support query
|
4066
4317
|
|
4067
4318
|
* 0.9.16
|
4068
4319
|
|
4069
|
-
|
4070
|
-
|
4320
|
+
* AoC transfers are now reported in activity app
|
4321
|
+
* new interface for Rest class authentication (keep backward compatibility)
|
4071
4322
|
|
4072
4323
|
* 0.9.15
|
4073
4324
|
|
4074
|
-
|
4075
|
-
|
4325
|
+
* new feature: "find" command in aspera files
|
4326
|
+
* sample code for transfer API
|
4076
4327
|
|
4077
4328
|
* 0.9.12
|
4078
4329
|
|
4079
|
-
|
4080
|
-
|
4330
|
+
* add nagios commands
|
4331
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
4081
4332
|
|
4082
4333
|
* 0.9.11
|
4083
4334
|
|
4084
|
-
|
4085
|
-
|
4335
|
+
* Breaking change: @stdin is now @stdin:
|
4336
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
4086
4337
|
|
4087
4338
|
|
4088
4339
|
* 0.9.10
|
4089
4340
|
|
4090
|
-
|
4091
|
-
|
4092
|
-
|
4093
|
-
|
4341
|
+
* Breaking change: parameter transfer-node becomes more generic: transfer-info
|
4342
|
+
* Display SaaS storage usage with command: aspera admin res node --id=nn info
|
4343
|
+
* cleaner way of specifying source file list for transfers
|
4344
|
+
* Breaking change: replaced download_mode option with http_download action
|
4094
4345
|
|
4095
4346
|
* 0.9.9
|
4096
4347
|
|
4097
|
-
|
4098
|
-
|
4348
|
+
* Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
|
4349
|
+
* Now supports "cargo" for Aspera on Cloud (automatic package download)
|
4099
4350
|
|
4100
4351
|
* 0.9.8
|
4101
4352
|
|
4102
|
-
|
4103
|
-
|
4353
|
+
* Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
|
4354
|
+
* AoC: share to share transfer with command "transfer"
|
4104
4355
|
|
4105
4356
|
* 0.9.7
|
4106
4357
|
|
4107
|
-
|
4108
|
-
|
4109
|
-
|
4110
|
-
|
4111
|
-
|
4358
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
|
4359
|
+
* preview persistency goes to unique file by default
|
4360
|
+
* catch mxf extension in preview as video
|
4361
|
+
* Faspex: possibility to download all paclages by specifying id=ALL
|
4362
|
+
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
4112
4363
|
|
4113
4364
|
* 0.9.6
|
4114
4365
|
|
4115
|
-
|
4116
|
-
|
4366
|
+
* Breaking change: `@param:`is now `@preset:` and is generic
|
4367
|
+
* AoC: added command to display current workspace information
|
4117
4368
|
|
4118
4369
|
* 0.9.5
|
4119
4370
|
|
4120
|
-
|
4121
|
-
|
4371
|
+
* new parameter: new_user_option used to choose between public_link and invite of external users.
|
4372
|
+
* fixed bug in wizard, and wizard uses now product detection
|
4122
4373
|
|
4123
4374
|
* 0.9.4
|
4124
4375
|
|
4125
|
-
|
4126
|
-
|
4127
|
-
|
4376
|
+
* Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
|
4377
|
+
* AoC Package send supports external users
|
4378
|
+
* new command to export AoC config to Aspera CLI config
|
4128
4379
|
|
4129
4380
|
* 0.9.3
|
4130
4381
|
|
4131
|
-
|
4132
|
-
|
4133
|
-
|
4134
|
-
|
4135
|
-
|
4136
|
-
|
4382
|
+
* REST error message show host and code
|
4383
|
+
* option for quiet display
|
4384
|
+
* modified transfer interface and allow token re-generation on error
|
4385
|
+
* async add admin command
|
4386
|
+
* async add db parameters
|
4387
|
+
* Breaking change: new option "sources" to specify files to transfer
|
4137
4388
|
|
4138
4389
|
* 0.9.2
|
4139
4390
|
|
4140
|
-
|
4391
|
+
* Breaking change: changed AoC package creation to match API, see AoC section
|
4141
4392
|
|
4142
4393
|
* 0.9.1
|
4143
4394
|
|
4144
|
-
|
4395
|
+
* Breaking change: changed faspex package creation to match API, see Faspex section
|
4145
4396
|
|
4146
4397
|
* 0.9
|
4147
4398
|
|
4148
|
-
|
4149
|
-
|
4399
|
+
* Renamed the CLI from aslmcli to `ascli`
|
4400
|
+
* Automatic rename and conversion of former config folder from aslmcli to `ascli`
|
4150
4401
|
|
4151
4402
|
* 0.7.6
|
4152
4403
|
|
4153
|
-
|
4404
|
+
* add "sync" plugin
|
4154
4405
|
|
4155
4406
|
* 0.7
|
4156
4407
|
|
4157
|
-
|
4158
|
-
|
4159
|
-
|
4160
|
-
|
4161
|
-
|
4408
|
+
* Breaking change: AoC package recv take option if for package instead of argument.
|
4409
|
+
* Breaking change: Rest class and Oauth class changed init parameters
|
4410
|
+
* AoC: receive package from public link
|
4411
|
+
* select by col value on output
|
4412
|
+
* added rename (AoC, node)
|
4162
4413
|
|
4163
4414
|
* 0.6.19
|
4164
4415
|
|
4165
4416
|
Breaking change:
|
4166
4417
|
|
4167
|
-
|
4168
|
-
|
4169
|
-
|
4170
|
-
|
4171
|
-
|
4172
|
-
|
4173
|
-
|
4174
|
-
|
4418
|
+
* ats server list provisioned → ats cluster list
|
4419
|
+
* ats server list clouds → ats cluster clouds
|
4420
|
+
* ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
|
4421
|
+
* ats server id xxx → ats cluster show --id=xxx
|
4422
|
+
* ats subscriptions → ats credential subscriptions
|
4423
|
+
* ats api_key repository list → ats credential cache list
|
4424
|
+
* ats api_key list → ats credential list
|
4425
|
+
* ats access_key id xxx → ats access_key --id=xxx
|
4175
4426
|
|
4176
4427
|
* 0.6.18
|
4177
4428
|
|
4178
|
-
|
4429
|
+
* some commands take now --id option instead of id command.
|
4179
4430
|
|
4180
4431
|
* 0.6.15
|
4181
4432
|
|
4182
|
-
|
4433
|
+
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
|
4183
4434
|
|
4184
4435
|
# BUGS, FEATURES, CONTRIBUTION
|
4185
4436
|
|
@@ -4232,8 +4483,8 @@ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "C
|
|
4232
4483
|
|
4233
4484
|
* remove rest and oauth classes and use ruby standard gems:
|
4234
4485
|
|
4235
|
-
|
4236
|
-
|
4486
|
+
* oauth
|
4487
|
+
* https://github.com/rest-client/rest-client
|
4237
4488
|
|
4238
4489
|
* use Thor or any standard Ruby CLI manager
|
4239
4490
|
|