aspera-cli 4.3.0 → 4.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +445 -194
- data/docs/README.erb.md +449 -239
- data/docs/doc_tools.rb +58 -0
- data/lib/aspera/aoc.rb +4 -3
- data/lib/aspera/cli/plugin.rb +13 -6
- data/lib/aspera/cli/plugins/aoc.rb +87 -51
- data/lib/aspera/cli/plugins/config.rb +79 -50
- data/lib/aspera/cli/plugins/faspex.rb +8 -6
- data/lib/aspera/cli/plugins/node.rb +51 -23
- data/lib/aspera/cli/plugins/preview.rb +9 -7
- data/lib/aspera/cli/plugins/server.rb +16 -5
- data/lib/aspera/cli/transfer_agent.rb +5 -20
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/fasp/http_gw.rb +47 -26
- data/lib/aspera/fasp/local.rb +0 -2
- data/lib/aspera/fasp/parameters.rb +16 -3
- data/lib/aspera/faspex_gw.rb +10 -9
- data/lib/aspera/node.rb +10 -0
- metadata +3 -3
- data/lib/aspera/fasp/aoc.rb +0 -24
data/README.md
CHANGED
@@ -1,11 +1,11 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
2
|
<font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
|
3
3
|
|
4
|
-
Version : 4.
|
4
|
+
Version : 4.4.0
|
5
5
|
|
6
6
|
_Laurent/2016-2021_
|
7
7
|
|
8
|
-
This gem provides `ascli
|
8
|
+
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
9
9
|
|
10
10
|
`ascli` is a also great tool to learn Aspera APIs.
|
11
11
|
|
@@ -57,7 +57,7 @@ Command line parameters in examples beginning with `my_`, like `my_param_value`
|
|
57
57
|
|
58
58
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
59
59
|
|
60
|
-
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
60
|
+
On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
61
61
|
|
62
62
|
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
63
63
|
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
@@ -82,7 +82,7 @@ Once the gem is installed, `ascli` shall be accessible:
|
|
82
82
|
|
83
83
|
```
|
84
84
|
$ ascli --version
|
85
|
-
4.
|
85
|
+
4.4.0
|
86
86
|
```
|
87
87
|
|
88
88
|
## First use
|
@@ -109,7 +109,7 @@ $ ascli server browse /
|
|
109
109
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
110
110
|
|
111
111
|
* create a [option preset](#lprt)
|
112
|
-
* define it as default for
|
112
|
+
* define it as default for `server` plugin
|
113
113
|
* list files in a folder
|
114
114
|
* download a file
|
115
115
|
|
@@ -166,6 +166,8 @@ Use this method only if you know what you do, else use the standard recommended
|
|
166
166
|
|
167
167
|
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
168
168
|
|
169
|
+
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
170
|
+
|
169
171
|
Ensure that you have Docker installed.
|
170
172
|
|
171
173
|
```
|
@@ -234,7 +236,7 @@ If you keep the same terminal (ont needed if re-login):
|
|
234
236
|
$ source ~/.rvm/scripts/rvm
|
235
237
|
```
|
236
238
|
|
237
|
-
It is advised to get one of the pre-compiled ruby version, you can list with:
|
239
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
238
240
|
|
239
241
|
```
|
240
242
|
$ rvm list --remote
|
@@ -430,8 +432,8 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
430
432
|
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
431
433
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
432
434
|
|
433
|
-
Several methods are provided
|
434
|
-
other methods are available. Refer to section: [Transfer Agents](#agents)
|
435
|
+
Several methods are provided to start a transfer.
|
436
|
+
Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
435
437
|
|
436
438
|
## <a name="offline_install"></a>Offline Installation (without internet)
|
437
439
|
|
@@ -477,7 +479,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
477
479
|
* Supports most Aspera server products (on-premise and SaaS)
|
478
480
|
* Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
479
481
|
* Supports Commands, Option values and Parameters shortcuts
|
480
|
-
* FASP [Transfer Agents](#agents) can be:
|
482
|
+
* FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
481
483
|
* Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
|
482
484
|
* Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
483
485
|
* Supports FaspStream creation (using Node API)
|
@@ -601,12 +603,12 @@ The style of output can be set using the `format` parameter, supporting:
|
|
601
603
|
* `yaml` : YAML
|
602
604
|
* `csv` : Comma Separated Values
|
603
605
|
|
604
|
-
###
|
606
|
+
### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
605
607
|
|
606
608
|
Table output can be filtered using the `select` parameter. Example:
|
607
609
|
|
608
610
|
```
|
609
|
-
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"
|
611
|
+
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
610
612
|
:...............................:..................................:...........:
|
611
613
|
: name : email : ats_admin :
|
612
614
|
:...............................:..................................:...........:
|
@@ -753,19 +755,15 @@ C:\Users\Kenji\.aspera\ascli
|
|
753
755
|
On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
|
754
756
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
755
757
|
|
756
|
-
Although the file is a standard YAML file, `ascli` provides commands to read and modify it
|
757
|
-
using the `config` command.
|
758
|
+
Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
|
758
759
|
|
759
|
-
All options for `ascli`
|
760
|
+
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
|
760
761
|
|
761
|
-
A configuration file provides a way to define default values, especially
|
762
|
-
for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
762
|
+
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
763
763
|
|
764
|
-
The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
|
765
|
-
(this can be overriden with option `--config-file=path` or equivalent env var).
|
764
|
+
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
|
766
765
|
|
767
|
-
|
768
|
-
called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
766
|
+
The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
769
767
|
|
770
768
|
### <a name="lprt"></a>Option preset
|
771
769
|
|
@@ -865,7 +863,7 @@ demo_server:
|
|
865
863
|
We can see here:
|
866
864
|
|
867
865
|
* The configuration was created with CLI version 0.3.7
|
868
|
-
* the default [option preset](#lprt) to load for plugin
|
866
|
+
* the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
869
867
|
* the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
870
868
|
* the default [option preset](#lprt) to load in any case is : `cli_default`
|
871
869
|
|
@@ -896,26 +894,26 @@ my_aoc_org:
|
|
896
894
|
|
897
895
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
898
896
|
|
897
|
+
### Options evaluation order
|
898
|
+
|
899
|
+
Some options are global, some options are available only for some plugins. (the plugin is the first level command).
|
900
|
+
|
899
901
|
Options are loaded using this algorithm:
|
900
902
|
|
901
|
-
*
|
902
|
-
|
903
|
-
|
904
|
-
*
|
905
|
-
*
|
903
|
+
* If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
904
|
+
* else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
905
|
+
* If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
906
|
+
* Environment variables are evaluated
|
907
|
+
* Command line options are evaluated
|
906
908
|
|
907
909
|
Parameters are evaluated in the order of command line.
|
908
910
|
|
909
|
-
To avoid loading the default [option preset](#lprt) for a plugin,
|
911
|
+
To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
|
910
912
|
|
911
913
|
On command line, words in parameter names are separated by a dash, in configuration file, separator
|
912
914
|
is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
|
913
915
|
|
914
|
-
|
915
|
-
convert olver versions, remove the leading ":" in front of keys.
|
916
|
-
|
917
|
-
The main plugin name is *config*, so it is possible to define a default [option preset](#lprt) for
|
918
|
-
the main plugin with:
|
916
|
+
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
919
917
|
|
920
918
|
```
|
921
919
|
$ ascli config id cli_default set interactive no
|
@@ -928,6 +926,18 @@ A [option preset](#lprt) value can be removed with `unset`:
|
|
928
926
|
$ ascli config id cli_default unset interactive
|
929
927
|
```
|
930
928
|
|
929
|
+
Example: Define options using command line:
|
930
|
+
|
931
|
+
```
|
932
|
+
$ ascli -N --url=x --password=y --username=y node --show-config
|
933
|
+
```
|
934
|
+
|
935
|
+
Example: Define options using a hash:
|
936
|
+
|
937
|
+
```
|
938
|
+
$ ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
939
|
+
```
|
940
|
+
|
931
941
|
### Examples
|
932
942
|
|
933
943
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
@@ -1056,7 +1066,7 @@ $ ascli config ascp info
|
|
1056
1066
|
...
|
1057
1067
|
```
|
1058
1068
|
|
1059
|
-
### Selection of
|
1069
|
+
### Selection of `ascp` location for [`direct`](#direct) agent
|
1060
1070
|
|
1061
1071
|
By default, `ascli` uses any found local product with ascp, including SDK.
|
1062
1072
|
|
@@ -1100,7 +1110,7 @@ $ ascli config ascp products list
|
|
1100
1110
|
:.........................................:................................................:
|
1101
1111
|
```
|
1102
1112
|
|
1103
|
-
### Selection of local client
|
1113
|
+
### Selection of local client for `ascp` for [`direct`](#direct) agent
|
1104
1114
|
|
1105
1115
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1106
1116
|
|
@@ -1155,7 +1165,7 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
1155
1165
|
|
1156
1166
|
There are currently 3 agents:
|
1157
1167
|
|
1158
|
-
* `direct` : a local execution of `ascp`
|
1168
|
+
* [`direct`](#direct) : a local execution of `ascp`
|
1159
1169
|
* `connect` : use of a local Connect Client
|
1160
1170
|
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
1161
1171
|
* `httpgw` : use of an Aspera HTTP Gateway
|
@@ -1167,7 +1177,7 @@ will effectively push files to the related server from the agent node.
|
|
1167
1177
|
`ascli` standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1168
1178
|
|
1169
1179
|
|
1170
|
-
### <a name="direct"></a>Direct (local ascp
|
1180
|
+
### <a name="direct"></a>Direct (local ascp execution)
|
1171
1181
|
|
1172
1182
|
By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
|
1173
1183
|
`ascli` will detect locally installed Aspera products.
|
@@ -1265,7 +1275,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
1265
1275
|
|
1266
1276
|
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
1267
1277
|
|
1268
|
-
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
1278
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
1269
1279
|
|
1270
1280
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
1271
1281
|
|
@@ -1298,7 +1308,7 @@ Columns:
|
|
1298
1308
|
* C=Connect Client
|
1299
1309
|
* arg=`ascp` argument or environment variable
|
1300
1310
|
|
1301
|
-
Fields with EX_ prefix are extensions to transfer agent `direct
|
1311
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in `ascli`).
|
1302
1312
|
|
1303
1313
|
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
1304
1314
|
|
@@ -1316,8 +1326,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1316
1326
|
|
1317
1327
|
### List of files for transfers
|
1318
1328
|
|
1319
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
1320
|
-
|
1329
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
1330
|
+
So, by default, the list of files to transfer will be simply specified on the command line:
|
1321
1331
|
|
1322
1332
|
```
|
1323
1333
|
$ ascli server upload ~/mysample.file secondfile
|
@@ -1329,7 +1339,7 @@ This is equivalent to:
|
|
1329
1339
|
$ ascli server upload --sources=@args ~/mysample.file secondfile
|
1330
1340
|
```
|
1331
1341
|
|
1332
|
-
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1342
|
+
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1333
1343
|
|
1334
1344
|
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
1335
1345
|
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
@@ -1352,13 +1362,15 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
1352
1362
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
1353
1363
|
```
|
1354
1364
|
|
1355
|
-
*
|
1365
|
+
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
1356
1366
|
|
1357
1367
|
```
|
1358
1368
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
1359
1369
|
```
|
1360
1370
|
|
1361
|
-
|
1371
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
1372
|
+
|
1373
|
+
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
1362
1374
|
|
1363
1375
|
* `list` : (default) the path of destination is the same as source
|
1364
1376
|
* `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
|
@@ -1369,6 +1381,8 @@ Example:
|
|
1369
1381
|
$ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
1370
1382
|
```
|
1371
1383
|
|
1384
|
+
Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
|
1385
|
+
|
1372
1386
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
1373
1387
|
|
1374
1388
|
* All files must be in the same source folder.
|
@@ -1433,15 +1447,154 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
1433
1447
|
|
1434
1448
|
|
1435
1449
|
|
1436
|
-
## <a name="scheduling"></a>
|
1450
|
+
## <a name="scheduling"></a>Lock for exclusive execution
|
1451
|
+
|
1452
|
+
In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
|
1453
|
+
|
1454
|
+
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
1455
|
+
|
1456
|
+
* Executing instances may pile-up and kill the system
|
1457
|
+
* The same file may be transfered by multiple instances at the same time.
|
1458
|
+
* `preview` may generate the same files in multiple instances.
|
1459
|
+
|
1460
|
+
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
1461
|
+
|
1462
|
+
* The Windows scheduler does this by default
|
1463
|
+
* Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
1464
|
+
|
1465
|
+
```
|
1466
|
+
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
1467
|
+
```
|
1468
|
+
|
1469
|
+
`ascli` natively supports a locking mechanism with option `lock_port`.
|
1470
|
+
(Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
|
1471
|
+
|
1472
|
+
Example:
|
1473
|
+
|
1474
|
+
Run this same command in two separate terminals within less than 30 seconds:
|
1475
|
+
|
1476
|
+
```
|
1477
|
+
ascli config echo @ruby:'sleep(30)' --lock-port=12345
|
1478
|
+
```
|
1479
|
+
|
1480
|
+
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
1481
|
+
|
1482
|
+
```
|
1483
|
+
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
1484
|
+
```
|
1485
|
+
|
1486
|
+
## "Provençale"
|
1487
|
+
|
1488
|
+
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
1489
|
+
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
1490
|
+
|
1491
|
+
The list of supported *PVCL* adapters can be retried with command:
|
1492
|
+
|
1493
|
+
```
|
1494
|
+
$ ascli conf ascp info
|
1495
|
+
+--------------------+-----------------------------------------------------------+
|
1496
|
+
| key | value |
|
1497
|
+
+--------------------+-----------------------------------------------------------+
|
1498
|
+
-----8<----snip---------
|
1499
|
+
| product_name | IBM Aspera SDK |
|
1500
|
+
| product_version | 4.0.1.182389 |
|
1501
|
+
| process | pvcl |
|
1502
|
+
| shares | pvcl |
|
1503
|
+
| noded | pvcl |
|
1504
|
+
| faux | pvcl |
|
1505
|
+
| file | pvcl |
|
1506
|
+
| stdio | pvcl |
|
1507
|
+
| stdio-tar | pvcl |
|
1508
|
+
+--------------------+-----------------------------------------------------------+
|
1509
|
+
```
|
1510
|
+
|
1511
|
+
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
1512
|
+
|
1513
|
+
Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
|
1514
|
+
|
1515
|
+
The simplified format is:
|
1516
|
+
|
1517
|
+
```
|
1518
|
+
<adapter>:///<sub file path>?<arg1>=<val1>&...
|
1519
|
+
```
|
1520
|
+
|
1521
|
+
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
1522
|
+
|
1523
|
+
## <a name="faux_testing"></a>`faux:` for testing
|
1524
|
+
|
1525
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`
|
1526
|
+
|
1527
|
+
This adapter can be used to simulate a file or a directory.
|
1528
|
+
|
1529
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
|
1530
|
+
|
1531
|
+
* `fname` is the name that will be assigned to the file on the destination
|
1532
|
+
* `fsize` is the number of bytes that will be sent (in decimal).
|
1533
|
+
|
1534
|
+
Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
|
1535
|
+
|
1536
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
|
1537
|
+
|
1538
|
+
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
1539
|
+
|
1540
|
+
```
|
1541
|
+
faux:///dirname?<arg1>=<val1>&...
|
1542
|
+
```
|
1543
|
+
|
1544
|
+
`dirname` is the folder name and can contain `/` to specify a subfolder.
|
1545
|
+
|
1546
|
+
Supported arguments are:
|
1547
|
+
|
1548
|
+
<table>
|
1549
|
+
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
1550
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
|
1551
|
+
<tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
|
1552
|
+
<tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
|
1553
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
|
1554
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
|
1555
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
|
1556
|
+
</table>
|
1557
|
+
|
1558
|
+
|
1559
|
+
The sequence parameter is applied as follows:
|
1560
|
+
|
1561
|
+
* If `seq` is `random` then each file size is:
|
1562
|
+
|
1563
|
+
* size +/- (inc * rand())
|
1564
|
+
* Where rand is a random number between 0 and 1
|
1565
|
+
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
1566
|
+
* Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
|
1437
1567
|
|
1438
|
-
|
1568
|
+
* If `seq` is `sequential` then each file size is:
|
1439
1569
|
|
1440
|
-
|
1570
|
+
* size + ((fileindex - 1) * inc)
|
1571
|
+
* Where first file is index 1
|
1572
|
+
* So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
|
1573
|
+
* As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
|
1441
1574
|
|
1442
|
-
|
1575
|
+
Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
|
1443
1576
|
|
1444
|
-
|
1577
|
+
To discard data at the destination, the destination argument is set to `faux://` .
|
1578
|
+
|
1579
|
+
Examples:
|
1580
|
+
|
1581
|
+
* Upload 20 gigabytes of random data to file myfile to directory /Upload
|
1582
|
+
|
1583
|
+
```
|
1584
|
+
$ ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
1585
|
+
```
|
1586
|
+
|
1587
|
+
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
1588
|
+
|
1589
|
+
```
|
1590
|
+
$ ascli server upload /tmp/sample --to-folder=faux://
|
1591
|
+
```
|
1592
|
+
|
1593
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
|
1594
|
+
|
1595
|
+
```
|
1596
|
+
$ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
1597
|
+
```
|
1445
1598
|
|
1446
1599
|
## <a name="commands"></a>Sample Commands
|
1447
1600
|
|
@@ -1462,7 +1615,7 @@ ascli aoc admin ats cluster clouds
|
|
1462
1615
|
ascli aoc admin ats cluster list
|
1463
1616
|
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
1464
1617
|
ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
|
1465
|
-
ascli aoc admin res
|
1618
|
+
ascli aoc admin res application list
|
1466
1619
|
ascli aoc admin res client list
|
1467
1620
|
ascli aoc admin res client_access_key list
|
1468
1621
|
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
@@ -1520,7 +1673,7 @@ ascli aoc files v3 info
|
|
1520
1673
|
ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
|
1521
1674
|
ascli aoc organization
|
1522
1675
|
ascli aoc packages list
|
1523
|
-
ascli aoc packages list --
|
1676
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
1524
1677
|
ascli aoc packages recv --id="my_package_id" --to-folder=.
|
1525
1678
|
ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
|
1526
1679
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
@@ -1530,6 +1683,8 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
|
|
1530
1683
|
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
1531
1684
|
ascli aoc user info modify @json:'{"name":"dummy change"}'
|
1532
1685
|
ascli aoc user info show
|
1686
|
+
ascli aoc user shared_inboxes
|
1687
|
+
ascli aoc user workspaces
|
1533
1688
|
ascli aoc workspace
|
1534
1689
|
ascli ats access_key --id=ak_aws delete
|
1535
1690
|
ascli ats access_key --id=akibmcloud --secret=somesecret cluster
|
@@ -1556,6 +1711,7 @@ ascli config ascp info
|
|
1556
1711
|
ascli config ascp install
|
1557
1712
|
ascli config ascp products list
|
1558
1713
|
ascli config ascp show
|
1714
|
+
ascli config ascp spec
|
1559
1715
|
ascli config check_update
|
1560
1716
|
ascli config doc
|
1561
1717
|
ascli config doc transfer-parameters
|
@@ -1680,7 +1836,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
1680
1836
|
```
|
1681
1837
|
$ ascli -h
|
1682
1838
|
NAME
|
1683
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
1839
|
+
ascli -- a command line tool for Aspera Applications (v4.4.0)
|
1684
1840
|
|
1685
1841
|
SYNOPSIS
|
1686
1842
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -1737,31 +1893,31 @@ OPTIONS:
|
|
1737
1893
|
--property=VALUE name of property to set
|
1738
1894
|
--id=VALUE resource identifier (modify,delete,show)
|
1739
1895
|
--config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
|
1740
|
-
--override=ENUM override existing value: yes, no
|
1741
1896
|
-N, --no-default do not load default configuration for plugin
|
1742
|
-
--
|
1743
|
-
--
|
1897
|
+
--override=ENUM Wizard: override existing value: yes, no
|
1898
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
|
1899
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
|
1900
|
+
--test-mode=ENUM Wizard: skip private key check step: yes, no
|
1901
|
+
--pkeypath=VALUE Wizard: path to private key for JWT
|
1744
1902
|
--ascp-path=VALUE path to ascp
|
1745
1903
|
--use-product=VALUE use ascp from specified product
|
1746
1904
|
--smtp=VALUE smtp configuration (extended value: hash)
|
1747
1905
|
--fpac=VALUE proxy auto configuration URL
|
1748
1906
|
-P, --presetVALUE load the named option preset from current config file
|
1749
|
-
--default=VALUE set as default configuration for specified plugin
|
1750
1907
|
--secret=VALUE default secret
|
1751
1908
|
--secrets=VALUE secret repository (Hash)
|
1752
1909
|
--sdk-url=VALUE URL to get SDK
|
1753
|
-
--sdk-folder=VALUE SDK folder
|
1910
|
+
--sdk-folder=VALUE SDK folder path
|
1754
1911
|
--notif-to=VALUE email recipient for notification of transfers
|
1755
1912
|
--notif-template=VALUE email ERB template for notification of transfers
|
1756
|
-
--
|
1757
|
-
--version-check-days=VALUE period to check neew version in days (zero to disable)
|
1913
|
+
--version-check-days=VALUE period in days to check new version (zero to disable)
|
1758
1914
|
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
1759
1915
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
1760
1916
|
--to-folder=VALUE destination folder for downloaded files
|
1761
1917
|
--sources=VALUE list of source files (see doc)
|
1762
|
-
--transfer-info=VALUE
|
1918
|
+
--transfer-info=VALUE parameters for transfer agent
|
1763
1919
|
--src-type=ENUM type of file list: list, pair
|
1764
|
-
--transfer=ENUM type of transfer: direct, httpgw, connect, node
|
1920
|
+
--transfer=ENUM type of transfer agent: direct, httpgw, connect, node
|
1765
1921
|
--progress=ENUM type of progress bar: none, native, multi
|
1766
1922
|
|
1767
1923
|
|
@@ -1782,7 +1938,7 @@ OPTIONS:
|
|
1782
1938
|
--validator=VALUE identifier of validator (optional for central)
|
1783
1939
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
1784
1940
|
--name=VALUE sync name
|
1785
|
-
--token=ENUM
|
1941
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
1786
1942
|
|
1787
1943
|
|
1788
1944
|
COMMAND: orchestrator
|
@@ -2213,19 +2369,55 @@ It allows actions (create, update, delete) on "resources": users, group, nodes,
|
|
2213
2369
|
|
2214
2370
|
Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2215
2371
|
|
2216
|
-
|
2372
|
+
#### Listing resources
|
2373
|
+
|
2374
|
+
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
2375
|
+
|
2376
|
+
The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
|
2377
|
+
|
2378
|
+
The following parameters are supported:
|
2379
|
+
|
2380
|
+
* `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
2381
|
+
* `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
2382
|
+
* `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
2383
|
+
* `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
2384
|
+
* `page` : native api parameter, in general do not use (added by
|
2385
|
+
* `per_page` : native api parameter, number of items par api call, in general do not use
|
2386
|
+
* Other specific parameters depending on resource type.
|
2387
|
+
|
2388
|
+
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
2389
|
+
|
2390
|
+
Other parameters are directly sent as parameters to the GET request on API.
|
2391
|
+
|
2392
|
+
`page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
2393
|
+
|
2394
|
+
`q` and `sort` are available on most resrouce types.
|
2395
|
+
|
2396
|
+
Other parameters depend on the type of entity (refer to AoC API).
|
2397
|
+
|
2398
|
+
Examples:
|
2399
|
+
|
2400
|
+
* List users with `laurent` in name:
|
2217
2401
|
|
2218
2402
|
```
|
2219
|
-
--query=@json:'{"
|
2403
|
+
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2220
2404
|
```
|
2221
2405
|
|
2222
|
-
|
2406
|
+
* List users who logded-in before a date:
|
2407
|
+
|
2223
2408
|
```
|
2224
|
-
--query=@json:'{"
|
2225
|
-
--query=@json:'{"q":"laurent"}'
|
2409
|
+
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2226
2410
|
```
|
2227
2411
|
|
2228
|
-
|
2412
|
+
* List external users and sort in reverse alphabetical order using name:
|
2413
|
+
|
2414
|
+
```
|
2415
|
+
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
2416
|
+
```
|
2417
|
+
|
2418
|
+
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
2419
|
+
|
2420
|
+
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
2229
2421
|
|
2230
2422
|
#### Access Key secrets
|
2231
2423
|
|
@@ -2326,17 +2518,20 @@ $ ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_k
|
|
2326
2518
|
$ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
2327
2519
|
```
|
2328
2520
|
|
2329
|
-
|
2330
|
-
|
2331
|
-
|
2332
|
-
|
2521
|
+
Examples of query (TODO: cleanup):
|
2522
|
+
|
2523
|
+
```
|
2524
|
+
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
2525
|
+
|
2526
|
+
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
2527
|
+
|
2333
2528
|
# filter= 'id', 'short_summary', or 'summary'
|
2334
2529
|
# count=nnn
|
2335
2530
|
# tag=x.y.z%3Dvalue
|
2336
2531
|
# iteration_token=nnn
|
2337
2532
|
# after_time=2016-05-01T23:53:09Z
|
2338
2533
|
# active_only=true|false
|
2339
|
-
|
2534
|
+
```
|
2340
2535
|
|
2341
2536
|
* Display node events (events)
|
2342
2537
|
|
@@ -2347,7 +2542,7 @@ $ ascli aoc admin res node --secret=_secret_ v3 events
|
|
2347
2542
|
* display members of a workspace
|
2348
2543
|
|
2349
2544
|
```
|
2350
|
-
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"
|
2545
|
+
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
2351
2546
|
:.............:.........:..................................:
|
2352
2547
|
: member_type : manager : member.email :
|
2353
2548
|
:.............:.........:..................................:
|
@@ -2368,27 +2563,27 @@ other query parameters:
|
|
2368
2563
|
|
2369
2564
|
* <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
|
2370
2565
|
|
2371
|
-
a-
|
2566
|
+
a- Get id of first workspace
|
2372
2567
|
|
2373
2568
|
```
|
2374
2569
|
WS1='First Workspace'
|
2375
2570
|
WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
2376
2571
|
```
|
2377
2572
|
|
2378
|
-
b-
|
2573
|
+
b- Get id of second workspace
|
2379
2574
|
|
2380
2575
|
```
|
2381
2576
|
WS2='Second Workspace'
|
2382
2577
|
WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
|
2383
2578
|
```
|
2384
2579
|
|
2385
|
-
c-
|
2580
|
+
c- Extract membership information
|
2386
2581
|
|
2387
2582
|
```
|
2388
|
-
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"
|
2583
|
+
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
2389
2584
|
```
|
2390
2585
|
|
2391
|
-
d-
|
2586
|
+
d- Convert to creation data for second workspace:
|
2392
2587
|
|
2393
2588
|
```
|
2394
2589
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
@@ -2400,16 +2595,16 @@ or, using jq:
|
|
2400
2595
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
2401
2596
|
```
|
2402
2597
|
|
2403
|
-
e-
|
2598
|
+
e- Add members to second workspace
|
2404
2599
|
|
2405
2600
|
```
|
2406
2601
|
$ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
2407
2602
|
```
|
2408
2603
|
|
2409
|
-
*
|
2604
|
+
* Get users who did not log since a date
|
2410
2605
|
|
2411
2606
|
```
|
2412
|
-
$ ascli aoc admin res user list --fields=email --query=@json:'{"
|
2607
|
+
$ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2413
2608
|
:...............................:
|
2414
2609
|
: email :
|
2415
2610
|
:...............................:
|
@@ -2418,10 +2613,10 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
|
|
2418
2613
|
:...............................:
|
2419
2614
|
```
|
2420
2615
|
|
2421
|
-
*
|
2616
|
+
* List "Limited" users
|
2422
2617
|
|
2423
2618
|
```
|
2424
|
-
$ ascli aoc admin res user list --fields=email --
|
2619
|
+
$ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
2425
2620
|
```
|
2426
2621
|
|
2427
2622
|
* Perform a multi Gbps transfer between two remote shared folders
|
@@ -2464,6 +2659,7 @@ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='
|
|
2464
2659
|
```
|
2465
2660
|
|
2466
2661
|
* create registration key to register a node
|
2662
|
+
|
2467
2663
|
```
|
2468
2664
|
$ ascli aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
2469
2665
|
jfqslfdjlfdjfhdjklqfhdkl
|
@@ -2483,6 +2679,19 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
|
|
2483
2679
|
+-----+---------+
|
2484
2680
|
```
|
2485
2681
|
|
2682
|
+
* List packages in a given shared inbox
|
2683
|
+
|
2684
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
2685
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2686
|
+
|
2687
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
2688
|
+
|
2689
|
+
```
|
2690
|
+
shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
2691
|
+
|
2692
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
2693
|
+
```
|
2694
|
+
|
2486
2695
|
## Shared folders
|
2487
2696
|
|
2488
2697
|
* list shared folders in node
|
@@ -2515,7 +2724,7 @@ Notes:
|
|
2515
2724
|
|
2516
2725
|
* the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
|
2517
2726
|
* to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". ascli will resolve the list of email addresses to expected user ids.
|
2518
|
-
* a
|
2727
|
+
* a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
|
2519
2728
|
* If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
2520
2729
|
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
|
2521
2730
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
@@ -2524,7 +2733,13 @@ Examples:
|
|
2524
2733
|
|
2525
2734
|
```
|
2526
2735
|
$ ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
|
2736
|
+
```
|
2737
|
+
|
2738
|
+
```
|
2527
2739
|
$ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
|
2740
|
+
```
|
2741
|
+
|
2742
|
+
```
|
2528
2743
|
$ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2529
2744
|
```
|
2530
2745
|
|
@@ -2540,7 +2755,7 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
2540
2755
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
2541
2756
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
2542
2757
|
|
2543
|
-
Typically, one would
|
2758
|
+
Typically, one would execute this command on a regular basis, using the method of your choice:
|
2544
2759
|
|
2545
2760
|
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2546
2761
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
@@ -2548,45 +2763,45 @@ Typically, one would regularly execute this command on a regular basis, using th
|
|
2548
2763
|
|
2549
2764
|
## Download Files
|
2550
2765
|
|
2551
|
-
Download of files is straightforward with a specific syntax for the `
|
2766
|
+
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2552
2767
|
|
2553
2768
|
* if only one source is provided, it is downloaded
|
2554
2769
|
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2555
2770
|
|
2556
2771
|
## Find Files
|
2557
2772
|
|
2558
|
-
The command `
|
2773
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2559
2774
|
|
2560
2775
|
The expression can be of 3 formats:
|
2561
2776
|
|
2562
|
-
* empty (default) : all files, equivalent to: `exec:true`
|
2563
|
-
* not starting with `exec:` : the expression is a regular expression, using ruby
|
2777
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
2778
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2564
2779
|
|
2565
2780
|
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2566
2781
|
|
2567
|
-
* starting with `exec:` : the
|
2782
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2568
2783
|
|
2569
|
-
Examples of expressions: (
|
2784
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
2570
2785
|
|
2571
|
-
*
|
2786
|
+
* Find files more recent than 100 days
|
2572
2787
|
|
2573
2788
|
```
|
2574
2789
|
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
2575
2790
|
```
|
2576
2791
|
|
2577
|
-
*
|
2792
|
+
* Find files older than 1 year on a given node and store in file list
|
2578
2793
|
|
2579
2794
|
```
|
2580
2795
|
$ ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
2581
2796
|
```
|
2582
2797
|
|
2583
|
-
*
|
2798
|
+
* Delete the files, one by one
|
2584
2799
|
|
2585
2800
|
```
|
2586
2801
|
$ cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
2587
2802
|
```
|
2588
2803
|
|
2589
|
-
*
|
2804
|
+
* Delete the files in bulk
|
2590
2805
|
|
2591
2806
|
```
|
2592
2807
|
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
@@ -2721,7 +2936,7 @@ Explanation:
|
|
2721
2936
|
* `|` the standard output of the first command is fed into the second one
|
2722
2937
|
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2723
2938
|
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2724
|
-
* `--transfer=node` use transfer agent type `node` instead of default `direct`
|
2939
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
|
2725
2940
|
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2726
2941
|
|
2727
2942
|
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
@@ -2770,6 +2985,14 @@ This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
|
|
2770
2985
|
|
2771
2986
|
Both password and SSH keys auth are supported.
|
2772
2987
|
|
2988
|
+
If not username is provided, the default transfer user `xfer` is used.
|
2989
|
+
|
2990
|
+
If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
|
2991
|
+
|
2992
|
+
```
|
2993
|
+
$ ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
2994
|
+
```
|
2995
|
+
|
2773
2996
|
Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
|
2774
2997
|
|
2775
2998
|
Examples:
|
@@ -2807,7 +3030,7 @@ This means that you dont have such an ssh agent running:
|
|
2807
3030
|
$ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
2808
3031
|
```
|
2809
3032
|
|
2810
|
-
This can also be set as default using a preset
|
3033
|
+
This can also be set as default using a preset.
|
2811
3034
|
|
2812
3035
|
## Example
|
2813
3036
|
|
@@ -2827,18 +3050,36 @@ This creates a [option preset](#lprt) "aspera_demo_server" and set it as default
|
|
2827
3050
|
|
2828
3051
|
This plugin gives access to capabilities provided by HSTS node API.
|
2829
3052
|
|
2830
|
-
##
|
3053
|
+
## File Operations
|
2831
3054
|
|
2832
3055
|
It is possible to:
|
2833
3056
|
* browse
|
2834
3057
|
* transfer (upload / download)
|
2835
3058
|
* ...
|
2836
3059
|
|
3060
|
+
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
3061
|
+
|
3062
|
+
* `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
|
3063
|
+
* `basic` : transfer spec is created like this:
|
3064
|
+
|
3065
|
+
```
|
3066
|
+
{
|
3067
|
+
"remote_host": address of node url,
|
3068
|
+
"remote_user": "xfer",
|
3069
|
+
"ssh_port": 33001,
|
3070
|
+
"token": "Basic <base 64 encoded user/pass>",
|
3071
|
+
"direction": send/recv
|
3072
|
+
}
|
3073
|
+
```
|
3074
|
+
|
3075
|
+
* `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
3076
|
+
|
2837
3077
|
## Central
|
2838
3078
|
|
2839
3079
|
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
|
2840
3080
|
|
2841
3081
|
Filtering can be applied:
|
3082
|
+
|
2842
3083
|
```
|
2843
3084
|
$ ascli node central file list
|
2844
3085
|
```
|
@@ -2864,7 +3105,6 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
2864
3105
|
* Start watchd and watchfolderd services running as a system user having access to files
|
2865
3106
|
* configure a watchfolder to define automated transfers
|
2866
3107
|
|
2867
|
-
|
2868
3108
|
```
|
2869
3109
|
$ ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
2870
3110
|
$ ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
@@ -3182,7 +3422,7 @@ $ ascli cos node info
|
|
3182
3422
|
$ ascli cos node upload 'faux:///sample1G?1g'
|
3183
3423
|
```
|
3184
3424
|
|
3185
|
-
Note: we generate a dummy file `sample1G`
|
3425
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
3186
3426
|
|
3187
3427
|
# Plugin: IBM Aspera Sync
|
3188
3428
|
|
@@ -3279,8 +3519,10 @@ yum install -y ImageMagick optipng
|
|
3279
3519
|
|
3280
3520
|
### Video: FFmpeg
|
3281
3521
|
|
3522
|
+
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
3523
|
+
|
3282
3524
|
```
|
3283
|
-
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt &&
|
3525
|
+
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
3284
3526
|
```
|
3285
3527
|
|
3286
3528
|
### Office: Unoconv and Libreoffice
|
@@ -3578,7 +3820,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
3578
3820
|
* `EX_loglevel` to change log level of the tool
|
3579
3821
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
3580
3822
|
|
3581
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the
|
3823
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
3582
3824
|
|
3583
3825
|
## Comparison of interfaces
|
3584
3826
|
|
@@ -3679,7 +3921,7 @@ Note that:
|
|
3679
3921
|
|
3680
3922
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
3681
3923
|
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3682
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the
|
3924
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
|
3683
3925
|
|
3684
3926
|
### server side and configuration
|
3685
3927
|
|
@@ -3698,7 +3940,7 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
|
|
3698
3940
|
|
3699
3941
|
```
|
3700
3942
|
|
3701
|
-
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3943
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3702
3944
|
|
3703
3945
|
# Aspera Health check and Nagios
|
3704
3946
|
|
@@ -3775,9 +4017,19 @@ So, it evolved into `ascli`:
|
|
3775
4017
|
|
3776
4018
|
# Changes (Release notes)
|
3777
4019
|
|
4020
|
+
* 4.4.0
|
4021
|
+
|
4022
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
4023
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
4024
|
+
* new: `preset` option can specify name or hash value
|
4025
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
4026
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
4027
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
4028
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
4029
|
+
|
3778
4030
|
* 4.3.0
|
3779
4031
|
|
3780
|
-
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on `direct` transfer agent.
|
4032
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
|
3781
4033
|
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
3782
4034
|
|
3783
4035
|
* 4.2.2
|
@@ -3802,20 +4054,19 @@ So, it evolved into `ascli`:
|
|
3802
4054
|
* new: command `aoc remind` to receive organization membership by email
|
3803
4055
|
* new: in `preview` option `value` to filter out on file name
|
3804
4056
|
* new: `initdemo` to initialize for demo server
|
3805
|
-
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
4057
|
+
* new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3806
4058
|
* fix: on Windows `conf ascp use` expects ascp.exe
|
3807
4059
|
* fix: (break) multi_session_threshold is Integer, not String
|
3808
4060
|
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3809
4061
|
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3810
4062
|
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3811
|
-
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
4063
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
|
3812
4064
|
|
3813
4065
|
* 4.1.0
|
3814
4066
|
|
3815
|
-
|
3816
|
-
|
3817
|
-
|
3818
|
-
* new: renamed command `nagios_check` to `health`
|
4067
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
4068
|
+
* new: update documentation with regard to offline and docker installation
|
4069
|
+
* new: renamed command `nagios_check` to `health`
|
3819
4070
|
* new: agent `http_gw` now supports upload
|
3820
4071
|
* new: added option `sdk_url` to install SDK from local file for offline install
|
3821
4072
|
* new: check new gem version periodically
|
@@ -3866,8 +4117,8 @@ So, it evolved into `ascli`:
|
|
3866
4117
|
* allow bulk delete in `aspera files` with option `bulk=yes`
|
3867
4118
|
* fix getting connect versions
|
3868
4119
|
* added section for Aix
|
3869
|
-
* support all ciphers for `
|
3870
|
-
* added transfer spec param `apply_local_docroot` for `
|
4120
|
+
* support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
|
4121
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#direct)
|
3871
4122
|
|
3872
4123
|
* 0.11.4
|
3873
4124
|
|
@@ -3922,7 +4173,7 @@ So, it evolved into `ascli`:
|
|
3922
4173
|
* 0.10.12
|
3923
4174
|
|
3924
4175
|
* added support for AoC node registration keys
|
3925
|
-
* replaced option : `local_resume` with `transfer_info` for agent `direct`
|
4176
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
|
3926
4177
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
3927
4178
|
* `@incps` : new extended value modifier
|
3928
4179
|
* ATS: no more provides access keys secrets: now user must provide it
|
@@ -3972,9 +4223,9 @@ So, it evolved into `ascli`:
|
|
3972
4223
|
|
3973
4224
|
* 0.10.2
|
3974
4225
|
|
3975
|
-
|
3976
|
-
|
3977
|
-
|
4226
|
+
* updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
|
4227
|
+
* added doc for "cargo" like actions
|
4228
|
+
* added doc for multi-session
|
3978
4229
|
|
3979
4230
|
* 0.10.1
|
3980
4231
|
|
@@ -4002,22 +4253,22 @@ So, it evolved into `ascli`:
|
|
4002
4253
|
|
4003
4254
|
* 0.9.33
|
4004
4255
|
|
4005
|
-
|
4006
|
-
|
4007
|
-
|
4008
|
-
|
4256
|
+
* new command to display basic token of node
|
4257
|
+
* new command to display bearer token of node in AoC
|
4258
|
+
* the --fields= option, support +_fieldname_ to add a field to default fields
|
4259
|
+
* many small changes
|
4009
4260
|
|
4010
4261
|
* 0.9.32
|
4011
4262
|
|
4012
|
-
|
4013
|
-
|
4014
|
-
|
4263
|
+
* all Faspex public links are now supported
|
4264
|
+
* removed faspex operation recv_publink
|
4265
|
+
* replaced with option `link` (consistent with AoC)
|
4015
4266
|
|
4016
4267
|
* 0.9.31
|
4017
4268
|
|
4018
|
-
|
4019
|
-
|
4020
|
-
|
4269
|
+
* added more support for public link: receive and send package, to user or dropbox and files view.
|
4270
|
+
* delete expired file lists
|
4271
|
+
* changed text table gem from text-table to terminal-table because it supports multiline values
|
4021
4272
|
|
4022
4273
|
* 0.9.27
|
4023
4274
|
|
@@ -4037,149 +4288,149 @@ So, it evolved into `ascli`:
|
|
4037
4288
|
|
4038
4289
|
* 0.9.24
|
4039
4290
|
|
4040
|
-
|
4041
|
-
|
4291
|
+
* fix bug where AoC node to node transfer did not work
|
4292
|
+
* fix bug on error if ED25519 private key is defined in .ssh
|
4042
4293
|
|
4043
4294
|
* 0.9.23
|
4044
4295
|
|
4045
|
-
|
4046
|
-
|
4296
|
+
* defined REST error handlers, more error conditions detected
|
4297
|
+
* commands to select specific ascp location
|
4047
4298
|
|
4048
4299
|
* 0.9.21
|
4049
4300
|
|
4050
|
-
|
4051
|
-
|
4301
|
+
* supports simplified wizard using global client
|
4302
|
+
* only ascp binary is required, other SDK (keys) files are now generated
|
4052
4303
|
|
4053
4304
|
* 0.9.20
|
4054
4305
|
|
4055
|
-
|
4056
|
-
|
4057
|
-
|
4306
|
+
* improved wizard (prepare for AoC global client id)
|
4307
|
+
* preview generator: addedoption : --skip-format=<png,mp4>
|
4308
|
+
* removed outdated pictures from this doc
|
4058
4309
|
|
4059
4310
|
* 0.9.19
|
4060
4311
|
|
4061
|
-
|
4312
|
+
* added command aspera bearer --scope=xx
|
4062
4313
|
|
4063
4314
|
* 0.9.18
|
4064
4315
|
|
4065
|
-
|
4316
|
+
* enhanced aspera admin events to support query
|
4066
4317
|
|
4067
4318
|
* 0.9.16
|
4068
4319
|
|
4069
|
-
|
4070
|
-
|
4320
|
+
* AoC transfers are now reported in activity app
|
4321
|
+
* new interface for Rest class authentication (keep backward compatibility)
|
4071
4322
|
|
4072
4323
|
* 0.9.15
|
4073
4324
|
|
4074
|
-
|
4075
|
-
|
4325
|
+
* new feature: "find" command in aspera files
|
4326
|
+
* sample code for transfer API
|
4076
4327
|
|
4077
4328
|
* 0.9.12
|
4078
4329
|
|
4079
|
-
|
4080
|
-
|
4330
|
+
* add nagios commands
|
4331
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
4081
4332
|
|
4082
4333
|
* 0.9.11
|
4083
4334
|
|
4084
|
-
|
4085
|
-
|
4335
|
+
* Breaking change: @stdin is now @stdin:
|
4336
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
4086
4337
|
|
4087
4338
|
|
4088
4339
|
* 0.9.10
|
4089
4340
|
|
4090
|
-
|
4091
|
-
|
4092
|
-
|
4093
|
-
|
4341
|
+
* Breaking change: parameter transfer-node becomes more generic: transfer-info
|
4342
|
+
* Display SaaS storage usage with command: aspera admin res node --id=nn info
|
4343
|
+
* cleaner way of specifying source file list for transfers
|
4344
|
+
* Breaking change: replaced download_mode option with http_download action
|
4094
4345
|
|
4095
4346
|
* 0.9.9
|
4096
4347
|
|
4097
|
-
|
4098
|
-
|
4348
|
+
* Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
|
4349
|
+
* Now supports "cargo" for Aspera on Cloud (automatic package download)
|
4099
4350
|
|
4100
4351
|
* 0.9.8
|
4101
4352
|
|
4102
|
-
|
4103
|
-
|
4353
|
+
* Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
|
4354
|
+
* AoC: share to share transfer with command "transfer"
|
4104
4355
|
|
4105
4356
|
* 0.9.7
|
4106
4357
|
|
4107
|
-
|
4108
|
-
|
4109
|
-
|
4110
|
-
|
4111
|
-
|
4358
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
|
4359
|
+
* preview persistency goes to unique file by default
|
4360
|
+
* catch mxf extension in preview as video
|
4361
|
+
* Faspex: possibility to download all paclages by specifying id=ALL
|
4362
|
+
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
4112
4363
|
|
4113
4364
|
* 0.9.6
|
4114
4365
|
|
4115
|
-
|
4116
|
-
|
4366
|
+
* Breaking change: `@param:`is now `@preset:` and is generic
|
4367
|
+
* AoC: added command to display current workspace information
|
4117
4368
|
|
4118
4369
|
* 0.9.5
|
4119
4370
|
|
4120
|
-
|
4121
|
-
|
4371
|
+
* new parameter: new_user_option used to choose between public_link and invite of external users.
|
4372
|
+
* fixed bug in wizard, and wizard uses now product detection
|
4122
4373
|
|
4123
4374
|
* 0.9.4
|
4124
4375
|
|
4125
|
-
|
4126
|
-
|
4127
|
-
|
4376
|
+
* Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
|
4377
|
+
* AoC Package send supports external users
|
4378
|
+
* new command to export AoC config to Aspera CLI config
|
4128
4379
|
|
4129
4380
|
* 0.9.3
|
4130
4381
|
|
4131
|
-
|
4132
|
-
|
4133
|
-
|
4134
|
-
|
4135
|
-
|
4136
|
-
|
4382
|
+
* REST error message show host and code
|
4383
|
+
* option for quiet display
|
4384
|
+
* modified transfer interface and allow token re-generation on error
|
4385
|
+
* async add admin command
|
4386
|
+
* async add db parameters
|
4387
|
+
* Breaking change: new option "sources" to specify files to transfer
|
4137
4388
|
|
4138
4389
|
* 0.9.2
|
4139
4390
|
|
4140
|
-
|
4391
|
+
* Breaking change: changed AoC package creation to match API, see AoC section
|
4141
4392
|
|
4142
4393
|
* 0.9.1
|
4143
4394
|
|
4144
|
-
|
4395
|
+
* Breaking change: changed faspex package creation to match API, see Faspex section
|
4145
4396
|
|
4146
4397
|
* 0.9
|
4147
4398
|
|
4148
|
-
|
4149
|
-
|
4399
|
+
* Renamed the CLI from aslmcli to `ascli`
|
4400
|
+
* Automatic rename and conversion of former config folder from aslmcli to `ascli`
|
4150
4401
|
|
4151
4402
|
* 0.7.6
|
4152
4403
|
|
4153
|
-
|
4404
|
+
* add "sync" plugin
|
4154
4405
|
|
4155
4406
|
* 0.7
|
4156
4407
|
|
4157
|
-
|
4158
|
-
|
4159
|
-
|
4160
|
-
|
4161
|
-
|
4408
|
+
* Breaking change: AoC package recv take option if for package instead of argument.
|
4409
|
+
* Breaking change: Rest class and Oauth class changed init parameters
|
4410
|
+
* AoC: receive package from public link
|
4411
|
+
* select by col value on output
|
4412
|
+
* added rename (AoC, node)
|
4162
4413
|
|
4163
4414
|
* 0.6.19
|
4164
4415
|
|
4165
4416
|
Breaking change:
|
4166
4417
|
|
4167
|
-
|
4168
|
-
|
4169
|
-
|
4170
|
-
|
4171
|
-
|
4172
|
-
|
4173
|
-
|
4174
|
-
|
4418
|
+
* ats server list provisioned → ats cluster list
|
4419
|
+
* ats server list clouds → ats cluster clouds
|
4420
|
+
* ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
|
4421
|
+
* ats server id xxx → ats cluster show --id=xxx
|
4422
|
+
* ats subscriptions → ats credential subscriptions
|
4423
|
+
* ats api_key repository list → ats credential cache list
|
4424
|
+
* ats api_key list → ats credential list
|
4425
|
+
* ats access_key id xxx → ats access_key --id=xxx
|
4175
4426
|
|
4176
4427
|
* 0.6.18
|
4177
4428
|
|
4178
|
-
|
4429
|
+
* some commands take now --id option instead of id command.
|
4179
4430
|
|
4180
4431
|
* 0.6.15
|
4181
4432
|
|
4182
|
-
|
4433
|
+
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
|
4183
4434
|
|
4184
4435
|
# BUGS, FEATURES, CONTRIBUTION
|
4185
4436
|
|
@@ -4232,8 +4483,8 @@ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "C
|
|
4232
4483
|
|
4233
4484
|
* remove rest and oauth classes and use ruby standard gems:
|
4234
4485
|
|
4235
|
-
|
4236
|
-
|
4486
|
+
* oauth
|
4487
|
+
* https://github.com/rest-client/rest-client
|
4237
4488
|
|
4238
4489
|
* use Thor or any standard Ruby CLI manager
|
4239
4490
|
|