aspera-cli 4.2.0 → 4.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +749 -353
- data/docs/Makefile +4 -4
- data/docs/README.erb.md +743 -283
- data/docs/doc_tools.rb +58 -0
- data/docs/test_env.conf +9 -1
- data/examples/aoc.rb +14 -3
- data/examples/faspex4.rb +89 -0
- data/lib/aspera/aoc.rb +24 -22
- data/lib/aspera/cli/main.rb +48 -20
- data/lib/aspera/cli/plugin.rb +13 -6
- data/lib/aspera/cli/plugins/aoc.rb +117 -78
- data/lib/aspera/cli/plugins/config.rb +127 -80
- data/lib/aspera/cli/plugins/faspex.rb +112 -63
- data/lib/aspera/cli/plugins/faspex5.rb +29 -25
- data/lib/aspera/cli/plugins/node.rb +54 -25
- data/lib/aspera/cli/plugins/preview.rb +94 -68
- data/lib/aspera/cli/plugins/server.rb +16 -5
- data/lib/aspera/cli/transfer_agent.rb +92 -72
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +48 -31
- data/lib/aspera/cos_node.rb +4 -3
- data/lib/aspera/fasp/http_gw.rb +47 -26
- data/lib/aspera/fasp/local.rb +31 -24
- data/lib/aspera/fasp/manager.rb +3 -0
- data/lib/aspera/fasp/node.rb +23 -1
- data/lib/aspera/fasp/parameters.rb +72 -89
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/faspex_gw.rb +10 -9
- data/lib/aspera/id_generator.rb +22 -0
- data/lib/aspera/node.rb +11 -3
- data/lib/aspera/oauth.rb +131 -135
- data/lib/aspera/persistency_action_once.rb +11 -7
- data/lib/aspera/persistency_folder.rb +6 -26
- data/lib/aspera/rest.rb +1 -1
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/timer_limiter.rb +22 -0
- data/lib/aspera/web_auth.rb +105 -0
- metadata +22 -4
- data/docs/transfer_spec.html +0 -99
- data/lib/aspera/fasp/aoc.rb +0 -24
data/README.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
|
2
|
-
|
|
2
|
+
<font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
|
|
3
3
|
|
|
4
|
-
Version : 4.
|
|
4
|
+
Version : 4.4.0
|
|
5
5
|
|
|
6
6
|
_Laurent/2016-2021_
|
|
7
7
|
|
|
8
|
-
This gem provides `ascli
|
|
8
|
+
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
|
9
9
|
|
|
10
10
|
`ascli` is a also great tool to learn Aspera APIs.
|
|
11
11
|
|
|
@@ -13,7 +13,7 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
|
13
13
|
|
|
14
14
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
|
15
15
|
|
|
16
|
-
Ruby version
|
|
16
|
+
Required Ruby version: > 2.4
|
|
17
17
|
|
|
18
18
|
# <a name="when_to_use"></a>When to use and when not to use
|
|
19
19
|
|
|
@@ -29,7 +29,8 @@ So it is designed for:
|
|
|
29
29
|
|
|
30
30
|
`ascli` can be seen as a command line tool integrating:
|
|
31
31
|
|
|
32
|
-
* a configuration file (config.yaml)
|
|
32
|
+
* a configuration file (config.yaml)
|
|
33
|
+
* advanced command line options
|
|
33
34
|
* cURL (for REST calls)
|
|
34
35
|
* Aspera transfer (ascp)
|
|
35
36
|
|
|
@@ -56,7 +57,7 @@ Command line parameters in examples beginning with `my_`, like `my_param_value`
|
|
|
56
57
|
|
|
57
58
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
|
58
59
|
|
|
59
|
-
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
|
60
|
+
On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
|
60
61
|
|
|
61
62
|
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
|
62
63
|
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
|
@@ -81,7 +82,7 @@ Once the gem is installed, `ascli` shall be accessible:
|
|
|
81
82
|
|
|
82
83
|
```
|
|
83
84
|
$ ascli --version
|
|
84
|
-
4.
|
|
85
|
+
4.4.0
|
|
85
86
|
```
|
|
86
87
|
|
|
87
88
|
## First use
|
|
@@ -108,7 +109,7 @@ $ ascli server browse /
|
|
|
108
109
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
|
109
110
|
|
|
110
111
|
* create a [option preset](#lprt)
|
|
111
|
-
* define it as default for
|
|
112
|
+
* define it as default for `server` plugin
|
|
112
113
|
* list files in a folder
|
|
113
114
|
* download a file
|
|
114
115
|
|
|
@@ -151,7 +152,7 @@ It is possible to install *either* directly on the host operating system (Linux,
|
|
|
151
152
|
|
|
152
153
|
The direct installation is recommended and consists in installing:
|
|
153
154
|
|
|
154
|
-
* [Ruby](#ruby) version
|
|
155
|
+
* [Ruby](#ruby) version > 2.4
|
|
155
156
|
* [aspera-cli](#the_gem)
|
|
156
157
|
* [Aspera SDK (ascp)](#fasp_prot)
|
|
157
158
|
|
|
@@ -165,6 +166,8 @@ Use this method only if you know what you do, else use the standard recommended
|
|
|
165
166
|
|
|
166
167
|
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
|
167
168
|
|
|
169
|
+
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
|
170
|
+
|
|
168
171
|
Ensure that you have Docker installed.
|
|
169
172
|
|
|
170
173
|
```
|
|
@@ -199,7 +202,7 @@ Use this method to install on the native host.
|
|
|
199
202
|
|
|
200
203
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
|
201
204
|
|
|
202
|
-
Ruby
|
|
205
|
+
Required Ruby version: > 2.4. Ruby version 3 is also supported.
|
|
203
206
|
|
|
204
207
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
|
205
208
|
|
|
@@ -207,7 +210,7 @@ Refer to the following sections for a proposed method for specific operating sys
|
|
|
207
210
|
|
|
208
211
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
|
|
209
212
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
|
210
|
-
If you have a simpler better way to install Ruby version
|
|
213
|
+
If you have a simpler better way to install Ruby version > 2.4 : use it !
|
|
211
214
|
|
|
212
215
|
### Generic: RVM: single user installation (not root)
|
|
213
216
|
|
|
@@ -233,7 +236,7 @@ If you keep the same terminal (ont needed if re-login):
|
|
|
233
236
|
$ source ~/.rvm/scripts/rvm
|
|
234
237
|
```
|
|
235
238
|
|
|
236
|
-
It is advised to get one of the pre-compiled ruby version, you can list with:
|
|
239
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
|
237
240
|
|
|
238
241
|
```
|
|
239
242
|
$ rvm list --remote
|
|
@@ -429,8 +432,8 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
|
429
432
|
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
|
430
433
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
|
431
434
|
|
|
432
|
-
Several methods are provided
|
|
433
|
-
other methods are available. Refer to section: [Transfer Agents](#agents)
|
|
435
|
+
Several methods are provided to start a transfer.
|
|
436
|
+
Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
|
434
437
|
|
|
435
438
|
## <a name="offline_install"></a>Offline Installation (without internet)
|
|
436
439
|
|
|
@@ -476,7 +479,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
|
476
479
|
* Supports most Aspera server products (on-premise and SaaS)
|
|
477
480
|
* Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
|
478
481
|
* Supports Commands, Option values and Parameters shortcuts
|
|
479
|
-
* FASP [Transfer Agents](#agents) can be:
|
|
482
|
+
* FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
|
480
483
|
* Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
|
|
481
484
|
* Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
|
482
485
|
* Supports FaspStream creation (using Node API)
|
|
@@ -600,12 +603,12 @@ The style of output can be set using the `format` parameter, supporting:
|
|
|
600
603
|
* `yaml` : YAML
|
|
601
604
|
* `csv` : Comma Separated Values
|
|
602
605
|
|
|
603
|
-
###
|
|
606
|
+
### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
|
604
607
|
|
|
605
608
|
Table output can be filtered using the `select` parameter. Example:
|
|
606
609
|
|
|
607
610
|
```
|
|
608
|
-
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"
|
|
611
|
+
$ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
|
609
612
|
:...............................:..................................:...........:
|
|
610
613
|
: name : email : ats_admin :
|
|
611
614
|
:...............................:..................................:...........:
|
|
@@ -752,19 +755,15 @@ C:\Users\Kenji\.aspera\ascli
|
|
|
752
755
|
On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
|
|
753
756
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
|
754
757
|
|
|
755
|
-
Although the file is a standard YAML file, `ascli` provides commands to read and modify it
|
|
756
|
-
using the `config` command.
|
|
758
|
+
Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
|
|
757
759
|
|
|
758
|
-
All options for `ascli`
|
|
760
|
+
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
|
|
759
761
|
|
|
760
|
-
A configuration file provides a way to define default values, especially
|
|
761
|
-
for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
|
762
|
+
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
|
762
763
|
|
|
763
|
-
The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
|
|
764
|
-
(this can be overriden with option `--config-file=path` or equivalent env var).
|
|
764
|
+
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
|
|
765
765
|
|
|
766
|
-
|
|
767
|
-
called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
|
766
|
+
The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
|
768
767
|
|
|
769
768
|
### <a name="lprt"></a>Option preset
|
|
770
769
|
|
|
@@ -864,7 +863,7 @@ demo_server:
|
|
|
864
863
|
We can see here:
|
|
865
864
|
|
|
866
865
|
* The configuration was created with CLI version 0.3.7
|
|
867
|
-
* the default [option preset](#lprt) to load for plugin
|
|
866
|
+
* the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
|
868
867
|
* the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
|
869
868
|
* the default [option preset](#lprt) to load in any case is : `cli_default`
|
|
870
869
|
|
|
@@ -895,26 +894,26 @@ my_aoc_org:
|
|
|
895
894
|
|
|
896
895
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
|
897
896
|
|
|
897
|
+
### Options evaluation order
|
|
898
|
+
|
|
899
|
+
Some options are global, some options are available only for some plugins. (the plugin is the first level command).
|
|
900
|
+
|
|
898
901
|
Options are loaded using this algorithm:
|
|
899
902
|
|
|
900
|
-
*
|
|
901
|
-
|
|
902
|
-
|
|
903
|
-
*
|
|
904
|
-
*
|
|
903
|
+
* If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
|
904
|
+
* else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
|
905
|
+
* If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
|
906
|
+
* Environment variables are evaluated
|
|
907
|
+
* Command line options are evaluated
|
|
905
908
|
|
|
906
909
|
Parameters are evaluated in the order of command line.
|
|
907
910
|
|
|
908
|
-
To avoid loading the default [option preset](#lprt) for a plugin,
|
|
911
|
+
To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
|
|
909
912
|
|
|
910
913
|
On command line, words in parameter names are separated by a dash, in configuration file, separator
|
|
911
914
|
is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
|
|
912
915
|
|
|
913
|
-
|
|
914
|
-
convert olver versions, remove the leading ":" in front of keys.
|
|
915
|
-
|
|
916
|
-
The main plugin name is *config*, so it is possible to define a default [option preset](#lprt) for
|
|
917
|
-
the main plugin with:
|
|
916
|
+
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
|
918
917
|
|
|
919
918
|
```
|
|
920
919
|
$ ascli config id cli_default set interactive no
|
|
@@ -927,6 +926,18 @@ A [option preset](#lprt) value can be removed with `unset`:
|
|
|
927
926
|
$ ascli config id cli_default unset interactive
|
|
928
927
|
```
|
|
929
928
|
|
|
929
|
+
Example: Define options using command line:
|
|
930
|
+
|
|
931
|
+
```
|
|
932
|
+
$ ascli -N --url=x --password=y --username=y node --show-config
|
|
933
|
+
```
|
|
934
|
+
|
|
935
|
+
Example: Define options using a hash:
|
|
936
|
+
|
|
937
|
+
```
|
|
938
|
+
$ ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
|
939
|
+
```
|
|
940
|
+
|
|
930
941
|
### Examples
|
|
931
942
|
|
|
932
943
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
|
@@ -1055,7 +1066,7 @@ $ ascli config ascp info
|
|
|
1055
1066
|
...
|
|
1056
1067
|
```
|
|
1057
1068
|
|
|
1058
|
-
### Selection of
|
|
1069
|
+
### Selection of `ascp` location for [`direct`](#direct) agent
|
|
1059
1070
|
|
|
1060
1071
|
By default, `ascli` uses any found local product with ascp, including SDK.
|
|
1061
1072
|
|
|
@@ -1099,7 +1110,7 @@ $ ascli config ascp products list
|
|
|
1099
1110
|
:.........................................:................................................:
|
|
1100
1111
|
```
|
|
1101
1112
|
|
|
1102
|
-
### Selection of local client
|
|
1113
|
+
### Selection of local client for `ascp` for [`direct`](#direct) agent
|
|
1103
1114
|
|
|
1104
1115
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
|
1105
1116
|
|
|
@@ -1154,7 +1165,7 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
|
1154
1165
|
|
|
1155
1166
|
There are currently 3 agents:
|
|
1156
1167
|
|
|
1157
|
-
* `direct` : a local execution of `ascp`
|
|
1168
|
+
* [`direct`](#direct) : a local execution of `ascp`
|
|
1158
1169
|
* `connect` : use of a local Connect Client
|
|
1159
1170
|
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
|
1160
1171
|
* `httpgw` : use of an Aspera HTTP Gateway
|
|
@@ -1166,7 +1177,7 @@ will effectively push files to the related server from the agent node.
|
|
|
1166
1177
|
`ascli` standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
|
1167
1178
|
|
|
1168
1179
|
|
|
1169
|
-
### <a name="direct"></a>Direct (local ascp
|
|
1180
|
+
### <a name="direct"></a>Direct (local ascp execution)
|
|
1170
1181
|
|
|
1171
1182
|
By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
|
|
1172
1183
|
`ascli` will detect locally installed Aspera products.
|
|
@@ -1185,11 +1196,19 @@ The `transfer-info` accepts the following optional parameters:
|
|
|
1185
1196
|
<tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
|
|
1186
1197
|
<tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
|
|
1187
1198
|
<tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
|
1199
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>true</td><td>Multi Session</td><td>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)</td></tr>
|
|
1188
1200
|
<tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
|
|
1189
1201
|
</table>
|
|
1190
1202
|
|
|
1191
1203
|
Resume parameters:
|
|
1192
1204
|
|
|
1205
|
+
In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
|
1206
|
+
Sleep between iteration is:
|
|
1207
|
+
|
|
1208
|
+
```
|
|
1209
|
+
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
|
1210
|
+
```
|
|
1211
|
+
|
|
1193
1212
|
<table>
|
|
1194
1213
|
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
|
1195
1214
|
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
|
@@ -1202,7 +1221,7 @@ Examples:
|
|
|
1202
1221
|
|
|
1203
1222
|
```
|
|
1204
1223
|
$ ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
|
1205
|
-
$ ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
|
|
1224
|
+
$ ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
|
1206
1225
|
```
|
|
1207
1226
|
|
|
1208
1227
|
### IBM Aspera Connect Client GUI
|
|
@@ -1256,7 +1275,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
|
1256
1275
|
|
|
1257
1276
|
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
|
1258
1277
|
|
|
1259
|
-
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
|
1278
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
|
1260
1279
|
|
|
1261
1280
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
1262
1281
|
|
|
@@ -1267,97 +1286,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
|
|
|
1267
1286
|
|
|
1268
1287
|
## <a name="transferparams"></a>Transfer Parameters
|
|
1269
1288
|
|
|
1270
|
-
All standard _transfer-spec_ parameters can be
|
|
1271
|
-
|
|
1272
|
-
also be saved/overridden in the config file.
|
|
1289
|
+
All standard _transfer-spec_ parameters can be speficied.
|
|
1290
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
|
1273
1291
|
|
|
1292
|
+
References:
|
|
1293
|
+
|
|
1294
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
|
1295
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
|
1296
|
+
|
|
1297
|
+
Parameters can be displayed with commands:
|
|
1298
|
+
|
|
1299
|
+
```
|
|
1300
|
+
$ ascli config ascp spec
|
|
1301
|
+
$ ascli config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
|
|
1302
|
+
```
|
|
1274
1303
|
|
|
1275
|
-
<p>
|
|
1276
1304
|
Columns:
|
|
1277
|
-
<ul>
|
|
1278
|
-
<li>F=Fasp Manager(local FASP execution)</li>
|
|
1279
|
-
<li>N=remote node(node API)</li>
|
|
1280
|
-
<li>C=Connect Client(web plugin)</li>
|
|
1281
|
-
</ul>
|
|
1282
|
-
</p>
|
|
1283
|
-
<p>
|
|
1284
|
-
Req/Def : Required or default value (- means emty)
|
|
1285
|
-
</p>
|
|
1286
|
-
<p>
|
|
1287
|
-
Fields with EX_ prefix are specific extensions to local mode.
|
|
1288
|
-
</p>
|
|
1289
|
-
<p>
|
|
1290
|
-
arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)
|
|
1291
|
-
</p>
|
|
1292
|
-
<p>
|
|
1293
|
-
UNDER CONSTRUCTION<br/>
|
|
1294
|
-
<a href="https://developer.ibm.com/apis/catalog/?search=aspera">Aspera API Documentation</a>→Node API→/opt/transfers<br/>
|
|
1295
|
-
</p>
|
|
1296
1305
|
|
|
1297
|
-
|
|
1298
|
-
|
|
1299
|
-
|
|
1300
|
-
|
|
1301
|
-
|
|
1302
|
-
|
|
1303
|
-
<tr><td>title</td><td>-</td><td>string</td><td class="no">N</td><td class="yes">Y</td><td class="yes">Y</td><td>-</td><td>Title of the transfer</td></tr>
|
|
1304
|
-
<tr><td>tags</td><td>-</td><td>hash</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--tags<br>--tags64</td><td>Metadata for transfer</td></tr>
|
|
1305
|
-
<tr><td>token</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>TOKEN<br/>-W</td><td>Authorization token: Bearer, Basic or ATM</td></tr>
|
|
1306
|
-
<tr><td>cookie</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>COOKIE</td><td>Metadata for transfer (older,string)</td></tr>
|
|
1307
|
-
<tr><td>remote_access_key</td><td>TODO</td><td>string</td><td></td><td></td><td></td><td>?</td><td>Node only?</td></tr>
|
|
1308
|
-
<tr><td>source_root</td><td>-</td><td>string</td><td></td><td></td><td></td><td>--source-prefix<br/>--source-prefix64</td><td>Source root directory.(TODO: verify option)</td></tr>
|
|
1309
|
-
<tr><td>fasp_port</td><td>33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-O</td><td>Specifies fasp (UDP) port.</td></tr>
|
|
1310
|
-
<tr><td>ssh_port</td><td>22 or 33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-P</td><td>Specifies ssh (TCP) port.</td></tr>
|
|
1311
|
-
<tr><td>rate_policy</td><td>server config</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--policy</td><td>Valid literals include "low","fair","high" and "fixed".</td></tr>
|
|
1312
|
-
<tr><td>symlink_policy</td><td>follow</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--symbolic-links</td><td>copy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).</td></tr>
|
|
1313
|
-
<tr><td>target_rate_kbps</td><td>-</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-l</td><td>Specifies desired speed for the transfer.</td></tr>
|
|
1314
|
-
<tr><td>min_rate_kbps</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-m</td><td>Set the minimum transfer rate in kilobits per second.</td></tr>
|
|
1315
|
-
<tr><td>cipher</td><td>none</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-c</td><td>in transit encryption type.<br/>none, aes-128, aes-256</td></tr>
|
|
1316
|
-
<tr><td>content_protection</td><td>encrypt<br/>decrypt</td><td>string</td><td></td><td></td><td></td><td>--file-crypt=</td><td>encryption at rest</td></tr>
|
|
1317
|
-
<tr><td>content_protection_password</td><td>-</td><td>string</td><td></td><td></td><td></td><td>PASS</td><td>Specifies a string password.</td></tr>
|
|
1318
|
-
<tr><td>overwrite</td><td>diff</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--overwrite</td><td>Overwrite destination files with the source files of the same name.<br/>never, always, diff, older, or diff+older</td></tr>
|
|
1319
|
-
<tr><td>retry_duration</td><td></td><td>string</td><td></td><td></td><td></td><td>TODO</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr>
|
|
1320
|
-
<tr><td>http_fallback</td><td></td><td>bool (node), integer</td><td></td><td></td><td></td><td>-y<br/>TODO</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.</td></tr>
|
|
1321
|
-
<tr><td>create_dir</td><td></td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-d</td><td>Specifies whether to create new directories.</td></tr>
|
|
1322
|
-
<tr><td>precalculate_job_size</td><td>srv. def.</td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--precalculate-job-size</td><td>Specifies whether to precalculate the job size.</td></tr>
|
|
1323
|
-
<tr><td>delete_source</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>?</td></tr>
|
|
1324
|
-
<tr><td>remove_after_transfer</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove file after transfer.</td></tr>
|
|
1325
|
-
<tr><td>remove_empty_directories</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove empty directories.</td></tr>
|
|
1326
|
-
<tr><td>multi_session</td><td>1</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-C</td><td>Specifies how many parts the transfer is in.</td></tr>
|
|
1327
|
-
<tr><td>multi_session_threshold</td><td>null</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-</td><td>in bytes</td></tr>
|
|
1328
|
-
<tr><td>exclude_newer_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-newer-than</td><td>-</td></tr>
|
|
1329
|
-
<tr><td>exclude_older_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-older-than</td><td>-</td></tr>
|
|
1330
|
-
<tr><td>preserve_acls</td><td></td><td>string</td><td class="yes">Y</td><td></td><td></td><td>--preserve-acls</td><td>-</td></tr>
|
|
1331
|
-
<tr><td>dgram_size</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-Z</td><td>in bytes</td></tr>
|
|
1332
|
-
<tr><td>compression</td><td></td><td>integer</td><td></td><td></td><td></td><td></td><td>ascp4 only, 0 / 1?</td></tr>
|
|
1333
|
-
<tr><td>read_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
|
1334
|
-
<tr><td>write_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
|
1335
|
-
<tr><td>use_ascp4</td><td>false</td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>-</td><td>specify version of protocol</td></tr>
|
|
1336
|
-
<tr><td>paths</td><td>source files (dest)</td><td>array</td><td></td><td></td><td></td><td>positional<br/>--file-list<br/>--file-pair-list</td><td>Contains a path to the source (required) and a path to the destination.</td></tr>
|
|
1337
|
-
<tr><td>http_fallback_port</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-t</td><td>Specifies http port.</td></tr>
|
|
1338
|
-
<tr><td>https_fallback_port</td><td></td><td>integer</td><td></td><td></td><td></td><td>todo</td><td>Specifies https port.</td></tr>
|
|
1339
|
-
<tr><td>cipher_allowed</td><td></td><td>string</td><td></td><td></td><td></td><td>-</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr>
|
|
1340
|
-
<tr><td>target_rate_cap_kbps</td><td></td><td></td><td class="no">N</td><td class="no">?</td><td class="yes">?</td><td>-</td><td>Returned by upload/download_setup node api.</td></tr>
|
|
1341
|
-
<tr><td>rate_policy_allowed</td><td></td><td></td><td></td><td></td><td></td><td>-</td><td>returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".</td></tr>
|
|
1342
|
-
<tr><td>ssh_private_key</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>KEY</td><td>Private key used for SSH authentication, Shall look like: `-----BEGIN RSA PRIVATE KEY-----\nMII`<br/>Note the JSON encoding `\` + `n` for newlines.</td></tr>
|
|
1343
|
-
<tr><td>remote_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>PASS</td><td>SSH session password</td></tr>
|
|
1344
|
-
<tr><td>resume_policy</td><td>faspmgr:<br/>none<br/>other:<br/>sparse_csum</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-k</td><td>none,attrs,sparse_csum,full_csum</td></tr>
|
|
1345
|
-
<tr><td>authentication</td><td>-</td><td class="no">N</td><td class="no">N</td><td class="yes">Y</td><td>-</td><td>token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.</td></tr>
|
|
1346
|
-
<tr><td>EX_ssh_key_paths</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-i</td><td>Use public key authentication and specify the private key file</td></tr>
|
|
1347
|
-
<tr><td>EX_at_rest_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>FILEPASS</td><td>Passphrase used for at rest encryption or decryption</td></tr>
|
|
1348
|
-
<tr><td>EX_proxy_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>PROXY_PASS</td><td>TODO</td></tr>
|
|
1349
|
-
<tr><td>EX_fasp_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--proxy</td><td>Specify the address of the Aspera high-speed proxy server</td></tr>
|
|
1350
|
-
<tr><td>EX_http_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-x</td><td>Specify the proxy server address used by HTTP Fallback</td></tr>
|
|
1351
|
-
<tr><td>EX_ascp_args</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>same</td><td>Add command line arguments to ascp</td></tr>
|
|
1352
|
-
<tr><td>EX_http_transfer_jpeg</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-j</td><td>HTTP transfers as JPEG file</td></tr>
|
|
1353
|
-
<tr><td>EX_license_text</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>LICENSE</td><td>license file text</td></tr>
|
|
1354
|
-
<tr><td>EX_file_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-list</td><td>source file list</td></tr>
|
|
1355
|
-
<tr><td>EX_file_pair_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-pair-list</td><td>source file pair list</td></tr>
|
|
1356
|
-
<tr><td>EX_multi_session_part</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-C</td><td>part for multisession</td></tr>
|
|
1357
|
-
<tr><td>EX_no_read</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-read</td><td>no read source</td></tr>
|
|
1358
|
-
<tr><td>EX_no_write</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-write</td><td>no write estination</td></tr>
|
|
1359
|
-
</table>
|
|
1306
|
+
* D=Direct (local `ascp` execution)
|
|
1307
|
+
* N=Node API
|
|
1308
|
+
* C=Connect Client
|
|
1309
|
+
* arg=`ascp` argument or environment variable
|
|
1310
|
+
|
|
1311
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in `ascli`).
|
|
1360
1312
|
|
|
1313
|
+
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
|
1361
1314
|
|
|
1362
1315
|
### Destination folder for transfers
|
|
1363
1316
|
|
|
@@ -1373,8 +1326,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
|
1373
1326
|
|
|
1374
1327
|
### List of files for transfers
|
|
1375
1328
|
|
|
1376
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
|
1377
|
-
|
|
1329
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
|
1330
|
+
So, by default, the list of files to transfer will be simply specified on the command line:
|
|
1378
1331
|
|
|
1379
1332
|
```
|
|
1380
1333
|
$ ascli server upload ~/mysample.file secondfile
|
|
@@ -1386,7 +1339,7 @@ This is equivalent to:
|
|
|
1386
1339
|
$ ascli server upload --sources=@args ~/mysample.file secondfile
|
|
1387
1340
|
```
|
|
1388
1341
|
|
|
1389
|
-
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
|
1342
|
+
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
|
1390
1343
|
|
|
1391
1344
|
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
|
1392
1345
|
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
|
@@ -1409,13 +1362,15 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
|
1409
1362
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
|
1410
1363
|
```
|
|
1411
1364
|
|
|
1412
|
-
*
|
|
1365
|
+
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
|
1413
1366
|
|
|
1414
1367
|
```
|
|
1415
1368
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
|
1416
1369
|
```
|
|
1417
1370
|
|
|
1418
|
-
|
|
1371
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
|
1372
|
+
|
|
1373
|
+
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
|
1419
1374
|
|
|
1420
1375
|
* `list` : (default) the path of destination is the same as source
|
|
1421
1376
|
* `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
|
|
@@ -1426,6 +1381,8 @@ Example:
|
|
|
1426
1381
|
$ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
|
1427
1382
|
```
|
|
1428
1383
|
|
|
1384
|
+
Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
|
|
1385
|
+
|
|
1429
1386
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
|
1430
1387
|
|
|
1431
1388
|
* All files must be in the same source folder.
|
|
@@ -1439,7 +1396,7 @@ Source files are located on "Aspera on cloud", when :
|
|
|
1439
1396
|
|
|
1440
1397
|
### <a name="multisession"></a>Support of multi-session
|
|
1441
1398
|
|
|
1442
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
|
1399
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
|
1443
1400
|
|
|
1444
1401
|
* when agent=node :
|
|
1445
1402
|
|
|
@@ -1460,6 +1417,7 @@ shall be preferred.
|
|
|
1460
1417
|
|
|
1461
1418
|
Multi-session spawn is done by `ascli`.
|
|
1462
1419
|
|
|
1420
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
|
1463
1421
|
|
|
1464
1422
|
### Examples
|
|
1465
1423
|
|
|
@@ -1489,15 +1447,154 @@ Multi-session spawn is done by `ascli`.
|
|
|
1489
1447
|
|
|
1490
1448
|
|
|
1491
1449
|
|
|
1492
|
-
## <a name="scheduling"></a>
|
|
1450
|
+
## <a name="scheduling"></a>Lock for exclusive execution
|
|
1451
|
+
|
|
1452
|
+
In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
|
|
1453
|
+
|
|
1454
|
+
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
|
1455
|
+
|
|
1456
|
+
* Executing instances may pile-up and kill the system
|
|
1457
|
+
* The same file may be transfered by multiple instances at the same time.
|
|
1458
|
+
* `preview` may generate the same files in multiple instances.
|
|
1459
|
+
|
|
1460
|
+
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
|
1461
|
+
|
|
1462
|
+
* The Windows scheduler does this by default
|
|
1463
|
+
* Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
|
1464
|
+
|
|
1465
|
+
```
|
|
1466
|
+
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
|
1467
|
+
```
|
|
1468
|
+
|
|
1469
|
+
`ascli` natively supports a locking mechanism with option `lock_port`.
|
|
1470
|
+
(Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
|
|
1471
|
+
|
|
1472
|
+
Example:
|
|
1473
|
+
|
|
1474
|
+
Run this same command in two separate terminals within less than 30 seconds:
|
|
1475
|
+
|
|
1476
|
+
```
|
|
1477
|
+
ascli config echo @ruby:'sleep(30)' --lock-port=12345
|
|
1478
|
+
```
|
|
1479
|
+
|
|
1480
|
+
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
|
1481
|
+
|
|
1482
|
+
```
|
|
1483
|
+
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
|
1484
|
+
```
|
|
1485
|
+
|
|
1486
|
+
## "Provençale"
|
|
1487
|
+
|
|
1488
|
+
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
|
1489
|
+
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
|
1490
|
+
|
|
1491
|
+
The list of supported *PVCL* adapters can be retried with command:
|
|
1492
|
+
|
|
1493
|
+
```
|
|
1494
|
+
$ ascli conf ascp info
|
|
1495
|
+
+--------------------+-----------------------------------------------------------+
|
|
1496
|
+
| key | value |
|
|
1497
|
+
+--------------------+-----------------------------------------------------------+
|
|
1498
|
+
-----8<----snip---------
|
|
1499
|
+
| product_name | IBM Aspera SDK |
|
|
1500
|
+
| product_version | 4.0.1.182389 |
|
|
1501
|
+
| process | pvcl |
|
|
1502
|
+
| shares | pvcl |
|
|
1503
|
+
| noded | pvcl |
|
|
1504
|
+
| faux | pvcl |
|
|
1505
|
+
| file | pvcl |
|
|
1506
|
+
| stdio | pvcl |
|
|
1507
|
+
| stdio-tar | pvcl |
|
|
1508
|
+
+--------------------+-----------------------------------------------------------+
|
|
1509
|
+
```
|
|
1510
|
+
|
|
1511
|
+
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
|
1512
|
+
|
|
1513
|
+
Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
|
|
1514
|
+
|
|
1515
|
+
The simplified format is:
|
|
1516
|
+
|
|
1517
|
+
```
|
|
1518
|
+
<adapter>:///<sub file path>?<arg1>=<val1>&...
|
|
1519
|
+
```
|
|
1520
|
+
|
|
1521
|
+
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
|
1522
|
+
|
|
1523
|
+
## <a name="faux_testing"></a>`faux:` for testing
|
|
1524
|
+
|
|
1525
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`
|
|
1493
1526
|
|
|
1494
|
-
|
|
1527
|
+
This adapter can be used to simulate a file or a directory.
|
|
1495
1528
|
|
|
1496
|
-
|
|
1529
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
|
|
1497
1530
|
|
|
1498
|
-
|
|
1531
|
+
* `fname` is the name that will be assigned to the file on the destination
|
|
1532
|
+
* `fsize` is the number of bytes that will be sent (in decimal).
|
|
1499
1533
|
|
|
1500
|
-
|
|
1534
|
+
Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
|
|
1535
|
+
|
|
1536
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
|
|
1537
|
+
|
|
1538
|
+
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
|
1539
|
+
|
|
1540
|
+
```
|
|
1541
|
+
faux:///dirname?<arg1>=<val1>&...
|
|
1542
|
+
```
|
|
1543
|
+
|
|
1544
|
+
`dirname` is the folder name and can contain `/` to specify a subfolder.
|
|
1545
|
+
|
|
1546
|
+
Supported arguments are:
|
|
1547
|
+
|
|
1548
|
+
<table>
|
|
1549
|
+
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
|
1550
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
|
|
1551
|
+
<tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
|
|
1552
|
+
<tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
|
|
1553
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
|
|
1554
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
|
|
1555
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
|
|
1556
|
+
</table>
|
|
1557
|
+
|
|
1558
|
+
|
|
1559
|
+
The sequence parameter is applied as follows:
|
|
1560
|
+
|
|
1561
|
+
* If `seq` is `random` then each file size is:
|
|
1562
|
+
|
|
1563
|
+
* size +/- (inc * rand())
|
|
1564
|
+
* Where rand is a random number between 0 and 1
|
|
1565
|
+
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
|
1566
|
+
* Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
|
|
1567
|
+
|
|
1568
|
+
* If `seq` is `sequential` then each file size is:
|
|
1569
|
+
|
|
1570
|
+
* size + ((fileindex - 1) * inc)
|
|
1571
|
+
* Where first file is index 1
|
|
1572
|
+
* So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
|
|
1573
|
+
* As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
|
|
1574
|
+
|
|
1575
|
+
Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
|
|
1576
|
+
|
|
1577
|
+
To discard data at the destination, the destination argument is set to `faux://` .
|
|
1578
|
+
|
|
1579
|
+
Examples:
|
|
1580
|
+
|
|
1581
|
+
* Upload 20 gigabytes of random data to file myfile to directory /Upload
|
|
1582
|
+
|
|
1583
|
+
```
|
|
1584
|
+
$ ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
|
1585
|
+
```
|
|
1586
|
+
|
|
1587
|
+
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
|
1588
|
+
|
|
1589
|
+
```
|
|
1590
|
+
$ ascli server upload /tmp/sample --to-folder=faux://
|
|
1591
|
+
```
|
|
1592
|
+
|
|
1593
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
|
|
1594
|
+
|
|
1595
|
+
```
|
|
1596
|
+
$ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
|
1597
|
+
```
|
|
1501
1598
|
|
|
1502
1599
|
## <a name="commands"></a>Sample Commands
|
|
1503
1600
|
|
|
@@ -1508,7 +1605,7 @@ ascli
|
|
|
1508
1605
|
ascli -h
|
|
1509
1606
|
ascli aoc -N remind --username=my_aoc_user_email
|
|
1510
1607
|
ascli aoc -N servers
|
|
1511
|
-
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}'
|
|
1608
|
+
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
|
1512
1609
|
ascli aoc admin ats access_key --id=akibmcloud --secret=somesecret node browse /
|
|
1513
1610
|
ascli aoc admin ats access_key --id=akibmcloud delete
|
|
1514
1611
|
ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
|
@@ -1518,7 +1615,7 @@ ascli aoc admin ats cluster clouds
|
|
|
1518
1615
|
ascli aoc admin ats cluster list
|
|
1519
1616
|
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
|
1520
1617
|
ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
|
|
1521
|
-
ascli aoc admin res
|
|
1618
|
+
ascli aoc admin res application list
|
|
1522
1619
|
ascli aoc admin res client list
|
|
1523
1620
|
ascli aoc admin res client_access_key list
|
|
1524
1621
|
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
|
@@ -1555,6 +1652,7 @@ ascli aoc automation workflow list --value=@json:'{"show_org_workflows":"true"}'
|
|
|
1555
1652
|
ascli aoc bearer_token --display=data --scope=user:all
|
|
1556
1653
|
ascli aoc faspex
|
|
1557
1654
|
ascli aoc files bearer /
|
|
1655
|
+
ascli aoc files bearer_token_node /
|
|
1558
1656
|
ascli aoc files browse /
|
|
1559
1657
|
ascli aoc files browse / -N --link=my_aoc_publink_folder
|
|
1560
1658
|
ascli aoc files delete /testsrc
|
|
@@ -1570,11 +1668,12 @@ ascli aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_li
|
|
|
1570
1668
|
ascli aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
|
|
1571
1669
|
ascli aoc files upload --to-folder=/testsrc testfile.bin
|
|
1572
1670
|
ascli aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
|
1671
|
+
ascli aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
|
1573
1672
|
ascli aoc files v3 info
|
|
1574
1673
|
ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
|
|
1575
1674
|
ascli aoc organization
|
|
1576
1675
|
ascli aoc packages list
|
|
1577
|
-
ascli aoc packages list --
|
|
1676
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
|
1578
1677
|
ascli aoc packages recv --id="my_package_id" --to-folder=.
|
|
1579
1678
|
ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
|
|
1580
1679
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
|
@@ -1584,6 +1683,8 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
|
|
|
1584
1683
|
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
|
1585
1684
|
ascli aoc user info modify @json:'{"name":"dummy change"}'
|
|
1586
1685
|
ascli aoc user info show
|
|
1686
|
+
ascli aoc user shared_inboxes
|
|
1687
|
+
ascli aoc user workspaces
|
|
1587
1688
|
ascli aoc workspace
|
|
1588
1689
|
ascli ats access_key --id=ak_aws delete
|
|
1589
1690
|
ascli ats access_key --id=akibmcloud --secret=somesecret cluster
|
|
@@ -1610,10 +1711,11 @@ ascli config ascp info
|
|
|
1610
1711
|
ascli config ascp install
|
|
1611
1712
|
ascli config ascp products list
|
|
1612
1713
|
ascli config ascp show
|
|
1714
|
+
ascli config ascp spec
|
|
1613
1715
|
ascli config check_update
|
|
1614
1716
|
ascli config doc
|
|
1615
1717
|
ascli config doc transfer-parameters
|
|
1616
|
-
ascli config email_test
|
|
1718
|
+
ascli config email_test --notif-to=my_recipient_email
|
|
1617
1719
|
ascli config export
|
|
1618
1720
|
ascli config genkey mykey
|
|
1619
1721
|
ascli config plugins
|
|
@@ -1709,7 +1811,7 @@ ascli server mkdir folder_1/target_hot
|
|
|
1709
1811
|
ascli server mv folder_1/200KB.2 folder_1/to.delete
|
|
1710
1812
|
ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
|
1711
1813
|
ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
|
1712
|
-
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername
|
|
1814
|
+
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
|
1713
1815
|
ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
|
1714
1816
|
ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
|
1715
1817
|
ascli shares admin share list
|
|
@@ -1734,7 +1836,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
|
1734
1836
|
```
|
|
1735
1837
|
$ ascli -h
|
|
1736
1838
|
NAME
|
|
1737
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
|
1839
|
+
ascli -- a command line tool for Aspera Applications (v4.4.0)
|
|
1738
1840
|
|
|
1739
1841
|
SYNOPSIS
|
|
1740
1842
|
ascli COMMANDS [OPTIONS] [ARGS]
|
|
@@ -1791,29 +1893,31 @@ OPTIONS:
|
|
|
1791
1893
|
--property=VALUE name of property to set
|
|
1792
1894
|
--id=VALUE resource identifier (modify,delete,show)
|
|
1793
1895
|
--config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
|
|
1794
|
-
--override=ENUM override existing value: yes, no
|
|
1795
1896
|
-N, --no-default do not load default configuration for plugin
|
|
1796
|
-
--
|
|
1797
|
-
--
|
|
1897
|
+
--override=ENUM Wizard: override existing value: yes, no
|
|
1898
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
|
|
1899
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
|
|
1900
|
+
--test-mode=ENUM Wizard: skip private key check step: yes, no
|
|
1901
|
+
--pkeypath=VALUE Wizard: path to private key for JWT
|
|
1798
1902
|
--ascp-path=VALUE path to ascp
|
|
1799
1903
|
--use-product=VALUE use ascp from specified product
|
|
1800
1904
|
--smtp=VALUE smtp configuration (extended value: hash)
|
|
1801
1905
|
--fpac=VALUE proxy auto configuration URL
|
|
1802
1906
|
-P, --presetVALUE load the named option preset from current config file
|
|
1803
|
-
--default=VALUE set as default configuration for specified plugin
|
|
1804
1907
|
--secret=VALUE default secret
|
|
1805
1908
|
--secrets=VALUE secret repository (Hash)
|
|
1806
1909
|
--sdk-url=VALUE URL to get SDK
|
|
1807
|
-
--sdk-folder=VALUE SDK folder
|
|
1808
|
-
--
|
|
1809
|
-
--
|
|
1910
|
+
--sdk-folder=VALUE SDK folder path
|
|
1911
|
+
--notif-to=VALUE email recipient for notification of transfers
|
|
1912
|
+
--notif-template=VALUE email ERB template for notification of transfers
|
|
1913
|
+
--version-check-days=VALUE period in days to check new version (zero to disable)
|
|
1810
1914
|
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
|
1811
1915
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
|
1812
1916
|
--to-folder=VALUE destination folder for downloaded files
|
|
1813
1917
|
--sources=VALUE list of source files (see doc)
|
|
1814
|
-
--transfer-info=VALUE
|
|
1918
|
+
--transfer-info=VALUE parameters for transfer agent
|
|
1815
1919
|
--src-type=ENUM type of file list: list, pair
|
|
1816
|
-
--transfer=ENUM type of transfer: direct, httpgw, connect, node
|
|
1920
|
+
--transfer=ENUM type of transfer agent: direct, httpgw, connect, node
|
|
1817
1921
|
--progress=ENUM type of progress bar: none, native, multi
|
|
1818
1922
|
|
|
1819
1923
|
|
|
@@ -1834,7 +1938,7 @@ OPTIONS:
|
|
|
1834
1938
|
--validator=VALUE identifier of validator (optional for central)
|
|
1835
1939
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
|
1836
1940
|
--name=VALUE sync name
|
|
1837
|
-
--token=ENUM
|
|
1941
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
|
1838
1942
|
|
|
1839
1943
|
|
|
1840
1944
|
COMMAND: orchestrator
|
|
@@ -1879,7 +1983,7 @@ OPTIONS:
|
|
|
1879
1983
|
|
|
1880
1984
|
|
|
1881
1985
|
COMMAND: faspex5
|
|
1882
|
-
SUBCOMMANDS: node package auth_client
|
|
1986
|
+
SUBCOMMANDS: node package auth_client jobs
|
|
1883
1987
|
OPTIONS:
|
|
1884
1988
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
1885
1989
|
--username=VALUE username to log in
|
|
@@ -1912,7 +2016,8 @@ OPTIONS:
|
|
|
1912
2016
|
--delivery-info=VALUE package delivery information (extended value)
|
|
1913
2017
|
--source-name=VALUE create package from remote source (by name)
|
|
1914
2018
|
--storage=VALUE Faspex local storage definition
|
|
1915
|
-
--
|
|
2019
|
+
--recipient=VALUE use if recipient is a dropbox (with *)
|
|
2020
|
+
--box=ENUM package box: inbox, archive, sent
|
|
1916
2021
|
|
|
1917
2022
|
|
|
1918
2023
|
COMMAND: shares2
|
|
@@ -1986,7 +2091,6 @@ OPTIONS:
|
|
|
1986
2091
|
--new-user-option=VALUE new user creation option
|
|
1987
2092
|
--from-folder=VALUE share to share source folder
|
|
1988
2093
|
--scope=VALUE OAuth scope for AoC API calls
|
|
1989
|
-
--notify=VALUE notify users that file was received
|
|
1990
2094
|
--bulk=ENUM bulk operation: yes, no
|
|
1991
2095
|
--default-ports=ENUM use standard FASP ports or get from node api: yes, no
|
|
1992
2096
|
|
|
@@ -2067,7 +2171,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
|
2067
2171
|
|
|
2068
2172
|
This will guide you through the steps to create.
|
|
2069
2173
|
|
|
2070
|
-
## <a name="
|
|
2174
|
+
## <a name="aocmanual"></a>Configuration: using manual setup
|
|
2071
2175
|
|
|
2072
2176
|
If you used the wizard (recommended): skip this section.
|
|
2073
2177
|
|
|
@@ -2233,7 +2337,7 @@ To activate default use of JWT authentication for `ascli` using the [option pres
|
|
|
2233
2337
|
|
|
2234
2338
|
* change auth method to JWT
|
|
2235
2339
|
* provide location of private key
|
|
2236
|
-
* provide username to login as (
|
|
2340
|
+
* provide username to login as (OAuth "subject")
|
|
2237
2341
|
|
|
2238
2342
|
Execute:
|
|
2239
2343
|
|
|
@@ -2265,19 +2369,55 @@ It allows actions (create, update, delete) on "resources": users, group, nodes,
|
|
|
2265
2369
|
|
|
2266
2370
|
Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
|
2267
2371
|
|
|
2268
|
-
|
|
2372
|
+
#### Listing resources
|
|
2373
|
+
|
|
2374
|
+
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
|
2375
|
+
|
|
2376
|
+
The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
|
|
2377
|
+
|
|
2378
|
+
The following parameters are supported:
|
|
2379
|
+
|
|
2380
|
+
* `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
|
2381
|
+
* `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
|
2382
|
+
* `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
|
2383
|
+
* `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
|
2384
|
+
* `page` : native api parameter, in general do not use (added by
|
|
2385
|
+
* `per_page` : native api parameter, number of items par api call, in general do not use
|
|
2386
|
+
* Other specific parameters depending on resource type.
|
|
2387
|
+
|
|
2388
|
+
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
|
2389
|
+
|
|
2390
|
+
Other parameters are directly sent as parameters to the GET request on API.
|
|
2391
|
+
|
|
2392
|
+
`page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
|
2393
|
+
|
|
2394
|
+
`q` and `sort` are available on most resrouce types.
|
|
2395
|
+
|
|
2396
|
+
Other parameters depend on the type of entity (refer to AoC API).
|
|
2397
|
+
|
|
2398
|
+
Examples:
|
|
2399
|
+
|
|
2400
|
+
* List users with `laurent` in name:
|
|
2401
|
+
|
|
2402
|
+
```
|
|
2403
|
+
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
|
2404
|
+
```
|
|
2405
|
+
|
|
2406
|
+
* List users who logded-in before a date:
|
|
2269
2407
|
|
|
2270
2408
|
```
|
|
2271
|
-
--query=@json:'{"
|
|
2409
|
+
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
|
2272
2410
|
```
|
|
2273
2411
|
|
|
2274
|
-
|
|
2412
|
+
* List external users and sort in reverse alphabetical order using name:
|
|
2413
|
+
|
|
2275
2414
|
```
|
|
2276
|
-
--query=@json:'{"member_of_any_workspace":
|
|
2277
|
-
--query=@json:'{"q":"laurent"}'
|
|
2415
|
+
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
|
2278
2416
|
```
|
|
2279
2417
|
|
|
2280
|
-
Refer to the AoC API for full list of query parameters.
|
|
2418
|
+
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
|
2419
|
+
|
|
2420
|
+
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
|
2281
2421
|
|
|
2282
2422
|
#### Access Key secrets
|
|
2283
2423
|
|
|
@@ -2331,10 +2471,10 @@ $ ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,em
|
|
|
2331
2471
|
: 98398 : dummyuser1@example.com :
|
|
2332
2472
|
: 98399 : dummyuser2@example.com :
|
|
2333
2473
|
:.......:........................:
|
|
2334
|
-
$ thelist=$(
|
|
2474
|
+
$ thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
|
2335
2475
|
$ echo $thelist
|
|
2336
|
-
|
|
2337
|
-
$ ascli aoc admin res user --bulk=yes --id=@json:
|
|
2476
|
+
["113501","354061"]
|
|
2477
|
+
$ ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
2338
2478
|
:.......:.........:
|
|
2339
2479
|
: id : status :
|
|
2340
2480
|
:.......:.........:
|
|
@@ -2343,6 +2483,14 @@ $ ascli aoc admin res user --bulk=yes --id=@json:[$thelist] delete
|
|
|
2343
2483
|
:.......:.........:
|
|
2344
2484
|
```
|
|
2345
2485
|
|
|
2486
|
+
* <a name="deactuser"></a>Find deactivated users since more than 2 years
|
|
2487
|
+
|
|
2488
|
+
```
|
|
2489
|
+
ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
|
2490
|
+
```
|
|
2491
|
+
|
|
2492
|
+
To delete them use the same method as before
|
|
2493
|
+
|
|
2346
2494
|
* Display current user's workspaces
|
|
2347
2495
|
|
|
2348
2496
|
```
|
|
@@ -2370,17 +2518,20 @@ $ ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_k
|
|
|
2370
2518
|
$ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
|
2371
2519
|
```
|
|
2372
2520
|
|
|
2373
|
-
|
|
2374
|
-
|
|
2375
|
-
|
|
2376
|
-
|
|
2521
|
+
Examples of query (TODO: cleanup):
|
|
2522
|
+
|
|
2523
|
+
```
|
|
2524
|
+
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
|
2525
|
+
|
|
2526
|
+
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
|
2527
|
+
|
|
2377
2528
|
# filter= 'id', 'short_summary', or 'summary'
|
|
2378
2529
|
# count=nnn
|
|
2379
2530
|
# tag=x.y.z%3Dvalue
|
|
2380
2531
|
# iteration_token=nnn
|
|
2381
2532
|
# after_time=2016-05-01T23:53:09Z
|
|
2382
2533
|
# active_only=true|false
|
|
2383
|
-
|
|
2534
|
+
```
|
|
2384
2535
|
|
|
2385
2536
|
* Display node events (events)
|
|
2386
2537
|
|
|
@@ -2391,7 +2542,7 @@ $ ascli aoc admin res node --secret=_secret_ v3 events
|
|
|
2391
2542
|
* display members of a workspace
|
|
2392
2543
|
|
|
2393
2544
|
```
|
|
2394
|
-
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"
|
|
2545
|
+
$ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
|
2395
2546
|
:.............:.........:..................................:
|
|
2396
2547
|
: member_type : manager : member.email :
|
|
2397
2548
|
:.............:.........:..................................:
|
|
@@ -2412,27 +2563,27 @@ other query parameters:
|
|
|
2412
2563
|
|
|
2413
2564
|
* <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
|
|
2414
2565
|
|
|
2415
|
-
a-
|
|
2566
|
+
a- Get id of first workspace
|
|
2416
2567
|
|
|
2417
2568
|
```
|
|
2418
2569
|
WS1='First Workspace'
|
|
2419
2570
|
WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
|
2420
2571
|
```
|
|
2421
2572
|
|
|
2422
|
-
b-
|
|
2573
|
+
b- Get id of second workspace
|
|
2423
2574
|
|
|
2424
2575
|
```
|
|
2425
2576
|
WS2='Second Workspace'
|
|
2426
2577
|
WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
|
|
2427
2578
|
```
|
|
2428
2579
|
|
|
2429
|
-
c-
|
|
2580
|
+
c- Extract membership information
|
|
2430
2581
|
|
|
2431
2582
|
```
|
|
2432
|
-
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"
|
|
2583
|
+
$ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
|
2433
2584
|
```
|
|
2434
2585
|
|
|
2435
|
-
d-
|
|
2586
|
+
d- Convert to creation data for second workspace:
|
|
2436
2587
|
|
|
2437
2588
|
```
|
|
2438
2589
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
|
@@ -2444,16 +2595,16 @@ or, using jq:
|
|
|
2444
2595
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
|
2445
2596
|
```
|
|
2446
2597
|
|
|
2447
|
-
e-
|
|
2598
|
+
e- Add members to second workspace
|
|
2448
2599
|
|
|
2449
2600
|
```
|
|
2450
2601
|
$ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
|
2451
2602
|
```
|
|
2452
2603
|
|
|
2453
|
-
*
|
|
2604
|
+
* Get users who did not log since a date
|
|
2454
2605
|
|
|
2455
2606
|
```
|
|
2456
|
-
$ ascli aoc admin res user list --fields=email --query=@json:'{"
|
|
2607
|
+
$ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
|
2457
2608
|
:...............................:
|
|
2458
2609
|
: email :
|
|
2459
2610
|
:...............................:
|
|
@@ -2462,10 +2613,10 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
|
|
|
2462
2613
|
:...............................:
|
|
2463
2614
|
```
|
|
2464
2615
|
|
|
2465
|
-
*
|
|
2616
|
+
* List "Limited" users
|
|
2466
2617
|
|
|
2467
2618
|
```
|
|
2468
|
-
$ ascli aoc admin res user list --fields=email --
|
|
2619
|
+
$ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
|
2469
2620
|
```
|
|
2470
2621
|
|
|
2471
2622
|
* Perform a multi Gbps transfer between two remote shared folders
|
|
@@ -2508,6 +2659,7 @@ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='
|
|
|
2508
2659
|
```
|
|
2509
2660
|
|
|
2510
2661
|
* create registration key to register a node
|
|
2662
|
+
|
|
2511
2663
|
```
|
|
2512
2664
|
$ ascli aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
|
2513
2665
|
jfqslfdjlfdjfhdjklqfhdkl
|
|
@@ -2527,6 +2679,19 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
|
|
|
2527
2679
|
+-----+---------+
|
|
2528
2680
|
```
|
|
2529
2681
|
|
|
2682
|
+
* List packages in a given shared inbox
|
|
2683
|
+
|
|
2684
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
|
2685
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
|
2686
|
+
|
|
2687
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
|
2688
|
+
|
|
2689
|
+
```
|
|
2690
|
+
shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
|
2691
|
+
|
|
2692
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
|
2693
|
+
```
|
|
2694
|
+
|
|
2530
2695
|
## Shared folders
|
|
2531
2696
|
|
|
2532
2697
|
* list shared folders in node
|
|
@@ -2559,7 +2724,7 @@ Notes:
|
|
|
2559
2724
|
|
|
2560
2725
|
* the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
|
|
2561
2726
|
* to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". ascli will resolve the list of email addresses to expected user ids.
|
|
2562
|
-
* a
|
|
2727
|
+
* a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
|
|
2563
2728
|
* If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
|
2564
2729
|
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
|
|
2565
2730
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
|
@@ -2568,7 +2733,13 @@ Examples:
|
|
|
2568
2733
|
|
|
2569
2734
|
```
|
|
2570
2735
|
$ ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
|
|
2736
|
+
```
|
|
2737
|
+
|
|
2738
|
+
```
|
|
2571
2739
|
$ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
|
|
2740
|
+
```
|
|
2741
|
+
|
|
2742
|
+
```
|
|
2572
2743
|
$ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
|
2573
2744
|
```
|
|
2574
2745
|
|
|
@@ -2584,7 +2755,7 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
|
2584
2755
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
|
2585
2756
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
|
2586
2757
|
|
|
2587
|
-
Typically, one would
|
|
2758
|
+
Typically, one would execute this command on a regular basis, using the method of your choice:
|
|
2588
2759
|
|
|
2589
2760
|
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
|
2590
2761
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
|
@@ -2592,45 +2763,45 @@ Typically, one would regularly execute this command on a regular basis, using th
|
|
|
2592
2763
|
|
|
2593
2764
|
## Download Files
|
|
2594
2765
|
|
|
2595
|
-
Download of files is straightforward with a specific syntax for the `
|
|
2766
|
+
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
|
2596
2767
|
|
|
2597
2768
|
* if only one source is provided, it is downloaded
|
|
2598
2769
|
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
|
2599
2770
|
|
|
2600
2771
|
## Find Files
|
|
2601
2772
|
|
|
2602
|
-
The command `
|
|
2773
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
|
2603
2774
|
|
|
2604
2775
|
The expression can be of 3 formats:
|
|
2605
2776
|
|
|
2606
|
-
* empty (default) : all files, equivalent to: `exec:true`
|
|
2607
|
-
* not starting with `exec:` : the expression is a regular expression, using ruby
|
|
2777
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
|
2778
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
|
2608
2779
|
|
|
2609
2780
|
For instance, to find files with a special extension, use `--value='\.myext$'`
|
|
2610
2781
|
|
|
2611
|
-
* starting with `exec:` : the
|
|
2782
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
|
2612
2783
|
|
|
2613
|
-
Examples of expressions: (
|
|
2784
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
|
2614
2785
|
|
|
2615
|
-
*
|
|
2786
|
+
* Find files more recent than 100 days
|
|
2616
2787
|
|
|
2617
2788
|
```
|
|
2618
2789
|
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
|
2619
2790
|
```
|
|
2620
2791
|
|
|
2621
|
-
*
|
|
2792
|
+
* Find files older than 1 year on a given node and store in file list
|
|
2622
2793
|
|
|
2623
2794
|
```
|
|
2624
2795
|
$ ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
|
2625
2796
|
```
|
|
2626
2797
|
|
|
2627
|
-
*
|
|
2798
|
+
* Delete the files, one by one
|
|
2628
2799
|
|
|
2629
2800
|
```
|
|
2630
2801
|
$ cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
|
2631
2802
|
```
|
|
2632
2803
|
|
|
2633
|
-
*
|
|
2804
|
+
* Delete the files in bulk
|
|
2634
2805
|
|
|
2635
2806
|
```
|
|
2636
2807
|
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
|
@@ -2644,19 +2815,40 @@ The activity app can be queried with:
|
|
|
2644
2815
|
$ ascli aoc admin analytics transfers
|
|
2645
2816
|
```
|
|
2646
2817
|
|
|
2647
|
-
It can also support filters and send notification
|
|
2818
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
|
2819
|
+
|
|
2820
|
+
`mytemplate.erb`:
|
|
2821
|
+
|
|
2822
|
+
```
|
|
2823
|
+
From: <%=from_name%> <<%=from_email%>>
|
|
2824
|
+
To: <<%=ev['user_email']%>>
|
|
2825
|
+
Subject: <%=ev['files_completed']%> files received
|
|
2826
|
+
|
|
2827
|
+
Dear <%=ev[:user_email.to_s]%>,
|
|
2828
|
+
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
|
|
2829
|
+
<%=ev['content']%>
|
|
2648
2830
|
|
|
2831
|
+
Thank you.
|
|
2649
2832
|
```
|
|
2650
|
-
|
|
2833
|
+
The environment provided contains the following additional variable:
|
|
2834
|
+
|
|
2835
|
+
* ev : all details on the transfer event
|
|
2836
|
+
|
|
2837
|
+
Example:
|
|
2838
|
+
|
|
2839
|
+
```
|
|
2840
|
+
$ ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
|
2651
2841
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
|
2652
|
-
--
|
|
2842
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
|
2653
2843
|
```
|
|
2654
2844
|
|
|
2845
|
+
Options:
|
|
2846
|
+
|
|
2655
2847
|
* `once_only` keep track of last date it was called, so next call will get only new events
|
|
2656
2848
|
* `query` filter (on API call)
|
|
2657
2849
|
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
|
2658
2850
|
|
|
2659
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months.
|
|
2851
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
|
2660
2852
|
|
|
2661
2853
|
## Using specific transfer ports
|
|
2662
2854
|
|
|
@@ -2717,7 +2909,37 @@ $ ascli ats api_key create
|
|
|
2717
2909
|
+--------+----------------------------------------------+
|
|
2718
2910
|
$ ascli config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
|
2719
2911
|
```
|
|
2912
|
+
## Cross Organization transfers
|
|
2913
|
+
|
|
2914
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
|
2915
|
+
|
|
2916
|
+
Although optional, the creation of [option preset](#lprt) is recommended to avoid placing all parameters in the command line.
|
|
2917
|
+
|
|
2918
|
+
Procedure to send a file from org1 to org2:
|
|
2919
|
+
|
|
2920
|
+
* Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
|
2921
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
|
2922
|
+
* Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
|
2923
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
|
2924
|
+
* execute the following:
|
|
2925
|
+
|
|
2926
|
+
```
|
|
2927
|
+
$ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
|
2928
|
+
```
|
|
2929
|
+
|
|
2930
|
+
Explanation:
|
|
2720
2931
|
|
|
2932
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
|
2933
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
|
2934
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
|
2935
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
|
2936
|
+
* `|` the standard output of the first command is fed into the second one
|
|
2937
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
|
2938
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
|
2939
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
|
|
2940
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
|
2941
|
+
|
|
2942
|
+
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
|
2721
2943
|
## Examples
|
|
2722
2944
|
|
|
2723
2945
|
Example: create access key on softlayer:
|
|
@@ -2763,6 +2985,14 @@ This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
|
|
|
2763
2985
|
|
|
2764
2986
|
Both password and SSH keys auth are supported.
|
|
2765
2987
|
|
|
2988
|
+
If not username is provided, the default transfer user `xfer` is used.
|
|
2989
|
+
|
|
2990
|
+
If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
|
|
2991
|
+
|
|
2992
|
+
```
|
|
2993
|
+
$ ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
|
2994
|
+
```
|
|
2995
|
+
|
|
2766
2996
|
Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
|
|
2767
2997
|
|
|
2768
2998
|
Examples:
|
|
@@ -2800,7 +3030,7 @@ This means that you dont have such an ssh agent running:
|
|
|
2800
3030
|
$ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
|
2801
3031
|
```
|
|
2802
3032
|
|
|
2803
|
-
This can also be set as default using a preset
|
|
3033
|
+
This can also be set as default using a preset.
|
|
2804
3034
|
|
|
2805
3035
|
## Example
|
|
2806
3036
|
|
|
@@ -2820,18 +3050,36 @@ This creates a [option preset](#lprt) "aspera_demo_server" and set it as default
|
|
|
2820
3050
|
|
|
2821
3051
|
This plugin gives access to capabilities provided by HSTS node API.
|
|
2822
3052
|
|
|
2823
|
-
##
|
|
3053
|
+
## File Operations
|
|
2824
3054
|
|
|
2825
3055
|
It is possible to:
|
|
2826
3056
|
* browse
|
|
2827
3057
|
* transfer (upload / download)
|
|
2828
3058
|
* ...
|
|
2829
3059
|
|
|
3060
|
+
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
|
3061
|
+
|
|
3062
|
+
* `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
|
|
3063
|
+
* `basic` : transfer spec is created like this:
|
|
3064
|
+
|
|
3065
|
+
```
|
|
3066
|
+
{
|
|
3067
|
+
"remote_host": address of node url,
|
|
3068
|
+
"remote_user": "xfer",
|
|
3069
|
+
"ssh_port": 33001,
|
|
3070
|
+
"token": "Basic <base 64 encoded user/pass>",
|
|
3071
|
+
"direction": send/recv
|
|
3072
|
+
}
|
|
3073
|
+
```
|
|
3074
|
+
|
|
3075
|
+
* `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
|
3076
|
+
|
|
2830
3077
|
## Central
|
|
2831
3078
|
|
|
2832
3079
|
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
|
|
2833
3080
|
|
|
2834
3081
|
Filtering can be applied:
|
|
3082
|
+
|
|
2835
3083
|
```
|
|
2836
3084
|
$ ascli node central file list
|
|
2837
3085
|
```
|
|
@@ -2857,7 +3105,6 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
|
2857
3105
|
* Start watchd and watchfolderd services running as a system user having access to files
|
|
2858
3106
|
* configure a watchfolder to define automated transfers
|
|
2859
3107
|
|
|
2860
|
-
|
|
2861
3108
|
```
|
|
2862
3109
|
$ ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
|
2863
3110
|
$ ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
|
@@ -2903,16 +3150,19 @@ to download files.
|
|
|
2903
3150
|
$ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
|
2904
3151
|
```
|
|
2905
3152
|
|
|
2906
|
-
# Plugin: IBM Aspera
|
|
3153
|
+
# Plugin: IBM Aspera Faspex5
|
|
2907
3154
|
|
|
2908
|
-
|
|
3155
|
+
3 authentication methods are supported:
|
|
3156
|
+
|
|
3157
|
+
* jwt
|
|
3158
|
+
* web
|
|
3159
|
+
* boot
|
|
2909
3160
|
|
|
2910
|
-
|
|
2911
|
-
* for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
|
|
3161
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
|
2912
3162
|
|
|
2913
|
-
|
|
3163
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
|
2914
3164
|
|
|
2915
|
-
|
|
3165
|
+
For boot method: (will be removed in future)
|
|
2916
3166
|
|
|
2917
3167
|
* open a browser
|
|
2918
3168
|
* start developer mode
|
|
@@ -2925,30 +3175,74 @@ Use it as password and use `--auth=boot`.
|
|
|
2925
3175
|
$ ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
|
2926
3176
|
```
|
|
2927
3177
|
|
|
2928
|
-
|
|
3178
|
+
Ready to use Faspex5 with CLI.
|
|
3179
|
+
|
|
3180
|
+
# Plugin: IBM Aspera Faspex (4.x)
|
|
3181
|
+
|
|
3182
|
+
Notes:
|
|
3183
|
+
|
|
3184
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
|
3185
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
|
3186
|
+
|
|
3187
|
+
## Listing Packages
|
|
3188
|
+
|
|
3189
|
+
Command: `faspex package list`
|
|
3190
|
+
|
|
3191
|
+
### Option `box`
|
|
3192
|
+
|
|
3193
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
|
3194
|
+
|
|
3195
|
+
### Option `recipient`
|
|
3196
|
+
|
|
3197
|
+
A user can receive a package because the recipient is:
|
|
3198
|
+
|
|
3199
|
+
* the user himself (default)
|
|
3200
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
|
3201
|
+
|
|
3202
|
+
### Option `query`
|
|
3203
|
+
|
|
3204
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
|
3205
|
+
|
|
3206
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
|
3207
|
+
* `page` : (native) id of page in call (default=0)
|
|
3208
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
|
3209
|
+
* `max` : maximum number of items
|
|
3210
|
+
* `pmax` : maximum number of pages
|
|
3211
|
+
|
|
3212
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
|
3213
|
+
|
|
3214
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
|
3215
|
+
|
|
3216
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
|
3217
|
+
|
|
3218
|
+
### Example
|
|
2929
3219
|
|
|
2930
3220
|
```
|
|
2931
|
-
$
|
|
2932
|
-
$ ascli faspex5 -Pf5boot auth_client create --value=@json:'{"name":"hello","client_type":"public","redirect_uris":["https://localhost:12345"],"allow_jwt_grant":true,"public_key":"'$jsonk'"}'
|
|
3221
|
+
$ ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
|
2933
3222
|
```
|
|
2934
3223
|
|
|
2935
|
-
|
|
3224
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
|
3225
|
+
|
|
3226
|
+
## Receiving a Package
|
|
3227
|
+
|
|
3228
|
+
The command is `package recv`, possible methods are:
|
|
3229
|
+
|
|
3230
|
+
* provide a package id with option `id`
|
|
3231
|
+
* provide a public link with option `link`
|
|
3232
|
+
* provide a `faspe:` URI with option `link`
|
|
2936
3233
|
|
|
2937
3234
|
```
|
|
2938
|
-
$
|
|
2939
|
-
$ ascli
|
|
3235
|
+
$ ascli faspex package recv --id=12345
|
|
3236
|
+
$ ascli faspex package recv --link=faspe://...
|
|
2940
3237
|
```
|
|
2941
3238
|
|
|
2942
|
-
|
|
3239
|
+
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
|
2943
3240
|
|
|
2944
3241
|
```
|
|
2945
|
-
$ ascli
|
|
2946
|
-
|
|
2947
|
-
```
|
|
2948
|
-
|
|
2949
|
-
Ready to use Faspex5 with CLI.
|
|
3242
|
+
$ ascli faspex package list --recipient='*thedropboxname'
|
|
3243
|
+
```
|
|
2950
3244
|
|
|
2951
|
-
|
|
3245
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
|
2952
3246
|
|
|
2953
3247
|
## Sending a Package
|
|
2954
3248
|
|
|
@@ -2967,7 +3261,19 @@ Additional optional parameters in `delivery_info`:
|
|
|
2967
3261
|
* Package Note: : `"note":"note this and that"`
|
|
2968
3262
|
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
|
2969
3263
|
|
|
2970
|
-
##
|
|
3264
|
+
## Email notification on transfer
|
|
3265
|
+
|
|
3266
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
|
3267
|
+
|
|
3268
|
+
Example:
|
|
3269
|
+
|
|
3270
|
+
```
|
|
3271
|
+
$ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
|
3272
|
+
```
|
|
3273
|
+
|
|
3274
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
|
3275
|
+
|
|
3276
|
+
## Operation on dropboxes
|
|
2971
3277
|
|
|
2972
3278
|
Example:
|
|
2973
3279
|
|
|
@@ -2977,7 +3283,7 @@ $ ascli faspex v4 dropbox list
|
|
|
2977
3283
|
$ ascli faspex v4 dropbox delete --id=36
|
|
2978
3284
|
```
|
|
2979
3285
|
|
|
2980
|
-
##
|
|
3286
|
+
## Remote sources
|
|
2981
3287
|
|
|
2982
3288
|
Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
|
|
2983
3289
|
the node API is used, for this it is required to add a section ":storage" that links
|
|
@@ -3028,29 +3334,38 @@ $ for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}'
|
|
|
3028
3334
|
# Plugin: IBM Cloud Object Storage
|
|
3029
3335
|
|
|
3030
3336
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
|
3031
|
-
It uses the same transfer service as Aspera on Cloud.
|
|
3032
|
-
|
|
3337
|
+
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
|
3338
|
+
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
|
3339
|
+
|
|
3340
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the forst method. If you dont have credentials but have access to the IBM Cloud console, then use the second method.
|
|
3033
3341
|
|
|
3034
|
-
|
|
3342
|
+
## Using endpoint, apikey and Ressource Instance ID (CRN)
|
|
3343
|
+
|
|
3344
|
+
If you have those parameters already, then following options shall be provided:
|
|
3035
3345
|
|
|
3036
3346
|
* `bucket` bucket name
|
|
3037
3347
|
* `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
|
|
3038
3348
|
* `apikey` API Key
|
|
3039
3349
|
* `crn` resource instance id
|
|
3040
3350
|
|
|
3041
|
-
|
|
3351
|
+
For example, let us create a default configuration:
|
|
3042
3352
|
|
|
3043
|
-
|
|
3044
|
-
|
|
3045
|
-
|
|
3353
|
+
```
|
|
3354
|
+
$ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
|
3355
|
+
$ ascli conf id default set cos mycos
|
|
3356
|
+
```
|
|
3046
3357
|
|
|
3047
|
-
|
|
3358
|
+
Then, jump to the transfer example.
|
|
3359
|
+
|
|
3360
|
+
## Using service credential file
|
|
3361
|
+
|
|
3362
|
+
If you are the COS administrator and dont have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
|
3048
3363
|
|
|
3049
3364
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
|
3050
3365
|
|
|
3051
3366
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
3052
3367
|
|
|
3053
|
-
or using the CLI:
|
|
3368
|
+
or using the IBM Cloud CLI:
|
|
3054
3369
|
|
|
3055
3370
|
```
|
|
3056
3371
|
$ ibmcloud resource service-keys
|
|
@@ -3081,33 +3396,38 @@ The field `resource_instance_id` is for option `crn`
|
|
|
3081
3396
|
|
|
3082
3397
|
The field `apikey` is for option `apikey`
|
|
3083
3398
|
|
|
3084
|
-
|
|
3399
|
+
(If needed: endpoints for regions can be found by querying the `endpoints` URL.)
|
|
3400
|
+
|
|
3401
|
+
The required options for this method are:
|
|
3402
|
+
|
|
3403
|
+
* `bucket` bucket name
|
|
3404
|
+
* `region` bucket region, e.g. eu-de
|
|
3405
|
+
* `service_credentials` see below
|
|
3085
3406
|
|
|
3086
|
-
For
|
|
3407
|
+
For example, let us create a default configuration:
|
|
3087
3408
|
|
|
3088
3409
|
```
|
|
3089
3410
|
$ ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
|
3090
3411
|
$ ascli conf id default set cos mycos
|
|
3091
3412
|
```
|
|
3092
3413
|
|
|
3093
|
-
|
|
3414
|
+
## Operations, transfers
|
|
3094
3415
|
|
|
3095
|
-
|
|
3096
|
-
$ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
|
3097
|
-
$ ascli conf id default set cos mycos
|
|
3098
|
-
```
|
|
3416
|
+
Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
|
|
3099
3417
|
|
|
3100
|
-
|
|
3418
|
+
A subset of `node` plugin operations are supported, basically node API:
|
|
3101
3419
|
|
|
3102
3420
|
```
|
|
3103
|
-
$ ascli cos node
|
|
3104
|
-
$ ascli cos node upload
|
|
3421
|
+
$ ascli cos node info
|
|
3422
|
+
$ ascli cos node upload 'faux:///sample1G?1g'
|
|
3105
3423
|
```
|
|
3106
3424
|
|
|
3425
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
|
3426
|
+
|
|
3107
3427
|
# Plugin: IBM Aspera Sync
|
|
3108
3428
|
|
|
3109
|
-
A basic plugin to start an "async" using `ascli`.
|
|
3110
|
-
to start from ma configuration file, using `ascli` standard options.
|
|
3429
|
+
A basic plugin to start an "async" using `ascli`.
|
|
3430
|
+
The main advantage is the possibility to start from ma configuration file, using `ascli` standard options.
|
|
3111
3431
|
|
|
3112
3432
|
# Plugin: Preview
|
|
3113
3433
|
|
|
@@ -3199,8 +3519,10 @@ yum install -y ImageMagick optipng
|
|
|
3199
3519
|
|
|
3200
3520
|
### Video: FFmpeg
|
|
3201
3521
|
|
|
3522
|
+
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
|
3523
|
+
|
|
3202
3524
|
```
|
|
3203
|
-
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt &&
|
|
3525
|
+
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
|
3204
3526
|
```
|
|
3205
3527
|
|
|
3206
3528
|
### Office: Unoconv and Libreoffice
|
|
@@ -3385,14 +3707,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
|
3385
3707
|
The `smtp` option is a hash table (extended value) with the following fields:
|
|
3386
3708
|
<table>
|
|
3387
3709
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
|
3388
|
-
<tr><td
|
|
3389
|
-
<tr><td
|
|
3390
|
-
<tr><td
|
|
3391
|
-
<tr><td
|
|
3392
|
-
<tr><td
|
|
3393
|
-
<tr><td
|
|
3394
|
-
<tr><td
|
|
3395
|
-
<tr><td
|
|
3710
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
|
3711
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
|
3712
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
|
3713
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
|
3714
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
|
3715
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
|
3716
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
|
3717
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
|
3396
3718
|
</table>
|
|
3397
3719
|
|
|
3398
3720
|
## Example of configuration:
|
|
@@ -3422,13 +3744,52 @@ $ ascli config id cli_default set smtp @val:@preset:smtp_google
|
|
|
3422
3744
|
$ ascli config id default set config cli_default
|
|
3423
3745
|
```
|
|
3424
3746
|
|
|
3747
|
+
## Email templates
|
|
3748
|
+
|
|
3749
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
|
3750
|
+
|
|
3751
|
+
The template is the full SMTP message, including headers.
|
|
3752
|
+
|
|
3753
|
+
The following variables are defined by default:
|
|
3754
|
+
|
|
3755
|
+
* from_name
|
|
3756
|
+
* from_email
|
|
3757
|
+
* to
|
|
3758
|
+
|
|
3759
|
+
Other variables are defined depending on context.
|
|
3760
|
+
|
|
3425
3761
|
## Test
|
|
3426
3762
|
|
|
3427
3763
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
|
3428
3764
|
|
|
3429
3765
|
```
|
|
3430
3766
|
$ ascli config --smtp=@preset:smtp_google smtp
|
|
3431
|
-
$ ascli config --smtp=@preset:smtp_google email sample.dest@example.com
|
|
3767
|
+
$ ascli config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
|
3768
|
+
```
|
|
3769
|
+
|
|
3770
|
+
## Notifications for transfer status
|
|
3771
|
+
|
|
3772
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
|
3773
|
+
|
|
3774
|
+
To activate, use option `notif_to`.
|
|
3775
|
+
|
|
3776
|
+
A default e-mail template is used, but it can be overriden with option `notif_template`.
|
|
3777
|
+
|
|
3778
|
+
The environment provided contains the following additional variables:
|
|
3779
|
+
|
|
3780
|
+
* subject
|
|
3781
|
+
* body
|
|
3782
|
+
* global_transfer_status
|
|
3783
|
+
* ts
|
|
3784
|
+
|
|
3785
|
+
Example of template:
|
|
3786
|
+
|
|
3787
|
+
```
|
|
3788
|
+
From: <%=from_name%> <<%=from_email%>>
|
|
3789
|
+
To: <<%=to%>>
|
|
3790
|
+
Subject: <%=subject%>
|
|
3791
|
+
|
|
3792
|
+
Transfer is: <%=global_transfer_status%>
|
|
3432
3793
|
```
|
|
3433
3794
|
|
|
3434
3795
|
# Tool: `asession`
|
|
@@ -3459,7 +3820,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
|
3459
3820
|
* `EX_loglevel` to change log level of the tool
|
|
3460
3821
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
|
3461
3822
|
|
|
3462
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the
|
|
3823
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
|
3463
3824
|
|
|
3464
3825
|
## Comparison of interfaces
|
|
3465
3826
|
|
|
@@ -3559,8 +3920,8 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
|
3559
3920
|
Note that:
|
|
3560
3921
|
|
|
3561
3922
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
|
3562
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
|
3563
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the
|
|
3923
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
|
3924
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
|
|
3564
3925
|
|
|
3565
3926
|
### server side and configuration
|
|
3566
3927
|
|
|
@@ -3579,7 +3940,7 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
|
|
|
3579
3940
|
|
|
3580
3941
|
```
|
|
3581
3942
|
|
|
3582
|
-
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
|
3943
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
|
3583
3944
|
|
|
3584
3945
|
# Aspera Health check and Nagios
|
|
3585
3946
|
|
|
@@ -3656,25 +4017,56 @@ So, it evolved into `ascli`:
|
|
|
3656
4017
|
|
|
3657
4018
|
# Changes (Release notes)
|
|
3658
4019
|
|
|
4020
|
+
* 4.4.0
|
|
4021
|
+
|
|
4022
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
|
4023
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
|
4024
|
+
* new: `preset` option can specify name or hash value
|
|
4025
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
|
4026
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
|
4027
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
|
4028
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
|
4029
|
+
|
|
4030
|
+
* 4.3.0
|
|
4031
|
+
|
|
4032
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
|
|
4033
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
|
4034
|
+
|
|
4035
|
+
* 4.2.2
|
|
4036
|
+
|
|
4037
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
|
4038
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
|
4039
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
|
4040
|
+
* new: `conf ascp spec` displays supported transfer spec
|
|
4041
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
|
4042
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
|
4043
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
|
4044
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then `ascli` exits with code 1 (instead of zero=success)
|
|
4045
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
|
4046
|
+
|
|
4047
|
+
* 4.2.1
|
|
4048
|
+
|
|
4049
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
|
4050
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
|
4051
|
+
|
|
3659
4052
|
* 4.2.0
|
|
3660
4053
|
|
|
3661
4054
|
* new: command `aoc remind` to receive organization membership by email
|
|
3662
4055
|
* new: in `preview` option `value` to filter out on file name
|
|
3663
4056
|
* new: `initdemo` to initialize for demo server
|
|
3664
|
-
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
|
4057
|
+
* new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
|
3665
4058
|
* fix: on Windows `conf ascp use` expects ascp.exe
|
|
3666
4059
|
* fix: (break) multi_session_threshold is Integer, not String
|
|
3667
4060
|
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
|
3668
4061
|
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
|
3669
4062
|
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
|
3670
|
-
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
|
4063
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
|
|
3671
4064
|
|
|
3672
4065
|
* 4.1.0
|
|
3673
4066
|
|
|
3674
|
-
|
|
3675
|
-
|
|
3676
|
-
|
|
3677
|
-
* new: renamed command `nagios_check` to `health`
|
|
4067
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
|
4068
|
+
* new: update documentation with regard to offline and docker installation
|
|
4069
|
+
* new: renamed command `nagios_check` to `health`
|
|
3678
4070
|
* new: agent `http_gw` now supports upload
|
|
3679
4071
|
* new: added option `sdk_url` to install SDK from local file for offline install
|
|
3680
4072
|
* new: check new gem version periodically
|
|
@@ -3725,8 +4117,8 @@ So, it evolved into `ascli`:
|
|
|
3725
4117
|
* allow bulk delete in `aspera files` with option `bulk=yes`
|
|
3726
4118
|
* fix getting connect versions
|
|
3727
4119
|
* added section for Aix
|
|
3728
|
-
* support all ciphers for `
|
|
3729
|
-
* added transfer spec param `apply_local_docroot` for `
|
|
4120
|
+
* support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
|
|
4121
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#direct)
|
|
3730
4122
|
|
|
3731
4123
|
* 0.11.4
|
|
3732
4124
|
|
|
@@ -3781,7 +4173,7 @@ So, it evolved into `ascli`:
|
|
|
3781
4173
|
* 0.10.12
|
|
3782
4174
|
|
|
3783
4175
|
* added support for AoC node registration keys
|
|
3784
|
-
* replaced option : `local_resume` with `transfer_info` for agent `direct`
|
|
4176
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
|
|
3785
4177
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
|
3786
4178
|
* `@incps` : new extended value modifier
|
|
3787
4179
|
* ATS: no more provides access keys secrets: now user must provide it
|
|
@@ -3831,9 +4223,9 @@ So, it evolved into `ascli`:
|
|
|
3831
4223
|
|
|
3832
4224
|
* 0.10.2
|
|
3833
4225
|
|
|
3834
|
-
|
|
3835
|
-
|
|
3836
|
-
|
|
4226
|
+
* updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
|
|
4227
|
+
* added doc for "cargo" like actions
|
|
4228
|
+
* added doc for multi-session
|
|
3837
4229
|
|
|
3838
4230
|
* 0.10.1
|
|
3839
4231
|
|
|
@@ -3861,22 +4253,22 @@ So, it evolved into `ascli`:
|
|
|
3861
4253
|
|
|
3862
4254
|
* 0.9.33
|
|
3863
4255
|
|
|
3864
|
-
|
|
3865
|
-
|
|
3866
|
-
|
|
3867
|
-
|
|
4256
|
+
* new command to display basic token of node
|
|
4257
|
+
* new command to display bearer token of node in AoC
|
|
4258
|
+
* the --fields= option, support +_fieldname_ to add a field to default fields
|
|
4259
|
+
* many small changes
|
|
3868
4260
|
|
|
3869
4261
|
* 0.9.32
|
|
3870
4262
|
|
|
3871
|
-
|
|
3872
|
-
|
|
3873
|
-
|
|
4263
|
+
* all Faspex public links are now supported
|
|
4264
|
+
* removed faspex operation recv_publink
|
|
4265
|
+
* replaced with option `link` (consistent with AoC)
|
|
3874
4266
|
|
|
3875
4267
|
* 0.9.31
|
|
3876
4268
|
|
|
3877
|
-
|
|
3878
|
-
|
|
3879
|
-
|
|
4269
|
+
* added more support for public link: receive and send package, to user or dropbox and files view.
|
|
4270
|
+
* delete expired file lists
|
|
4271
|
+
* changed text table gem from text-table to terminal-table because it supports multiline values
|
|
3880
4272
|
|
|
3881
4273
|
* 0.9.27
|
|
3882
4274
|
|
|
@@ -3896,149 +4288,149 @@ So, it evolved into `ascli`:
|
|
|
3896
4288
|
|
|
3897
4289
|
* 0.9.24
|
|
3898
4290
|
|
|
3899
|
-
|
|
3900
|
-
|
|
4291
|
+
* fix bug where AoC node to node transfer did not work
|
|
4292
|
+
* fix bug on error if ED25519 private key is defined in .ssh
|
|
3901
4293
|
|
|
3902
4294
|
* 0.9.23
|
|
3903
4295
|
|
|
3904
|
-
|
|
3905
|
-
|
|
4296
|
+
* defined REST error handlers, more error conditions detected
|
|
4297
|
+
* commands to select specific ascp location
|
|
3906
4298
|
|
|
3907
4299
|
* 0.9.21
|
|
3908
4300
|
|
|
3909
|
-
|
|
3910
|
-
|
|
4301
|
+
* supports simplified wizard using global client
|
|
4302
|
+
* only ascp binary is required, other SDK (keys) files are now generated
|
|
3911
4303
|
|
|
3912
4304
|
* 0.9.20
|
|
3913
4305
|
|
|
3914
|
-
|
|
3915
|
-
|
|
3916
|
-
|
|
4306
|
+
* improved wizard (prepare for AoC global client id)
|
|
4307
|
+
* preview generator: addedoption : --skip-format=<png,mp4>
|
|
4308
|
+
* removed outdated pictures from this doc
|
|
3917
4309
|
|
|
3918
4310
|
* 0.9.19
|
|
3919
4311
|
|
|
3920
|
-
|
|
4312
|
+
* added command aspera bearer --scope=xx
|
|
3921
4313
|
|
|
3922
4314
|
* 0.9.18
|
|
3923
4315
|
|
|
3924
|
-
|
|
4316
|
+
* enhanced aspera admin events to support query
|
|
3925
4317
|
|
|
3926
4318
|
* 0.9.16
|
|
3927
4319
|
|
|
3928
|
-
|
|
3929
|
-
|
|
4320
|
+
* AoC transfers are now reported in activity app
|
|
4321
|
+
* new interface for Rest class authentication (keep backward compatibility)
|
|
3930
4322
|
|
|
3931
4323
|
* 0.9.15
|
|
3932
4324
|
|
|
3933
|
-
|
|
3934
|
-
|
|
4325
|
+
* new feature: "find" command in aspera files
|
|
4326
|
+
* sample code for transfer API
|
|
3935
4327
|
|
|
3936
4328
|
* 0.9.12
|
|
3937
4329
|
|
|
3938
|
-
|
|
3939
|
-
|
|
4330
|
+
* add nagios commands
|
|
4331
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
|
3940
4332
|
|
|
3941
4333
|
* 0.9.11
|
|
3942
4334
|
|
|
3943
|
-
|
|
3944
|
-
|
|
4335
|
+
* Breaking change: @stdin is now @stdin:
|
|
4336
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
|
3945
4337
|
|
|
3946
4338
|
|
|
3947
4339
|
* 0.9.10
|
|
3948
4340
|
|
|
3949
|
-
|
|
3950
|
-
|
|
3951
|
-
|
|
3952
|
-
|
|
4341
|
+
* Breaking change: parameter transfer-node becomes more generic: transfer-info
|
|
4342
|
+
* Display SaaS storage usage with command: aspera admin res node --id=nn info
|
|
4343
|
+
* cleaner way of specifying source file list for transfers
|
|
4344
|
+
* Breaking change: replaced download_mode option with http_download action
|
|
3953
4345
|
|
|
3954
4346
|
* 0.9.9
|
|
3955
4347
|
|
|
3956
|
-
|
|
3957
|
-
|
|
4348
|
+
* Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
|
|
4349
|
+
* Now supports "cargo" for Aspera on Cloud (automatic package download)
|
|
3958
4350
|
|
|
3959
4351
|
* 0.9.8
|
|
3960
4352
|
|
|
3961
|
-
|
|
3962
|
-
|
|
4353
|
+
* Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
|
|
4354
|
+
* AoC: share to share transfer with command "transfer"
|
|
3963
4355
|
|
|
3964
4356
|
* 0.9.7
|
|
3965
4357
|
|
|
3966
|
-
|
|
3967
|
-
|
|
3968
|
-
|
|
3969
|
-
|
|
3970
|
-
|
|
4358
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
|
|
4359
|
+
* preview persistency goes to unique file by default
|
|
4360
|
+
* catch mxf extension in preview as video
|
|
4361
|
+
* Faspex: possibility to download all paclages by specifying id=ALL
|
|
4362
|
+
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
|
3971
4363
|
|
|
3972
4364
|
* 0.9.6
|
|
3973
4365
|
|
|
3974
|
-
|
|
3975
|
-
|
|
4366
|
+
* Breaking change: `@param:`is now `@preset:` and is generic
|
|
4367
|
+
* AoC: added command to display current workspace information
|
|
3976
4368
|
|
|
3977
4369
|
* 0.9.5
|
|
3978
4370
|
|
|
3979
|
-
|
|
3980
|
-
|
|
4371
|
+
* new parameter: new_user_option used to choose between public_link and invite of external users.
|
|
4372
|
+
* fixed bug in wizard, and wizard uses now product detection
|
|
3981
4373
|
|
|
3982
4374
|
* 0.9.4
|
|
3983
4375
|
|
|
3984
|
-
|
|
3985
|
-
|
|
3986
|
-
|
|
4376
|
+
* Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
|
|
4377
|
+
* AoC Package send supports external users
|
|
4378
|
+
* new command to export AoC config to Aspera CLI config
|
|
3987
4379
|
|
|
3988
4380
|
* 0.9.3
|
|
3989
4381
|
|
|
3990
|
-
|
|
3991
|
-
|
|
3992
|
-
|
|
3993
|
-
|
|
3994
|
-
|
|
3995
|
-
|
|
4382
|
+
* REST error message show host and code
|
|
4383
|
+
* option for quiet display
|
|
4384
|
+
* modified transfer interface and allow token re-generation on error
|
|
4385
|
+
* async add admin command
|
|
4386
|
+
* async add db parameters
|
|
4387
|
+
* Breaking change: new option "sources" to specify files to transfer
|
|
3996
4388
|
|
|
3997
4389
|
* 0.9.2
|
|
3998
4390
|
|
|
3999
|
-
|
|
4391
|
+
* Breaking change: changed AoC package creation to match API, see AoC section
|
|
4000
4392
|
|
|
4001
4393
|
* 0.9.1
|
|
4002
4394
|
|
|
4003
|
-
|
|
4395
|
+
* Breaking change: changed faspex package creation to match API, see Faspex section
|
|
4004
4396
|
|
|
4005
4397
|
* 0.9
|
|
4006
4398
|
|
|
4007
|
-
|
|
4008
|
-
|
|
4399
|
+
* Renamed the CLI from aslmcli to `ascli`
|
|
4400
|
+
* Automatic rename and conversion of former config folder from aslmcli to `ascli`
|
|
4009
4401
|
|
|
4010
4402
|
* 0.7.6
|
|
4011
4403
|
|
|
4012
|
-
|
|
4404
|
+
* add "sync" plugin
|
|
4013
4405
|
|
|
4014
4406
|
* 0.7
|
|
4015
4407
|
|
|
4016
|
-
|
|
4017
|
-
|
|
4018
|
-
|
|
4019
|
-
|
|
4020
|
-
|
|
4408
|
+
* Breaking change: AoC package recv take option if for package instead of argument.
|
|
4409
|
+
* Breaking change: Rest class and Oauth class changed init parameters
|
|
4410
|
+
* AoC: receive package from public link
|
|
4411
|
+
* select by col value on output
|
|
4412
|
+
* added rename (AoC, node)
|
|
4021
4413
|
|
|
4022
4414
|
* 0.6.19
|
|
4023
4415
|
|
|
4024
4416
|
Breaking change:
|
|
4025
4417
|
|
|
4026
|
-
|
|
4027
|
-
|
|
4028
|
-
|
|
4029
|
-
|
|
4030
|
-
|
|
4031
|
-
|
|
4032
|
-
|
|
4033
|
-
|
|
4418
|
+
* ats server list provisioned → ats cluster list
|
|
4419
|
+
* ats server list clouds → ats cluster clouds
|
|
4420
|
+
* ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
|
|
4421
|
+
* ats server id xxx → ats cluster show --id=xxx
|
|
4422
|
+
* ats subscriptions → ats credential subscriptions
|
|
4423
|
+
* ats api_key repository list → ats credential cache list
|
|
4424
|
+
* ats api_key list → ats credential list
|
|
4425
|
+
* ats access_key id xxx → ats access_key --id=xxx
|
|
4034
4426
|
|
|
4035
4427
|
* 0.6.18
|
|
4036
4428
|
|
|
4037
|
-
|
|
4429
|
+
* some commands take now --id option instead of id command.
|
|
4038
4430
|
|
|
4039
4431
|
* 0.6.15
|
|
4040
4432
|
|
|
4041
|
-
|
|
4433
|
+
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
|
|
4042
4434
|
|
|
4043
4435
|
# BUGS, FEATURES, CONTRIBUTION
|
|
4044
4436
|
|
|
@@ -4075,20 +4467,24 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
|
4075
4467
|
|
|
4076
4468
|
## Error "Remote host is not who we expected"
|
|
4077
4469
|
|
|
4078
|
-
`ascp`
|
|
4470
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
|
4471
|
+
|
|
4472
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
|
4079
4473
|
|
|
4080
4474
|
```
|
|
4081
4475
|
--ts=@json:'{"sshfp":null}'
|
|
4082
4476
|
```
|
|
4083
4477
|
|
|
4084
|
-
|
|
4478
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
|
4479
|
+
|
|
4480
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
|
4085
4481
|
|
|
4086
4482
|
## Miscelaneous
|
|
4087
4483
|
|
|
4088
4484
|
* remove rest and oauth classes and use ruby standard gems:
|
|
4089
4485
|
|
|
4090
|
-
|
|
4091
|
-
|
|
4486
|
+
* oauth
|
|
4487
|
+
* https://github.com/rest-client/rest-client
|
|
4092
4488
|
|
|
4093
4489
|
* use Thor or any standard Ruby CLI manager
|
|
4094
4490
|
|