aspera-cli 4.2.0 → 4.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +749 -353
  3. data/docs/Makefile +4 -4
  4. data/docs/README.erb.md +743 -283
  5. data/docs/doc_tools.rb +58 -0
  6. data/docs/test_env.conf +9 -1
  7. data/examples/aoc.rb +14 -3
  8. data/examples/faspex4.rb +89 -0
  9. data/lib/aspera/aoc.rb +24 -22
  10. data/lib/aspera/cli/main.rb +48 -20
  11. data/lib/aspera/cli/plugin.rb +13 -6
  12. data/lib/aspera/cli/plugins/aoc.rb +117 -78
  13. data/lib/aspera/cli/plugins/config.rb +127 -80
  14. data/lib/aspera/cli/plugins/faspex.rb +112 -63
  15. data/lib/aspera/cli/plugins/faspex5.rb +29 -25
  16. data/lib/aspera/cli/plugins/node.rb +54 -25
  17. data/lib/aspera/cli/plugins/preview.rb +94 -68
  18. data/lib/aspera/cli/plugins/server.rb +16 -5
  19. data/lib/aspera/cli/transfer_agent.rb +92 -72
  20. data/lib/aspera/cli/version.rb +1 -1
  21. data/lib/aspera/command_line_builder.rb +48 -31
  22. data/lib/aspera/cos_node.rb +4 -3
  23. data/lib/aspera/fasp/http_gw.rb +47 -26
  24. data/lib/aspera/fasp/local.rb +31 -24
  25. data/lib/aspera/fasp/manager.rb +3 -0
  26. data/lib/aspera/fasp/node.rb +23 -1
  27. data/lib/aspera/fasp/parameters.rb +72 -89
  28. data/lib/aspera/fasp/parameters.yaml +531 -0
  29. data/lib/aspera/fasp/uri.rb +1 -1
  30. data/lib/aspera/faspex_gw.rb +10 -9
  31. data/lib/aspera/id_generator.rb +22 -0
  32. data/lib/aspera/node.rb +11 -3
  33. data/lib/aspera/oauth.rb +131 -135
  34. data/lib/aspera/persistency_action_once.rb +11 -7
  35. data/lib/aspera/persistency_folder.rb +6 -26
  36. data/lib/aspera/rest.rb +1 -1
  37. data/lib/aspera/sync.rb +40 -35
  38. data/lib/aspera/timer_limiter.rb +22 -0
  39. data/lib/aspera/web_auth.rb +105 -0
  40. metadata +22 -4
  41. data/docs/transfer_spec.html +0 -99
  42. data/lib/aspera/fasp/aoc.rb +0 -24
data/README.md CHANGED
@@ -1,11 +1,11 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
- # `ascli` : Command Line Interface for IBM Aspera products
2
+ <font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
3
3
 
4
- Version : 4.2.0
4
+ Version : 4.4.0
5
5
 
6
6
  _Laurent/2016-2021_
7
7
 
8
- This gem provides `ascli`: a command line interface to Aspera Applications.
8
+ This gem provides the `ascli` Command Line Interface to IBM Aspera software.
9
9
 
10
10
  `ascli` is a also great tool to learn Aspera APIs.
11
11
 
@@ -13,7 +13,7 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
13
13
 
14
14
  Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
15
15
 
16
- Ruby version must be >= > 2.4
16
+ Required Ruby version: > 2.4
17
17
 
18
18
  # <a name="when_to_use"></a>When to use and when not to use
19
19
 
@@ -29,7 +29,8 @@ So it is designed for:
29
29
 
30
30
  `ascli` can be seen as a command line tool integrating:
31
31
 
32
- * a configuration file (config.yaml) and advanced command line options
32
+ * a configuration file (config.yaml)
33
+ * advanced command line options
33
34
  * cURL (for REST calls)
34
35
  * Aspera transfer (ascp)
35
36
 
@@ -56,7 +57,7 @@ Command line parameters in examples beginning with `my_`, like `my_param_value`
56
57
 
57
58
  On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
58
59
 
59
- On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
60
+ On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
60
61
 
61
62
  * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
62
63
  * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
@@ -81,7 +82,7 @@ Once the gem is installed, `ascli` shall be accessible:
81
82
 
82
83
  ```
83
84
  $ ascli --version
84
- 4.2.0
85
+ 4.4.0
85
86
  ```
86
87
 
87
88
  ## First use
@@ -108,7 +109,7 @@ $ ascli server browse /
108
109
  If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
109
110
 
110
111
  * create a [option preset](#lprt)
111
- * define it as default for "server" plugin
112
+ * define it as default for `server` plugin
112
113
  * list files in a folder
113
114
  * download a file
114
115
 
@@ -151,7 +152,7 @@ It is possible to install *either* directly on the host operating system (Linux,
151
152
 
152
153
  The direct installation is recommended and consists in installing:
153
154
 
154
- * [Ruby](#ruby) version >= > 2.4
155
+ * [Ruby](#ruby) version > 2.4
155
156
  * [aspera-cli](#the_gem)
156
157
  * [Aspera SDK (ascp)](#fasp_prot)
157
158
 
@@ -165,6 +166,8 @@ Use this method only if you know what you do, else use the standard recommended
165
166
 
166
167
  This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
167
168
 
169
+ The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
170
+
168
171
  Ensure that you have Docker installed.
169
172
 
170
173
  ```
@@ -199,7 +202,7 @@ Use this method to install on the native host.
199
202
 
200
203
  A ruby interpreter is required to run the tool or to use the gem and tool.
201
204
 
202
- Ruby minimum version: > 2.4. Ruby version 3 is also supported.
205
+ Required Ruby version: > 2.4. Ruby version 3 is also supported.
203
206
 
204
207
  *Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
205
208
 
@@ -207,7 +210,7 @@ Refer to the following sections for a proposed method for specific operating sys
207
210
 
208
211
  The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
209
212
  If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
210
- If you have a simpler better way to install Ruby version >= > 2.4 : use it !
213
+ If you have a simpler better way to install Ruby version > 2.4 : use it !
211
214
 
212
215
  ### Generic: RVM: single user installation (not root)
213
216
 
@@ -233,7 +236,7 @@ If you keep the same terminal (ont needed if re-login):
233
236
  $ source ~/.rvm/scripts/rvm
234
237
  ```
235
238
 
236
- It is advised to get one of the pre-compiled ruby version, you can list with:
239
+ It is advised to get one of the pre-compiled ruby version, you can list with:
237
240
 
238
241
  ```
239
242
  $ rvm list --remote
@@ -429,8 +432,8 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
429
432
  `ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
430
433
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
431
434
 
432
- Several methods are provided on how to start a transfer. Use of a local client is one of them, but
433
- other methods are available. Refer to section: [Transfer Agents](#agents)
435
+ Several methods are provided to start a transfer.
436
+ Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
434
437
 
435
438
  ## <a name="offline_install"></a>Offline Installation (without internet)
436
439
 
@@ -476,7 +479,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
476
479
  * Supports most Aspera server products (on-premise and SaaS)
477
480
  * Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
478
481
  * Supports Commands, Option values and Parameters shortcuts
479
- * FASP [Transfer Agents](#agents) can be: FaspManager (local ascp), or Connect Client, or any transfer node
482
+ * FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
480
483
  * Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
481
484
  * Allows transfers from products to products, essentially at node level (using the node transfer agent)
482
485
  * Supports FaspStream creation (using Node API)
@@ -600,12 +603,12 @@ The style of output can be set using the `format` parameter, supporting:
600
603
  * `yaml` : YAML
601
604
  * `csv` : Comma Separated Values
602
605
 
603
- ### Filtering columns for `object_list`
606
+ ### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
604
607
 
605
608
  Table output can be filtered using the `select` parameter. Example:
606
609
 
607
610
  ```
608
- $ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"per_page":1000,"page":1,"sort":"name"}' --select=@json:'{"ats_admin":true}'
611
+ $ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
609
612
  :...............................:..................................:...........:
610
613
  : name : email : ats_admin :
611
614
  :...............................:..................................:...........:
@@ -752,19 +755,15 @@ C:\Users\Kenji\.aspera\ascli
752
755
  On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
753
756
  Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
754
757
 
755
- Although the file is a standard YAML file, `ascli` provides commands to read and modify it
756
- using the `config` command.
758
+ Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
757
759
 
758
- All options for `ascli` commands can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
760
+ All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
759
761
 
760
- A configuration file provides a way to define default values, especially
761
- for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
762
+ A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
762
763
 
763
- The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
764
- (this can be overriden with option `--config-file=path` or equivalent env var).
764
+ The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
765
765
 
766
- So, finally, the configuration file is simply a catalog of pre-defined lists of options,
767
- called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
766
+ The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
768
767
 
769
768
  ### <a name="lprt"></a>Option preset
770
769
 
@@ -864,7 +863,7 @@ demo_server:
864
863
  We can see here:
865
864
 
866
865
  * The configuration was created with CLI version 0.3.7
867
- * the default [option preset](#lprt) to load for plugin "server" is : `demo_server`
866
+ * the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
868
867
  * the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
869
868
  * the default [option preset](#lprt) to load in any case is : `cli_default`
870
869
 
@@ -895,26 +894,26 @@ my_aoc_org:
895
894
 
896
895
  So, the key file will be read only at execution time, but not be embedded in the configuration file.
897
896
 
897
+ ### Options evaluation order
898
+
899
+ Some options are global, some options are available only for some plugins. (the plugin is the first level command).
900
+
898
901
  Options are loaded using this algorithm:
899
902
 
900
- * if option '--preset=xxxx' is specified (or -Pxxxx), this reads the [option preset](#lprt) specified from the configuration file.
901
- * else if option --no-default (or -N) is specified, then dont load default
902
- * else it looks for the name of the default [option preset](#lprt) in section "default" and loads it
903
- * environment variables are evaluated
904
- * command line options are evaluated
903
+ * If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
904
+ * else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
905
+ * If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
906
+ * Environment variables are evaluated
907
+ * Command line options are evaluated
905
908
 
906
909
  Parameters are evaluated in the order of command line.
907
910
 
908
- To avoid loading the default [option preset](#lprt) for a plugin, just specify a non existing configuration: `-Pnone`
911
+ To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
909
912
 
910
913
  On command line, words in parameter names are separated by a dash, in configuration file, separator
911
914
  is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
912
915
 
913
- Note: before version 0.4.5, some keys could be ruby symbols, from 0.4.5 all keys are strings. To
914
- convert olver versions, remove the leading ":" in front of keys.
915
-
916
- The main plugin name is *config*, so it is possible to define a default [option preset](#lprt) for
917
- the main plugin with:
916
+ The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
918
917
 
919
918
  ```
920
919
  $ ascli config id cli_default set interactive no
@@ -927,6 +926,18 @@ A [option preset](#lprt) value can be removed with `unset`:
927
926
  $ ascli config id cli_default unset interactive
928
927
  ```
929
928
 
929
+ Example: Define options using command line:
930
+
931
+ ```
932
+ $ ascli -N --url=x --password=y --username=y node --show-config
933
+ ```
934
+
935
+ Example: Define options using a hash:
936
+
937
+ ```
938
+ $ ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
939
+ ```
940
+
930
941
  ### Examples
931
942
 
932
943
  For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
@@ -1055,7 +1066,7 @@ $ ascli config ascp info
1055
1066
  ...
1056
1067
  ```
1057
1068
 
1058
- ### Selection of local `ascp`
1069
+ ### Selection of `ascp` location for [`direct`](#direct) agent
1059
1070
 
1060
1071
  By default, `ascli` uses any found local product with ascp, including SDK.
1061
1072
 
@@ -1099,7 +1110,7 @@ $ ascli config ascp products list
1099
1110
  :.........................................:................................................:
1100
1111
  ```
1101
1112
 
1102
- ### Selection of local client
1113
+ ### Selection of local client for `ascp` for [`direct`](#direct) agent
1103
1114
 
1104
1115
  If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
1105
1116
 
@@ -1154,7 +1165,7 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
1154
1165
 
1155
1166
  There are currently 3 agents:
1156
1167
 
1157
- * `direct` : a local execution of `ascp`
1168
+ * [`direct`](#direct) : a local execution of `ascp`
1158
1169
  * `connect` : use of a local Connect Client
1159
1170
  * `node` : use of an Aspera Transfer Node (potentially _remote_).
1160
1171
  * `httpgw` : use of an Aspera HTTP Gateway
@@ -1166,7 +1177,7 @@ will effectively push files to the related server from the agent node.
1166
1177
  `ascli` standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1167
1178
 
1168
1179
 
1169
- ### <a name="direct"></a>Direct (local ascp using FASPManager API)
1180
+ ### <a name="direct"></a>Direct (local ascp execution)
1170
1181
 
1171
1182
  By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
1172
1183
  `ascli` will detect locally installed Aspera products.
@@ -1185,11 +1196,19 @@ The `transfer-info` accepts the following optional parameters:
1185
1196
  <tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
1186
1197
  <tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
1187
1198
  <tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1199
+ <tr><td>multi_incr_udp</td><td>Bool</td><td>true</td><td>Multi Session</td><td>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)</td></tr>
1188
1200
  <tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
1189
1201
  </table>
1190
1202
 
1191
1203
  Resume parameters:
1192
1204
 
1205
+ In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1206
+ Sleep between iteration is:
1207
+
1208
+ ```
1209
+ max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
1210
+ ```
1211
+
1193
1212
  <table>
1194
1213
  <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1195
1214
  <tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
@@ -1202,7 +1221,7 @@ Examples:
1202
1221
 
1203
1222
  ```
1204
1223
  $ ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1205
- $ ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
1224
+ $ ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1206
1225
  ```
1207
1226
 
1208
1227
  ### IBM Aspera Connect Client GUI
@@ -1256,7 +1275,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
1256
1275
 
1257
1276
  If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1258
1277
 
1259
- It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1278
+ It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1260
1279
 
1261
1280
  The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1262
1281
 
@@ -1267,97 +1286,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
1267
1286
 
1268
1287
  ## <a name="transferparams"></a>Transfer Parameters
1269
1288
 
1270
- All standard _transfer-spec_ parameters can be overloaded. To display parameters,
1271
- run in debug mode (--log-level=debug). [_transfer-spec_](#transferspec) can
1272
- also be saved/overridden in the config file.
1289
+ All standard _transfer-spec_ parameters can be speficied.
1290
+ [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1273
1291
 
1292
+ References:
1293
+
1294
+ * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1295
+ * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1296
+
1297
+ Parameters can be displayed with commands:
1298
+
1299
+ ```
1300
+ $ ascli config ascp spec
1301
+ $ ascli config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
1302
+ ```
1274
1303
 
1275
- <p>
1276
1304
  Columns:
1277
- <ul>
1278
- <li>F=Fasp Manager(local FASP execution)</li>
1279
- <li>N=remote node(node API)</li>
1280
- <li>C=Connect Client(web plugin)</li>
1281
- </ul>
1282
- </p>
1283
- <p>
1284
- Req/Def : Required or default value (- means emty)
1285
- </p>
1286
- <p>
1287
- Fields with EX_ prefix are specific extensions to local mode.
1288
- </p>
1289
- <p>
1290
- arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)
1291
- </p>
1292
- <p>
1293
- UNDER CONSTRUCTION<br/>
1294
- <a href="https://developer.ibm.com/apis/catalog/?search=aspera">Aspera API Documentation</a>&rarr;Node API&rarr;/opt/transfers<br/>
1295
- </p>
1296
1305
 
1297
- <table>
1298
- <tr><th>Field</th><th>Req/Def</th><th>Type</th><th>F</th><th>N</th><th>C</th><th>arg</th><th>Description</th></tr>
1299
- <tr><td>direction</td><td>Required</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--mode</td><td>Direction: "send" or "receive"</td></tr>
1300
- <tr><td>remote_host</td><td>Required</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--host</td><td>IP or fully qualified domain name of the remote server</td></tr>
1301
- <tr><td>remote_user</td><td>Required</td><td>string</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--user</td><td>Remote user. Default value is "xfer" on node or connect.</td></tr>
1302
- <tr><td>destination_root</td><td>Required</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>last arg</td><td>Destination root directory.</td></tr>
1303
- <tr><td>title</td><td>-</td><td>string</td><td class="no">N</td><td class="yes">Y</td><td class="yes">Y</td><td>-</td><td>Title of the transfer</td></tr>
1304
- <tr><td>tags</td><td>-</td><td>hash</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--tags<br>--tags64</td><td>Metadata for transfer</td></tr>
1305
- <tr><td>token</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>TOKEN<br/>-W</td><td>Authorization token: Bearer, Basic or ATM</td></tr>
1306
- <tr><td>cookie</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>COOKIE</td><td>Metadata for transfer (older,string)</td></tr>
1307
- <tr><td>remote_access_key</td><td>TODO</td><td>string</td><td></td><td></td><td></td><td>?</td><td>Node only?</td></tr>
1308
- <tr><td>source_root</td><td>-</td><td>string</td><td></td><td></td><td></td><td>--source-prefix<br/>--source-prefix64</td><td>Source root directory.(TODO: verify option)</td></tr>
1309
- <tr><td>fasp_port</td><td>33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-O</td><td>Specifies fasp (UDP) port.</td></tr>
1310
- <tr><td>ssh_port</td><td>22 or 33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-P</td><td>Specifies ssh (TCP) port.</td></tr>
1311
- <tr><td>rate_policy</td><td>server config</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--policy</td><td>Valid literals include "low","fair","high" and "fixed".</td></tr>
1312
- <tr><td>symlink_policy</td><td>follow</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--symbolic-links</td><td>copy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).</td></tr>
1313
- <tr><td>target_rate_kbps</td><td>-</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-l</td><td>Specifies desired speed for the transfer.</td></tr>
1314
- <tr><td>min_rate_kbps</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-m</td><td>Set the minimum transfer rate in kilobits per second.</td></tr>
1315
- <tr><td>cipher</td><td>none</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-c</td><td>in transit encryption type.<br/>none, aes-128, aes-256</td></tr>
1316
- <tr><td>content_protection</td><td>encrypt<br/>decrypt</td><td>string</td><td></td><td></td><td></td><td>--file-crypt=</td><td>encryption at rest</td></tr>
1317
- <tr><td>content_protection_password</td><td>-</td><td>string</td><td></td><td></td><td></td><td>PASS</td><td>Specifies a string password.</td></tr>
1318
- <tr><td>overwrite</td><td>diff</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--overwrite</td><td>Overwrite destination files with the source files of the same name.<br/>never, always, diff, older, or diff+older</td></tr>
1319
- <tr><td>retry_duration</td><td></td><td>string</td><td></td><td></td><td></td><td>TODO</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr>
1320
- <tr><td>http_fallback</td><td></td><td>bool (node), integer</td><td></td><td></td><td></td><td>-y<br/>TODO</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.</td></tr>
1321
- <tr><td>create_dir</td><td></td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-d</td><td>Specifies whether to create new directories.</td></tr>
1322
- <tr><td>precalculate_job_size</td><td>srv. def.</td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--precalculate-job-size</td><td>Specifies whether to precalculate the job size.</td></tr>
1323
- <tr><td>delete_source</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>?</td></tr>
1324
- <tr><td>remove_after_transfer</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove file after transfer.</td></tr>
1325
- <tr><td>remove_empty_directories</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove empty directories.</td></tr>
1326
- <tr><td>multi_session</td><td>1</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-C</td><td>Specifies how many parts the transfer is in.</td></tr>
1327
- <tr><td>multi_session_threshold</td><td>null</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-</td><td>in bytes</td></tr>
1328
- <tr><td>exclude_newer_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-newer-than</td><td>-</td></tr>
1329
- <tr><td>exclude_older_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-older-than</td><td>-</td></tr>
1330
- <tr><td>preserve_acls</td><td></td><td>string</td><td class="yes">Y</td><td></td><td></td><td>--preserve-acls</td><td>-</td></tr>
1331
- <tr><td>dgram_size</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-Z</td><td>in bytes</td></tr>
1332
- <tr><td>compression</td><td></td><td>integer</td><td></td><td></td><td></td><td></td><td>ascp4 only, 0 / 1?</td></tr>
1333
- <tr><td>read_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
1334
- <tr><td>write_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
1335
- <tr><td>use_ascp4</td><td>false</td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>-</td><td>specify version of protocol</td></tr>
1336
- <tr><td>paths</td><td>source files (dest)</td><td>array</td><td></td><td></td><td></td><td>positional<br/>--file-list<br/>--file-pair-list</td><td>Contains a path to the source (required) and a path to the destination.</td></tr>
1337
- <tr><td>http_fallback_port</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-t</td><td>Specifies http port.</td></tr>
1338
- <tr><td>https_fallback_port</td><td></td><td>integer</td><td></td><td></td><td></td><td>todo</td><td>Specifies https port.</td></tr>
1339
- <tr><td>cipher_allowed</td><td></td><td>string</td><td></td><td></td><td></td><td>-</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr>
1340
- <tr><td>target_rate_cap_kbps</td><td></td><td></td><td class="no">N</td><td class="no">?</td><td class="yes">?</td><td>-</td><td>Returned by upload/download_setup node api.</td></tr>
1341
- <tr><td>rate_policy_allowed</td><td></td><td></td><td></td><td></td><td></td><td>-</td><td>returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".</td></tr>
1342
- <tr><td>ssh_private_key</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>KEY</td><td>Private key used for SSH authentication, Shall look like: `-----BEGIN RSA PRIVATE KEY-----\nMII`<br/>Note the JSON encoding `\` + `n` for newlines.</td></tr>
1343
- <tr><td>remote_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>PASS</td><td>SSH session password</td></tr>
1344
- <tr><td>resume_policy</td><td>faspmgr:<br/>none<br/>other:<br/>sparse_csum</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-k</td><td>none,attrs,sparse_csum,full_csum</td></tr>
1345
- <tr><td>authentication</td><td>-</td><td class="no">N</td><td class="no">N</td><td class="yes">Y</td><td>-</td><td>token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.</td></tr>
1346
- <tr><td>EX_ssh_key_paths</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-i</td><td>Use public key authentication and specify the private key file</td></tr>
1347
- <tr><td>EX_at_rest_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>FILEPASS</td><td>Passphrase used for at rest encryption or decryption</td></tr>
1348
- <tr><td>EX_proxy_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>PROXY_PASS</td><td>TODO</td></tr>
1349
- <tr><td>EX_fasp_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--proxy</td><td>Specify the address of the Aspera high-speed proxy server</td></tr>
1350
- <tr><td>EX_http_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-x</td><td>Specify the proxy server address used by HTTP Fallback</td></tr>
1351
- <tr><td>EX_ascp_args</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>same</td><td>Add command line arguments to ascp</td></tr>
1352
- <tr><td>EX_http_transfer_jpeg</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-j</td><td>HTTP transfers as JPEG file</td></tr>
1353
- <tr><td>EX_license_text</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>LICENSE</td><td>license file text</td></tr>
1354
- <tr><td>EX_file_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-list</td><td>source file list</td></tr>
1355
- <tr><td>EX_file_pair_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-pair-list</td><td>source file pair list</td></tr>
1356
- <tr><td>EX_multi_session_part</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-C</td><td>part for multisession</td></tr>
1357
- <tr><td>EX_no_read</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-read</td><td>no read source</td></tr>
1358
- <tr><td>EX_no_write</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-write</td><td>no write estination</td></tr>
1359
- </table>
1306
+ * D=Direct (local `ascp` execution)
1307
+ * N=Node API
1308
+ * C=Connect Client
1309
+ * arg=`ascp` argument or environment variable
1310
+
1311
+ Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in `ascli`).
1360
1312
 
1313
+ <table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td>&nbsp;</td><td>Y</td><td>&nbsp;</td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td>&nbsp;</td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td>&nbsp;</td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>rate_policy_allowed</td><td>string</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>authentication</td><td>string</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr></table>
1361
1314
 
1362
1315
  ### Destination folder for transfers
1363
1316
 
@@ -1373,8 +1326,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
1373
1326
 
1374
1327
  ### List of files for transfers
1375
1328
 
1376
- When uploading, downloading or sending files, the user must specify the list of files to transfer.
1377
- Most of the time, the list of files to transfer will be simply specified on the command line:
1329
+ When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1330
+ So, by default, the list of files to transfer will be simply specified on the command line:
1378
1331
 
1379
1332
  ```
1380
1333
  $ ascli server upload ~/mysample.file secondfile
@@ -1386,7 +1339,7 @@ This is equivalent to:
1386
1339
  $ ascli server upload --sources=@args ~/mysample.file secondfile
1387
1340
  ```
1388
1341
 
1389
- More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1342
+ More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1390
1343
 
1391
1344
  Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
1392
1345
  all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
@@ -1409,13 +1362,15 @@ For ease of use and flexibility, the list of files to transfer is specified by t
1409
1362
  --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
1410
1363
  ```
1411
1364
 
1412
- * Although not recommended, because it applies *only* to the `local` transfer agent (i.e. bare ascp), it is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`. In that case, one must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided.
1365
+ * Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
1413
1366
 
1414
1367
  ```
1415
1368
  --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
1416
1369
  ```
1417
1370
 
1418
- In case the file list is provided on the command line (i.e. using `--sources=@args` or `--sources=<Array>`, but not `--sources=@ts`), the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1371
+ This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1372
+
1373
+ In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1419
1374
 
1420
1375
  * `list` : (default) the path of destination is the same as source
1421
1376
  * `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
@@ -1426,6 +1381,8 @@ Example:
1426
1381
  $ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1427
1382
  ```
1428
1383
 
1384
+ Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
1385
+
1429
1386
  Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
1430
1387
 
1431
1388
  * All files must be in the same source folder.
@@ -1439,7 +1396,7 @@ Source files are located on "Aspera on cloud", when :
1439
1396
 
1440
1397
  ### <a name="multisession"></a>Support of multi-session
1441
1398
 
1442
- Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
1399
+ Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1443
1400
 
1444
1401
  * when agent=node :
1445
1402
 
@@ -1460,6 +1417,7 @@ shall be preferred.
1460
1417
 
1461
1418
  Multi-session spawn is done by `ascli`.
1462
1419
 
1420
+ When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
1463
1421
 
1464
1422
  ### Examples
1465
1423
 
@@ -1489,15 +1447,154 @@ Multi-session spawn is done by `ascli`.
1489
1447
 
1490
1448
 
1491
1449
 
1492
- ## <a name="scheduling"></a>Scheduling an exclusive execution
1450
+ ## <a name="scheduling"></a>Lock for exclusive execution
1451
+
1452
+ In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
1453
+
1454
+ For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
1455
+
1456
+ * Executing instances may pile-up and kill the system
1457
+ * The same file may be transfered by multiple instances at the same time.
1458
+ * `preview` may generate the same files in multiple instances.
1459
+
1460
+ Usually the OS native scheduler already provides some sort of protection against parallel execution:
1461
+
1462
+ * The Windows scheduler does this by default
1463
+ * Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
1464
+
1465
+ ```
1466
+ /usr/bin/flock -w 0 /var/cron.lock ascli ...
1467
+ ```
1468
+
1469
+ `ascli` natively supports a locking mechanism with option `lock_port`.
1470
+ (Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
1471
+
1472
+ Example:
1473
+
1474
+ Run this same command in two separate terminals within less than 30 seconds:
1475
+
1476
+ ```
1477
+ ascli config echo @ruby:'sleep(30)' --lock-port=12345
1478
+ ```
1479
+
1480
+ The first instance will sleep 30 seconds, the second one will immediately exit like this:
1481
+
1482
+ ```
1483
+ WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
1484
+ ```
1485
+
1486
+ ## "Proven&ccedil;ale"
1487
+
1488
+ `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
1489
+ , some are provided om shared libraries and must be activated. (e.g. using `trapd`)
1490
+
1491
+ The list of supported *PVCL* adapters can be retried with command:
1492
+
1493
+ ```
1494
+ $ ascli conf ascp info
1495
+ +--------------------+-----------------------------------------------------------+
1496
+ | key | value |
1497
+ +--------------------+-----------------------------------------------------------+
1498
+ -----8<----snip---------
1499
+ | product_name | IBM Aspera SDK |
1500
+ | product_version | 4.0.1.182389 |
1501
+ | process | pvcl |
1502
+ | shares | pvcl |
1503
+ | noded | pvcl |
1504
+ | faux | pvcl |
1505
+ | file | pvcl |
1506
+ | stdio | pvcl |
1507
+ | stdio-tar | pvcl |
1508
+ +--------------------+-----------------------------------------------------------+
1509
+ ```
1510
+
1511
+ Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
1512
+
1513
+ Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
1514
+
1515
+ The simplified format is:
1516
+
1517
+ ```
1518
+ <adapter>:///<sub file path>?<arg1>=<val1>&...
1519
+ ```
1520
+
1521
+ One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
1522
+
1523
+ ## <a name="faux_testing"></a>`faux:` for testing
1524
+
1525
+ This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`
1493
1526
 
1494
- It is possible to ensure that a given command is only run once at a time with parameter: `--lock-port=nnnn`. This is especially usefull when scheduling a command on a regular basis, for instance involving transfers, and a transfer may last longer than the execution period.
1527
+ This adapter can be used to simulate a file or a directory.
1495
1528
 
1496
- This opens a local TCP server port, and fails if this port is already used, providing a local lock.
1529
+ To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
1497
1530
 
1498
- This option is used when the tools is executed automatically, for instance with "preview" generation.
1531
+ * `fname` is the name that will be assigned to the file on the destination
1532
+ * `fsize` is the number of bytes that will be sent (in decimal).
1499
1533
 
1500
- Usually the OS native scheduler shall already provide some sort of such protection (windows scheduler has it natively, linux cron can leverage `flock`).
1534
+ Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
1535
+
1536
+ For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
1537
+
1538
+ To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
1539
+
1540
+ ```
1541
+ faux:///dirname?<arg1>=<val1>&...
1542
+ ```
1543
+
1544
+ `dirname` is the folder name and can contain `/` to specify a subfolder.
1545
+
1546
+ Supported arguments are:
1547
+
1548
+ <table>
1549
+ <tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
1550
+ <tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
1551
+ <tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
1552
+ <tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
1553
+ <tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
1554
+ <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
1555
+ <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
1556
+ </table>
1557
+
1558
+
1559
+ The sequence parameter is applied as follows:
1560
+
1561
+ * If `seq` is `random` then each file size is:
1562
+
1563
+ * size +/- (inc * rand())
1564
+ * Where rand is a random number between 0 and 1
1565
+ * Note that file size must not be negative, inc will be set to size if it is greater than size
1566
+ * Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
1567
+
1568
+ * If `seq` is `sequential` then each file size is:
1569
+
1570
+ * size + ((fileindex - 1) * inc)
1571
+ * Where first file is index 1
1572
+ * So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
1573
+ * As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
1574
+
1575
+ Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
1576
+
1577
+ To discard data at the destination, the destination argument is set to `faux://` .
1578
+
1579
+ Examples:
1580
+
1581
+ * Upload 20 gigabytes of random data to file myfile to directory /Upload
1582
+
1583
+ ```
1584
+ $ ascli server upload faux:///myfile\?20g --to-folder=/Upload
1585
+ ```
1586
+
1587
+ * Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
1588
+
1589
+ ```
1590
+ $ ascli server upload /tmp/sample --to-folder=faux://
1591
+ ```
1592
+
1593
+ * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
1594
+
1595
+ ```
1596
+ $ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1597
+ ```
1501
1598
 
1502
1599
  ## <a name="commands"></a>Sample Commands
1503
1600
 
@@ -1508,7 +1605,7 @@ ascli
1508
1605
  ascli -h
1509
1606
  ascli aoc -N remind --username=my_aoc_user_email
1510
1607
  ascli aoc -N servers
1511
- ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}'
1608
+ ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
1512
1609
  ascli aoc admin ats access_key --id=akibmcloud --secret=somesecret node browse /
1513
1610
  ascli aoc admin ats access_key --id=akibmcloud delete
1514
1611
  ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
@@ -1518,7 +1615,7 @@ ascli aoc admin ats cluster clouds
1518
1615
  ascli aoc admin ats cluster list
1519
1616
  ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
1520
1617
  ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
1521
- ascli aoc admin res apps_new list
1618
+ ascli aoc admin res application list
1522
1619
  ascli aoc admin res client list
1523
1620
  ascli aoc admin res client_access_key list
1524
1621
  ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
@@ -1555,6 +1652,7 @@ ascli aoc automation workflow list --value=@json:'{"show_org_workflows":"true"}'
1555
1652
  ascli aoc bearer_token --display=data --scope=user:all
1556
1653
  ascli aoc faspex
1557
1654
  ascli aoc files bearer /
1655
+ ascli aoc files bearer_token_node /
1558
1656
  ascli aoc files browse /
1559
1657
  ascli aoc files browse / -N --link=my_aoc_publink_folder
1560
1658
  ascli aoc files delete /testsrc
@@ -1570,11 +1668,12 @@ ascli aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_li
1570
1668
  ascli aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
1571
1669
  ascli aoc files upload --to-folder=/testsrc testfile.bin
1572
1670
  ascli aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
1671
+ ascli aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
1573
1672
  ascli aoc files v3 info
1574
1673
  ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
1575
1674
  ascli aoc organization
1576
1675
  ascli aoc packages list
1577
- ascli aoc packages list --format=csv --fields=id --display=data|head -n 1);\
1676
+ ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
1578
1677
  ascli aoc packages recv --id="my_package_id" --to-folder=.
1579
1678
  ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
1580
1679
  ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
@@ -1584,6 +1683,8 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
1584
1683
  ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
1585
1684
  ascli aoc user info modify @json:'{"name":"dummy change"}'
1586
1685
  ascli aoc user info show
1686
+ ascli aoc user shared_inboxes
1687
+ ascli aoc user workspaces
1587
1688
  ascli aoc workspace
1588
1689
  ascli ats access_key --id=ak_aws delete
1589
1690
  ascli ats access_key --id=akibmcloud --secret=somesecret cluster
@@ -1610,10 +1711,11 @@ ascli config ascp info
1610
1711
  ascli config ascp install
1611
1712
  ascli config ascp products list
1612
1713
  ascli config ascp show
1714
+ ascli config ascp spec
1613
1715
  ascli config check_update
1614
1716
  ascli config doc
1615
1717
  ascli config doc transfer-parameters
1616
- ascli config email_test aspera.user1@gmail.com
1718
+ ascli config email_test --notif-to=my_recipient_email
1617
1719
  ascli config export
1618
1720
  ascli config genkey mykey
1619
1721
  ascli config plugins
@@ -1709,7 +1811,7 @@ ascli server mkdir folder_1/target_hot
1709
1811
  ascli server mv folder_1/200KB.2 folder_1/to.delete
1710
1812
  ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
1711
1813
  ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
1712
- ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername
1814
+ ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
1713
1815
  ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
1714
1816
  ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
1715
1817
  ascli shares admin share list
@@ -1734,7 +1836,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
1734
1836
  ```
1735
1837
  $ ascli -h
1736
1838
  NAME
1737
- ascli -- a command line tool for Aspera Applications (v4.2.0)
1839
+ ascli -- a command line tool for Aspera Applications (v4.4.0)
1738
1840
 
1739
1841
  SYNOPSIS
1740
1842
  ascli COMMANDS [OPTIONS] [ARGS]
@@ -1791,29 +1893,31 @@ OPTIONS:
1791
1893
  --property=VALUE name of property to set
1792
1894
  --id=VALUE resource identifier (modify,delete,show)
1793
1895
  --config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
1794
- --override=ENUM override existing value: yes, no
1795
1896
  -N, --no-default do not load default configuration for plugin
1796
- --use-generic-client=ENUM wizard: AoC: use global or org specific jwt client id: yes, no
1797
- --pkeypath=VALUE path to private key for JWT (wizard)
1897
+ --override=ENUM Wizard: override existing value: yes, no
1898
+ --use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
1899
+ --default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
1900
+ --test-mode=ENUM Wizard: skip private key check step: yes, no
1901
+ --pkeypath=VALUE Wizard: path to private key for JWT
1798
1902
  --ascp-path=VALUE path to ascp
1799
1903
  --use-product=VALUE use ascp from specified product
1800
1904
  --smtp=VALUE smtp configuration (extended value: hash)
1801
1905
  --fpac=VALUE proxy auto configuration URL
1802
1906
  -P, --presetVALUE load the named option preset from current config file
1803
- --default=VALUE set as default configuration for specified plugin
1804
1907
  --secret=VALUE default secret
1805
1908
  --secrets=VALUE secret repository (Hash)
1806
1909
  --sdk-url=VALUE URL to get SDK
1807
- --sdk-folder=VALUE SDK folder location
1808
- --test-mode=ENUM skip user validation in wizard mode: yes, no
1809
- --version-check-days=VALUE period to check neew version in days (zero to disable)
1910
+ --sdk-folder=VALUE SDK folder path
1911
+ --notif-to=VALUE email recipient for notification of transfers
1912
+ --notif-template=VALUE email ERB template for notification of transfers
1913
+ --version-check-days=VALUE period in days to check new version (zero to disable)
1810
1914
  --ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
1811
1915
  --local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
1812
1916
  --to-folder=VALUE destination folder for downloaded files
1813
1917
  --sources=VALUE list of source files (see doc)
1814
- --transfer-info=VALUE additional information for transfer client
1918
+ --transfer-info=VALUE parameters for transfer agent
1815
1919
  --src-type=ENUM type of file list: list, pair
1816
- --transfer=ENUM type of transfer: direct, httpgw, connect, node, aoc
1920
+ --transfer=ENUM type of transfer agent: direct, httpgw, connect, node
1817
1921
  --progress=ENUM type of progress bar: none, native, multi
1818
1922
 
1819
1923
 
@@ -1834,7 +1938,7 @@ OPTIONS:
1834
1938
  --validator=VALUE identifier of validator (optional for central)
1835
1939
  --asperabrowserurl=VALUE URL for simple aspera web ui
1836
1940
  --name=VALUE sync name
1837
- --token=ENUM todo: type of token used for transfers: aspera, basic, auto
1941
+ --token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
1838
1942
 
1839
1943
 
1840
1944
  COMMAND: orchestrator
@@ -1879,7 +1983,7 @@ OPTIONS:
1879
1983
 
1880
1984
 
1881
1985
  COMMAND: faspex5
1882
- SUBCOMMANDS: node package auth_client
1986
+ SUBCOMMANDS: node package auth_client jobs
1883
1987
  OPTIONS:
1884
1988
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1885
1989
  --username=VALUE username to log in
@@ -1912,7 +2016,8 @@ OPTIONS:
1912
2016
  --delivery-info=VALUE package delivery information (extended value)
1913
2017
  --source-name=VALUE create package from remote source (by name)
1914
2018
  --storage=VALUE Faspex local storage definition
1915
- --box=ENUM package box: inbox, sent, archive
2019
+ --recipient=VALUE use if recipient is a dropbox (with *)
2020
+ --box=ENUM package box: inbox, archive, sent
1916
2021
 
1917
2022
 
1918
2023
  COMMAND: shares2
@@ -1986,7 +2091,6 @@ OPTIONS:
1986
2091
  --new-user-option=VALUE new user creation option
1987
2092
  --from-folder=VALUE share to share source folder
1988
2093
  --scope=VALUE OAuth scope for AoC API calls
1989
- --notify=VALUE notify users that file was received
1990
2094
  --bulk=ENUM bulk operation: yes, no
1991
2095
  --default-ports=ENUM use standard FASP ports or get from node api: yes, no
1992
2096
 
@@ -2067,7 +2171,7 @@ For this, specify the option: `--use-generic-client=no`.
2067
2171
 
2068
2172
  This will guide you through the steps to create.
2069
2173
 
2070
- ## <a name="aocwizard"></a>Configuration: using manual setup
2174
+ ## <a name="aocmanual"></a>Configuration: using manual setup
2071
2175
 
2072
2176
  If you used the wizard (recommended): skip this section.
2073
2177
 
@@ -2233,7 +2337,7 @@ To activate default use of JWT authentication for `ascli` using the [option pres
2233
2337
 
2234
2338
  * change auth method to JWT
2235
2339
  * provide location of private key
2236
- * provide username to login as (OAuthg "subject")
2340
+ * provide username to login as (OAuth "subject")
2237
2341
 
2238
2342
  Execute:
2239
2343
 
@@ -2265,19 +2369,55 @@ It allows actions (create, update, delete) on "resources": users, group, nodes,
2265
2369
 
2266
2370
  Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2267
2371
 
2268
- To get more resources when doing request add:
2372
+ #### Listing resources
2373
+
2374
+ The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
2375
+
2376
+ The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
2377
+
2378
+ The following parameters are supported:
2379
+
2380
+ * `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
2381
+ * `sort`: name of fields to sort results, prefix with `-` for reverse order.
2382
+ * `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
2383
+ * `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
2384
+ * `page` : native api parameter, in general do not use (added by
2385
+ * `per_page` : native api parameter, number of items par api call, in general do not use
2386
+ * Other specific parameters depending on resource type.
2387
+
2388
+ Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
2389
+
2390
+ Other parameters are directly sent as parameters to the GET request on API.
2391
+
2392
+ `page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
2393
+
2394
+ `q` and `sort` are available on most resrouce types.
2395
+
2396
+ Other parameters depend on the type of entity (refer to AoC API).
2397
+
2398
+ Examples:
2399
+
2400
+ * List users with `laurent` in name:
2401
+
2402
+ ```
2403
+ ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
2404
+ ```
2405
+
2406
+ * List users who logded-in before a date:
2269
2407
 
2270
2408
  ```
2271
- --query=@json:'{"per_page":10000}'
2409
+ ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2272
2410
  ```
2273
2411
 
2274
- other query parameters can be used:
2412
+ * List external users and sort in reverse alphabetical order using name:
2413
+
2275
2414
  ```
2276
- --query=@json:'{"member_of_any_workspace":true}'
2277
- --query=@json:'{"q":"laurent"}'
2415
+ ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
2278
2416
  ```
2279
2417
 
2280
- Refer to the AoC API for full list of query parameters.
2418
+ Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
2419
+
2420
+ Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
2281
2421
 
2282
2422
  #### Access Key secrets
2283
2423
 
@@ -2331,10 +2471,10 @@ $ ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,em
2331
2471
  : 98398 : dummyuser1@example.com :
2332
2472
  : 98399 : dummyuser2@example.com :
2333
2473
  :.......:........................:
2334
- $ thelist=$(echo $(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email --field=id --format=csv)|tr ' ' ,)
2474
+ $ thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
2335
2475
  $ echo $thelist
2336
- 98398,98399
2337
- $ ascli aoc admin res user --bulk=yes --id=@json:[$thelist] delete
2476
+ ["113501","354061"]
2477
+ $ ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
2338
2478
  :.......:.........:
2339
2479
  : id : status :
2340
2480
  :.......:.........:
@@ -2343,6 +2483,14 @@ $ ascli aoc admin res user --bulk=yes --id=@json:[$thelist] delete
2343
2483
  :.......:.........:
2344
2484
  ```
2345
2485
 
2486
+ * <a name="deactuser"></a>Find deactivated users since more than 2 years
2487
+
2488
+ ```
2489
+ ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
2490
+ ```
2491
+
2492
+ To delete them use the same method as before
2493
+
2346
2494
  * Display current user's workspaces
2347
2495
 
2348
2496
  ```
@@ -2370,17 +2518,20 @@ $ ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_k
2370
2518
  $ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2371
2519
  ```
2372
2520
 
2373
- # page=1&per_page=10&q=type:(file_upload+OR+file_delete+OR+file_download+OR+file_rename+OR+folder_create+OR+folder_delete+OR+folder_share+OR+folder_share_via_public_link)&sort=-date
2374
- #events=@api_files.read('events',{'q'=>'type:(file_upload OR file_download)'})[:data]
2375
- # can add filters: tag=aspera.files.package_id%3DLA8OU3p8w
2376
- #'tag'=>'aspera.files.package_id%3DJvbl0w-5A'
2521
+ Examples of query (TODO: cleanup):
2522
+
2523
+ ```
2524
+ {"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
2525
+
2526
+ {"tag":"aspera.files.package_id=LA8OU3p8w"}
2527
+
2377
2528
  # filter= 'id', 'short_summary', or 'summary'
2378
2529
  # count=nnn
2379
2530
  # tag=x.y.z%3Dvalue
2380
2531
  # iteration_token=nnn
2381
2532
  # after_time=2016-05-01T23:53:09Z
2382
2533
  # active_only=true|false
2383
-
2534
+ ```
2384
2535
 
2385
2536
  * Display node events (events)
2386
2537
 
@@ -2391,7 +2542,7 @@ $ ascli aoc admin res node --secret=_secret_ v3 events
2391
2542
  * display members of a workspace
2392
2543
 
2393
2544
  ```
2394
- $ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2545
+ $ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2395
2546
  :.............:.........:..................................:
2396
2547
  : member_type : manager : member.email :
2397
2548
  :.............:.........:..................................:
@@ -2412,27 +2563,27 @@ other query parameters:
2412
2563
 
2413
2564
  * <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
2414
2565
 
2415
- a- get id of first workspace
2566
+ a- Get id of first workspace
2416
2567
 
2417
2568
  ```
2418
2569
  WS1='First Workspace'
2419
2570
  WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
2420
2571
  ```
2421
2572
 
2422
- b- get id of second workspace
2573
+ b- Get id of second workspace
2423
2574
 
2424
2575
  ```
2425
2576
  WS2='Second Workspace'
2426
2577
  WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
2427
2578
  ```
2428
2579
 
2429
- c- extract membership information and change workspace id
2580
+ c- Extract membership information
2430
2581
 
2431
2582
  ```
2432
- $ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"per_page":10000,"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2583
+ $ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2433
2584
  ```
2434
2585
 
2435
- d- convert to creation data for second workspace:
2586
+ d- Convert to creation data for second workspace:
2436
2587
 
2437
2588
  ```
2438
2589
  grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
@@ -2444,16 +2595,16 @@ or, using jq:
2444
2595
  jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
2445
2596
  ```
2446
2597
 
2447
- e- add members to second workspace
2598
+ e- Add members to second workspace
2448
2599
 
2449
2600
  ```
2450
2601
  $ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2451
2602
  ```
2452
2603
 
2453
- * get users who did not log since a date
2604
+ * Get users who did not log since a date
2454
2605
 
2455
2606
  ```
2456
- $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,"q":"last_login_at:<2018-05-28"}'
2607
+ $ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2457
2608
  :...............................:
2458
2609
  : email :
2459
2610
  :...............................:
@@ -2462,10 +2613,10 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
2462
2613
  :...............................:
2463
2614
  ```
2464
2615
 
2465
- * list "Limited" users
2616
+ * List "Limited" users
2466
2617
 
2467
2618
  ```
2468
- $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000}' --select=@json:'{"member_of_any_workspace":false}'
2619
+ $ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
2469
2620
  ```
2470
2621
 
2471
2622
  * Perform a multi Gbps transfer between two remote shared folders
@@ -2508,6 +2659,7 @@ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='
2508
2659
  ```
2509
2660
 
2510
2661
  * create registration key to register a node
2662
+
2511
2663
  ```
2512
2664
  $ ascli aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2513
2665
  jfqslfdjlfdjfhdjklqfhdkl
@@ -2527,6 +2679,19 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
2527
2679
  +-----+---------+
2528
2680
  ```
2529
2681
 
2682
+ * List packages in a given shared inbox
2683
+
2684
+ First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2685
+ (To find out available filters, consult the API definition, or use the web interface in developer mode).
2686
+
2687
+ Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
2688
+
2689
+ ```
2690
+ shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
2691
+
2692
+ ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2693
+ ```
2694
+
2530
2695
  ## Shared folders
2531
2696
 
2532
2697
  * list shared folders in node
@@ -2559,7 +2724,7 @@ Notes:
2559
2724
 
2560
2725
  * the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
2561
2726
  * to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". ascli will resolve the list of email addresses to expected user ids.
2562
- * a recipîent can be a shared inbox, in this case just use the name of the shared inbox as recipient.
2727
+ * a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
2563
2728
  * If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
2564
2729
  * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
2565
2730
  * if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
@@ -2568,7 +2733,13 @@ Examples:
2568
2733
 
2569
2734
  ```
2570
2735
  $ ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
2736
+ ```
2737
+
2738
+ ```
2571
2739
  $ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
2740
+ ```
2741
+
2742
+ ```
2572
2743
  $ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2573
2744
  ```
2574
2745
 
@@ -2584,7 +2755,7 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2584
2755
  * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
2585
2756
  * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
2586
2757
 
2587
- Typically, one would regularly execute this command on a regular basis, using the method of your choice:
2758
+ Typically, one would execute this command on a regular basis, using the method of your choice:
2588
2759
 
2589
2760
  * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2590
2761
  * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
@@ -2592,45 +2763,45 @@ Typically, one would regularly execute this command on a regular basis, using th
2592
2763
 
2593
2764
  ## Download Files
2594
2765
 
2595
- Download of files is straightforward with a specific syntax for the `aspera files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2766
+ Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2596
2767
 
2597
2768
  * if only one source is provided, it is downloaded
2598
2769
  * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2599
2770
 
2600
2771
  ## Find Files
2601
2772
 
2602
- The command `aspera files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2773
+ The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2603
2774
 
2604
2775
  The expression can be of 3 formats:
2605
2776
 
2606
- * empty (default) : all files, equivalent to: `exec:true`
2607
- * not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
2777
+ * empty (default) : all files, equivalent to value: `exec:true`
2778
+ * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2608
2779
 
2609
2780
  For instance, to find files with a special extension, use `--value='\.myext$'`
2610
2781
 
2611
- * starting with `exec:` : the ruby code after the prefix is executed for each entry found. the entry variable name is `f`. the file is displayed if the result is true;
2782
+ * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2612
2783
 
2613
- Examples of expressions: (think to prefix with `exec:` and put in single quotes using bash)
2784
+ Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2614
2785
 
2615
- * find files more recent than 100 days
2786
+ * Find files more recent than 100 days
2616
2787
 
2617
2788
  ```
2618
2789
  f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2619
2790
  ```
2620
2791
 
2621
- * expression to find files older than 1 year on a given node and store in file list
2792
+ * Find files older than 1 year on a given node and store in file list
2622
2793
 
2623
2794
  ```
2624
2795
  $ ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2625
2796
  ```
2626
2797
 
2627
- * delete the files, one by one
2798
+ * Delete the files, one by one
2628
2799
 
2629
2800
  ```
2630
2801
  $ cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2631
2802
  ```
2632
2803
 
2633
- * delete the files in bulk
2804
+ * Delete the files in bulk
2634
2805
 
2635
2806
  ```
2636
2807
  cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
@@ -2644,19 +2815,40 @@ The activity app can be queried with:
2644
2815
  $ ascli aoc admin analytics transfers
2645
2816
  ```
2646
2817
 
2647
- It can also support filters and send notification email with a template:
2818
+ It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2819
+
2820
+ `mytemplate.erb`:
2821
+
2822
+ ```
2823
+ From: <%=from_name%> <<%=from_email%>>
2824
+ To: <<%=ev['user_email']%>>
2825
+ Subject: <%=ev['files_completed']%> files received
2826
+
2827
+ Dear <%=ev[:user_email.to_s]%>,
2828
+ We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
2829
+ <%=ev['content']%>
2648
2830
 
2831
+ Thank you.
2649
2832
  ```
2650
- $ ascli aoc admin analytics transfers --once-only=yes --lock-port=123455 \
2833
+ The environment provided contains the following additional variable:
2834
+
2835
+ * ev : all details on the transfer event
2836
+
2837
+ Example:
2838
+
2839
+ ```
2840
+ $ ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2651
2841
  --query=@json:'{"status":"completed","direction":"receive"}' \
2652
- --notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
2842
+ --notif-to=active --notif-template=@file:mytemplate.erb
2653
2843
  ```
2654
2844
 
2845
+ Options:
2846
+
2655
2847
  * `once_only` keep track of last date it was called, so next call will get only new events
2656
2848
  * `query` filter (on API call)
2657
2849
  * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2658
2850
 
2659
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
2851
+ Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
2660
2852
 
2661
2853
  ## Using specific transfer ports
2662
2854
 
@@ -2717,7 +2909,37 @@ $ ascli ats api_key create
2717
2909
  +--------+----------------------------------------------+
2718
2910
  $ ascli config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2719
2911
  ```
2912
+ ## Cross Organization transfers
2913
+
2914
+ It is possible to transfer files directly between organizations without having to first download locally and then upload...
2915
+
2916
+ Although optional, the creation of [option preset](#lprt) is recommended to avoid placing all parameters in the command line.
2917
+
2918
+ Procedure to send a file from org1 to org2:
2919
+
2920
+ * Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2921
+ * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2922
+ * Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
2923
+ * Check that access works and locate the destination folder `mydestfolder`
2924
+ * execute the following:
2925
+
2926
+ ```
2927
+ $ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2928
+ ```
2929
+
2930
+ Explanation:
2720
2931
 
2932
+ * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2933
+ * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2934
+ * `--format=json` format the output in JSON (instead of default text table)
2935
+ * `--display=data` display only the result, and remove other information, such as workspace name
2936
+ * `|` the standard output of the first command is fed into the second one
2937
+ * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2938
+ * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2939
+ * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
2940
+ * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2941
+
2942
+ Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
2721
2943
  ## Examples
2722
2944
 
2723
2945
  Example: create access key on softlayer:
@@ -2763,6 +2985,14 @@ This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
2763
2985
 
2764
2986
  Both password and SSH keys auth are supported.
2765
2987
 
2988
+ If not username is provided, the default transfer user `xfer` is used.
2989
+
2990
+ If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
2991
+
2992
+ ```
2993
+ $ ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2994
+ ```
2995
+
2766
2996
  Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
2767
2997
 
2768
2998
  Examples:
@@ -2800,7 +3030,7 @@ This means that you dont have such an ssh agent running:
2800
3030
  $ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
2801
3031
  ```
2802
3032
 
2803
- This can also be set as default using a preset
3033
+ This can also be set as default using a preset.
2804
3034
 
2805
3035
  ## Example
2806
3036
 
@@ -2820,18 +3050,36 @@ This creates a [option preset](#lprt) "aspera_demo_server" and set it as default
2820
3050
 
2821
3051
  This plugin gives access to capabilities provided by HSTS node API.
2822
3052
 
2823
- ## Simple Operations
3053
+ ## File Operations
2824
3054
 
2825
3055
  It is possible to:
2826
3056
  * browse
2827
3057
  * transfer (upload / download)
2828
3058
  * ...
2829
3059
 
3060
+ For transfers, it is possible to control how transfer is authorized using option: `token_type`:
3061
+
3062
+ * `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
3063
+ * `basic` : transfer spec is created like this:
3064
+
3065
+ ```
3066
+ {
3067
+ "remote_host": address of node url,
3068
+ "remote_user": "xfer",
3069
+ "ssh_port": 33001,
3070
+ "token": "Basic <base 64 encoded user/pass>",
3071
+ "direction": send/recv
3072
+ }
3073
+ ```
3074
+
3075
+ * `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
3076
+
2830
3077
  ## Central
2831
3078
 
2832
3079
  The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
2833
3080
 
2834
3081
  Filtering can be applied:
3082
+
2835
3083
  ```
2836
3084
  $ ascli node central file list
2837
3085
  ```
@@ -2857,7 +3105,6 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
2857
3105
  * Start watchd and watchfolderd services running as a system user having access to files
2858
3106
  * configure a watchfolder to define automated transfers
2859
3107
 
2860
-
2861
3108
  ```
2862
3109
  $ ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2863
3110
  $ ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
@@ -2903,16 +3150,19 @@ to download files.
2903
3150
  $ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2904
3151
  ```
2905
3152
 
2906
- # Plugin: IBM Aspera Faspex
3153
+ # Plugin: IBM Aspera Faspex5
2907
3154
 
2908
- Notes:
3155
+ 3 authentication methods are supported:
3156
+
3157
+ * jwt
3158
+ * web
3159
+ * boot
2909
3160
 
2910
- * the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2911
- * for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
3161
+ For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
2912
3162
 
2913
- ## Faspex 5 Beta1
3163
+ For web method, create an API client in Faspex, and use: --auth=web
2914
3164
 
2915
- As the web UI does not yet allow adding API client yet, the way to use CLI is:
3165
+ For boot method: (will be removed in future)
2916
3166
 
2917
3167
  * open a browser
2918
3168
  * start developer mode
@@ -2925,30 +3175,74 @@ Use it as password and use `--auth=boot`.
2925
3175
  $ ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2926
3176
  ```
2927
3177
 
2928
- An JWT client can then be created with a private key:
3178
+ Ready to use Faspex5 with CLI.
3179
+
3180
+ # Plugin: IBM Aspera Faspex (4.x)
3181
+
3182
+ Notes:
3183
+
3184
+ * The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
3185
+ * For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
3186
+
3187
+ ## Listing Packages
3188
+
3189
+ Command: `faspex package list`
3190
+
3191
+ ### Option `box`
3192
+
3193
+ By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
3194
+
3195
+ ### Option `recipient`
3196
+
3197
+ A user can receive a package because the recipient is:
3198
+
3199
+ * the user himself (default)
3200
+ * the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
3201
+
3202
+ ### Option `query`
3203
+
3204
+ As inboxes may be large, it is possible to use the following query parameters:
3205
+
3206
+ * `count` : (native) number items in one API call (default=0, equivalent to 10)
3207
+ * `page` : (native) id of page in call (default=0)
3208
+ * `startIndex` : (native) index of item to start, default=0, oldest index=0
3209
+ * `max` : maximum number of items
3210
+ * `pmax` : maximum number of pages
3211
+
3212
+ (SQL query is `LIMIT <startIndex>, <count>`)
3213
+
3214
+ The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
3215
+
3216
+ If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
3217
+
3218
+ ### Example
2929
3219
 
2930
3220
  ```
2931
- $ jsonk=$(openssl rsa -in ~/.aspera/ascli/aspera_on_cloud_key -pubout 2> /dev/null | sed -e :a -e N -e '$!ba' -e 's/\n/\\n/g')
2932
- $ ascli faspex5 -Pf5boot auth_client create --value=@json:'{"name":"hello","client_type":"public","redirect_uris":["https://localhost:12345"],"allow_jwt_grant":true,"public_key":"'$jsonk'"}'
3221
+ $ ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2933
3222
  ```
2934
3223
 
2935
- or deleted by name:
3224
+ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
3225
+
3226
+ ## Receiving a Package
3227
+
3228
+ The command is `package recv`, possible methods are:
3229
+
3230
+ * provide a package id with option `id`
3231
+ * provide a public link with option `link`
3232
+ * provide a `faspe:` URI with option `link`
2936
3233
 
2937
3234
  ```
2938
- $ id=$(ascli faspex5 auth_client list --select=@json:'{"name":"hello"}' --fields=client_id --format=csv)
2939
- $ ascli faspex5 auth_client delete --id=$id
3235
+ $ ascli faspex package recv --id=12345
3236
+ $ ascli faspex package recv --link=faspe://...
2940
3237
  ```
2941
3238
 
2942
- Once the API client is created with a client_id and secret (result of create command), create a configuration:
3239
+ If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
2943
3240
 
2944
3241
  ```
2945
- $ ascli conf id f5 update --url=https://localhost/aspera/faspex --auth=jwt --client-id=abcd --client-secret=def --username=pierre@example.com --private-key=@val:@file:~/.aspera/ascli/aspera_on_cloud_key
2946
- $ ascli conf id default set faspex5 f5
2947
- ```
2948
-
2949
- Ready to use Faspex5 with CLI.
3242
+ $ ascli faspex package list --recipient='*thedropboxname'
3243
+ ```
2950
3244
 
2951
- Once the graphical registration form exist, ther bootstrap method can be removed.
3245
+ if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
2952
3246
 
2953
3247
  ## Sending a Package
2954
3248
 
@@ -2967,7 +3261,19 @@ Additional optional parameters in `delivery_info`:
2967
3261
  * Package Note: : `"note":"note this and that"`
2968
3262
  * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2969
3263
 
2970
- ## operation on dropboxes
3264
+ ## Email notification on transfer
3265
+
3266
+ Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
3267
+
3268
+ Example:
3269
+
3270
+ ```
3271
+ $ ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
3272
+ ```
3273
+
3274
+ In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
3275
+
3276
+ ## Operation on dropboxes
2971
3277
 
2972
3278
  Example:
2973
3279
 
@@ -2977,7 +3283,7 @@ $ ascli faspex v4 dropbox list
2977
3283
  $ ascli faspex v4 dropbox delete --id=36
2978
3284
  ```
2979
3285
 
2980
- ## remote sources
3286
+ ## Remote sources
2981
3287
 
2982
3288
  Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
2983
3289
  the node API is used, for this it is required to add a section ":storage" that links
@@ -3028,29 +3334,38 @@ $ for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}'
3028
3334
  # Plugin: IBM Cloud Object Storage
3029
3335
 
3030
3336
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
3031
- It uses the same transfer service as Aspera on Cloud.
3032
- see [https://status.aspera.io](https://status.aspera.io)
3337
+ It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
3338
+ Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
3339
+
3340
+ There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the forst method. If you dont have credentials but have access to the IBM Cloud console, then use the second method.
3033
3341
 
3034
- Required options are either:
3342
+ ## Using endpoint, apikey and Ressource Instance ID (CRN)
3343
+
3344
+ If you have those parameters already, then following options shall be provided:
3035
3345
 
3036
3346
  * `bucket` bucket name
3037
3347
  * `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
3038
3348
  * `apikey` API Key
3039
3349
  * `crn` resource instance id
3040
3350
 
3041
- or:
3351
+ For example, let us create a default configuration:
3042
3352
 
3043
- * `bucket` bucket name
3044
- * `region` bucket region, e.g. eu-de
3045
- * `service_credentials` see below
3353
+ ```
3354
+ $ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
3355
+ $ ascli conf id default set cos mycos
3356
+ ```
3046
3357
 
3047
- Service credentials are directly created using the IBM cloud web ui. Navigate to:
3358
+ Then, jump to the transfer example.
3359
+
3360
+ ## Using service credential file
3361
+
3362
+ If you are the COS administrator and dont have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
3048
3363
 
3049
3364
  Navigation Menu &rarr; Resource List &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
3050
3365
 
3051
3366
  Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
3052
3367
 
3053
- or using the CLI:
3368
+ or using the IBM Cloud CLI:
3054
3369
 
3055
3370
  ```
3056
3371
  $ ibmcloud resource service-keys
@@ -3081,33 +3396,38 @@ The field `resource_instance_id` is for option `crn`
3081
3396
 
3082
3397
  The field `apikey` is for option `apikey`
3083
3398
 
3084
- Endpoints for regions can be found by querying the `endpoints` URL.
3399
+ (If needed: endpoints for regions can be found by querying the `endpoints` URL.)
3400
+
3401
+ The required options for this method are:
3402
+
3403
+ * `bucket` bucket name
3404
+ * `region` bucket region, e.g. eu-de
3405
+ * `service_credentials` see below
3085
3406
 
3086
- For convenience, let us create a default configuration, for example:
3407
+ For example, let us create a default configuration:
3087
3408
 
3088
3409
  ```
3089
3410
  $ ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
3090
3411
  $ ascli conf id default set cos mycos
3091
3412
  ```
3092
3413
 
3093
- or using direct parameters:
3414
+ ## Operations, transfers
3094
3415
 
3095
- ```
3096
- $ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
3097
- $ ascli conf id default set cos mycos
3098
- ```
3416
+ Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
3099
3417
 
3100
- Now, Ready to do operations, a subset of "node" plugin operations are supported, basically node API:
3418
+ A subset of `node` plugin operations are supported, basically node API:
3101
3419
 
3102
3420
  ```
3103
- $ ascli cos node browse /
3104
- $ ascli cos node upload myfile.txt
3421
+ $ ascli cos node info
3422
+ $ ascli cos node upload 'faux:///sample1G?1g'
3105
3423
  ```
3106
3424
 
3425
+ Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
3426
+
3107
3427
  # Plugin: IBM Aspera Sync
3108
3428
 
3109
- A basic plugin to start an "async" using `ascli`. The main advantage is the possibility
3110
- to start from ma configuration file, using `ascli` standard options.
3429
+ A basic plugin to start an "async" using `ascli`.
3430
+ The main advantage is the possibility to start from ma configuration file, using `ascli` standard options.
3111
3431
 
3112
3432
  # Plugin: Preview
3113
3433
 
@@ -3199,8 +3519,10 @@ yum install -y ImageMagick optipng
3199
3519
 
3200
3520
  ### Video: FFmpeg
3201
3521
 
3522
+ The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
3523
+
3202
3524
  ```
3203
- curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && tar xJvf - && rm -f /opt/ffmpeg /usr/bin/{ffmpeg,ffprobe} && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3525
+ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3204
3526
  ```
3205
3527
 
3206
3528
  ### Office: Unoconv and Libreoffice
@@ -3385,14 +3707,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
3385
3707
  The `smtp` option is a hash table (extended value) with the following fields:
3386
3708
  <table>
3387
3709
  <tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
3388
- <tr><td>server</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
3389
- <tr><td>tls</td><td>true</td><td>false</td><td>use of TLS</td></tr>
3390
- <tr><td>port</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
3391
- <tr><td>domain</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
3392
- <tr><td>username</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
3393
- <tr><td>password</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
3394
- <tr><td>from\_email</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
3395
- <tr><td>from\_name</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
3710
+ <tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
3711
+ <tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
3712
+ <tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
3713
+ <tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
3714
+ <tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
3715
+ <tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
3716
+ <tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
3717
+ <tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
3396
3718
  </table>
3397
3719
 
3398
3720
  ## Example of configuration:
@@ -3422,13 +3744,52 @@ $ ascli config id cli_default set smtp @val:@preset:smtp_google
3422
3744
  $ ascli config id default set config cli_default
3423
3745
  ```
3424
3746
 
3747
+ ## Email templates
3748
+
3749
+ Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
3750
+
3751
+ The template is the full SMTP message, including headers.
3752
+
3753
+ The following variables are defined by default:
3754
+
3755
+ * from_name
3756
+ * from_email
3757
+ * to
3758
+
3759
+ Other variables are defined depending on context.
3760
+
3425
3761
  ## Test
3426
3762
 
3427
3763
  Check settings with `smtp_settings` command. Send test email with `email_test`.
3428
3764
 
3429
3765
  ```
3430
3766
  $ ascli config --smtp=@preset:smtp_google smtp
3431
- $ ascli config --smtp=@preset:smtp_google email sample.dest@example.com
3767
+ $ ascli config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3768
+ ```
3769
+
3770
+ ## Notifications for transfer status
3771
+
3772
+ An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
3773
+
3774
+ To activate, use option `notif_to`.
3775
+
3776
+ A default e-mail template is used, but it can be overriden with option `notif_template`.
3777
+
3778
+ The environment provided contains the following additional variables:
3779
+
3780
+ * subject
3781
+ * body
3782
+ * global_transfer_status
3783
+ * ts
3784
+
3785
+ Example of template:
3786
+
3787
+ ```
3788
+ From: <%=from_name%> <<%=from_email%>>
3789
+ To: <<%=to%>>
3790
+ Subject: <%=subject%>
3791
+
3792
+ Transfer is: <%=global_transfer_status%>
3432
3793
  ```
3433
3794
 
3434
3795
  # Tool: `asession`
@@ -3459,7 +3820,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
3459
3820
  * `EX_loglevel` to change log level of the tool
3460
3821
  * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
3461
3822
 
3462
- Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the "local" transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3823
+ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3463
3824
 
3464
3825
  ## Comparison of interfaces
3465
3826
 
@@ -3559,8 +3920,8 @@ Interesting ascp features are found in its arguments: (see ascp manual):
3559
3920
  Note that:
3560
3921
 
3561
3922
  * `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
3562
- * not all native ascp arguments are available as standard transfer_spec parameters
3563
- * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
3923
+ * most, but not all native ascp arguments are available as standard transfer_spec parameters
3924
+ * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
3564
3925
 
3565
3926
  ### server side and configuration
3566
3927
 
@@ -3579,7 +3940,7 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
3579
3940
 
3580
3941
  ```
3581
3942
 
3582
- The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3943
+ The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3583
3944
 
3584
3945
  # Aspera Health check and Nagios
3585
3946
 
@@ -3656,25 +4017,56 @@ So, it evolved into `ascli`:
3656
4017
 
3657
4018
  # Changes (Release notes)
3658
4019
 
4020
+ * 4.4.0
4021
+
4022
+ * new: `aoc packages list` add possibility to add filter with option `query`
4023
+ * new: `aoc admin res xxx list` now get all items by default #50
4024
+ * new: `preset` option can specify name or hash value
4025
+ * new: `node` plugin accepts bearer token and access key as credential
4026
+ * new: `node` option `token_type` allows using basic token in addition to aspera type.
4027
+ * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
4028
+ * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
4029
+
4030
+ * 4.3.0
4031
+
4032
+ * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
4033
+ * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
4034
+
4035
+ * 4.2.2
4036
+
4037
+ * new: `faspex package list` retrieves the whole list, not just first page
4038
+ * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
4039
+ * new: the error "Remote host is not who we expected" displays a special remediation message
4040
+ * new: `conf ascp spec` displays supported transfer spec
4041
+ * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
4042
+ * fix: space character in `faspe:` url are precent encoded if needed
4043
+ * fix: `preview scan`: if file_id is unknown, ignore and continue scan
4044
+ * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then `ascli` exits with code 1 (instead of zero=success)
4045
+ * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
4046
+
4047
+ * 4.2.1
4048
+
4049
+ * new: command `faspex package recv` supports link of type: `faspe:`
4050
+ * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
4051
+
3659
4052
  * 4.2.0
3660
4053
 
3661
4054
  * new: command `aoc remind` to receive organization membership by email
3662
4055
  * new: in `preview` option `value` to filter out on file name
3663
4056
  * new: `initdemo` to initialize for demo server
3664
- * new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
4057
+ * new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3665
4058
  * fix: on Windows `conf ascp use` expects ascp.exe
3666
4059
  * fix: (break) multi_session_threshold is Integer, not String
3667
4060
  * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3668
4061
  * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3669
4062
  * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3670
- * change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
4063
+ * change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
3671
4064
 
3672
4065
  * 4.1.0
3673
4066
 
3674
- * fix: remove keys from transfer spec and command line when not needed
3675
- * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3676
- * new: update documentation with regard to offline and docker installation
3677
- * new: renamed command `nagios_check` to `health`
4067
+ * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
4068
+ * new: update documentation with regard to offline and docker installation
4069
+ * new: renamed command `nagios_check` to `health`
3678
4070
  * new: agent `http_gw` now supports upload
3679
4071
  * new: added option `sdk_url` to install SDK from local file for offline install
3680
4072
  * new: check new gem version periodically
@@ -3725,8 +4117,8 @@ So, it evolved into `ascli`:
3725
4117
  * allow bulk delete in `aspera files` with option `bulk=yes`
3726
4118
  * fix getting connect versions
3727
4119
  * added section for Aix
3728
- * support all ciphers for `local`ascp (incl. gcm, etc..)
3729
- * added transfer spec param `apply_local_docroot` for `local`
4120
+ * support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
4121
+ * added transfer spec param `apply_local_docroot` for [`direct`](#direct)
3730
4122
 
3731
4123
  * 0.11.4
3732
4124
 
@@ -3781,7 +4173,7 @@ So, it evolved into `ascli`:
3781
4173
  * 0.10.12
3782
4174
 
3783
4175
  * added support for AoC node registration keys
3784
- * replaced option : `local_resume` with `transfer_info` for agent `direct`
4176
+ * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
3785
4177
  * Transfer agent is no more a Singleton instance, but only one is used in CLI
3786
4178
  * `@incps` : new extended value modifier
3787
4179
  * ATS: no more provides access keys secrets: now user must provide it
@@ -3831,9 +4223,9 @@ So, it evolved into `ascli`:
3831
4223
 
3832
4224
  * 0.10.2
3833
4225
 
3834
- * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
3835
- * added doc for "cargo" like actions
3836
- * added doc for multi-session
4226
+ * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
4227
+ * added doc for "cargo" like actions
4228
+ * added doc for multi-session
3837
4229
 
3838
4230
  * 0.10.1
3839
4231
 
@@ -3861,22 +4253,22 @@ So, it evolved into `ascli`:
3861
4253
 
3862
4254
  * 0.9.33
3863
4255
 
3864
- * new command to display basic token of node
3865
- * new command to display bearer token of node in AoC
3866
- * the --fields= option, support +_fieldname_ to add a field to default fields
3867
- * many small changes
4256
+ * new command to display basic token of node
4257
+ * new command to display bearer token of node in AoC
4258
+ * the --fields= option, support +_fieldname_ to add a field to default fields
4259
+ * many small changes
3868
4260
 
3869
4261
  * 0.9.32
3870
4262
 
3871
- * all Faspex public links are now supported
3872
- * removed faspex operation recv_publink
3873
- * replaced with option `link` (consistent with AoC)
4263
+ * all Faspex public links are now supported
4264
+ * removed faspex operation recv_publink
4265
+ * replaced with option `link` (consistent with AoC)
3874
4266
 
3875
4267
  * 0.9.31
3876
4268
 
3877
- * added more support for public link: receive and send package, to user or dropbox and files view.
3878
- * delete expired file lists
3879
- * changed text table gem from text-table to terminal-table because it supports multiline values
4269
+ * added more support for public link: receive and send package, to user or dropbox and files view.
4270
+ * delete expired file lists
4271
+ * changed text table gem from text-table to terminal-table because it supports multiline values
3880
4272
 
3881
4273
  * 0.9.27
3882
4274
 
@@ -3896,149 +4288,149 @@ So, it evolved into `ascli`:
3896
4288
 
3897
4289
  * 0.9.24
3898
4290
 
3899
- * fix bug where AoC node to node transfer did not work
3900
- * fix bug on error if ED25519 private key is defined in .ssh
4291
+ * fix bug where AoC node to node transfer did not work
4292
+ * fix bug on error if ED25519 private key is defined in .ssh
3901
4293
 
3902
4294
  * 0.9.23
3903
4295
 
3904
- * defined REST error handlers, more error conditions detected
3905
- * commands to select specific ascp location
4296
+ * defined REST error handlers, more error conditions detected
4297
+ * commands to select specific ascp location
3906
4298
 
3907
4299
  * 0.9.21
3908
4300
 
3909
- * supports simplified wizard using global client
3910
- * only ascp binary is required, other SDK (keys) files are now generated
4301
+ * supports simplified wizard using global client
4302
+ * only ascp binary is required, other SDK (keys) files are now generated
3911
4303
 
3912
4304
  * 0.9.20
3913
4305
 
3914
- * improved wizard (prepare for AoC global client id)
3915
- * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
3916
- * removed outdated pictures from this doc
4306
+ * improved wizard (prepare for AoC global client id)
4307
+ * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
4308
+ * removed outdated pictures from this doc
3917
4309
 
3918
4310
  * 0.9.19
3919
4311
 
3920
- * added command aspera bearer --scope=xx
4312
+ * added command aspera bearer --scope=xx
3921
4313
 
3922
4314
  * 0.9.18
3923
4315
 
3924
- * enhanced aspera admin events to support query
4316
+ * enhanced aspera admin events to support query
3925
4317
 
3926
4318
  * 0.9.16
3927
4319
 
3928
- * AoC transfers are now reported in activity app
3929
- * new interface for Rest class authentication (keep backward compatibility)
4320
+ * AoC transfers are now reported in activity app
4321
+ * new interface for Rest class authentication (keep backward compatibility)
3930
4322
 
3931
4323
  * 0.9.15
3932
4324
 
3933
- * new feature: "find" command in aspera files
3934
- * sample code for transfer API
4325
+ * new feature: "find" command in aspera files
4326
+ * sample code for transfer API
3935
4327
 
3936
4328
  * 0.9.12
3937
4329
 
3938
- * add nagios commands
3939
- * support of ATS for IBM Cloud, removed old version based on aspera id
4330
+ * add nagios commands
4331
+ * support of ATS for IBM Cloud, removed old version based on aspera id
3940
4332
 
3941
4333
  * 0.9.11
3942
4334
 
3943
- * Breaking change: @stdin is now @stdin:
3944
- * support of ATS for IBM Cloud, removed old version based on aspera id
4335
+ * Breaking change: @stdin is now @stdin:
4336
+ * support of ATS for IBM Cloud, removed old version based on aspera id
3945
4337
 
3946
4338
 
3947
4339
  * 0.9.10
3948
4340
 
3949
- * Breaking change: parameter transfer-node becomes more generic: transfer-info
3950
- * Display SaaS storage usage with command: aspera admin res node --id=nn info
3951
- * cleaner way of specifying source file list for transfers
3952
- * Breaking change: replaced download_mode option with http_download action
4341
+ * Breaking change: parameter transfer-node becomes more generic: transfer-info
4342
+ * Display SaaS storage usage with command: aspera admin res node --id=nn info
4343
+ * cleaner way of specifying source file list for transfers
4344
+ * Breaking change: replaced download_mode option with http_download action
3953
4345
 
3954
4346
  * 0.9.9
3955
4347
 
3956
- * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
3957
- * Now supports "cargo" for Aspera on Cloud (automatic package download)
4348
+ * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
4349
+ * Now supports "cargo" for Aspera on Cloud (automatic package download)
3958
4350
 
3959
4351
  * 0.9.8
3960
4352
 
3961
- * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
3962
- * AoC: share to share transfer with command "transfer"
4353
+ * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
4354
+ * AoC: share to share transfer with command "transfer"
3963
4355
 
3964
4356
  * 0.9.7
3965
4357
 
3966
- * homogeneous [_transfer-spec_](#transferspec) for node and local
3967
- * preview persistency goes to unique file by default
3968
- * catch mxf extension in preview as video
3969
- * Faspex: possibility to download all paclages by specifying id=ALL
3970
- * Faspex: to come: cargo-like function to download only new packages with id=NEW
4358
+ * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
4359
+ * preview persistency goes to unique file by default
4360
+ * catch mxf extension in preview as video
4361
+ * Faspex: possibility to download all paclages by specifying id=ALL
4362
+ * Faspex: to come: cargo-like function to download only new packages with id=NEW
3971
4363
 
3972
4364
  * 0.9.6
3973
4365
 
3974
- * Breaking change: `@param:`is now `@preset:` and is generic
3975
- * AoC: added command to display current workspace information
4366
+ * Breaking change: `@param:`is now `@preset:` and is generic
4367
+ * AoC: added command to display current workspace information
3976
4368
 
3977
4369
  * 0.9.5
3978
4370
 
3979
- * new parameter: new_user_option used to choose between public_link and invite of external users.
3980
- * fixed bug in wizard, and wizard uses now product detection
4371
+ * new parameter: new_user_option used to choose between public_link and invite of external users.
4372
+ * fixed bug in wizard, and wizard uses now product detection
3981
4373
 
3982
4374
  * 0.9.4
3983
4375
 
3984
- * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
3985
- * AoC Package send supports external users
3986
- * new command to export AoC config to Aspera CLI config
4376
+ * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
4377
+ * AoC Package send supports external users
4378
+ * new command to export AoC config to Aspera CLI config
3987
4379
 
3988
4380
  * 0.9.3
3989
4381
 
3990
- * REST error message show host and code
3991
- * option for quiet display
3992
- * modified transfer interface and allow token re-generation on error
3993
- * async add admin command
3994
- * async add db parameters
3995
- * Breaking change: new option "sources" to specify files to transfer
4382
+ * REST error message show host and code
4383
+ * option for quiet display
4384
+ * modified transfer interface and allow token re-generation on error
4385
+ * async add admin command
4386
+ * async add db parameters
4387
+ * Breaking change: new option "sources" to specify files to transfer
3996
4388
 
3997
4389
  * 0.9.2
3998
4390
 
3999
- * Breaking change: changed AoC package creation to match API, see AoC section
4391
+ * Breaking change: changed AoC package creation to match API, see AoC section
4000
4392
 
4001
4393
  * 0.9.1
4002
4394
 
4003
- * Breaking change: changed faspex package creation to match API, see Faspex section
4395
+ * Breaking change: changed faspex package creation to match API, see Faspex section
4004
4396
 
4005
4397
  * 0.9
4006
4398
 
4007
- * Renamed the CLI from aslmcli to `ascli`
4008
- * Automatic rename and conversion of former config folder from aslmcli to `ascli`
4399
+ * Renamed the CLI from aslmcli to `ascli`
4400
+ * Automatic rename and conversion of former config folder from aslmcli to `ascli`
4009
4401
 
4010
4402
  * 0.7.6
4011
4403
 
4012
- * add "sync" plugin
4404
+ * add "sync" plugin
4013
4405
 
4014
4406
  * 0.7
4015
4407
 
4016
- * Breaking change: AoC package recv take option if for package instead of argument.
4017
- * Breaking change: Rest class and Oauth class changed init parameters
4018
- * AoC: receive package from public link
4019
- * select by col value on output
4020
- * added rename (AoC, node)
4408
+ * Breaking change: AoC package recv take option if for package instead of argument.
4409
+ * Breaking change: Rest class and Oauth class changed init parameters
4410
+ * AoC: receive package from public link
4411
+ * select by col value on output
4412
+ * added rename (AoC, node)
4021
4413
 
4022
4414
  * 0.6.19
4023
4415
 
4024
4416
  Breaking change:
4025
4417
 
4026
- * ats server list provisioned &rarr; ats cluster list
4027
- * ats server list clouds &rarr; ats cluster clouds
4028
- * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4029
- * ats server id xxx &rarr; ats cluster show --id=xxx
4030
- * ats subscriptions &rarr; ats credential subscriptions
4031
- * ats api_key repository list &rarr; ats credential cache list
4032
- * ats api_key list &rarr; ats credential list
4033
- * ats access_key id xxx &rarr; ats access_key --id=xxx
4418
+ * ats server list provisioned &rarr; ats cluster list
4419
+ * ats server list clouds &rarr; ats cluster clouds
4420
+ * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4421
+ * ats server id xxx &rarr; ats cluster show --id=xxx
4422
+ * ats subscriptions &rarr; ats credential subscriptions
4423
+ * ats api_key repository list &rarr; ats credential cache list
4424
+ * ats api_key list &rarr; ats credential list
4425
+ * ats access_key id xxx &rarr; ats access_key --id=xxx
4034
4426
 
4035
4427
  * 0.6.18
4036
4428
 
4037
- * some commands take now --id option instead of id command.
4429
+ * some commands take now --id option instead of id command.
4038
4430
 
4039
4431
  * 0.6.15
4040
4432
 
4041
- * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
4433
+ * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
4042
4434
 
4043
4435
  # BUGS, FEATURES, CONTRIBUTION
4044
4436
 
@@ -4075,20 +4467,24 @@ You may either install the suggested Gems, or remove your ed25519 key from your
4075
4467
 
4076
4468
  ## Error "Remote host is not who we expected"
4077
4469
 
4078
- `ascp` version 4.x changed the algorithm used to check the SSH server certificate. To ignore the certificate (SSH fingerprint) add option on client side:
4470
+ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
4471
+
4472
+ Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
4079
4473
 
4080
4474
  ```
4081
4475
  --ts=@json:'{"sshfp":null}'
4082
4476
  ```
4083
4477
 
4084
- Refer to ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server): if you have access to server side, basically disable other SSH host keys than RSA.
4478
+ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
4479
+
4480
+ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
4085
4481
 
4086
4482
  ## Miscelaneous
4087
4483
 
4088
4484
  * remove rest and oauth classes and use ruby standard gems:
4089
4485
 
4090
- * oauth
4091
- * https://github.com/rest-client/rest-client
4486
+ * oauth
4487
+ * https://github.com/rest-client/rest-client
4092
4488
 
4093
4489
  * use Thor or any standard Ruby CLI manager
4094
4490