aspera-cli 4.3.0 → 4.4.0

Sign up to get free protection for your applications and to get access to all the features.
data/README.md CHANGED
@@ -1,11 +1,11 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
2
  <font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
3
3
 
4
- Version : 4.3.0
4
+ Version : 4.4.0
5
5
 
6
6
  _Laurent/2016-2021_
7
7
 
8
- This gem provides `ascli`: a command line interface to Aspera Applications.
8
+ This gem provides the `ascli` Command Line Interface to IBM Aspera software.
9
9
 
10
10
  `ascli` is a also great tool to learn Aspera APIs.
11
11
 
@@ -57,7 +57,7 @@ Command line parameters in examples beginning with `my_`, like `my_param_value`
57
57
 
58
58
  On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
59
59
 
60
- On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
60
+ On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
61
61
 
62
62
  * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
63
63
  * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
@@ -82,7 +82,7 @@ Once the gem is installed, `ascli` shall be accessible:
82
82
 
83
83
  ```
84
84
  $ ascli --version
85
- 4.3.0
85
+ 4.4.0
86
86
  ```
87
87
 
88
88
  ## First use
@@ -109,7 +109,7 @@ $ ascli server browse /
109
109
  If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
110
110
 
111
111
  * create a [option preset](#lprt)
112
- * define it as default for "server" plugin
112
+ * define it as default for `server` plugin
113
113
  * list files in a folder
114
114
  * download a file
115
115
 
@@ -166,6 +166,8 @@ Use this method only if you know what you do, else use the standard recommended
166
166
 
167
167
  This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
168
168
 
169
+ The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
170
+
169
171
  Ensure that you have Docker installed.
170
172
 
171
173
  ```
@@ -234,7 +236,7 @@ If you keep the same terminal (ont needed if re-login):
234
236
  $ source ~/.rvm/scripts/rvm
235
237
  ```
236
238
 
237
- It is advised to get one of the pre-compiled ruby version, you can list with:
239
+ It is advised to get one of the pre-compiled ruby version, you can list with:
238
240
 
239
241
  ```
240
242
  $ rvm list --remote
@@ -430,8 +432,8 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
430
432
  `ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
431
433
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
432
434
 
433
- Several methods are provided on how to start a transfer. Use of a local client is one of them, but
434
- other methods are available. Refer to section: [Transfer Agents](#agents)
435
+ Several methods are provided to start a transfer.
436
+ Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
435
437
 
436
438
  ## <a name="offline_install"></a>Offline Installation (without internet)
437
439
 
@@ -477,7 +479,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
477
479
  * Supports most Aspera server products (on-premise and SaaS)
478
480
  * Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
479
481
  * Supports Commands, Option values and Parameters shortcuts
480
- * FASP [Transfer Agents](#agents) can be: FaspManager (local ascp), or Connect Client, or any transfer node
482
+ * FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
481
483
  * Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
482
484
  * Allows transfers from products to products, essentially at node level (using the node transfer agent)
483
485
  * Supports FaspStream creation (using Node API)
@@ -601,12 +603,12 @@ The style of output can be set using the `format` parameter, supporting:
601
603
  * `yaml` : YAML
602
604
  * `csv` : Comma Separated Values
603
605
 
604
- ### Filtering columns for `object_list`
606
+ ### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
605
607
 
606
608
  Table output can be filtered using the `select` parameter. Example:
607
609
 
608
610
  ```
609
- $ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"per_page":1000,"page":1,"sort":"name"}' --select=@json:'{"ats_admin":true}'
611
+ $ ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
610
612
  :...............................:..................................:...........:
611
613
  : name : email : ats_admin :
612
614
  :...............................:..................................:...........:
@@ -753,19 +755,15 @@ C:\Users\Kenji\.aspera\ascli
753
755
  On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
754
756
  Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
755
757
 
756
- Although the file is a standard YAML file, `ascli` provides commands to read and modify it
757
- using the `config` command.
758
+ Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
758
759
 
759
- All options for `ascli` commands can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
760
+ All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuratin file.
760
761
 
761
- A configuration file provides a way to define default values, especially
762
- for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
762
+ A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
763
763
 
764
- The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
765
- (this can be overriden with option `--config-file=path` or equivalent env var).
764
+ The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
766
765
 
767
- So, finally, the configuration file is simply a catalog of pre-defined lists of options,
768
- called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
766
+ The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
769
767
 
770
768
  ### <a name="lprt"></a>Option preset
771
769
 
@@ -865,7 +863,7 @@ demo_server:
865
863
  We can see here:
866
864
 
867
865
  * The configuration was created with CLI version 0.3.7
868
- * the default [option preset](#lprt) to load for plugin "server" is : `demo_server`
866
+ * the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
869
867
  * the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
870
868
  * the default [option preset](#lprt) to load in any case is : `cli_default`
871
869
 
@@ -896,26 +894,26 @@ my_aoc_org:
896
894
 
897
895
  So, the key file will be read only at execution time, but not be embedded in the configuration file.
898
896
 
897
+ ### Options evaluation order
898
+
899
+ Some options are global, some options are available only for some plugins. (the plugin is the first level command).
900
+
899
901
  Options are loaded using this algorithm:
900
902
 
901
- * if option '--preset=xxxx' is specified (or -Pxxxx), this reads the [option preset](#lprt) specified from the configuration file.
902
- * else if option --no-default (or -N) is specified, then dont load default
903
- * else it looks for the name of the default [option preset](#lprt) in section "default" and loads it
904
- * environment variables are evaluated
905
- * command line options are evaluated
903
+ * If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
904
+ * else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
905
+ * If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
906
+ * Environment variables are evaluated
907
+ * Command line options are evaluated
906
908
 
907
909
  Parameters are evaluated in the order of command line.
908
910
 
909
- To avoid loading the default [option preset](#lprt) for a plugin, just specify a non existing configuration: `-Pnone`
911
+ To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
910
912
 
911
913
  On command line, words in parameter names are separated by a dash, in configuration file, separator
912
914
  is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
913
915
 
914
- Note: before version 0.4.5, some keys could be ruby symbols, from 0.4.5 all keys are strings. To
915
- convert olver versions, remove the leading ":" in front of keys.
916
-
917
- The main plugin name is *config*, so it is possible to define a default [option preset](#lprt) for
918
- the main plugin with:
916
+ The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
919
917
 
920
918
  ```
921
919
  $ ascli config id cli_default set interactive no
@@ -928,6 +926,18 @@ A [option preset](#lprt) value can be removed with `unset`:
928
926
  $ ascli config id cli_default unset interactive
929
927
  ```
930
928
 
929
+ Example: Define options using command line:
930
+
931
+ ```
932
+ $ ascli -N --url=x --password=y --username=y node --show-config
933
+ ```
934
+
935
+ Example: Define options using a hash:
936
+
937
+ ```
938
+ $ ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
939
+ ```
940
+
931
941
  ### Examples
932
942
 
933
943
  For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
@@ -1056,7 +1066,7 @@ $ ascli config ascp info
1056
1066
  ...
1057
1067
  ```
1058
1068
 
1059
- ### Selection of local `ascp`
1069
+ ### Selection of `ascp` location for [`direct`](#direct) agent
1060
1070
 
1061
1071
  By default, `ascli` uses any found local product with ascp, including SDK.
1062
1072
 
@@ -1100,7 +1110,7 @@ $ ascli config ascp products list
1100
1110
  :.........................................:................................................:
1101
1111
  ```
1102
1112
 
1103
- ### Selection of local client
1113
+ ### Selection of local client for `ascp` for [`direct`](#direct) agent
1104
1114
 
1105
1115
  If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
1106
1116
 
@@ -1155,7 +1165,7 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
1155
1165
 
1156
1166
  There are currently 3 agents:
1157
1167
 
1158
- * `direct` : a local execution of `ascp`
1168
+ * [`direct`](#direct) : a local execution of `ascp`
1159
1169
  * `connect` : use of a local Connect Client
1160
1170
  * `node` : use of an Aspera Transfer Node (potentially _remote_).
1161
1171
  * `httpgw` : use of an Aspera HTTP Gateway
@@ -1167,7 +1177,7 @@ will effectively push files to the related server from the agent node.
1167
1177
  `ascli` standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1168
1178
 
1169
1179
 
1170
- ### <a name="direct"></a>Direct (local ascp using FASPManager API)
1180
+ ### <a name="direct"></a>Direct (local ascp execution)
1171
1181
 
1172
1182
  By default `ascli` uses a local ascp, equivalent to specifying `--transfer=direct`.
1173
1183
  `ascli` will detect locally installed Aspera products.
@@ -1265,7 +1275,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
1265
1275
 
1266
1276
  If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1267
1277
 
1268
- It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1278
+ It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1269
1279
 
1270
1280
  The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1271
1281
 
@@ -1298,7 +1308,7 @@ Columns:
1298
1308
  * C=Connect Client
1299
1309
  * arg=`ascp` argument or environment variable
1300
1310
 
1301
- Fields with EX_ prefix are extensions to transfer agent `direct`. (only in `ascli`).
1311
+ Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in `ascli`).
1302
1312
 
1303
1313
  <table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td>&nbsp;</td><td>Y</td><td>&nbsp;</td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td>&nbsp;</td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td>&nbsp;</td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td>&nbsp;</td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>rate_policy_allowed</td><td>string</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>authentication</td><td>string</td><td>&nbsp;</td><td>&nbsp;</td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td>&nbsp;</td><td>&nbsp;</td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>&nbsp;</td></tr></table>
1304
1314
 
@@ -1316,8 +1326,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
1316
1326
 
1317
1327
  ### List of files for transfers
1318
1328
 
1319
- When uploading, downloading or sending files, the user must specify the list of files to transfer.
1320
- Most of the time, the list of files to transfer will be simply specified on the command line:
1329
+ When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1330
+ So, by default, the list of files to transfer will be simply specified on the command line:
1321
1331
 
1322
1332
  ```
1323
1333
  $ ascli server upload ~/mysample.file secondfile
@@ -1329,7 +1339,7 @@ This is equivalent to:
1329
1339
  $ ascli server upload --sources=@args ~/mysample.file secondfile
1330
1340
  ```
1331
1341
 
1332
- More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1342
+ More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1333
1343
 
1334
1344
  Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
1335
1345
  all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
@@ -1352,13 +1362,15 @@ For ease of use and flexibility, the list of files to transfer is specified by t
1352
1362
  --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
1353
1363
  ```
1354
1364
 
1355
- * Although not recommended, because it applies *only* to the `local` transfer agent (i.e. bare ascp), it is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`. In that case, one must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided.
1365
+ * Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
1356
1366
 
1357
1367
  ```
1358
1368
  --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
1359
1369
  ```
1360
1370
 
1361
- In case the file list is provided on the command line (i.e. using `--sources=@args` or `--sources=<Array>`, but not `--sources=@ts`), the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1371
+ This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1372
+
1373
+ In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1362
1374
 
1363
1375
  * `list` : (default) the path of destination is the same as source
1364
1376
  * `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
@@ -1369,6 +1381,8 @@ Example:
1369
1381
  $ ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1370
1382
  ```
1371
1383
 
1384
+ Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
1385
+
1372
1386
  Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
1373
1387
 
1374
1388
  * All files must be in the same source folder.
@@ -1433,15 +1447,154 @@ When multi-session is used, one separate UDP port is used per session (refer to
1433
1447
 
1434
1448
 
1435
1449
 
1436
- ## <a name="scheduling"></a>Scheduling an exclusive execution
1450
+ ## <a name="scheduling"></a>Lock for exclusive execution
1451
+
1452
+ In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
1453
+
1454
+ For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
1455
+
1456
+ * Executing instances may pile-up and kill the system
1457
+ * The same file may be transfered by multiple instances at the same time.
1458
+ * `preview` may generate the same files in multiple instances.
1459
+
1460
+ Usually the OS native scheduler already provides some sort of protection against parallel execution:
1461
+
1462
+ * The Windows scheduler does this by default
1463
+ * Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
1464
+
1465
+ ```
1466
+ /usr/bin/flock -w 0 /var/cron.lock ascli ...
1467
+ ```
1468
+
1469
+ `ascli` natively supports a locking mechanism with option `lock_port`.
1470
+ (Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
1471
+
1472
+ Example:
1473
+
1474
+ Run this same command in two separate terminals within less than 30 seconds:
1475
+
1476
+ ```
1477
+ ascli config echo @ruby:'sleep(30)' --lock-port=12345
1478
+ ```
1479
+
1480
+ The first instance will sleep 30 seconds, the second one will immediately exit like this:
1481
+
1482
+ ```
1483
+ WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
1484
+ ```
1485
+
1486
+ ## "Proven&ccedil;ale"
1487
+
1488
+ `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
1489
+ , some are provided om shared libraries and must be activated. (e.g. using `trapd`)
1490
+
1491
+ The list of supported *PVCL* adapters can be retried with command:
1492
+
1493
+ ```
1494
+ $ ascli conf ascp info
1495
+ +--------------------+-----------------------------------------------------------+
1496
+ | key | value |
1497
+ +--------------------+-----------------------------------------------------------+
1498
+ -----8<----snip---------
1499
+ | product_name | IBM Aspera SDK |
1500
+ | product_version | 4.0.1.182389 |
1501
+ | process | pvcl |
1502
+ | shares | pvcl |
1503
+ | noded | pvcl |
1504
+ | faux | pvcl |
1505
+ | file | pvcl |
1506
+ | stdio | pvcl |
1507
+ | stdio-tar | pvcl |
1508
+ +--------------------+-----------------------------------------------------------+
1509
+ ```
1510
+
1511
+ Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
1512
+
1513
+ Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
1514
+
1515
+ The simplified format is:
1516
+
1517
+ ```
1518
+ <adapter>:///<sub file path>?<arg1>=<val1>&...
1519
+ ```
1520
+
1521
+ One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
1522
+
1523
+ ## <a name="faux_testing"></a>`faux:` for testing
1524
+
1525
+ This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`
1526
+
1527
+ This adapter can be used to simulate a file or a directory.
1528
+
1529
+ To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
1530
+
1531
+ * `fname` is the name that will be assigned to the file on the destination
1532
+ * `fsize` is the number of bytes that will be sent (in decimal).
1533
+
1534
+ Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
1535
+
1536
+ For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
1537
+
1538
+ To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
1539
+
1540
+ ```
1541
+ faux:///dirname?<arg1>=<val1>&...
1542
+ ```
1543
+
1544
+ `dirname` is the folder name and can contain `/` to specify a subfolder.
1545
+
1546
+ Supported arguments are:
1547
+
1548
+ <table>
1549
+ <tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
1550
+ <tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
1551
+ <tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
1552
+ <tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
1553
+ <tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
1554
+ <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
1555
+ <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
1556
+ </table>
1557
+
1558
+
1559
+ The sequence parameter is applied as follows:
1560
+
1561
+ * If `seq` is `random` then each file size is:
1562
+
1563
+ * size +/- (inc * rand())
1564
+ * Where rand is a random number between 0 and 1
1565
+ * Note that file size must not be negative, inc will be set to size if it is greater than size
1566
+ * Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
1437
1567
 
1438
- It is possible to ensure that a given command is only run once at a time with parameter: `--lock-port=nnnn`. This is especially usefull when scheduling a command on a regular basis, for instance involving transfers, and a transfer may last longer than the execution period.
1568
+ * If `seq` is `sequential` then each file size is:
1439
1569
 
1440
- This opens a local TCP server port, and fails if this port is already used, providing a local lock.
1570
+ * size + ((fileindex - 1) * inc)
1571
+ * Where first file is index 1
1572
+ * So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
1573
+ * As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
1441
1574
 
1442
- This option is used when the tools is executed automatically, for instance with "preview" generation.
1575
+ Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
1443
1576
 
1444
- Usually the OS native scheduler shall already provide some sort of such protection (windows scheduler has it natively, linux cron can leverage `flock`).
1577
+ To discard data at the destination, the destination argument is set to `faux://` .
1578
+
1579
+ Examples:
1580
+
1581
+ * Upload 20 gigabytes of random data to file myfile to directory /Upload
1582
+
1583
+ ```
1584
+ $ ascli server upload faux:///myfile\?20g --to-folder=/Upload
1585
+ ```
1586
+
1587
+ * Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
1588
+
1589
+ ```
1590
+ $ ascli server upload /tmp/sample --to-folder=faux://
1591
+ ```
1592
+
1593
+ * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
1594
+
1595
+ ```
1596
+ $ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1597
+ ```
1445
1598
 
1446
1599
  ## <a name="commands"></a>Sample Commands
1447
1600
 
@@ -1462,7 +1615,7 @@ ascli aoc admin ats cluster clouds
1462
1615
  ascli aoc admin ats cluster list
1463
1616
  ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
1464
1617
  ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
1465
- ascli aoc admin res apps_new list
1618
+ ascli aoc admin res application list
1466
1619
  ascli aoc admin res client list
1467
1620
  ascli aoc admin res client_access_key list
1468
1621
  ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
@@ -1520,7 +1673,7 @@ ascli aoc files v3 info
1520
1673
  ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
1521
1674
  ascli aoc organization
1522
1675
  ascli aoc packages list
1523
- ascli aoc packages list --format=csv --fields=id --display=data|head -n 1);\
1676
+ ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
1524
1677
  ascli aoc packages recv --id="my_package_id" --to-folder=.
1525
1678
  ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
1526
1679
  ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
@@ -1530,6 +1683,8 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
1530
1683
  ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
1531
1684
  ascli aoc user info modify @json:'{"name":"dummy change"}'
1532
1685
  ascli aoc user info show
1686
+ ascli aoc user shared_inboxes
1687
+ ascli aoc user workspaces
1533
1688
  ascli aoc workspace
1534
1689
  ascli ats access_key --id=ak_aws delete
1535
1690
  ascli ats access_key --id=akibmcloud --secret=somesecret cluster
@@ -1556,6 +1711,7 @@ ascli config ascp info
1556
1711
  ascli config ascp install
1557
1712
  ascli config ascp products list
1558
1713
  ascli config ascp show
1714
+ ascli config ascp spec
1559
1715
  ascli config check_update
1560
1716
  ascli config doc
1561
1717
  ascli config doc transfer-parameters
@@ -1680,7 +1836,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
1680
1836
  ```
1681
1837
  $ ascli -h
1682
1838
  NAME
1683
- ascli -- a command line tool for Aspera Applications (v4.3.0)
1839
+ ascli -- a command line tool for Aspera Applications (v4.4.0)
1684
1840
 
1685
1841
  SYNOPSIS
1686
1842
  ascli COMMANDS [OPTIONS] [ARGS]
@@ -1737,31 +1893,31 @@ OPTIONS:
1737
1893
  --property=VALUE name of property to set
1738
1894
  --id=VALUE resource identifier (modify,delete,show)
1739
1895
  --config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
1740
- --override=ENUM override existing value: yes, no
1741
1896
  -N, --no-default do not load default configuration for plugin
1742
- --use-generic-client=ENUM wizard: AoC: use global or org specific jwt client id: yes, no
1743
- --pkeypath=VALUE path to private key for JWT (wizard)
1897
+ --override=ENUM Wizard: override existing value: yes, no
1898
+ --use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
1899
+ --default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
1900
+ --test-mode=ENUM Wizard: skip private key check step: yes, no
1901
+ --pkeypath=VALUE Wizard: path to private key for JWT
1744
1902
  --ascp-path=VALUE path to ascp
1745
1903
  --use-product=VALUE use ascp from specified product
1746
1904
  --smtp=VALUE smtp configuration (extended value: hash)
1747
1905
  --fpac=VALUE proxy auto configuration URL
1748
1906
  -P, --presetVALUE load the named option preset from current config file
1749
- --default=VALUE set as default configuration for specified plugin
1750
1907
  --secret=VALUE default secret
1751
1908
  --secrets=VALUE secret repository (Hash)
1752
1909
  --sdk-url=VALUE URL to get SDK
1753
- --sdk-folder=VALUE SDK folder location
1910
+ --sdk-folder=VALUE SDK folder path
1754
1911
  --notif-to=VALUE email recipient for notification of transfers
1755
1912
  --notif-template=VALUE email ERB template for notification of transfers
1756
- --test-mode=ENUM skip user validation in wizard mode: yes, no
1757
- --version-check-days=VALUE period to check neew version in days (zero to disable)
1913
+ --version-check-days=VALUE period in days to check new version (zero to disable)
1758
1914
  --ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
1759
1915
  --local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
1760
1916
  --to-folder=VALUE destination folder for downloaded files
1761
1917
  --sources=VALUE list of source files (see doc)
1762
- --transfer-info=VALUE additional information for transfer client
1918
+ --transfer-info=VALUE parameters for transfer agent
1763
1919
  --src-type=ENUM type of file list: list, pair
1764
- --transfer=ENUM type of transfer: direct, httpgw, connect, node, aoc
1920
+ --transfer=ENUM type of transfer agent: direct, httpgw, connect, node
1765
1921
  --progress=ENUM type of progress bar: none, native, multi
1766
1922
 
1767
1923
 
@@ -1782,7 +1938,7 @@ OPTIONS:
1782
1938
  --validator=VALUE identifier of validator (optional for central)
1783
1939
  --asperabrowserurl=VALUE URL for simple aspera web ui
1784
1940
  --name=VALUE sync name
1785
- --token=ENUM todo: type of token used for transfers: aspera, basic, auto
1941
+ --token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
1786
1942
 
1787
1943
 
1788
1944
  COMMAND: orchestrator
@@ -2213,19 +2369,55 @@ It allows actions (create, update, delete) on "resources": users, group, nodes,
2213
2369
 
2214
2370
  Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2215
2371
 
2216
- To get more resources when doing request add:
2372
+ #### Listing resources
2373
+
2374
+ The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
2375
+
2376
+ The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
2377
+
2378
+ The following parameters are supported:
2379
+
2380
+ * `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
2381
+ * `sort`: name of fields to sort results, prefix with `-` for reverse order.
2382
+ * `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
2383
+ * `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
2384
+ * `page` : native api parameter, in general do not use (added by
2385
+ * `per_page` : native api parameter, number of items par api call, in general do not use
2386
+ * Other specific parameters depending on resource type.
2387
+
2388
+ Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
2389
+
2390
+ Other parameters are directly sent as parameters to the GET request on API.
2391
+
2392
+ `page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
2393
+
2394
+ `q` and `sort` are available on most resrouce types.
2395
+
2396
+ Other parameters depend on the type of entity (refer to AoC API).
2397
+
2398
+ Examples:
2399
+
2400
+ * List users with `laurent` in name:
2217
2401
 
2218
2402
  ```
2219
- --query=@json:'{"per_page":10000}'
2403
+ ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
2220
2404
  ```
2221
2405
 
2222
- other query parameters can be used:
2406
+ * List users who logded-in before a date:
2407
+
2223
2408
  ```
2224
- --query=@json:'{"member_of_any_workspace":true}'
2225
- --query=@json:'{"q":"laurent"}'
2409
+ ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2226
2410
  ```
2227
2411
 
2228
- Refer to the AoC API for full list of query parameters.
2412
+ * List external users and sort in reverse alphabetical order using name:
2413
+
2414
+ ```
2415
+ ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
2416
+ ```
2417
+
2418
+ Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
2419
+
2420
+ Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
2229
2421
 
2230
2422
  #### Access Key secrets
2231
2423
 
@@ -2326,17 +2518,20 @@ $ ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_k
2326
2518
  $ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2327
2519
  ```
2328
2520
 
2329
- # page=1&per_page=10&q=type:(file_upload+OR+file_delete+OR+file_download+OR+file_rename+OR+folder_create+OR+folder_delete+OR+folder_share+OR+folder_share_via_public_link)&sort=-date
2330
- #events=@api_files.read('events',{'q'=>'type:(file_upload OR file_download)'})[:data]
2331
- # can add filters: tag=aspera.files.package_id%3DLA8OU3p8w
2332
- #'tag'=>'aspera.files.package_id%3DJvbl0w-5A'
2521
+ Examples of query (TODO: cleanup):
2522
+
2523
+ ```
2524
+ {"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
2525
+
2526
+ {"tag":"aspera.files.package_id=LA8OU3p8w"}
2527
+
2333
2528
  # filter= 'id', 'short_summary', or 'summary'
2334
2529
  # count=nnn
2335
2530
  # tag=x.y.z%3Dvalue
2336
2531
  # iteration_token=nnn
2337
2532
  # after_time=2016-05-01T23:53:09Z
2338
2533
  # active_only=true|false
2339
-
2534
+ ```
2340
2535
 
2341
2536
  * Display node events (events)
2342
2537
 
@@ -2347,7 +2542,7 @@ $ ascli aoc admin res node --secret=_secret_ v3 events
2347
2542
  * display members of a workspace
2348
2543
 
2349
2544
  ```
2350
- $ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2545
+ $ ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2351
2546
  :.............:.........:..................................:
2352
2547
  : member_type : manager : member.email :
2353
2548
  :.............:.........:..................................:
@@ -2368,27 +2563,27 @@ other query parameters:
2368
2563
 
2369
2564
  * <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
2370
2565
 
2371
- a- get id of first workspace
2566
+ a- Get id of first workspace
2372
2567
 
2373
2568
  ```
2374
2569
  WS1='First Workspace'
2375
2570
  WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
2376
2571
  ```
2377
2572
 
2378
- b- get id of second workspace
2573
+ b- Get id of second workspace
2379
2574
 
2380
2575
  ```
2381
2576
  WS2='Second Workspace'
2382
2577
  WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
2383
2578
  ```
2384
2579
 
2385
- c- extract membership information and change workspace id
2580
+ c- Extract membership information
2386
2581
 
2387
2582
  ```
2388
- $ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"per_page":10000,"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2583
+ $ ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2389
2584
  ```
2390
2585
 
2391
- d- convert to creation data for second workspace:
2586
+ d- Convert to creation data for second workspace:
2392
2587
 
2393
2588
  ```
2394
2589
  grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
@@ -2400,16 +2595,16 @@ or, using jq:
2400
2595
  jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
2401
2596
  ```
2402
2597
 
2403
- e- add members to second workspace
2598
+ e- Add members to second workspace
2404
2599
 
2405
2600
  ```
2406
2601
  $ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2407
2602
  ```
2408
2603
 
2409
- * get users who did not log since a date
2604
+ * Get users who did not log since a date
2410
2605
 
2411
2606
  ```
2412
- $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,"q":"last_login_at:<2018-05-28"}'
2607
+ $ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2413
2608
  :...............................:
2414
2609
  : email :
2415
2610
  :...............................:
@@ -2418,10 +2613,10 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
2418
2613
  :...............................:
2419
2614
  ```
2420
2615
 
2421
- * list "Limited" users
2616
+ * List "Limited" users
2422
2617
 
2423
2618
  ```
2424
- $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000}' --select=@json:'{"member_of_any_workspace":false}'
2619
+ $ ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
2425
2620
  ```
2426
2621
 
2427
2622
  * Perform a multi Gbps transfer between two remote shared folders
@@ -2464,6 +2659,7 @@ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='
2464
2659
  ```
2465
2660
 
2466
2661
  * create registration key to register a node
2662
+
2467
2663
  ```
2468
2664
  $ ascli aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2469
2665
  jfqslfdjlfdjfhdjklqfhdkl
@@ -2483,6 +2679,19 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
2483
2679
  +-----+---------+
2484
2680
  ```
2485
2681
 
2682
+ * List packages in a given shared inbox
2683
+
2684
+ First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2685
+ (To find out available filters, consult the API definition, or use the web interface in developer mode).
2686
+
2687
+ Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
2688
+
2689
+ ```
2690
+ shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
2691
+
2692
+ ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2693
+ ```
2694
+
2486
2695
  ## Shared folders
2487
2696
 
2488
2697
  * list shared folders in node
@@ -2515,7 +2724,7 @@ Notes:
2515
2724
 
2516
2725
  * the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
2517
2726
  * to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". ascli will resolve the list of email addresses to expected user ids.
2518
- * a recipîent can be a shared inbox, in this case just use the name of the shared inbox as recipient.
2727
+ * a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
2519
2728
  * If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
2520
2729
  * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
2521
2730
  * if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
@@ -2524,7 +2733,13 @@ Examples:
2524
2733
 
2525
2734
  ```
2526
2735
  $ ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
2736
+ ```
2737
+
2738
+ ```
2527
2739
  $ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
2740
+ ```
2741
+
2742
+ ```
2528
2743
  $ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2529
2744
  ```
2530
2745
 
@@ -2540,7 +2755,7 @@ $ ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2540
2755
  * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
2541
2756
  * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
2542
2757
 
2543
- Typically, one would regularly execute this command on a regular basis, using the method of your choice:
2758
+ Typically, one would execute this command on a regular basis, using the method of your choice:
2544
2759
 
2545
2760
  * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2546
2761
  * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
@@ -2548,45 +2763,45 @@ Typically, one would regularly execute this command on a regular basis, using th
2548
2763
 
2549
2764
  ## Download Files
2550
2765
 
2551
- Download of files is straightforward with a specific syntax for the `aspera files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2766
+ Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2552
2767
 
2553
2768
  * if only one source is provided, it is downloaded
2554
2769
  * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2555
2770
 
2556
2771
  ## Find Files
2557
2772
 
2558
- The command `aspera files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2773
+ The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2559
2774
 
2560
2775
  The expression can be of 3 formats:
2561
2776
 
2562
- * empty (default) : all files, equivalent to: `exec:true`
2563
- * not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
2777
+ * empty (default) : all files, equivalent to value: `exec:true`
2778
+ * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2564
2779
 
2565
2780
  For instance, to find files with a special extension, use `--value='\.myext$'`
2566
2781
 
2567
- * starting with `exec:` : the ruby code after the prefix is executed for each entry found. the entry variable name is `f`. the file is displayed if the result is true;
2782
+ * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2568
2783
 
2569
- Examples of expressions: (think to prefix with `exec:` and put in single quotes using bash)
2784
+ Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2570
2785
 
2571
- * find files more recent than 100 days
2786
+ * Find files more recent than 100 days
2572
2787
 
2573
2788
  ```
2574
2789
  f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2575
2790
  ```
2576
2791
 
2577
- * expression to find files older than 1 year on a given node and store in file list
2792
+ * Find files older than 1 year on a given node and store in file list
2578
2793
 
2579
2794
  ```
2580
2795
  $ ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2581
2796
  ```
2582
2797
 
2583
- * delete the files, one by one
2798
+ * Delete the files, one by one
2584
2799
 
2585
2800
  ```
2586
2801
  $ cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2587
2802
  ```
2588
2803
 
2589
- * delete the files in bulk
2804
+ * Delete the files in bulk
2590
2805
 
2591
2806
  ```
2592
2807
  cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
@@ -2721,7 +2936,7 @@ Explanation:
2721
2936
  * `|` the standard output of the first command is fed into the second one
2722
2937
  * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2723
2938
  * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2724
- * `--transfer=node` use transfer agent type `node` instead of default `direct`
2939
+ * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
2725
2940
  * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2726
2941
 
2727
2942
  Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
@@ -2770,6 +2985,14 @@ This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
2770
2985
 
2771
2986
  Both password and SSH keys auth are supported.
2772
2987
 
2988
+ If not username is provided, the default transfer user `xfer` is used.
2989
+
2990
+ If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
2991
+
2992
+ ```
2993
+ $ ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2994
+ ```
2995
+
2773
2996
  Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
2774
2997
 
2775
2998
  Examples:
@@ -2807,7 +3030,7 @@ This means that you dont have such an ssh agent running:
2807
3030
  $ ascli server --ssh-options=@ruby:'{use_agent: false}' ...
2808
3031
  ```
2809
3032
 
2810
- This can also be set as default using a preset
3033
+ This can also be set as default using a preset.
2811
3034
 
2812
3035
  ## Example
2813
3036
 
@@ -2827,18 +3050,36 @@ This creates a [option preset](#lprt) "aspera_demo_server" and set it as default
2827
3050
 
2828
3051
  This plugin gives access to capabilities provided by HSTS node API.
2829
3052
 
2830
- ## Simple Operations
3053
+ ## File Operations
2831
3054
 
2832
3055
  It is possible to:
2833
3056
  * browse
2834
3057
  * transfer (upload / download)
2835
3058
  * ...
2836
3059
 
3060
+ For transfers, it is possible to control how transfer is authorized using option: `token_type`:
3061
+
3062
+ * `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
3063
+ * `basic` : transfer spec is created like this:
3064
+
3065
+ ```
3066
+ {
3067
+ "remote_host": address of node url,
3068
+ "remote_user": "xfer",
3069
+ "ssh_port": 33001,
3070
+ "token": "Basic <base 64 encoded user/pass>",
3071
+ "direction": send/recv
3072
+ }
3073
+ ```
3074
+
3075
+ * `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
3076
+
2837
3077
  ## Central
2838
3078
 
2839
3079
  The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
2840
3080
 
2841
3081
  Filtering can be applied:
3082
+
2842
3083
  ```
2843
3084
  $ ascli node central file list
2844
3085
  ```
@@ -2864,7 +3105,6 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
2864
3105
  * Start watchd and watchfolderd services running as a system user having access to files
2865
3106
  * configure a watchfolder to define automated transfers
2866
3107
 
2867
-
2868
3108
  ```
2869
3109
  $ ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2870
3110
  $ ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
@@ -3182,7 +3422,7 @@ $ ascli cos node info
3182
3422
  $ ascli cos node upload 'faux:///sample1G?1g'
3183
3423
  ```
3184
3424
 
3185
- Note: we generate a dummy file `sample1G` if size 2GB using the `faux` PVCL (man ascp), but you can of course send a real file by specifying a real file instead.
3425
+ Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
3186
3426
 
3187
3427
  # Plugin: IBM Aspera Sync
3188
3428
 
@@ -3279,8 +3519,10 @@ yum install -y ImageMagick optipng
3279
3519
 
3280
3520
  ### Video: FFmpeg
3281
3521
 
3522
+ The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
3523
+
3282
3524
  ```
3283
- curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && tar xJvf - && rm -f /opt/ffmpeg /usr/bin/{ffmpeg,ffprobe} && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3525
+ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3284
3526
  ```
3285
3527
 
3286
3528
  ### Office: Unoconv and Libreoffice
@@ -3578,7 +3820,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
3578
3820
  * `EX_loglevel` to change log level of the tool
3579
3821
  * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
3580
3822
 
3581
- Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the "local" transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3823
+ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3582
3824
 
3583
3825
  ## Comparison of interfaces
3584
3826
 
@@ -3679,7 +3921,7 @@ Note that:
3679
3921
 
3680
3922
  * `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
3681
3923
  * most, but not all native ascp arguments are available as standard transfer_spec parameters
3682
- * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
3924
+ * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
3683
3925
 
3684
3926
  ### server side and configuration
3685
3927
 
@@ -3698,7 +3940,7 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
3698
3940
 
3699
3941
  ```
3700
3942
 
3701
- The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3943
+ The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3702
3944
 
3703
3945
  # Aspera Health check and Nagios
3704
3946
 
@@ -3775,9 +4017,19 @@ So, it evolved into `ascli`:
3775
4017
 
3776
4018
  # Changes (Release notes)
3777
4019
 
4020
+ * 4.4.0
4021
+
4022
+ * new: `aoc packages list` add possibility to add filter with option `query`
4023
+ * new: `aoc admin res xxx list` now get all items by default #50
4024
+ * new: `preset` option can specify name or hash value
4025
+ * new: `node` plugin accepts bearer token and access key as credential
4026
+ * new: `node` option `token_type` allows using basic token in addition to aspera type.
4027
+ * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
4028
+ * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
4029
+
3778
4030
  * 4.3.0
3779
4031
 
3780
- * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on `direct` transfer agent.
4032
+ * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
3781
4033
  * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3782
4034
 
3783
4035
  * 4.2.2
@@ -3802,20 +4054,19 @@ So, it evolved into `ascli`:
3802
4054
  * new: command `aoc remind` to receive organization membership by email
3803
4055
  * new: in `preview` option `value` to filter out on file name
3804
4056
  * new: `initdemo` to initialize for demo server
3805
- * new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
4057
+ * new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3806
4058
  * fix: on Windows `conf ascp use` expects ascp.exe
3807
4059
  * fix: (break) multi_session_threshold is Integer, not String
3808
4060
  * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3809
4061
  * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3810
4062
  * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3811
- * change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
4063
+ * change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
3812
4064
 
3813
4065
  * 4.1.0
3814
4066
 
3815
- * fix: remove keys from transfer spec and command line when not needed
3816
- * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3817
- * new: update documentation with regard to offline and docker installation
3818
- * new: renamed command `nagios_check` to `health`
4067
+ * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
4068
+ * new: update documentation with regard to offline and docker installation
4069
+ * new: renamed command `nagios_check` to `health`
3819
4070
  * new: agent `http_gw` now supports upload
3820
4071
  * new: added option `sdk_url` to install SDK from local file for offline install
3821
4072
  * new: check new gem version periodically
@@ -3866,8 +4117,8 @@ So, it evolved into `ascli`:
3866
4117
  * allow bulk delete in `aspera files` with option `bulk=yes`
3867
4118
  * fix getting connect versions
3868
4119
  * added section for Aix
3869
- * support all ciphers for `local`ascp (incl. gcm, etc..)
3870
- * added transfer spec param `apply_local_docroot` for `local`
4120
+ * support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
4121
+ * added transfer spec param `apply_local_docroot` for [`direct`](#direct)
3871
4122
 
3872
4123
  * 0.11.4
3873
4124
 
@@ -3922,7 +4173,7 @@ So, it evolved into `ascli`:
3922
4173
  * 0.10.12
3923
4174
 
3924
4175
  * added support for AoC node registration keys
3925
- * replaced option : `local_resume` with `transfer_info` for agent `direct`
4176
+ * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
3926
4177
  * Transfer agent is no more a Singleton instance, but only one is used in CLI
3927
4178
  * `@incps` : new extended value modifier
3928
4179
  * ATS: no more provides access keys secrets: now user must provide it
@@ -3972,9 +4223,9 @@ So, it evolved into `ascli`:
3972
4223
 
3973
4224
  * 0.10.2
3974
4225
 
3975
- * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
3976
- * added doc for "cargo" like actions
3977
- * added doc for multi-session
4226
+ * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
4227
+ * added doc for "cargo" like actions
4228
+ * added doc for multi-session
3978
4229
 
3979
4230
  * 0.10.1
3980
4231
 
@@ -4002,22 +4253,22 @@ So, it evolved into `ascli`:
4002
4253
 
4003
4254
  * 0.9.33
4004
4255
 
4005
- * new command to display basic token of node
4006
- * new command to display bearer token of node in AoC
4007
- * the --fields= option, support +_fieldname_ to add a field to default fields
4008
- * many small changes
4256
+ * new command to display basic token of node
4257
+ * new command to display bearer token of node in AoC
4258
+ * the --fields= option, support +_fieldname_ to add a field to default fields
4259
+ * many small changes
4009
4260
 
4010
4261
  * 0.9.32
4011
4262
 
4012
- * all Faspex public links are now supported
4013
- * removed faspex operation recv_publink
4014
- * replaced with option `link` (consistent with AoC)
4263
+ * all Faspex public links are now supported
4264
+ * removed faspex operation recv_publink
4265
+ * replaced with option `link` (consistent with AoC)
4015
4266
 
4016
4267
  * 0.9.31
4017
4268
 
4018
- * added more support for public link: receive and send package, to user or dropbox and files view.
4019
- * delete expired file lists
4020
- * changed text table gem from text-table to terminal-table because it supports multiline values
4269
+ * added more support for public link: receive and send package, to user or dropbox and files view.
4270
+ * delete expired file lists
4271
+ * changed text table gem from text-table to terminal-table because it supports multiline values
4021
4272
 
4022
4273
  * 0.9.27
4023
4274
 
@@ -4037,149 +4288,149 @@ So, it evolved into `ascli`:
4037
4288
 
4038
4289
  * 0.9.24
4039
4290
 
4040
- * fix bug where AoC node to node transfer did not work
4041
- * fix bug on error if ED25519 private key is defined in .ssh
4291
+ * fix bug where AoC node to node transfer did not work
4292
+ * fix bug on error if ED25519 private key is defined in .ssh
4042
4293
 
4043
4294
  * 0.9.23
4044
4295
 
4045
- * defined REST error handlers, more error conditions detected
4046
- * commands to select specific ascp location
4296
+ * defined REST error handlers, more error conditions detected
4297
+ * commands to select specific ascp location
4047
4298
 
4048
4299
  * 0.9.21
4049
4300
 
4050
- * supports simplified wizard using global client
4051
- * only ascp binary is required, other SDK (keys) files are now generated
4301
+ * supports simplified wizard using global client
4302
+ * only ascp binary is required, other SDK (keys) files are now generated
4052
4303
 
4053
4304
  * 0.9.20
4054
4305
 
4055
- * improved wizard (prepare for AoC global client id)
4056
- * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
4057
- * removed outdated pictures from this doc
4306
+ * improved wizard (prepare for AoC global client id)
4307
+ * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
4308
+ * removed outdated pictures from this doc
4058
4309
 
4059
4310
  * 0.9.19
4060
4311
 
4061
- * added command aspera bearer --scope=xx
4312
+ * added command aspera bearer --scope=xx
4062
4313
 
4063
4314
  * 0.9.18
4064
4315
 
4065
- * enhanced aspera admin events to support query
4316
+ * enhanced aspera admin events to support query
4066
4317
 
4067
4318
  * 0.9.16
4068
4319
 
4069
- * AoC transfers are now reported in activity app
4070
- * new interface for Rest class authentication (keep backward compatibility)
4320
+ * AoC transfers are now reported in activity app
4321
+ * new interface for Rest class authentication (keep backward compatibility)
4071
4322
 
4072
4323
  * 0.9.15
4073
4324
 
4074
- * new feature: "find" command in aspera files
4075
- * sample code for transfer API
4325
+ * new feature: "find" command in aspera files
4326
+ * sample code for transfer API
4076
4327
 
4077
4328
  * 0.9.12
4078
4329
 
4079
- * add nagios commands
4080
- * support of ATS for IBM Cloud, removed old version based on aspera id
4330
+ * add nagios commands
4331
+ * support of ATS for IBM Cloud, removed old version based on aspera id
4081
4332
 
4082
4333
  * 0.9.11
4083
4334
 
4084
- * Breaking change: @stdin is now @stdin:
4085
- * support of ATS for IBM Cloud, removed old version based on aspera id
4335
+ * Breaking change: @stdin is now @stdin:
4336
+ * support of ATS for IBM Cloud, removed old version based on aspera id
4086
4337
 
4087
4338
 
4088
4339
  * 0.9.10
4089
4340
 
4090
- * Breaking change: parameter transfer-node becomes more generic: transfer-info
4091
- * Display SaaS storage usage with command: aspera admin res node --id=nn info
4092
- * cleaner way of specifying source file list for transfers
4093
- * Breaking change: replaced download_mode option with http_download action
4341
+ * Breaking change: parameter transfer-node becomes more generic: transfer-info
4342
+ * Display SaaS storage usage with command: aspera admin res node --id=nn info
4343
+ * cleaner way of specifying source file list for transfers
4344
+ * Breaking change: replaced download_mode option with http_download action
4094
4345
 
4095
4346
  * 0.9.9
4096
4347
 
4097
- * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
4098
- * Now supports "cargo" for Aspera on Cloud (automatic package download)
4348
+ * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
4349
+ * Now supports "cargo" for Aspera on Cloud (automatic package download)
4099
4350
 
4100
4351
  * 0.9.8
4101
4352
 
4102
- * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
4103
- * AoC: share to share transfer with command "transfer"
4353
+ * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
4354
+ * AoC: share to share transfer with command "transfer"
4104
4355
 
4105
4356
  * 0.9.7
4106
4357
 
4107
- * homogeneous [_transfer-spec_](#transferspec) for node and local
4108
- * preview persistency goes to unique file by default
4109
- * catch mxf extension in preview as video
4110
- * Faspex: possibility to download all paclages by specifying id=ALL
4111
- * Faspex: to come: cargo-like function to download only new packages with id=NEW
4358
+ * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
4359
+ * preview persistency goes to unique file by default
4360
+ * catch mxf extension in preview as video
4361
+ * Faspex: possibility to download all paclages by specifying id=ALL
4362
+ * Faspex: to come: cargo-like function to download only new packages with id=NEW
4112
4363
 
4113
4364
  * 0.9.6
4114
4365
 
4115
- * Breaking change: `@param:`is now `@preset:` and is generic
4116
- * AoC: added command to display current workspace information
4366
+ * Breaking change: `@param:`is now `@preset:` and is generic
4367
+ * AoC: added command to display current workspace information
4117
4368
 
4118
4369
  * 0.9.5
4119
4370
 
4120
- * new parameter: new_user_option used to choose between public_link and invite of external users.
4121
- * fixed bug in wizard, and wizard uses now product detection
4371
+ * new parameter: new_user_option used to choose between public_link and invite of external users.
4372
+ * fixed bug in wizard, and wizard uses now product detection
4122
4373
 
4123
4374
  * 0.9.4
4124
4375
 
4125
- * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
4126
- * AoC Package send supports external users
4127
- * new command to export AoC config to Aspera CLI config
4376
+ * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
4377
+ * AoC Package send supports external users
4378
+ * new command to export AoC config to Aspera CLI config
4128
4379
 
4129
4380
  * 0.9.3
4130
4381
 
4131
- * REST error message show host and code
4132
- * option for quiet display
4133
- * modified transfer interface and allow token re-generation on error
4134
- * async add admin command
4135
- * async add db parameters
4136
- * Breaking change: new option "sources" to specify files to transfer
4382
+ * REST error message show host and code
4383
+ * option for quiet display
4384
+ * modified transfer interface and allow token re-generation on error
4385
+ * async add admin command
4386
+ * async add db parameters
4387
+ * Breaking change: new option "sources" to specify files to transfer
4137
4388
 
4138
4389
  * 0.9.2
4139
4390
 
4140
- * Breaking change: changed AoC package creation to match API, see AoC section
4391
+ * Breaking change: changed AoC package creation to match API, see AoC section
4141
4392
 
4142
4393
  * 0.9.1
4143
4394
 
4144
- * Breaking change: changed faspex package creation to match API, see Faspex section
4395
+ * Breaking change: changed faspex package creation to match API, see Faspex section
4145
4396
 
4146
4397
  * 0.9
4147
4398
 
4148
- * Renamed the CLI from aslmcli to `ascli`
4149
- * Automatic rename and conversion of former config folder from aslmcli to `ascli`
4399
+ * Renamed the CLI from aslmcli to `ascli`
4400
+ * Automatic rename and conversion of former config folder from aslmcli to `ascli`
4150
4401
 
4151
4402
  * 0.7.6
4152
4403
 
4153
- * add "sync" plugin
4404
+ * add "sync" plugin
4154
4405
 
4155
4406
  * 0.7
4156
4407
 
4157
- * Breaking change: AoC package recv take option if for package instead of argument.
4158
- * Breaking change: Rest class and Oauth class changed init parameters
4159
- * AoC: receive package from public link
4160
- * select by col value on output
4161
- * added rename (AoC, node)
4408
+ * Breaking change: AoC package recv take option if for package instead of argument.
4409
+ * Breaking change: Rest class and Oauth class changed init parameters
4410
+ * AoC: receive package from public link
4411
+ * select by col value on output
4412
+ * added rename (AoC, node)
4162
4413
 
4163
4414
  * 0.6.19
4164
4415
 
4165
4416
  Breaking change:
4166
4417
 
4167
- * ats server list provisioned &rarr; ats cluster list
4168
- * ats server list clouds &rarr; ats cluster clouds
4169
- * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4170
- * ats server id xxx &rarr; ats cluster show --id=xxx
4171
- * ats subscriptions &rarr; ats credential subscriptions
4172
- * ats api_key repository list &rarr; ats credential cache list
4173
- * ats api_key list &rarr; ats credential list
4174
- * ats access_key id xxx &rarr; ats access_key --id=xxx
4418
+ * ats server list provisioned &rarr; ats cluster list
4419
+ * ats server list clouds &rarr; ats cluster clouds
4420
+ * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4421
+ * ats server id xxx &rarr; ats cluster show --id=xxx
4422
+ * ats subscriptions &rarr; ats credential subscriptions
4423
+ * ats api_key repository list &rarr; ats credential cache list
4424
+ * ats api_key list &rarr; ats credential list
4425
+ * ats access_key id xxx &rarr; ats access_key --id=xxx
4175
4426
 
4176
4427
  * 0.6.18
4177
4428
 
4178
- * some commands take now --id option instead of id command.
4429
+ * some commands take now --id option instead of id command.
4179
4430
 
4180
4431
  * 0.6.15
4181
4432
 
4182
- * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
4433
+ * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
4183
4434
 
4184
4435
  # BUGS, FEATURES, CONTRIBUTION
4185
4436
 
@@ -4232,8 +4483,8 @@ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "C
4232
4483
 
4233
4484
  * remove rest and oauth classes and use ruby standard gems:
4234
4485
 
4235
- * oauth
4236
- * https://github.com/rest-client/rest-client
4486
+ * oauth
4487
+ * https://github.com/rest-client/rest-client
4237
4488
 
4238
4489
  * use Thor or any standard Ruby CLI manager
4239
4490