aspera-cli 4.18.1 → 4.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 98185fd9f823fc0d85ebf7183d5c88cf6e0aac7cafd8325802bfa0724c52ddf4
-  data.tar.gz: 848811c07aa60060cae1320cd0512e72a73bb78fee9e1837245ef856af3883de
+  metadata.gz: 89df3ebc2bf7a146a16e59154a73efcfc2e11f7767aba28725f9443de04772b7
+  data.tar.gz: 33469a2bc4991b106ab3002927e823f280c016fc070a2bdcbb3edb286506041d
 SHA512:
-  metadata.gz: ede08e6e6c6ce3ef33d0c402730ee1854b5d0364da3d47ef736f0e7680a9f8cc1098316fe5945c15aab98ed931c26dd625ea7fb9673e70068dfd2755bb7a053a
-  data.tar.gz: 1bf901163167844fbada435d0cbac759075b51e9f2da22f6f2b97e99be53c202d69e83ec38f0ee4cade3f937a29c502e22e75ac4452b5e0bb426b0fafa7d883c
+  metadata.gz: d57c69a340b866cfe5c3003c94ecb3ebc8c61aba7011ee4b8683da60d9615dd060fb94ac153e16eb98bd11f7dfa1deb117b1ca2ecd5bfe5d46e5a390ec290a01
+  data.tar.gz: fa364474177bc6d2f648474b1eed12b45733e84a5d4896d3075225ed1212aed27f64725ba38a400a3f9c5da2e7e2acf46d6a41164e5d92c1e2ce3a94498c5401

data/CHANGELOG.md

@@ -1,6 +1,19 @@
 # Changes (Release notes)
 
+* 4.19.0
+  * New Features:
+    * `server`: add support for `async` (Aspera Sync) from Transfer SDK
+    * **global**: #156 support sending folders with httpgw
+    * **global**: new value for option `format`: `multi`
+  * Issues Fixed:
+    * `aoc`: #157 fix problem with `files browse` on a link
+    * `sync`: better documentation and handling of options.
+  * Breaking Changes:
+    * **global**: Default value for direct agent option `transfer_info.multi_incr_udp` is `true` on Windows, and now `false` on other platforms.
+    * **global**: Token based transfers now use the RSA key only. Direct agent option `transfer_info.client_ssh_key` allows changing this behavior.
+  
 * 4.18.1
+
   * New Features:
     * none
   * Issues Fixed:

data/CONTRIBUTING.md

@@ -228,7 +228,7 @@ Then procedure is as follows:
 
 ## Docker image build
 
-The Dockerfile template allows customizing the optional copy of gem file , versus install from rubygems,org as well as customizing the retrieval of the SDK.
+The `Dockerfile.tmpl.erb` template allows customization of the optional copying of the `aspera-cli.gem` file, as opposed to installing from rubygems.org. It also supports customizing the retrieval of the SDK.
 
 ### Default image build
 
@@ -278,19 +278,11 @@ make docker_push_beta
 
 ## Single executable build
 
-It is possible to build `ascli` as a single native executable using `rubyc` (gem `ruby-packer`).
+Initially, `rubyc` (gem [`ruby-packer`](https://github.com/pmq20/ruby-packer) and [here](https://github.com/you54f/ruby-packer)) was used to build a single executable.
 
-```bash
-make single
-```
-
-This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
+https://www.tebako.org/
 
-> **Note:** Perl `Findbin` is needed, install on Linux with:
-
-```bash
-sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
-```
+A modern version of this is now used: [`tebako`](https://github.com/tamatebako/tebako) for which a container is provided.
 
 ## Long Term Implementation and delivery improvements
 
@@ -299,3 +291,4 @@ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
   - <https://github.com/oauth-xx/oauth2>
 - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
 - Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (`rubyc`)
+- look at <https://github.com/phusion/traveling-ruby>

data/README.md

@@ -10,7 +10,7 @@
 
 ## Introduction
 
-Version : 4.18.1
+Version : 4.19.0
 
 Laurent/2016-2024
 
@@ -92,7 +92,7 @@ Once the gem is installed, `ascli` shall be accessible:
 
 ```console
 $ ascli --version
-4.18.1
+4.19.0
 ```
 
 ### First use
@@ -706,7 +706,7 @@ ascli -v
 ```
 
 ```text
-4.18.1
+4.19.0
 ```
 
 In order to keep persistency of configuration on the host, you should specify your user's configuration folder as a volume for the container.
@@ -1497,6 +1497,7 @@ If transposition of single object is not desired, use option: `transpose_single`
 The style of output can be set using the `format` option, supporting:
 
 - `table` : Text table (default)
+- `multi` : List of elements are displayed as a list of tables
 - `text` : Value as String
 - `ruby` : Ruby code
 - `json` : JSON code
@@ -1924,7 +1925,8 @@ ascp use /usr/bin/ascp
 check_update
 coffee
 coffee --ui=text
-coffee --ui=text --query=@json:'{"text":"true"}'
+coffee --ui=text --image=@json:'{"text":true,"double":false}'
+coffee --ui=text --image=@json:'{"text":true}'
 detect https://faspex4.example.com/path
 detect https://faspex5.example.com/path
 detect https://node.example.com/path
@@ -2745,19 +2747,21 @@ Refer to section [FASP](#fasp-configuration).
 
 The `transfer_info` option accepts the following optional parameters to control multi-session, Web Socket Session, Resume policy and add any argument to `ascp`:
 
-| Name                   | Type  | Description |
-|------------------------|-------|-------------|
-| `wss`                  | Bool  | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
-| `ascp_args`            | Array | Array of strings with native `ascp` arguments.<br/>Default: [] |
-| `spawn_timeout_sec`    | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
-| `spawn_delay_sec`      | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
-| `multi_incr_udp`       | Bool  | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
-| `trusted_certs`        | Array | List of repositories for trusted certificates. |
-| `resume`               | Hash  | Resume parameters. See below |
-| `resume.iter_max`      | int   | Max number of retry on error<br/>Default: 7 |
-| `resume.sleep_initial` | int   | First Sleep before retry<br/>Default: 2 |
-| `resume.sleep_factor`  | int   | Multiplier of sleep period between attempts<br/>Default: 2 |
-| `resume.sleep_max`     | int   | Default: 60 |
+| Name                   | Type    | Description |
+|------------------------|---------|-------------|
+| `wss`                  | Bool    | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
+| `quiet`                | Bool    | If `true`, then `ascp` progress bar is not shown.<br/>Default: `false` |
+| `trusted_certs`        | Array   | List of repositories for trusted certificates. |
+| `client_ssh_key`       | String  | SSH Keys to use for token-based transfers. One of: `dsa_rsa`, `rsa`, `per_client`. Default: `rsa` |
+| `ascp_args`            | Array   | Array of strings with native `ascp` arguments.<br/>Default: `[]` |
+| `spawn_timeout_sec`    | Float   | Multi session<br/>Verification time that `ascp` is running<br/>Default: `3` |
+| `spawn_delay_sec`      | Float   | Multi session<br/>Delay between startup of sessions<br/>Default: `2` |
+| `multi_incr_udp`       | Bool    | Multi Session<br/>Increment UDP port on multi-session<br/>If `true`, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: `true` on Windows, else `false` |
+| `resume`               | Hash    | Resume parameters. See below |
+| `resume.iter_max`      | Integer | Max number of retry on error<br/>Default: `7` |
+| `resume.sleep_initial` | Integer | First Sleep before retry<br/>Default: `2` |
+| `resume.sleep_factor`  | Integer | Multiplier of sleep period between attempts<br/>Default: `2` |
+| `resume.sleep_max`     | Integer | Default: `60` |
 
 In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` time.
 Sleep between iterations is given by the following formula where `iter_index` is the current iteration index, starting at 0:
@@ -2767,7 +2771,7 @@ max( sleep_max , sleep_initial * sleep_factor ^ iter_index )
 ```
 
 By default, Ruby's root CA store is used to validate any HTTPS endpoint used by `ascp` (e.g. WSS).
-In order to use a custom certificate store, use the `trusted_certs` option.
+In order to use a custom certificate store, use the `trusted_certs` option of direct agent's option `transfer_info`.
 To use `ascp`'s default, use option: `--transfer-info=@json:'{"trusted_certs":null}'`.
 
 Some transfer errors are considered **retry-able** (e.g. timeout) and some other not (e.g. wrong password).
@@ -3561,7 +3565,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
 ```text
 ascli -h
 NAME
-        ascli -- a command line tool for Aspera Applications (v4.18.1)
+        ascli -- a command line tool for Aspera Applications (v4.19.0)
 
 SYNOPSIS
         ascli COMMANDS [OPTIONS] [ARGS]
@@ -3593,7 +3597,7 @@ OPTIONS: global
         --interactive=ENUM           Use interactive input of missing params: [no], yes
         --ask-options=ENUM           Ask even optional options: [no], yes
         --struct-parser=ENUM         Default parser when expected value is a struct: json, ruby
-        --format=ENUM                Output format: text, nagios, ruby, json, jsonpp, yaml, [table], csv, image
+        --format=ENUM                Output format: text, nagios, ruby, json, jsonpp, yaml, [table], multi, csv, image
         --output=VALUE               Destination for results (String)
         --display=ENUM               Output only some information: [info], data, error
         --fields=VALUE               Comma separated list of: fields, or ALL, or DEF (String, Array, Regexp, Proc)
@@ -3617,7 +3621,7 @@ OPTIONS: global
         --pid-file=VALUE             Write process identifier to file, delete on exit (String)
 
 COMMAND: config
-SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test file flush_tokens folder gem genkey image initdemo open plugins preset proxy_check pubkey remote_certificate smtp_settings throw vault wizard
+SUBCOMMANDS: ascp check_update coffee detect documentation echo email_test file flush_tokens folder gem genkey image initdemo open platform plugins preset proxy_check pubkey remote_certificate smtp_settings throw vault wizard
 OPTIONS:
         --home=VALUE                 Home folder for tool (String)
         --config-file=VALUE          Path to YAML file with preset configuration
@@ -5061,8 +5065,11 @@ files show /
 files show testdst/test_file.bin
 files sync admin status --sync-info=@json:'{"name":"my_aoc_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
 files sync admin status --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/testdst","reset":true}]}'
+files sync start --sync-info=@json:'{"name":"my_aoc_sync1","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
 files sync start --sync-info=@json:'{"name":"my_aoc_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/testdst"}}'
 files sync start --sync-info=@json:'{"sessions":[{"name":"my_aoc_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/testdst","reset":true}]}'
+files sync start pull /testdst /data/local_sync --sync-info=@json:'{"reset":true,"quiet":false}'
+files sync start pull /testdst /data/local_sync --sync-info=@json:'{"sessions":[{"reset":true}]}'
 files thumbnail my_test_folder/video_file.mpg
 files thumbnail my_test_folder/video_file.mpg --query=@json:'{"text":true,"double":true}'
 files transfer push /testsrc --to-folder=/testdst test_file.bin
@@ -5830,8 +5837,8 @@ ssync summary %name:my_node_sync
 stream list
 sync admin status --sync-info=@json:'{"name":"my_node_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/aspera-test-dir-tiny"}}'
 sync admin status --sync-info=@json:'{"sessions":[{"name":"my_node_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/aspera-test-dir-tiny","reset":true}]}'
+sync start --sync-info=@json:'{"name":"my_node_sync1","reset":true,"quiet":false,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/aspera-test-dir-tiny"}}'
 sync start --sync-info=@json:'{"name":"my_node_sync2","reset":true,"direction":"pull","local":{"path":"/data/local_sync"},"remote":{"path":"/aspera-test-dir-tiny"}}'
-sync start --sync-info=@json:'{"sessions":[{"name":"my_node_sync1","direction":"pull","local_dir":"/data/local_sync","remote_dir":"/aspera-test-dir-tiny","reset":true}]}'
 transfer list --query=@json:'{"active_only":true}'
 transfer sessions
 transport
@@ -7124,28 +7131,52 @@ An interface for the `async` utility is provided in the following plugins:
 - `aoc files sync` (uses node)
 - `shares files sync` (uses node)
 
-The main advantage over the `async` command line when using `server` is the possibility to use a configuration file, using standard options of `ascli`.
+One advantage over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`.
+Moreover `ascli` supports sync with application requiring token-based authorization.
 
-In this case, some of the `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
+Some of the `sync` parameters are filled by the related plugin using transfer spec parameters (e.g. including token).
 
-> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
+> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). The Transfer SDK 1.3 includes this.
 
-Two JSON syntax are supported for option `sync_info`.
+`sync` supports 0 or 3 arguments:
 
-### `async` JSON: API format
+- direction
+- local folder
+- remote folder
+
+Additional options can be provided with option `sync_info`, for which two syntax are possible, as follows.
+
+### `async` API and conf format
 
 It is the same payload as specified on the option `--conf` of `async` or in node API `/asyncs`.
 This is the preferred syntax and allows a single session definition.
-But there is no progress output nor error messages.
+
+> **Note:** No progress, nor error messages is provided on terminal.
 
 Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
 
-### `async` JSON: Options mapping
+If 3 arguments are provided they are mapped to:
+
+- `direction`
+- `local.path`
+- `remote.path`
+
+### `async` options mapping
 
 `ascli` defines a JSON equivalent to regular `async`options.
 It is based on a JSON representation of `async` command line options.
 It allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
 
+This is the mode selection if there is either keys `sessions` or `instance` in option `sync_info`.
+
+> **Note:** Progress and error messages are provided on terminal like regular command line invocation of `async`.
+
+If 3 arguments are provided, they are applied to the first (and only) session and mapped to:
+
+- `direction`
+- `local_dir`
+- `remote_dir`
+
 ## Hot folder
 
 ### Requirements

data/examples/build_exec

@@ -0,0 +1,85 @@
+#!/bin/bash
+
+set -e
+
+if test $# -ne 2; then
+    echo "Usage: $0 <CLI_TARGET_EXECUTABLE> <CLI_ENTRY_SCRIPT>" 1>&2
+    exit 1
+fi
+
+CLI_TARGET_EXECUTABLE=$1
+CLI_ENTRY_SCRIPT=$(realpath $2)
+CLI_EXEC_NAME=$(basename $CLI_TARGET_EXECUTABLE)
+CLI_ROOT=$(dirname $(dirname $CLI_ENTRY_SCRIPT))
+
+echo "Project folder: $CLI_ROOT"
+echo "Executable:     $CLI_TARGET_EXECUTABLE"
+echo "Exec Name:      $CLI_EXEC_NAME"
+echo "Entry:          $CLI_ENTRY_SCRIPT"
+
+# tebako press --root=$(TB_WORK_DIR) --entry-point=ascli --output=$(TB_WORK_DIR)/$(CLI_TARGET_EXECUTABLE) --Ruby=3.2.4
+
+if true;then
+    tebako_command=tebako
+    tebako_args="--root=$CLI_ROOT --entry-point=ascli --output=$CLI_TARGET_EXECUTABLE --Ruby=3.3.3"
+    case $(uname -s) in
+        Darwin)
+            brew update
+            brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib \
+                ncurses double-conversion boost jemalloc  glog libevent libsodium lz4 xz \
+                libyaml openssl@3
+            # brew install m4
+            # boost fmt
+            #export PATH=$(dirname $(brew ls m4|grep '/m4$'|head -n 1)):$PATH
+            #export PATH=$(dirname $(brew ls binutils|grep '/ar$'|head -n 1)):$PATH
+            #tebako_bin=$CLI_ROOT/tmp/tbkbin
+            tebako_root=$CLI_ROOT/../tebako-tmp
+            #export PATH=$tebako_bin:$PATH
+            mkdir -p $tebako_root
+            #mkdir -p $tebako_bin
+            # need ar from apple
+            #rm -f $tebako_bin/ar
+            #ln -s /usr/bin/ar $tebako_bin
+            export PATH=$(echo $PATH|tr : \\n|grep -v /binutils/|tr \\n :)
+            #tebako setup --Ruby=3.3.4 --prefix=$tebako_root
+            tebako press --Ruby=3.3.4 --prefix=$tebako_root --root=$CLI_ROOT/examples/tebako --entry-point=ascli --output=$CLI_TARGET_EXECUTABLE
+            #tebako press --Ruby=3.3.4 --prefix=$tebako_root --root=$CLI_ROOT --entry-point=$CLI_ENTRY_SCRIPT --output=$CLI_TARGET_EXECUTABLE
+            exit 0
+            ;;
+        Linux)
+            work_dir=/mnt/w
+            tebako press --root=$work_dir --entry-point=ascli --output=$work_dir/ascli.exe --Ruby=3.3.3 
+            tebako_prefix="podman run -it --rm -v $PWD:$work_dir ghcr.io/tamatebako/tebako-ubuntu-20.04:0.8.6"
+            tebako_opts=--patchelf
+            ;;
+        *)
+            echo "This OS is not supported." >&2
+            exit 1
+            ;;
+    esac
+    command $tebako_command $tebako_args
+else
+    # https://github.com/you54f/ruby-packer
+    # https://github.com/YOU54F/ruby-packer/releases
+    set -e
+    FOLDER="$(dirname $0)/../tmp"
+    RUBYC="$FOLDER/rubyc"
+    if test ! -e "$RUBYC"; then
+        mkdir -p "$FOLDER"
+        case $(uname -sm|tr ' ' -) in
+            Darwin-arm64)
+                curl -L https://github.com/YOU54F/ruby-packer/releases/download/rel-20230812/rubyc-Darwin-arm64.tar.gz | tar -xz -C "$FOLDER"
+                mv "$FOLDER/rubyc-Darwin-arm64" "$RUBYC"
+                ;;
+            Linux-x86_64)
+                curl -L https://github.com/YOU54F/ruby-packer/releases/download/rel-20230812/rubyc-Linux-x86_64.tar.gz | tar -xz -C "$FOLDER"
+                mv "$FOLDER/rubyc-Linux-x86_64" "$RUBYC"
+                ;;
+            *)
+                echo "This architecture is not supported." >&2
+                exit 1
+                ;;
+        esac
+    fi
+    exec "$RUBYC" -o "$CLI_TARGET_EXECUTABLE" "$CLI_ENTRY_SCRIPT"
+fi

data/lib/aspera/agent/base.rb

@@ -38,6 +38,8 @@ module Aspera
           end.map{|file|file[0..(-1 - RUBY_EXT.length)].to_sym}
         end
       end
+
+      # Wait for all sessions to terminate and return the status of each session
       def wait_for_completion
         # list of: :success or "error message string"
         statuses = wait_for_transfers_completion