aspera-cli 4.21.1 → 4.22.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94d8abaf1bf8e938fe9ef79a7f0ecc06870b799d44acb0783a8638e88be8585d
-  data.tar.gz: b8aeca6b8c34446aac0f4b54fac0ee52db502a6e28149583e30536cc7ba3ab58
+  metadata.gz: 4142ef4954b98f748d8d0d19c63a53e0faace015b8286480b31b652b994fa15c
+  data.tar.gz: 2c77b1badfd80ddbc7c8d98acdb303039c87ee2e7fe64b6fbcecd7935101426c
 SHA512:
-  metadata.gz: e90b2de02a88fc84ca2e97df962f054200e14c8c2020dd1a1f42f271b15e0f8ca1e67a4ffaefda7c10f6a69c28611a83d400e140c8ccc9e4edb6f56399db8fe8
-  data.tar.gz: 6688107fb04c162b4c18bcc45c1450c73170ecae5d430af782ba596e5879fdaf0ecd01020e8ba48c2b39a543d3f0bb3b88f689242ee98714cfa6bd658b92b520
+  metadata.gz: 524c91bde0ca7c7267a18be627ea00f89de4fd6376fc7d93eb75eb204c0507eb15f3800e5d51536f16996fddf3a26831e5dc3ce7a516701db9098b658d098116
+  data.tar.gz: 4c37acd0e5834e723367594a972dea4506416b3b49ce2c7fc4b0b9d6caa9719272f0fa0158d627a8d5bfc02913aba863a40bff346330ff07c10dc1a6754b9aa6

data/BUGS.md

@@ -12,7 +12,7 @@ Please, make sure to include the following information:
 
 IMPORTANT: Remove any confidential information from the logs before posting them.
 
-[Use Github (Public Issue Reporting)](https://github.com/IBM/aspera-cli/issues)
+[Use GitHub (Public Issue Reporting)](https://github.com/IBM/aspera-cli/issues)
 
 ## Scope of Support
 

data/CHANGELOG.md

@@ -1,30 +1,60 @@
 # Changes (Release notes)
 
+* 4.22.0
+
+  * New Features:
+    * `faspex5`: Support paging for Faspex5 browsing.
+    * `aoc`: #196 Command `packages list` now also supports option `once_only`.
+    * `vault`: Support for IBM HashiCorp Vault to store secrets.
+    * `wizard`: Preset name can now be specified as optional positional parameter.
+    * `config`: New command `ascp schema` displays JSON schema for transfer spec for all, or just one agent.
+    * `node`: #198 By default do not allow creation of folder if a link exists with the same name. Use option `query` with parameter `check` set to `false` to disable.
+    * `node`: In gen4 operations, also used in `aoc files`, new commands: `mklink`, `mkfile`.
+  * Issues Fixed:
+    * `aoc`: #195 `package receive ALL` for shared inbox without workspace now works.
+  * Breaking Changes:
+    * `faspex5`, `aoc`: `gateway` now takes argument as `Hash` with `url` instead of only `String`.
+    * `faspex5 postprocessing`: Now takes a flat `Hash`, instead of multi-level `Hash`.
+    * HTTP: More retry parameters.
+    * `node`: renamed command `http_node_download` to `cat`, and it directly displays the content of the file in the terminal unless option `--output` is specified.
+
+* 4.21.2
+
+  * New Features:
+    * **container**: Updated Ruby to 3.4.2
+  * Issues Fixed:
+    * **global**: #185 `@val:` shall stop processing extended values
+    * **global**: #186 Removed dependency on OpenSSL 3.3 gem to avoid MSYS2 dep on Windows.
+    * `echo`: Display of list (Array) was showing only first element of it.
+    * `transferd`: support for version 1.1.5+
+  * Breaking Changes:
+    * `preview`: Updated Image Magick to v7+
+    * `aoc`: `admin subscription` split into `admin subscription account` and `admin subscription usage`
+    * **agent**: `alpha` renamed to `desktop`
+
 * 4.21.1
 
   * New Features:
     * `config`: New command: `transferd` to list and install specific version of `asperatransferd` and `ascp`
-    * `config`: New command: `token` to list and flush cached tokens.
-    * `faspex5`: New command: `admin contact reset_password`
     * `config`: New command: `tokens` with `list`, `show`, `flush` (replace `flush_tokens`)
+    * `faspex5`: New command: `admin contact reset_password`
     * `aoc`: #178 packages can be browsed, and individual files can be downloaded now.
   * Issues Fixed:
+    * `config`: #175 `ascli config preset set GLOBAL version_check_days 0` causes a bad `config.yaml` to be written
     * `config`: #180 problem in `ascp install`
-    * `aoc`: #184 token cache shall be different per aoc org.
     * `config`: Soft links in transfer SDK archive are correctly extracted
+    * `aoc`: #184 token cache shall be different per AoC org.
     * `aoc`: Fix `packages delete` not working.
     * `direct` agent: #174 Race condition fix with `ascp`: timeout waiting mgt port connect (select not readable)
-    * `config`: #175 `ascli config preset set GLOBAL version_check_days 0` causes a bad `config.yaml` to be written
     * `preview`: #177 fix bug that prevents preview generation to work.
   * Breaking Changes:
-    * **global** : Ruby minimum versions is now 3.1 (mainly due to switch to openssl 3). Future minimum is 3.2. Recommended is 3.4.
     * `transferd`: Use of Aspera Transfer Daemon requires minimum version 1.1.4. agent `trsdk` renamed `transferd`.
     * `ascp`: Default SDK version is now 1.1.4. Removes support for ascp4.
-    * **global** : Options `transpose_single` and `multi_table` replaced with single option `multi_single` and values: `no`, `yes`, `single`.
-    * **global** : Column name for single object is now `field` instead of `key`.
-    * **global** : Ruby minimum versions is now 2.7 (2.6 previously, that removes macOS default ruby support, but newer ruby version can be installed on macOS)
     * `node`: Removed deprecated command prefix `exec:`, use `@ruby:` instead.
-    * **global** : Now uses openssl 3.
+    * **global**: Now uses OpenSSL 3.
+    * **global**: Ruby minimum versions is now 3.1 (mainly due to switch to OpenSSL 3). Future minimum is 3.2. Recommended is 3.4. (that removes macOS default ruby support. Newer Ruby version shall be installed on macOS with `brew`)
+    * **global**: Options `transpose_single` and `multi_table` replaced with single option `multi_single` and values: `no`, `yes`, `single`.
+    * **global**: Column name for single object is now `field` instead of `key`.
 
 * 4.20.0
 
@@ -33,15 +63,15 @@
   * New Features:
     * `aoc`: Improved usability for creation of Admin shared folders.
     * `node`: New option `node_cache` (bool) for gen4 operations.
-    * `node`: Option `root_id` now works always for node gen4, as well as `%id:` for file selection in addition to path.
+    * `node`: Option `root_id` now always works for node gen4, as well as `%id:` for file selection in addition to path.
     * `node`: `transfer list` now uses the `iteration_token` to retrieve all values. Option `once_only` is now supported.
-    * **global** : option `http_options` now include retry options.
+    * **global**: option `http_options` now include retry options.
   * Issues Fixed:
     * `aoc`: Fixed `find` command not working. (undefined variable)
-    * `aoc`: #165 aoc mkdir now follows the last link of containing folder
+    * `aoc`: #165 AoC mkdir now follows the last link of containing folder
   * Breaking Changes:
     * Internal: Basic REST calls now return data directly. (no more `data` key). For advanced calls, use `call`.
-    * Internal: Transfer SDK download is now a 2-step procedure: First get the YAML file from github with URLS for the various platforms and versions, and then download the archive from the official IBM repository.
+    * Internal: Transfer SDK download is now a 2-step procedure: First get the YAML file from GitHub with URLs for the various platforms and versions, and then download the archive from the official IBM repository.
     **global**: option `format=multi` is replaced with option `multi_table=yes`
     * `faspex5`: removed deprecated option `value` replaced with positional parameter.
   
@@ -49,7 +79,7 @@
 
   * New Features:
     * `server`: add support for `async` (Aspera Sync) from Transfer SDK
-    * **global**: #156 support sending folders with httpgw
+    * **global**: #156 support sending folders with `httpgw`
     * **global**: new value for option `format`: `multi`
   * Issues Fixed:
     * `aoc`: #157 fix problem with `files browse` on a link
@@ -160,7 +190,7 @@
     * **global**: option `pid_file` to write tool's PID during execution, deleted on exit
     * `config`: command `remote_certificate` to retrieve a remote certificate
     * `config`: added logger level `trace1` and `trace2`
-    * `config`: `wizard` can detect multiple applications at the same address or url.
+    * `config`: `wizard` can detect multiple applications at the same address or URL.
     * `aoc`: wizard accepts public links
     * `aoc`: support private links, and possibility to list shared folder with workspace `@json:null`
     * `orchestrator`: error analysis for workflow start
@@ -168,7 +198,7 @@
     * `node`: added command `transfer sessions` to list all sessions of all transfers
     * `node`: generate bearer token from private key and user information
     * `node`: access node API with bearer token as credentials
-    * **global**: agent `direct` allows ignoring certificate for wss using http options
+    * **global**: agent `direct` allows ignoring certificate for WSS using HTTP options
     * `preview`: command `show` generates a preview and displays it in terminal
   * Issues Fixed:
     * Ruby warning: `net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError` solved by removing dependency on `net-smtp` from gem spec (already in base ruby).
@@ -233,7 +263,7 @@
     * `faspex5`: list content of package, and allow partial download of package
     * `faspex5`: list packages support multiple pages and items limitations (`max` and `pmax`)
     * `aoc`: files operations with workspace-less user (#109)
-    * `node`: async with gen3 token (#110)
+    * `node`: `async` with gen3 token (#110)
     * `node`: display of preview of file in terminal for access keys
   * Issues Fixed:
     * `cos`: do not use refresh token when not supported
@@ -249,10 +279,10 @@
     * **global**: `delete` operation supports option `value` for deletion parameters
     * `aoc`: command `aoc packages recv` accepts option `query` to specify a shared inbox
     * `faspex`: (v4) user delete accepts option `value` with value `{"destroy":true}` to delete users permanently
-    * `faspex`: (v4) gateway to faspex 5 for package send
+    * `faspex`: (v4) gateway to Faspex 5 for package send
     * `faspex5`: possibility to change email templates
     * `faspex5`: shared folder list and browse
-    * `faspex5`: emulate faspex 4 postprocessing, plugin: `faspex5` command: `postprocessing`
+    * `faspex5`: emulate Faspex 4 post-processing, plugin: `faspex5` command: `postprocessing`
     * `faspex5`: send package from remote source
     * `shares`: option `type` for command `shares admin user`
     * `shares`: full support for shares admin operations
@@ -268,14 +298,14 @@
   * New Features:
     * **global**: `vault`: secret finder, migration from config file
     * **global**: allow removal of transfer spec parameter by setting value to `null`
-    * **global**: option `ascp_opts` allows to provide native `ascp` options on command line
+    * **global**: option `ascp_opts` allows providing native `ascp` options on command line
     * `node`, `server`: command `sync` added to `node` (gen4) and `server` plugins, also available in `aoc`
   * Issues Fixed:
     * **global**: security: no shell interpolation
     * **global**: agent `node`: when WSS is used: no localhost (certificate)
     * `aoc`: #99 `file download` for single shared folder
-    * `faspex5`: change of API in faspex 5 for send package (paths is mandatory for any type of transfer now)
-    * **global**: Oauth web authentication was broken, fixed now
+    * `faspex5`: change of API in Faspex 5 for send package (paths is mandatory for any type of transfer now)
+    * **global**: OAuth web authentication was broken, fixed now
   * Breaking Changes:
     * **container**: image has entry point
     * `aoc`: `admin res node` commands `v3` and `v4` replaced with `do` and command `v3` moved inside `do`

data/CONTRIBUTING.md

@@ -9,34 +9,34 @@ Before you go ahead please search existing issues for your problem.
 To make sure that we can help you quickly please include and check the following information:
 
 - Include the `ascli` version you are running in your report.
-- If you are not running the latest version (please check), update.
+- If you are not running the latest version (please check) then update.
 - Include your `ruby -e "puts RUBY_DESCRIPTION"`.
 
 Thanks!
 
 ## Making Contributions
 
-To fetch & test the gem for development, do:
+To fetch and test the gem for development, do:
 
 ```bash
 git clone https://github.com/IBM/aspera-cli.git
 cd aspera-cli
-bundle install
-make test
 ```
 
+Then see [Running Tests](#running-tests).
+
 If you want to contribute, please:
 
 - Fork the project.
 - Make your feature addition or bug fix.
-- Add tests for it. This is important so I don't break it in a future version unintentionally.
-- **Bonus Points** go out to anyone who also updates `CHANGELOG.md` :)
+- Add tests for it. This is important, so it doesn't break in a future version unintentionally.
+- Run `rubocop` to comply for coding standards
+- Update `CHANGELOG.md`
 - Send a pull request on GitHub.
-- run `rubocop` to comply for coding standards
 
 ## Architecture
 
-A list of classes are provided in <docs/ml.png>
+A list of classes are provided in <docs/uml.png>
 
 Architecture:
 
@@ -46,22 +46,49 @@ The entry point is: `lib/aspera/cli/main.rb`.
 
 Plugins are located in: `lib/aspera/cli/plugins`.
 
-Transfer agents, in: `lib/aspera/fasp`.
+Transfer agents, in: `lib/aspera/agent`.
 
 ## Ruby version
 
-Install ruby with [RVM](https://rvm.io).
+Install Ruby using your preferred method.
 
-Cleanup installed gems in `rvm` to start fresh:
+To clean up installed gems to start fresh:
 
 ```bash
-make cleanup_gems
+make clean_gems
 ```
 
 ## Tool chain
 
 TODO: document installation of tool chain.
 
+Build system uses GNU Make.
+
+### Environment
+
+A few macros/envvars control some aspects:
+
+| macro                       | description                          |
+|-----------------------------|--------------------------------------|
+| `ASPERA_CLI_TEST_CONF_FILE` | Path to configuration file with secrets for tests |
+| `ASPERA_CLI_TEST_MACOS`     | Set to `true` if local HSTS running on macOS      |
+| `ASPERA_CLI_TEST_PRIVATE`   | Path to private folder               |
+| `ENABLE_COVERAGE`           | Tests with coverage analysis if set. |
+| `SIGNING_KEY`               | Path to signing key to build Gem.    |
+| `GEM_VERSION`               | Gem version to build container       |
+
+Those macros can be set either in an env var, or on the `make` command line.
+
+> **Note:** vars `ASPERA_CLI_TEST_` are typically set in user's shell profile.
+
+To use the CLI directly from the development environment, add this to your shell profile:
+
+```bash
+dev_ascli=$HOME/github/aspera-cli
+export PATH=$dev_ascli/bin:$PATH
+export RUBYLIB=$dev_ascli/lib:$RUBYLIB
+```
+
 ### Documentation
 
 Documentation is generated with `pandoc` and `LaTeX`.
@@ -83,7 +110,8 @@ tlmgr install fvextra selnolig lualatex-math
 
 ## Running Tests
 
-First, a testing configuration file must be created, from project top folder:
+First, a testing configuration file must be created.
+From project top folder, execute:
 
 ```bash
 mkdir ~/some_secure_folder
@@ -104,7 +132,7 @@ This project uses a `Makefile` for tests:
 make test
 ```
 
-When new commands are added to the CLI, new tests shall be added to the test suite.
+When new commands are added to the CLI, new tests shall be added to the test suite in `tests/Makefile`.
 
 ### Special tests
 
@@ -121,110 +149,40 @@ To run every test: `make full`
 
 For preparation of a release, do the following:
 
-1. Select a ruby version to test with: `rvm use 3.2.2`
+1. Select a ruby version to test with.
 2. Remove all gems: `make clean_gems`
-3. Starts tests: `make test`
-4. Install optional gems: `make install_optional_gems`
-5. Test remaining cases: `cd tests` and then `make optional` and `make interactive`
-
-This ensures that optional gems do not prevent `ascli` to run.
+3. `cd tests && make full`
 
 To test additional Ruby version, repeat the procedure with other Ruby versions.
 
 ## Coverage
 
 A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
-Enable coverage monitoring using env var `ENABLE_COVERAGE`.
+Enable coverage monitoring using macro/envvar `ENABLE_COVERAGE`.
 
 ```bash
-ENABLE_COVERAGE=1 make test
+cd tests
+make ENABLE_COVERAGE=1
 ```
 
 Once tests are completed, or during test, consult the page: [coverage/index.html](coverage/index.html)
 
 ## Build
 
-By default the gem is built signed: `make`.
-The appropriate signing key is required, and its path must be set to env var `SIGNING_KEY`.
-It is also possible to build a non-signed version for development purpose: `make unsigned_gem`.
-
-### Gem Signature
-
-Refer to:
-
-- <https://guides.rubygems.org/security/>
-- <https://ruby-doc.org/current/stdlibs/rubygems/Gem/Security.html>
-- `gem cert --help`
-
-Then procedure is as follows:
-
-- The maintainer creates the initial certificate and a private key:
-
-  ```bash
-  cd /path/to/vault
-  gem cert --build maintainer@example.com
-  ```
-
-  > **Note:** The email must match the field `spec.email` in `aspera-cli.gemspec`
-
-  This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
-  
-  - `gem-private_key.pem` : This file shall be kept secret in a vault.
-  - `gem-public_cert.pem` : This file is copied to a public place, here in folder `certs`
-
-  > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
-
-- The maintainer builds the signed gem.
-
-  The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
-
-  To build the signed gem:
-
-  ```bash
-  SIGNING_KEY=/path/to/vault/gem-private_key.pem make
-  ```
-
-- The user can activate gem signature verification on installation:
-
-  Add the certificate to gem trusted certificates:
-
-  ```bash
-  curl https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem -so aspera-cli-certificate.pem
-  gem cert --add aspera-cli-certificate.pem
-  rm aspera-cli-certificate.pem
-  ```
+By default, the gem is built signed: `make`.
+A private key is required to generate a signed Gem.
+Its path must be set using macro/envvar `SIGNING_KEY`, see below.
+The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
 
-  - The user installs the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted.
-
-  ```bash
-  gem install -P HighSecurity aspera-cli
-  ```
-
-  - The maintainer can renew the certificate when it is expired using the same private key:
-
-  ```bash
-  SIGNING_KEY=/path/to/vault/gem-private_key.pem make update-cert
-  ```
-
-  Alternatively, to generate a new certificate with the same key:
-
-  ```bash
-  SIGNING_KEY=/path/to/vault/gem-private_key.pem make new-cert
-  ```
-
-  - Show the current certificate contents
-
-  ```bash
-  make show-cert
-  ```
+```bash
+make SIGNING_KEY=/path/to/vault/gem-private_key.pem
+```
 
-  > Note: to provide a passphrase add argument: `-passin pass:_value_` to `openssl`
+It is also possible to build an unsigned version for development purpose: `make unsigned_gem`.
 
-  - Check that the signing key is the same as used to sign the certificate:
+### Gem Signature
 
-  ```bash
-  SIGNING_KEY=/path/to/vault/gem-private_key.pem make check-cert-key
-  ```
+Refer to <certs/README.md>
 
 ### gRPC stubs for transfer SDK
 
@@ -236,73 +194,36 @@ make grpc
 
 It downloads the latest proto file and then compiles it.
 
-## Docker image build
+## Container image build
 
-The `Dockerfile.tmpl.erb` template allows customization of the optional copying of the `aspera-cli.gem` file, as opposed to installing from rubygems.org. It also supports customizing the retrieval of the SDK.
+See [Container build](./container/README.md).
 
-### Default image build
-
-Build the image:
+For operations, move to the folder:
 
 ```bash
-make docker
+cd container
 ```
 
-This does the following:
-
-- Install the official gem version directly from <rubygems.org>.
-- Build the image for the version number in the current repository
-- creates tags for both the version and `latest`
-
-> **Note:** This target creates the `Dockerfile` from an `ERB` (embedded Ruby) template (Makefile target `dockerfile`).
-A template is used as it allows some level of customization to tell where to take the gem from, as as for the SDK file.
-
-Then, to push to the image registry (both tags: version and `latest`):
-
-```bash
-make docker_push
-```
-
-### Specific version image build
-
-To build a specific version: override `make` macro `GEM_VERSION`:
-
-```bash
-make docker GEM_VERSION=4.11.0
-make docker_push GEM_VERSION=4.11.0
-```
-
-> **Note:** This does not use the locally generated gem file.
-Only the local docker file is used.
-The gem is installed from rubygems.org. This also sets the `latest` tag.
+## Single executable build
 
-### Development version image build
+See [Executable build](./binary/README.md).
 
-To build/push a beta/development container:
-it does not create the `latest` tag, it uses the gem file generated locally with a special version number.
+For operations, move to the folder:
 
 ```bash
-make docker_beta
-make docker_push_beta
+cd binary
 ```
 
-## Single executable build
-
-Initially, `rubyc` (gem [`ruby-packer`](https://github.com/pmq20/ruby-packer) and [here](https://github.com/you54f/ruby-packer)) was used to build a single executable.
+## Development check list for new release
 
-<https://www.tebako.org/>
+When preparing for a new release do the following:
 
-A modern version of this is now used: [`tebako`](https://github.com/tamatebako/tebako) for which a container is provided.
-
-```bash
-make single GEM_VERSION=4.11.0
-```
+- TODO
 
 ## Long Term Implementation and delivery improvements
 
-- replace rest and oauth classes with ruby standard gems:
+- replace Rest and OAuth classes with ruby standard gems:
   - <https://github.com/rest-client/rest-client>
   - <https://github.com/oauth-xx/oauth2>
 - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
-- Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (`rubyc`)
 - look at <https://github.com/phusion/traveling-ruby>