aspera-cli 4.1.0 → 4.2.0

data/docs/README.erb.md

@@ -106,9 +106,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
 
 If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
 
-If you want to test with Aspera demo transfer server, a default configuration is created on first use:
+To test with Aspera demo transfer server, setup the environment and then test:
 
 ```
+$ <%=cmd%> config initdemo
 $ <%=cmd%> server browse /
 :............:...........:......:........:...........................:.......................:
 :   zmode    :   zuid    : zgid :  size  :           mtime           :         name          :
@@ -128,7 +129,7 @@ If you want to use <%=tool%> with another server, and in order to make further c
 * download a file
 
 ```
-$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
+$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
 updated: myserver
 $ <%=cmd%> config id default set server myserver
 updated: default&rarr;server to myserver
@@ -162,18 +163,22 @@ Then, follow the section relative to the product you want to interact with ( Asp
 
 # <a name="installation"></a>Installation
 
-It is possible to install *either* as a docker container or step by step on the native host:
+It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
+
+The direct installation is recommended and consists in installing:
 
 * [Ruby](#ruby) version >= <%= gemspec.required_ruby_version %>
 * [<%= gemspec.name %>](#the_gem)
 * [Aspera SDK (ascp)](#fasp_prot)
 
-The following sections provide information on the installation.
+The following sections provide information on the various installation methods.
 
 An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
 
 ## Docker container
 
+Use this method only if you know what you do, else use the standard recommended method as described here above.
+
 This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
 
 Ensure that you have Docker installed.
@@ -197,11 +202,12 @@ $ ./<%=cmd%> install
 
 Start using it !
 
-Note that the tool is run in the container.
+Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
 
-The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>`.
+The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
 
 To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
+To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
 
 ## <a name="ruby"></a>Ruby
 
@@ -211,7 +217,7 @@ A ruby interpreter is required to run the tool or to use the gem and tool.
 
 Ruby minimum version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
 
-*Any type of Ruby installation can be used* : rpm, yum, dnf, rvm, brew, windows installer, ... .
+*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
 
 Refer to the following sections for a proposed method for specific operating systems.
 
@@ -789,7 +795,7 @@ $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
 The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
 
 ```
-$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera --ts=@json:'{"precalculate_job_size":true}'
+$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
 ```
 
 * This creates a <%=prst%> `demo_server` with all provided options.
@@ -797,13 +803,13 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
 The command `set` allows setting individual options in a <%=prst%>.
 
 ```
-$ <%=cmd%> config id demo_server set password demoaspera
+$ <%=cmd%> config id demo_server set password _demo_pass_
 ```
 
 The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
 
 ```
-$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"demoaspera","ts":{"precalculate_job_size":true}}'
+$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
 ```
 
 A good practice is to not manually edit the configuration file and use modification commands instead.
@@ -840,11 +846,19 @@ Note that special plugin name: `config` can be associated with a preset that is
 Operations on this preset are done using regular `config` operations:
 
 ```
-$ <%=cmd%> config id default set _plugin_name_ _defauklt_preset_for_plugin_
+$ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
 $ <%=cmd%> config id default get _plugin_name_
-"_defauklt_preset_for_plugin_"
+"_default_preset_for_plugin_"
 ```
 
+### <a name="lprtdef"></a>Special Plugin: config
+
+Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
+
+When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
+
+If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
+
 ### Format of file
 
 The configuration file is a hash in a YAML file. Example:
@@ -860,7 +874,7 @@ cli_default:
 demo_server:
   url: ssh://demo.asperasoft.com:33001
   username: asperaweb
-  password: demoaspera
+  password: _demo_pass_
 ```
 
 We can see here:
@@ -929,7 +943,6 @@ A <%=prst%> value can be removed with `unset`:
 $ <%=cmd%> config id cli_default unset interactive
 ```
 
-
 ### Examples
 
 For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
@@ -1049,21 +1062,42 @@ The `config` plugin also allows specification for the use of a local FASP client
 
 ```
 $ <%=cmd%> config ascp show
-/Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
+/Users/laurent/.aspera/ascli/sdk/ascp
+$ <%=cmd%> config ascp info
++--------------------+-----------------------------------------------------------+
+| key                | value                                                     |
++--------------------+-----------------------------------------------------------+
+| ascp               | /Users/laurent/.aspera/ascli/sdk/ascp                     |
+...
 ```
 
 ### Selection of local `ascp`
 
+By default, <%=tool%> uses any found local product with ascp, including SDK.
+
 To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
 
-To permanently use another ascp:
+For a permanent change, the command `config ascp use` sets the same parameter for the global default.
+
+Using a POSIX shell:
 
 ```
 $ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
-saved to default global preset /Users/laurent/Applications/Aspera CLI/bin/ascp
+ascp version: 4.0.0.182279
+Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
+Saved to default global preset global_common_defaults
 ```
 
-This sets up a global default.
+Windows:
+
+```
+$ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
+ascp version: 4.0.0.182279
+Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
+Saved to default global preset global_common_defaults
+```
+
+If the path has spaces, read section: [Shell and Command line parsing](#parsing).
 
 ### List locally installed Aspera Transfer products
 
@@ -1150,7 +1184,7 @@ will effectively push files to the related server from the agent node.
 
 ### <a name="direct"></a>Direct (local ascp using FASPManager API)
 
-By default the CLI will use a local FASP protocol, equivalent to specifying `--transfer=direct`.
+By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
 <%=tool%> will detect locally installed Aspera products.
 Refer to section [FASP](#client).
 
@@ -1160,17 +1194,33 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
 * `EX_http_proxy_url` (proxy for legacy http fallback)
 * `EX_ascp_args`
 
-The `transfer-info` optionally provides the following auto resume parameters:
+The `transfer-info` accepts the following optional parameters:
+
+<table>
+<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
+<tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
+<tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
+<tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
+<tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
+</table>
+
+Resume parameters:
 
 <table>
-<tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
-<tr><td>iter_max</td>.    <td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
-<tr><td>sleep_initial</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
-<tr><td>sleep_factor</td> <td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
-<tr><td>sleep_max</td>.   <td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
-<tr><td>wss</td>          <td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
+<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
+<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
+<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
+<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
+<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
 </table>
 
+Examples:
+
+```
+$ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
+$ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
+```
+
 ### IBM Aspera Connect Client GUI
 
 By specifying option: `--transfer=connect`, <%=tool%> will start transfers
@@ -1220,7 +1270,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
 
 <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
 
-If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
+If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
 
 It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
 
@@ -1966,10 +2016,10 @@ $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
 * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
 * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
 
-Typically, one would regularly execute this command on a regular basis, using the method oif your choice:
+Typically, one would regularly execute this command on a regular basis, using the method of your choice:
 
-* Windows scheduler
-* cron
+* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
+* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
 * etc...
 
 ## Download Files
@@ -2189,7 +2239,7 @@ This can also be set as default using a preset
 One can test the "server" application using the well known demo server:
 
 ```
-$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
+$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
 $ <%=cmd%> config id default set server aspera_demo_server
 $ <%=cmd%> server browse /aspera-test-dir-large
 $ <%=cmd%> server download /aspera-test-dir-large/200MB
@@ -2493,7 +2543,7 @@ to start from ma configuration file, using <%=tool%> standard options.
 
 # Plugin: Preview
 
-The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on an Aspera HSTS for use primarily in the Aspera on Cloud application.
+The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
 This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
 Several parameters can be used to tune several aspects:
 
@@ -2622,9 +2672,10 @@ $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=
 $ <%=cmd%> config id default set preview previewconf
 ```
 
-Here we assume that Office file generation is disabled, else remove the option. `lock_port` prevents concurrent execution of generation when using a scheduler.
+Here we assume that Office file generation is disabled, else remove this option.
+`lock_port` prevents concurrent execution of generation when using a scheduler.
 
-Once can check if the access key is well configured using:
+One can check if the access key is well configured using:
 
 ```
 $ <%=cmd%> -Ppreviewconf node browse /
@@ -2710,6 +2761,14 @@ $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
 
 The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
 
+When scanning the option `value` has the same behaviour as for the `node find` command.
+
+For instance to filter out files beginning with `._` do:
+
+```
+... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
+```
+
 ## Preview File types
 
 Two types of preview can be generated:
@@ -2847,7 +2906,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
 ## Simple session
 
 ```
-MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
+MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
 
 echo "${MY_TSPEC}"|asession
 ```
@@ -2860,7 +2919,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
 
 ```
 $ asession
-{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"demoaspera","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
+{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
 {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
 {"type":"DONE"}
 ```
@@ -3009,7 +3068,20 @@ So, it evolved into <%=tool%>:
 
 # Changes (Release notes)
 
-* 4.x
+* <%= gemspec.version.to_s %>
+
+	* new: command `aoc remind` to receive organization membership by email
+	* new: in `preview` option `value` to filter out on file name
+	* new: `initdemo` to initialize for demo server
+	* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
+	* fix: on Windows `conf ascp use` expects ascp.exe
+	* fix: (break) multi_session_threshold is Integer, not String
+	* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
+	* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
+	* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
+	* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
+
+* 4.1.0
 
   	* fix: remove keys from transfer spec and command line when not needed
   	* fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist 
@@ -3380,11 +3452,15 @@ Breaking change:
 
   * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
 
-# BUGS
+# BUGS, FEATURES, CONTRIBUTION
+
+For issues or feature requests use the Github repository and issues.
+
+You can also contribute to this open source project.
 
-* This is best effort code without official support, dont expect full capabilities. This code is not supported by IBM/Aspera. You can contact the author for bugs or features.
+One can also create one's own command nplugin.
 
-## only one value for any option
+## Only one value for any option
 
 Some commands and sub commands may ask for the same option name.
 Currently, since option definition is position independant (last one wins), it is not possible
@@ -3398,7 +3474,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
 
 ## ED255519 key not supported
 
-ED255519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
+ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
+
 Without this deactivation, if such key was present the following error was generated:
 
 ```
@@ -3408,7 +3485,17 @@ OpenSSH keys only supported if ED25519 is available
 Which meant that you do not have ruby support for ED25519 SSH keys.
 You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
 
-# TODO
+## Error "Remote host is not who we expected"
+
+`ascp` version 4.x changed the algorithm used to check the SSH server certificate. To ignore the certificate (SSH fingerprint) add option on client side:
+
+```
+--ts=@json:'{"sshfp":null}'
+```
+
+Refer to ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server): if you have access to server side, basically disable other SSH host keys than RSA.
+
+## Miscelaneous
 
 * remove rest and oauth classes and use ruby standard gems:
 
@@ -3424,9 +3511,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
 * Going through proxy: use env var http_proxy and https_proxy, no_proxy
 
 * easier use with https://github.com/pmq20/ruby-packer
-
-# Contribution
-
-Send comments !
-
-Create your own plugin !

data/docs/test_env.conf

@@ -5,8 +5,8 @@ default:
   config: cli_default
   aoc: tst_aoc1
   faspex: tst_faspex
-  faspex5: tst_faspex5
-  shares: tst_shares
+  faspex5: tst_faspex5_boot
+  shares: tst_shares_1
   shares2: tst_shares2
   node: tst_node
   server: tst_server
@@ -47,11 +47,18 @@ tst_node_faspex:
   url: your value here
   username: your value here
   password: your value here
-tst_faspex5:
+tst_faspex5_boot:
   url: your value here
   auth: your value here
   username: your value here
   password: your value here
+tst_faspex5:
+  url: your value here
+  auth: your value here
+  client_id: your value here
+  client_secret: your value here
+  private_key: your value here
+  username: your value here
 tst_shares:
   url: your value here
   username: your value here
@@ -111,6 +118,8 @@ misc:
   faspex_publink_send_to_fxuser: your value here
   faspex_publink_send_to_dropbox: your value here
   shares_upload: your value here
+  console_smart_id: your value here
+  console_smart_file: your value here
   orch_workflow_id: your value here
   file_dcm: your value here
   file_pdf: your value here
@@ -139,5 +148,5 @@ misc:
   email_internal: your value here
   email_external: your value here
   aoc_org: your value here
-  aoc_user: your value here
+  aoc_user_email: your value here
   http_gw_fqdn_port: your value here

data/lib/aspera/aoc.rb

@@ -9,6 +9,8 @@ module Aspera
     private
     @@use_standard_ports = true
 
+    API_V1='api/v1'
+
     PRODUCT_NAME='Aspera on Cloud'
     # Production domain of AoC
     PROD_DOMAIN='ibmaspera.com'
@@ -42,11 +44,6 @@ module Aspera
     FILES_APP='files'
     PACKAGES_APP='packages'
 
-    CLIENT_RANDOM={
-      'aspera.global-cli-client' => 'frpmsRsG4mjZ0PlxCgdJlvONqBg4Vlpz_IX7gXmBMAfsgMLy2FO6CXLodKfKAuhqnCqSptLbe_wdmnm9JRuEPO-PpFqpq_Kb',
-      'aspera.drive' => 'UegzQ3LcbLht5dLYAXaR-7ZMnJ6-kwPEXWEXaqLSOMGmtzNA9r6kPFLElqBfq66BfgMabdO96k5sPXV-H8M3vsx9LbGlewF1'
-    }
-
     def self.get_client_info(client_name=CLIENT_APPS.first)
       client_index=CLIENT_APPS.index(client_name)
       raise "no such pre-defined client: #{client_name}" if client_index.nil?
@@ -68,9 +65,14 @@ module Aspera
       return organization,instance_domain
     end
 
+    # base API url depends on domain, which could be "qa.xxx"
+    def self.api_base_url(api_domain=PROD_DOMAIN)
+      return "https://api.#{api_domain}"
+    end
+
     def self.metering_api(entitlement_id,customer_id,api_domain=PROD_DOMAIN)
       return Rest.new({
-        :base_url => "https://api.#{api_domain}/metering/v1",
+        :base_url => "#{api_base_url(api_domain)}/metering/v1",
         :headers  => {'X-Aspera-Entitlement-Authorization' => Rest.basic_creds(entitlement_id,customer_id)}
       })
     end
@@ -126,7 +128,7 @@ module Aspera
       # access key secrets are provided out of band to get node api access
       # key: access key
       # value: associated secret
-      @secrets={}
+      @key_chain=nil
 
       # init rest params
       aoc_rest_p={:auth=>{:type =>:oauth2}}
@@ -147,12 +149,12 @@ module Aspera
 
       # get org name and domain from url
       organization,instance_domain=self.class.parse_url(opt[:url])
-      # this is the base API url (depends on domain, which could be "qa.xxx")
-      api_base_url="https://api.#{instance_domain}"
-      # base API URL
-      aoc_rest_p[:base_url]="#{api_base_url}/#{opt[:subpath]}"
+      # this is the base API url
+      api_url_base=self.class.api_base_url(instance_domain)
+      # API URL, including subpath (version ...)
+      aoc_rest_p[:base_url]="#{api_url_base}/#{opt[:subpath]}"
       # base auth URL
-      aoc_auth_p[:base_url] = "#{api_base_url}/#{OAUTH_API_SUBPATH}/#{organization}"
+      aoc_auth_p[:base_url] = "#{api_url_base}/#{OAUTH_API_SUBPATH}/#{organization}"
       aoc_auth_p[:client_id]=opt[:client_id]
       aoc_auth_p[:client_secret] = opt[:client_secret]
 
@@ -192,17 +194,12 @@ module Aspera
       super(aoc_rest_p)
     end
 
-    def add_secrets(secrets)
-      @secrets.merge!(secrets)
-      Log.log.debug("now secrets:#{secrets}")
+    def key_chain=(keychain)
+      raise "keychain already set" unless @key_chain.nil?
+      @key_chain=keychain
       nil
     end
 
-    def has_secret(ak)
-      Log.log.debug("has key:#{ak} -> #{@secrets.has_key?(ak)}")
-      return @secrets.has_key?(ak)
-    end
-
     # additional transfer spec (tags) for package information
     def self.package_tags(package_info,operation)
       return {'tags'=>{'aspera'=>{'files'=>{
@@ -299,9 +296,9 @@ module Aspera
         :base_url => node_info['url'],
         :headers  => {'X-Aspera-AccessKey'=>node_info['access_key']},
       }
-      ak_secret=@secrets[node_info['access_key']]
+      ak_secret=@key_chain.get_secret(node_info['access_key'],false)
       if ak_secret.nil? and node_scope.nil?
-        raise 'There must be at least one of: secret, node scope'
+        raise "There must be at least one of: 'secret' or 'scope' for access key #{node_info['access_key']}"
       end
       # if secret provided on command line or if there is no scope
       if !ak_secret.nil? or node_scope.nil?