aspera-cli 4.4.0 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +1042 -787
  3. data/bin/ascli +1 -1
  4. data/bin/asession +3 -5
  5. data/docs/Makefile +4 -7
  6. data/docs/README.erb.md +988 -740
  7. data/examples/faspex4.rb +4 -6
  8. data/examples/transfer.rb +2 -2
  9. data/lib/aspera/aoc.rb +139 -118
  10. data/lib/aspera/cli/listener/progress_multi.rb +5 -5
  11. data/lib/aspera/cli/main.rb +64 -34
  12. data/lib/aspera/cli/manager.rb +19 -20
  13. data/lib/aspera/cli/plugin.rb +9 -1
  14. data/lib/aspera/cli/plugins/aoc.rb +156 -143
  15. data/lib/aspera/cli/plugins/ats.rb +11 -10
  16. data/lib/aspera/cli/plugins/bss.rb +2 -2
  17. data/lib/aspera/cli/plugins/config.rb +236 -112
  18. data/lib/aspera/cli/plugins/faspex.rb +29 -7
  19. data/lib/aspera/cli/plugins/faspex5.rb +21 -8
  20. data/lib/aspera/cli/plugins/node.rb +21 -9
  21. data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
  22. data/lib/aspera/cli/plugins/preview.rb +2 -2
  23. data/lib/aspera/cli/plugins/server.rb +3 -3
  24. data/lib/aspera/cli/plugins/shares.rb +17 -0
  25. data/lib/aspera/cli/transfer_agent.rb +47 -85
  26. data/lib/aspera/cli/version.rb +1 -1
  27. data/lib/aspera/environment.rb +4 -4
  28. data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
  29. data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
  30. data/lib/aspera/fasp/{local.rb → agent_direct.rb} +14 -17
  31. data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +4 -4
  32. data/lib/aspera/fasp/{node.rb → agent_node.rb} +25 -8
  33. data/lib/aspera/fasp/agent_trsdk.rb +106 -0
  34. data/lib/aspera/fasp/default.rb +17 -0
  35. data/lib/aspera/fasp/installation.rb +64 -48
  36. data/lib/aspera/fasp/parameters.rb +7 -3
  37. data/lib/aspera/faspex_gw.rb +6 -6
  38. data/lib/aspera/keychain/encrypted_hash.rb +120 -0
  39. data/lib/aspera/keychain/macos_security.rb +94 -0
  40. data/lib/aspera/log.rb +45 -32
  41. data/lib/aspera/node.rb +3 -6
  42. data/lib/aspera/rest.rb +65 -49
  43. metadata +68 -27
  44. data/lib/aspera/api_detector.rb +0 -60
  45. data/lib/aspera/secrets.rb +0 -20
data/docs/README.erb.md CHANGED
@@ -16,7 +16,9 @@ Ruby Doc: [<%=gemspec.metadata['documentation_uri']%>](<%=gemspec.metadata['docu
16
16
 
17
17
  Required Ruby version: <%=gemspec.required_ruby_version%>
18
18
 
19
- # <a name="when_to_use"></a>When to use and when not to use
19
+ [Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
20
+
21
+ # <a id="when_to_use"></a>When to use and when not to use
20
22
 
21
23
  <%=tool%> is designed to be used as a command line tool to:
22
24
 
@@ -39,21 +41,18 @@ One might be tempted to use it as an integration element, e.g. by building a com
39
41
  For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
40
42
 
41
43
  * Product APIs (REST) : e.g. AoC, Faspex, node
42
- * Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
44
+ * Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
43
45
 
44
46
  Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
45
47
 
46
48
  For scripting and ad'hoc command line operations, <%=tool%> is perfect.
47
49
 
48
- # Notations
50
+ # <a id="parsing"></a>Notations, Shell and Command line parsing
49
51
 
50
- In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
51
- Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
52
+ In examples, command line operations are shown using a shell such: `bash` or `zsh`.
52
53
 
53
54
  Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
54
55
 
55
- # <a name="parsing"></a>Shell and Command line parsing
56
-
57
56
  <%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
58
57
 
59
58
  On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
@@ -66,12 +65,15 @@ On Windows, `cmd.exe` is typically used. Windows process creation does not recei
66
65
  In case of doubt of argument values after parsing test like this:
67
66
 
68
67
  ```
69
- $ <%=cmd%> conf echo "Hello World" arg2 3
68
+ <%=cmd%> conf echo "Hello World" arg2 3
69
+ ```
70
+
71
+ ```
70
72
  "Hello World"
71
73
  ERROR: Argument: unprocessed values: ["arg2", "3"]
72
74
  ```
73
75
 
74
- `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
76
+ `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (<%=tool%>), next command line arguments are shown in the error message.
75
77
 
76
78
  # Quick Start
77
79
 
@@ -82,7 +84,10 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
82
84
  Once the gem is installed, <%=tool%> shall be accessible:
83
85
 
84
86
  ```
85
- $ <%=cmd%> --version
87
+ <%=cmd%> --version
88
+ ```
89
+
90
+ ```
86
91
  <%=gemspec.version.to_s%>
87
92
  ```
88
93
 
@@ -95,8 +100,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
95
100
  To test with Aspera demo transfer server, setup the environment and then test:
96
101
 
97
102
  ```
98
- $ <%=cmd%> config initdemo
99
- $ <%=cmd%> server browse /
103
+ <%=cmd%> config initdemo
104
+ ```
105
+
106
+ ```
107
+ <%=cmd%> server browse /
108
+ ```
109
+
110
+ ```
100
111
  :............:...........:......:........:...........................:.......................:
101
112
  : zmode : zuid : zgid : size : mtime : name :
102
113
  :............:...........:......:........:...........................:.......................:
@@ -115,11 +126,26 @@ If you want to use <%=tool%> with another server, and in order to make further c
115
126
  * download a file
116
127
 
117
128
  ```
118
- $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
129
+ <%=cmd%> config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
130
+ ```
131
+
132
+ ```
119
133
  updated: myserver
120
- $ <%=cmd%> config id default set server myserver
134
+ ```
135
+
136
+ ```
137
+ <%=cmd%> config preset set default server myserver
138
+ ```
139
+
140
+ ```
121
141
  updated: default&rarr;server to myserver
122
- $ <%=cmd%> server browse /aspera-test-dir-large
142
+ ```
143
+
144
+ ```
145
+ <%=cmd%> server browse /aspera-test-dir-large
146
+ ```
147
+
148
+ ```
123
149
  :............:...........:......:..............:...........................:............................:
124
150
  : zmode : zuid : zgid : size : mtime : name :
125
151
  :............:...........:......:..............:...........................:............................:
@@ -136,7 +162,13 @@ $ <%=cmd%> server browse /aspera-test-dir-large
136
162
  : -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
137
163
  : -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
138
164
  :............:...........:......:..............:...........................:............................:
139
- $ <%=cmd%> server download /aspera-test-dir-large/200MB
165
+ ```
166
+
167
+ ```
168
+ <%=cmd%> server download /aspera-test-dir-large/200MB
169
+ ```
170
+
171
+ ```
140
172
  Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
141
173
  complete
142
174
  ```
@@ -147,9 +179,9 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
147
179
 
148
180
  Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
149
181
 
150
- # <a name="installation"></a>Installation
182
+ # <a id="installation"></a>Installation
151
183
 
152
- It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
184
+ It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
153
185
 
154
186
  The direct installation is recommended and consists in installing:
155
187
 
@@ -159,33 +191,36 @@ The direct installation is recommended and consists in installing:
159
191
 
160
192
  The following sections provide information on the various installation methods.
161
193
 
162
- An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
194
+ An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
163
195
 
164
196
  ## Docker container
165
197
 
166
198
  Use this method only if you know what you do, else use the standard recommended method as described here above.
167
199
 
168
- This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
200
+ This method installs a docker image that contains: Ruby, <%=tool%> and the FASP sdk.
169
201
 
170
202
  The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
171
203
 
172
204
  Ensure that you have Docker installed.
173
205
 
174
206
  ```
175
- $ docker --version
207
+ docker --version
176
208
  ```
177
209
 
178
210
  Download the wrapping script:
179
211
 
180
212
  ```
181
- $ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
182
- $ chmod a+x <%=cmd%>
213
+ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
214
+ ```
215
+
216
+ ```
217
+ chmod a+x <%=cmd%>
183
218
  ```
184
219
 
185
220
  Install the container image:
186
221
 
187
222
  ```
188
- $ ./<%=cmd%> install
223
+ ./<%=cmd%> install
189
224
  ```
190
225
 
191
226
  Start using it !
@@ -195,9 +230,9 @@ Note that the tool is run in the container, so transfers are also executed in th
195
230
  The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
196
231
 
197
232
  To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
198
- To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
233
+ To add local storage as a volume edit the script: <%=tool%> and add a `--volume` stanza.
199
234
 
200
- ## <a name="ruby"></a>Ruby
235
+ ## <a id="ruby"></a>Ruby
201
236
 
202
237
  Use this method to install on the native host.
203
238
 
@@ -209,7 +244,7 @@ Required Ruby version: <%=gemspec.required_ruby_version%>. Ruby version 3 is als
209
244
 
210
245
  Refer to the following sections for a proposed method for specific operating systems.
211
246
 
212
- The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
247
+ The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
213
248
  If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
214
249
  If you have a simpler better way to install Ruby version <%=gemspec.required_ruby_version%> : use it !
215
250
 
@@ -222,31 +257,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
222
257
  Install the 2 keys
223
258
 
224
259
  ```
225
- $ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
260
+ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
226
261
  ```
227
262
 
228
263
  Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
229
264
 
230
265
  ```
231
- $ \curl -sSL https://get.rvm.io | bash -s stable
266
+ \curl -sSL https://get.rvm.io | bash -s stable
232
267
  ```
233
268
 
234
- If you keep the same terminal (ont needed if re-login):
269
+ If you keep the same terminal (not needed if re-login):
235
270
 
236
271
  ```
237
- $ source ~/.rvm/scripts/rvm
272
+ source ~/.rvm/scripts/rvm
238
273
  ```
239
274
 
240
275
  It is advised to get one of the pre-compiled ruby version, you can list with:
241
276
 
242
277
  ```
243
- $ rvm list --remote
278
+ rvm list --remote
244
279
  ```
245
280
 
246
281
  Install the chosen pre-compiled Ruby version:
247
282
 
248
283
  ```
249
- $ rvm install 2.7.2 --binary
284
+ rvm install 2.7.2 --binary
250
285
  ```
251
286
 
252
287
  Ruby is now installed for the user, go on to Gem installation.
@@ -259,7 +294,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
259
294
  One can install in another location with :
260
295
 
261
296
  ```
262
- # curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
297
+ curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
263
298
  ```
264
299
 
265
300
  As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
@@ -267,8 +302,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
267
302
  To activate ruby (and <%=cmd%>) later, source it:
268
303
 
269
304
  ```
270
- # source /etc/profile.d/rvm.sh.ok
271
- # rvm version
305
+ source /etc/profile.d/rvm.sh.ok
306
+ ```
307
+
308
+ ```
309
+ rvm version
272
310
  ```
273
311
 
274
312
  ### Windows: Installer
@@ -281,16 +319,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
281
319
 
282
320
  ### macOS: pre-installed or `brew`
283
321
 
284
- MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
322
+ macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
285
323
 
286
324
  ```
287
- $ sudo gem install <%=gemspec.name%><%=geminstadd%>
325
+ sudo gem install <%=gemspec.name%><%=geminstadd%>
288
326
  ```
289
327
 
290
328
  Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
291
329
 
292
330
  ```
293
- $ brew install ruby
331
+ brew install ruby
294
332
  ```
295
333
 
296
334
  ### Linux: package
@@ -300,90 +338,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
300
338
  Example:
301
339
 
302
340
  ```
303
- # yum install -y ruby rubygems ruby-json
341
+ yum install -y ruby rubygems ruby-json
304
342
  ```
305
343
 
306
344
  One can cleanup the whole yum-installed ruby environment like this to uninstall:
307
345
 
308
346
  ```
309
347
  gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
348
+ ```
349
+
350
+ ```
310
351
  yum remove -y ruby ruby-libs
311
352
  ```
312
353
 
313
354
  ### Other Unixes: Aix, etc...
314
355
 
315
- If your unix do not provide a pre-built ruby, you can get it using one of those
356
+ If your Unix does not provide a pre-built ruby, you can get it using one of those
316
357
  [methods](https://www.ruby-lang.org/en/documentation/installation/)
317
358
 
318
359
  For instance to build from source, and install in `/opt/ruby` :
319
360
 
320
361
  ```
321
- # wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
322
- # gzip -d ruby-2.7.2.tar.gz
323
- # tar xvf ruby-2.7.2.tar
324
- # cd ruby-2.7.2
325
- # ./configure --prefix=/opt/ruby
326
- # make ruby.imp
327
- # make
328
- # make install
329
- ```
330
-
331
- ### <a name="offline_install"></a>Installation without internet access
332
-
333
- Note that currently no pre-packaged version exist yet.
334
- A method to build one provided here:
335
-
336
- On a server with the same OS version and with internet access follow the "Generic single user installation" method.
337
-
338
- Then create an archive:
339
-
340
- ```
341
- $ cd
342
- $ tar zcvf rvm-<%=cmd%>.tgz .rvm
343
- ```
344
-
345
- Get the Aspera SDK. Execute:
346
-
347
- ```
348
- $ <%=cmd%> conf --show-config|grep sdk_url
349
- ```
362
+ wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
350
363
 
351
- Then download the SDK archive from that URL.
364
+ gzip -d ruby-2.7.2.tar.gz
352
365
 
353
- Another method for the SDK is to install the SDK (`<%=cmd%> conf ascp install`) on the first system, and archive `$HOME/.aspera`.
366
+ tar xvf ruby-2.7.2.tar
354
367
 
355
- Transfer those 2 archives to the target system without internet access.
368
+ cd ruby-2.7.2
356
369
 
357
- On the target system:
370
+ ./configure --prefix=/opt/ruby
358
371
 
359
- * Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
360
- * in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
372
+ make ruby.imp
361
373
 
362
- ```
363
- source path_to_rvm_root/scripts/rvm
364
- rvm use 2.7.2
365
- ```
366
-
367
- For the SDK, either install from archive:
374
+ make
368
375
 
369
- ```
370
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
376
+ make install
371
377
  ```
372
378
 
373
- or restore the `$HOME/.aspera` folder for the user.
374
-
375
- ## <a name="the_gem"></a>`<%=gemspec.name%>` gem
379
+ ## <a id="the_gem"></a>`<%=gemspec.name%>` gem
376
380
 
377
381
  Once you have Ruby and rights to install gems: Install the gem and its dependencies:
378
382
 
379
383
  ```
380
- # gem install <%=gemspec.name%><%=geminstadd%>
384
+ gem install <%=gemspec.name%><%=geminstadd%>
381
385
  ```
382
386
 
383
387
  To upgrade to the latest version:
384
388
 
385
389
  ```
386
- # gem update <%=gemspec.name%>
390
+ gem update <%=gemspec.name%>
387
391
  ```
388
392
 
389
393
  <%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
@@ -391,12 +395,10 @@ To upgrade to the latest version:
391
395
  To check manually:
392
396
 
393
397
  ```
394
- # <%=cmd%> conf check_update
398
+ <%=cmd%> conf check_update
395
399
  ```
396
400
 
397
-
398
-
399
- ## <a name="fasp_prot"></a>FASP Protocol
401
+ ## <a id="fasp_prot"></a>FASP Protocol
400
402
 
401
403
  Most file transfers will be done using the FASP protocol, using `ascp`.
402
404
  Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
@@ -404,17 +406,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
404
406
  * ascp
405
407
  * aspera-license (in same folder, or ../etc)
406
408
 
407
- This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
409
+ This can be installed either be installing an Aspera transfer software, or using an embedded command:
408
410
 
409
411
  ```
410
- $ <%=cmd%> conf ascp install
412
+ <%=cmd%> conf ascp install
411
413
  ```
412
414
 
413
- If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
415
+ If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
414
416
 
415
417
  ```
416
- $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
417
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
418
+ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
419
+ ```
420
+
421
+ ```
422
+ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
418
423
  ```
419
424
 
420
425
  The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
@@ -434,36 +439,55 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
434
439
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
435
440
 
436
441
  Several methods are provided to start a transfer.
437
- Use of a local client ([`direct`](#direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
442
+ Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
438
443
 
439
- ## <a name="offline_install"></a>Offline Installation (without internet)
444
+ ## <a id="offline_install"></a>Installation in air gapped environment
440
445
 
441
- The procedure consists in:
446
+ Note that currently no pre-packaged version exist yet.
447
+ A method to build one is provided here:
448
+
449
+ The procedure:
442
450
 
443
451
  * Follow the non-root installation procedure with RVM, including gem
444
- * archive (zip, tar) the main RVM folder (includes <%=cmd%>):
452
+ * Archive (zip, tar) the main RVM folder (includes <%=cmd%>):
445
453
 
446
454
  ```
447
- $ cd ~
448
- $ tar zcvf rvm_<%=cmd%>.tgz .rvm
455
+ cd $HOME && tar zcvf rvm-<%=cmd%>.tgz .rvm
449
456
  ```
450
457
 
451
- * retrieve the SDK:
458
+ * Get the Aspera SDK.
452
459
 
453
460
  ```
454
- $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
461
+ <%=cmd%> conf --show-config --fields=sdk_url
462
+ ```
463
+
464
+ * Download the SDK archive from that URL.
465
+
466
+ ```
467
+ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
468
+ ```
469
+
470
+ * Transfer those 2 files to the target system
471
+
472
+ * On target system
473
+
474
+ ```
475
+ cd $HOME
476
+
477
+ tar zxvf rvm-<%=cmd%>.tgz
478
+
479
+ source ~/.rvm/scripts/rvm
480
+
481
+ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
455
482
  ```
456
483
 
457
- * on the system without internet access:
484
+ * Add those lines to shell init (`.profile`)
458
485
 
459
486
  ```
460
- $ cd ~
461
- $ tar zxvf rvm_<%=cmd%>.tgz
462
- $ source ~/.rvm/scripts/rvm
463
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
487
+ source ~/.rvm/scripts/rvm
464
488
  ```
465
489
 
466
- # <a name="cli"></a>Command Line Interface: <%=tool%>
490
+ # <a id="cli"></a>Command Line Interface: <%=tool%>
467
491
 
468
492
  The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
469
493
 
@@ -492,7 +516,7 @@ The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which intera
492
516
  Basic usage is displayed by executing:
493
517
 
494
518
  ```
495
- $ <%=cmd%> -h
519
+ <%=cmd%> -h
496
520
  ```
497
521
 
498
522
  Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
@@ -506,7 +530,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
506
530
  There are two types of arguments: Commands and Options. Example :
507
531
 
508
532
  ```
509
- $ <%=cmd%> command --option-name=VAL1 VAL2
533
+ <%=cmd%> command --option-name=VAL1 VAL2
510
534
  ```
511
535
 
512
536
  * executes _command_: `command`
@@ -534,13 +558,16 @@ Exceptions:
534
558
  * the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
535
559
 
536
560
  ```
537
- $ <%=cmd%> config echo -- --sample
561
+ <%=cmd%> config echo -- --sample
562
+ ```
563
+
564
+ ```
538
565
  "--sample"
539
566
  ```
540
567
 
541
568
  Note that `--sample` is taken as an argument, and not option.
542
569
 
543
- Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on comand line and evaluated in order.
570
+ Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
544
571
 
545
572
  The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
546
573
 
@@ -561,7 +588,7 @@ Command line arguments that are not options are either commands or arguments. If
561
588
 
562
589
  Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
563
590
 
564
- The behaviour can be controlled with:
591
+ The behavior can be controlled with:
565
592
 
566
593
  * --interactive=&lt;yes|no&gt; (default=yes if STDIN is a terminal, else no)
567
594
  * yes : missing mandatory parameters/options are asked to the user
@@ -578,8 +605,8 @@ The information displayed depends on the action.
578
605
 
579
606
  Depending on action, the output will contain:
580
607
 
581
- * `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is atteribute value. Nested hashes are collapsed.
582
- * `object_list` : displayed as a 2 dimensional table: one line per item, one colum per attribute.
608
+ * `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
609
+ * `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
583
610
  * `value_list` : a table with one column.
584
611
  * `empty` : nothing
585
612
  * `status` : a message
@@ -604,12 +631,15 @@ The style of output can be set using the `format` parameter, supporting:
604
631
  * `yaml` : YAML
605
632
  * `csv` : Comma Separated Values
606
633
 
607
- ### <a name="option_select"></a>Option: `select`: Filter on columns values for `object_list`
634
+ ### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
608
635
 
609
636
  Table output can be filtered using the `select` parameter. Example:
610
637
 
611
638
  ```
612
- $ <%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
639
+ <%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
640
+ ```
641
+
642
+ ```
613
643
  :...............................:..................................:...........:
614
644
  : name : email : ats_admin :
615
645
  :...............................:..................................:...........:
@@ -622,7 +652,7 @@ Note that `select` filters selected elements from the result of API calls, while
622
652
 
623
653
  ### Verbosity of output
624
654
 
625
- Outpout messages are categorized in 3 types:
655
+ Output messages are categorized in 3 types:
626
656
 
627
657
  * `info` output contain additional information, such as number of elements in a table
628
658
  * `data` output contain the actual output of the command (object, or list of objects)
@@ -636,7 +666,7 @@ The option `display` controls the level of output:
636
666
 
637
667
  ### Selection of output object properties
638
668
 
639
- By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific colums to be displayed, by setting the `fields` option to one of the following value:
669
+ By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
640
670
 
641
671
  * DEF : default display of columns (that's the default, when not set)
642
672
  * ALL : all columns available
@@ -645,7 +675,7 @@ By default, a table output will display one line per entry, and columns for each
645
675
  * +a,b,c : add selected properties to the default selection.
646
676
  * -a,b,c : remove selected properties from the default selection.
647
677
 
648
- ## <a name="extended"></a>Extended Value Syntax
678
+ ## <a id="extended"></a>Extended Value Syntax
649
679
 
650
680
  Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
651
681
 
@@ -682,23 +712,32 @@ To display the result of an extended value, use the `config echo` command.
682
712
  Example: read the content of the specified file, then, base64 decode, then unzip:
683
713
 
684
714
  ```
685
- $ <%=cmd%> config echo @zlib:@base64:@file:myfile.dat
715
+ <%=cmd%> config echo @zlib:@base64:@file:myfile.dat
686
716
  ```
687
717
 
688
718
  Example: create a value as a hash, with one key and the value is read from a file:
689
719
 
690
720
  ```
691
- $ <%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
721
+ <%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
692
722
  ```
693
723
 
694
724
  Example: read a csv file and create a list of hash for bulk provisioning:
695
725
 
696
726
  ```
697
- $ cat test.csv
727
+ cat test.csv
728
+ ```
729
+
730
+ ```
698
731
  name,email
699
732
  lolo,laurent@example.com
700
733
  toto,titi@tutu.tata
701
- $ <%=cmd%> config echo @csvt:@file:test.csv
734
+ ```
735
+
736
+ ```
737
+ <%=cmd%> config echo @csvt:@file:test.csv
738
+ ```
739
+
740
+ ```
702
741
  :......:.....................:
703
742
  : name : email :
704
743
  :......:.....................:
@@ -710,13 +749,16 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
710
749
  Example: create a hash and include values from preset named "config" of config file in this hash
711
750
 
712
751
  ```
713
- $ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
752
+ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
753
+ ```
754
+
755
+ ```
714
756
  {"version"=>"0.9", "hello"=>true}
715
757
  ```
716
758
 
717
759
  Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
718
760
 
719
- ## <a name="native"></a>Structured Value
761
+ ## <a id="native"></a>Structured Value
720
762
 
721
763
  Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
722
764
 
@@ -727,7 +769,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
727
769
 
728
770
  It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
729
771
 
730
- ## <a name="conffolder"></a>Configuration and Persistency Folder
772
+ ## <a id="conffolder"></a>Configuration and Persistency Folder
731
773
 
732
774
  <%=tool%> configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/<%=cmd%>`.
733
775
 
@@ -737,49 +779,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
737
779
  The main folder can be displayed using :
738
780
 
739
781
  ```
740
- $ <%=cmd%> config folder
782
+ <%=cmd%> config folder
783
+ ```
784
+
785
+ ```
741
786
  /Users/kenji/.aspera/<%=cmd%>
742
787
  ```
743
788
 
744
- It can be overriden using the envinonment variable `<%=evp%>HOME`.
789
+ It can be overridden using the environment variable `<%=evp%>HOME`.
745
790
 
746
791
  Example (Windows):
747
792
 
748
793
  ```
749
- $ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
750
- $ <%=cmd%> config folder
794
+ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
795
+ <%=cmd%> config folder
751
796
  C:\Users\Kenji\.aspera\<%=cmd%>
752
797
  ```
753
798
 
754
- ## <a name="configfile"></a>Configuration file
799
+ ## <a id="configfile"></a>Configuration file
755
800
 
756
801
  On the first execution of <%=tool%>, an empty configuration file is created in the configuration folder.
757
802
  Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
758
803
 
759
804
  Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it using the `config` command.
760
805
 
761
- All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuratin file.
806
+ All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuration file.
762
807
 
763
808
  A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
764
809
 
765
- The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overriden with option `--config-file=path` or equivalent env var).
810
+ The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
766
811
 
767
812
  The configuration file is simply a catalog of pre-defined lists of options, called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
768
813
 
769
- ### <a name="lprt"></a><%=prstt%>
814
+ ### <a id="lprt"></a><%=prstt%>
770
815
 
771
816
  A <%=prst%> is simply a collection of parameters and their associated values in a named section in the configuration file.
772
817
 
773
818
  A named <%=prst%> can be modified directly using <%=tool%>, which will update the configuration file :
774
819
 
775
820
  ```
776
- $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
821
+ <%=cmd%> config preset set|delete|show|initialize|update <<%=opprst%>>
777
822
  ```
778
823
 
779
824
  The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
780
825
 
781
826
  ```
782
- $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
827
+ <%=cmd%> config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
783
828
  ```
784
829
 
785
830
  * This creates a <%=prst%> `demo_server` with all provided options.
@@ -787,39 +832,48 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
787
832
  The command `set` allows setting individual options in a <%=prst%>.
788
833
 
789
834
  ```
790
- $ <%=cmd%> config id demo_server set password _demo_pass_
835
+ <%=cmd%> config preset set demo_server password _demo_pass_
791
836
  ```
792
837
 
793
838
  The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
794
839
 
795
840
  ```
796
- $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
841
+ <%=cmd%> config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
797
842
  ```
798
843
 
799
- A good practice is to not manually edit the configuration file and use modification commands instead.
800
- If necessary, the configuration file can be edited (or simply consulted) with:
844
+ A full terminal based overview of the configuration can be displayed using:
801
845
 
802
846
  ```
803
- $ <%=cmd%> config open
847
+ <%=cmd%> config preset over
804
848
  ```
805
849
 
806
- A full terminal based overview of the configuration can be displayed using:
850
+ A list of <%=prst%> can be displayed using:
807
851
 
808
852
  ```
809
- $ <%=cmd%> config over
853
+ <%=cmd%> config preset list
810
854
  ```
811
855
 
812
- A list of <%=prst%> can be displayed using:
856
+ A good practice is to not manually edit the configuration file and use modification commands instead.
857
+ If necessary, the configuration file can opened in a text editor with:
813
858
 
814
859
  ```
815
- $ <%=cmd%> config list
860
+ <%=cmd%> config open
816
861
  ```
817
862
 
818
- ### <a name="lprtconf"></a>Special <%=prstt%>: config
863
+ Older format for commands are still supported:
864
+
865
+ ```
866
+ <%=cmd%> config id <name> set|delete|show|initialize|update
867
+ <%=cmd%> config over
868
+ <%=cmd%> config list
869
+ ```
870
+
871
+
872
+ ### <a id="lprtconf"></a>Special <%=prstt%>: config
819
873
 
820
874
  This preset name is reserved and contains a single key: `version`. This is the version of <%=tool%> which created the file.
821
875
 
822
- ### <a name="lprtdef"></a>Special <%=prstt%>: default
876
+ ### <a id="lprtdef"></a>Special <%=prstt%>: default
823
877
 
824
878
  This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
825
879
 
@@ -830,16 +884,16 @@ Note that special plugin name: `config` can be associated with a preset that is
830
884
  Operations on this preset are done using regular `config` operations:
831
885
 
832
886
  ```
833
- $ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
834
- $ <%=cmd%> config id default get _plugin_name_
887
+ <%=cmd%> config preset set default _plugin_name_ _default_preset_for_plugin_
888
+ <%=cmd%> config preset get default _plugin_name_
835
889
  "_default_preset_for_plugin_"
836
890
  ```
837
891
 
838
- ### <a name="lprtdef"></a>Special Plugin: config
892
+ ### <a id="lplugconf"></a>Special Plugin: config
839
893
 
840
894
  Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
841
895
 
842
- When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
896
+ When <%=tool%> starts, it looks for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
843
897
 
844
898
  If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
845
899
 
@@ -878,10 +932,10 @@ The user may create as many <%=prsts%> as needed. For instance, a particular <%=
878
932
 
879
933
  Values in the configuration also follow the [Extended Value Syntax](#extended).
880
934
 
881
- Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config id update` command, the user shall use the `@val:` prefix. Example:
935
+ Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
882
936
 
883
937
  ```
884
- $ <%=cmd%> config id my_aoc_org set private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
938
+ <%=cmd%> config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
885
939
  ```
886
940
 
887
941
  This creates the <%=prst%>:
@@ -917,26 +971,29 @@ is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration
917
971
  The main plugin name is `config`, so it is possible to define a default <%=prst%> for the main plugin with:
918
972
 
919
973
  ```
920
- $ <%=cmd%> config id cli_default set interactive no
921
- $ <%=cmd%> config id default set config cli_default
974
+ <%=cmd%> config preset set cli_default interactive no
975
+ ```
976
+
977
+ ```
978
+ <%=cmd%> config preset set default config cli_default
922
979
  ```
923
980
 
924
981
  A <%=prst%> value can be removed with `unset`:
925
982
 
926
983
  ```
927
- $ <%=cmd%> config id cli_default unset interactive
984
+ <%=cmd%> config preset unset cli_default interactive
928
985
  ```
929
986
 
930
987
  Example: Define options using command line:
931
988
 
932
989
  ```
933
- $ <%=cmd%> -N --url=x --password=y --username=y node --show-config
990
+ <%=cmd%> -N --url=x --password=y --username=y node --show-config
934
991
  ```
935
992
 
936
993
  Example: Define options using a hash:
937
994
 
938
995
  ```
939
- $ <%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
996
+ <%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
940
997
  ```
941
998
 
942
999
  ### Examples
@@ -946,44 +1003,120 @@ only username/password and url are required (either on command line, or from con
946
1003
  Those can usually be provided on the command line:
947
1004
 
948
1005
  ```
949
- $ <%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
1006
+ <%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
950
1007
  ```
951
1008
 
952
1009
  This can also be provisioned in a config file:
953
1010
 
1011
+ * Build <%=prst%>
1012
+
954
1013
  ```
955
- 1$ <%=cmd%> config id shares06 set url https://10.25.0.6
956
- 2$ <%=cmd%> config id shares06 set username john
957
- 3$ <%=cmd%> config id shares06 set password 4sp3ra
958
- 4$ <%=cmd%> config id default set shares shares06
959
- 5$ <%=cmd%> config overview
960
- 6$ <%=cmd%> shares repo browse /
1014
+ <%=cmd%> config preset set shares06 url https://10.25.0.6
1015
+ <%=cmd%> config preset set shares06 username john
1016
+ <%=cmd%> config preset set shares06 password 4sp3ra
961
1017
  ```
962
1018
 
963
- The three first commands build a <%=prst%>.
964
1019
  Note that this can also be done with one single command:
965
1020
 
966
1021
  ```
967
- $ <%=cmd%> config id shares06 init @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
1022
+ <%=cmd%> config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
1023
+ ```
1024
+
1025
+ or
1026
+
1027
+ ```
1028
+ <%=cmd%> config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
1029
+ ```
1030
+
1031
+ * Define this <%=prst%> as the default <%=prst%> for the specified plugin (`shares`)
1032
+
1033
+ ```
1034
+ <%=cmd%> config preset set default shares shares06
1035
+ ```
1036
+
1037
+ * Display the content of configuration file in table format
1038
+
1039
+ ```
1040
+ <%=cmd%> config overview
1041
+ ```
1042
+
1043
+ * Execute a command on the shares application using default parameters
1044
+
1045
+ ```
1046
+ <%=cmd%> shares repo browse /
1047
+ ```
1048
+
1049
+ ## <a id="vault"></a>Secret Vault
1050
+
1051
+ When a secret or password is needed, it is possible to store in the secret vault.
1052
+
1053
+ By default the vault is defined using option `secrets`.
1054
+
1055
+ ### Using system keychain
1056
+
1057
+ Only on macOS.
1058
+
1059
+ It is possible to store secrets in macOS keychain (only read supported currently).
1060
+
1061
+ Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
1062
+
1063
+ ### Modern config file format: encrypted in config file
1064
+
1065
+ It is possible to store and use secrets encrypted.
1066
+ For this use the `config vault` command.
1067
+
1068
+ The vault can be initialized with `config vault init`
1069
+
1070
+ Then secrets can be manipulated using commands:
1071
+
1072
+ * `set`
1073
+ * `get`
1074
+ * `list`
1075
+ * `delete`
1076
+
1077
+ Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
1078
+
1079
+ ### Legacy config file format
1080
+
1081
+ The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
1082
+
1083
+ For example, choose a repository name, for example `my_secrets`, and populate it like this:
1084
+
1085
+ ```
1086
+ <%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
1087
+
1088
+ <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
1089
+
1090
+ <%=cmd%> conf id default get config
1091
+
1092
+ cli_default
968
1093
  ```
969
1094
 
970
- The fourth command defines this <%=prst%> as the default <%=prst%> for the
971
- specified application ("shares"). The 5th command displays the content of configuration file in table format.
972
- Alternative <%=prsts%> can be used with option "-P&lt;<%=prst%>&gt;"
973
- (or --preset=&lt;<%=prst%>&gt;)
1095
+ Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
1096
+ So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
974
1097
 
975
- Eventually, the last command shows a call to the shares application using default parameters.
1098
+ ```
1099
+ <%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
1100
+ ```
1101
+
1102
+ A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
976
1103
 
1104
+ To test if a secret can be found use:
1105
+
1106
+ ```
1107
+ <%=cmd%> conf vault get --username=access_key1
1108
+ ```
977
1109
 
978
1110
  ## Plugins
979
1111
 
980
1112
  The CLI tool uses a plugin mechanism. The first level command (just after <%=tool%> on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
981
1113
  For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
982
1114
 
983
- ### Create your own plugin
1115
+ ### <a id="createownplugin"></a>Create your own plugin
1116
+
984
1117
  ```
985
- $ mkdir -p ~/.aspera/<%=cmd%>/plugins
986
- $ cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
1118
+ mkdir -p ~/.aspera/<%=cmd%>/plugins
1119
+ cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
987
1120
  require 'aspera/cli/plugin'
988
1121
  module Aspera
989
1122
  module Cli
@@ -998,39 +1131,78 @@ end # Aspera
998
1131
  EOF
999
1132
  ```
1000
1133
 
1001
- ## Debugging
1134
+ ### <a id="plugins"></a>Plugins: Application URL and Authentication
1135
+
1136
+ <%=tool%> comes with several Aspera application plugins.
1137
+
1138
+ REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
1139
+
1140
+ Those are using options:
1002
1141
 
1003
- The gem is equipped with traces. By default logging level is "warn". To increase debug level, use parameter `log_level`, so either command line `--log-level=xx` or env var `<%=evp%>LOG_LEVEL`.
1142
+ * url
1143
+ * username
1144
+ * password
1004
1145
 
1005
- It is also possible to activate traces before initialisation using env var `AS_LOG_LEVEL`.
1146
+ Those can be provided using command line, parameter set, env var, see section above.
1147
+
1148
+ Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
1149
+
1150
+ ## Logging, Debugging
1151
+
1152
+ The gem is equipped with traces. By default logging level is `warn`.
1153
+ To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `<%=evp%>LOG_LEVEL`, or parameter in con file).
1154
+
1155
+ It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
1156
+
1157
+ By default passwords and secrets are removed from logs.
1158
+ Use option `log_passwords` to change this behaviour.
1006
1159
 
1007
1160
  ## Learning Aspera Product APIs (REST)
1008
1161
 
1009
1162
  This CLI uses REST APIs.
1010
- To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
1011
- exact content or HTTP requests and responses.
1163
+ To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
1012
1164
 
1013
1165
  In order to get traces of execution, use argument : `--log-level=debug`
1014
1166
 
1015
- ## <a name="graphical"></a>Graphical Interactions: Browser and Text Editor
1167
+ ## <a id="http_options"></a>HTTP socket parameters
1168
+
1169
+ If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
1170
+
1171
+ Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
1172
+
1173
+ * `read_timeout` 60 sec
1174
+ * `write_timeout` 60 sec
1175
+
1176
+ Default values are the ones of Ruby.
1177
+
1178
+ Example:
1179
+
1180
+ ```
1181
+ <%=cmd%> aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
1182
+ ```
1183
+
1184
+ ## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
1016
1185
 
1017
1186
  Some actions may require the use of a graphical tool:
1018
1187
 
1019
1188
  * a browser for Aspera on Cloud authentication (web auth method)
1020
1189
  * a text editor for configuration file edition
1021
1190
 
1022
- By default the CLI will assume that a graphical environment is available on windows,
1023
- and on other systems, rely on the presence of the "DISPLAY" environment variable.
1191
+ By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
1024
1192
  It is also possible to force the graphical mode with option --ui :
1025
1193
 
1026
- * `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
1027
- a text editor for file edition.
1028
- * `--ui=text` forces a text environment, the URL or file path to open is displayed on
1029
- terminal.
1194
+ * `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
1195
+ * `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
1030
1196
 
1031
1197
  ## HTTP proxy for REST
1032
1198
 
1033
- To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honoured by Ruby when calling REST APIs.
1199
+ To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
1200
+
1201
+ ## <a id="certificates"></a>SSL CA certificate bundle
1202
+
1203
+ <%=tool%> uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
1204
+
1205
+ `ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). <%=tool%> overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
1034
1206
 
1035
1207
  ## Proxy auto config
1036
1208
 
@@ -1039,13 +1211,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
1039
1211
  The PAC file can be tested with command: `config proxy_check` , example:
1040
1212
 
1041
1213
  ```
1042
- $ <%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
1214
+ <%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
1043
1215
  PROXY proxy.example.com:8080
1044
1216
  ```
1045
1217
 
1046
1218
  This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
1047
1219
 
1048
- ## <a name="client"></a>FASP configuration
1220
+ ## <a id="client"></a>FASP configuration
1049
1221
 
1050
1222
  The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
1051
1223
 
@@ -1057,9 +1229,9 @@ The `config` plugin also allows specification for the use of a local FASP client
1057
1229
  ### Show path of currently used `ascp`
1058
1230
 
1059
1231
  ```
1060
- $ <%=cmd%> config ascp show
1232
+ <%=cmd%> config ascp show
1061
1233
  /Users/laurent/.aspera/ascli/sdk/ascp
1062
- $ <%=cmd%> config ascp info
1234
+ <%=cmd%> config ascp info
1063
1235
  +--------------------+-----------------------------------------------------------+
1064
1236
  | key | value |
1065
1237
  +--------------------+-----------------------------------------------------------+
@@ -1067,7 +1239,7 @@ $ <%=cmd%> config ascp info
1067
1239
  ...
1068
1240
  ```
1069
1241
 
1070
- ### Selection of `ascp` location for [`direct`](#direct) agent
1242
+ ### Selection of `ascp` location for [`direct`](#agt_direct) agent
1071
1243
 
1072
1244
  By default, <%=tool%> uses any found local product with ascp, including SDK.
1073
1245
 
@@ -1078,7 +1250,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
1078
1250
  Using a POSIX shell:
1079
1251
 
1080
1252
  ```
1081
- $ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
1253
+ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
1082
1254
  ascp version: 4.0.0.182279
1083
1255
  Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
1084
1256
  Saved to default global preset global_common_defaults
@@ -1087,7 +1259,7 @@ Saved to default global preset global_common_defaults
1087
1259
  Windows:
1088
1260
 
1089
1261
  ```
1090
- $ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1262
+ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1091
1263
  ascp version: 4.0.0.182279
1092
1264
  Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1093
1265
  Saved to default global preset global_common_defaults
@@ -1100,7 +1272,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
1100
1272
  Locally installed Aspera products can be listed with:
1101
1273
 
1102
1274
  ```
1103
- $ <%=cmd%> config ascp products list
1275
+ <%=cmd%> config ascp products list
1104
1276
  :.........................................:................................................:
1105
1277
  : name : app_root :
1106
1278
  :.........................................:................................................:
@@ -1111,7 +1283,7 @@ $ <%=cmd%> config ascp products list
1111
1283
  :.........................................:................................................:
1112
1284
  ```
1113
1285
 
1114
- ### Selection of local client for `ascp` for [`direct`](#direct) agent
1286
+ ### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
1115
1287
 
1116
1288
  If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
1117
1289
 
@@ -1120,14 +1292,14 @@ Using the option use_product finds the ascp binary of the selected product.
1120
1292
  To permanently use the ascp of a product:
1121
1293
 
1122
1294
  ```
1123
- $ <%=cmd%> config ascp products use 'Aspera Connect'
1295
+ <%=cmd%> config ascp products use 'Aspera Connect'
1124
1296
  saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
1125
1297
  ```
1126
1298
 
1127
1299
  ### Installation of Connect Client on command line
1128
1300
 
1129
1301
  ```
1130
- $ <%=cmd%> config ascp connect list
1302
+ <%=cmd%> config ascp connect list
1131
1303
  :...............................................:......................................:..............:
1132
1304
  : id : title : version :
1133
1305
  :...............................................:......................................:..............:
@@ -1140,7 +1312,7 @@ $ <%=cmd%> config ascp connect list
1140
1312
  : urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
1141
1313
  : urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
1142
1314
  :...............................................:......................................:..............:
1143
- $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1315
+ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1144
1316
  :.............................................:..........................:.......................................................................:..........:...............:
1145
1317
  : title : type : href : hreflang : rel :
1146
1318
  :.............................................:..........................:.......................................................................:..........:...............:
@@ -1153,11 +1325,11 @@ $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1153
1325
  : Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
1154
1326
  : Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
1155
1327
  :.............................................:..........................:.......................................................................:..........:...............:
1156
- $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
1328
+ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
1157
1329
  downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
1158
1330
  ```
1159
1331
 
1160
- ## <a name="agents"></a>Transfer Agents
1332
+ ## <a id="agents"></a>Transfer Agents
1161
1333
 
1162
1334
  Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
1163
1335
 
@@ -1166,102 +1338,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
1166
1338
 
1167
1339
  There are currently 3 agents:
1168
1340
 
1169
- * [`direct`](#direct) : a local execution of `ascp`
1170
- * `connect` : use of a local Connect Client
1171
- * `node` : use of an Aspera Transfer Node (potentially _remote_).
1172
- * `httpgw` : use of an Aspera HTTP Gateway
1341
+ * [`direct`](#agt_direct) : a local execution of `ascp`
1342
+ * [`connect`](#agt_connect) : use of a local Connect Client
1343
+ * [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
1344
+ * [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
1345
+ * [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
1173
1346
 
1174
1347
  Note that all transfer operation are seen from the point of view of the agent.
1175
1348
  For instance, a node agent making an "upload", or "package send" operation,
1176
1349
  will effectively push files to the related server from the agent node.
1177
1350
 
1178
- <%=tool%> standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1351
+ <%=tool%> standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1179
1352
 
1180
1353
 
1181
- ### <a name="direct"></a>Direct (local ascp execution)
1354
+ ### <a id="agt_direct"></a>Direct
1182
1355
 
1183
- By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
1184
- <%=tool%> will detect locally installed Aspera products.
1356
+ The `direct` agent directly executes a local ascp.
1357
+ This is the default for <%=tool%>.
1358
+ This is equivalent to specifying `--transfer=direct`.
1359
+ <%=tool%> will detect locally installed Aspera products, including SDK.
1185
1360
  Refer to section [FASP](#client).
1186
1361
 
1187
- To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
1188
-
1189
- * `EX_fasp_proxy_url`
1190
- * `EX_http_proxy_url` (proxy for legacy http fallback)
1191
- * `EX_ascp_args`
1192
-
1193
- The `transfer-info` accepts the following optional parameters:
1362
+ The `transfer-info` accepts the following optional parameters to control multi-session, WSS
1194
1363
 
1195
1364
  <table>
1196
- <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1197
- <tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
1198
- <tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
1199
- <tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1200
- <tr><td>multi_incr_udp</td><td>Bool</td><td>true</td><td>Multi Session</td><td>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)</td></tr>
1201
- <tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
1365
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1366
+ <tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
1367
+ <tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
1368
+ <tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
1369
+ <tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
1370
+ <tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
1371
+ <tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
1372
+ <tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
1373
+ <tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
1374
+ <tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
1202
1375
  </table>
1203
1376
 
1204
- Resume parameters:
1205
-
1206
- In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1207
- Sleep between iteration is:
1377
+ Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1378
+ Sleep between iterations is:
1208
1379
 
1209
1380
  ```
1210
1381
  max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
1211
1382
  ```
1212
1383
 
1213
- <table>
1214
- <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1215
- <tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
1216
- <tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
1217
- <tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
1218
- <tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
1219
- </table>
1384
+ Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
1220
1385
 
1221
1386
  Examples:
1222
1387
 
1223
1388
  ```
1224
- $ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1225
- $ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1389
+ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1390
+ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1226
1391
  ```
1227
1392
 
1228
- ### IBM Aspera Connect Client GUI
1393
+ To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
1229
1394
 
1230
- By specifying option: `--transfer=connect`, <%=tool%> will start transfers
1231
- using the locally installed Aspera Connect Client.
1395
+ * `EX_fasp_proxy_url`
1396
+ * `EX_http_proxy_url` (proxy for legacy http fallback)
1397
+ * `EX_ascp_args`
1232
1398
 
1233
- ### Aspera Node API : Node to node transfers
1399
+ ### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
1400
+
1401
+ By specifying option: `--transfer=connect`, <%=tool%> will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
1402
+
1403
+ ### <a id="agt_node"></a>Aspera Node API : Node to node transfers
1234
1404
 
1235
1405
  By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
1236
1406
  Transfer Server using the Node API, either on a local or remote node.
1407
+ Parameters provided in option `transfer_info` are:
1237
1408
 
1238
- If a default node has been configured
1239
- in the configuration file, then this node is used by default else the parameter
1240
- `--transfer-info` is required. The node specification shall be a hash table with
1241
- three keys: url, username and password, corresponding to the URL of the node API
1242
- and associated credentials (node user or access key).
1409
+ <table>
1410
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1411
+ <tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
1412
+ <tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
1413
+ <tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
1414
+ <tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
1415
+ </table>
1243
1416
 
1244
- The `--transfer-info` parameter can directly specify a pre-configured <%=prst%> :
1245
- `--transfer-info=@preset:<psetname>` or specified using the option syntax :
1417
+ Like any other option, `transfer_info` can get its value from a pre-configured <%=prst%> :
1418
+ `--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
1246
1419
  `--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
1247
1420
 
1248
- ### <a name="trinfoaoc"></a>Aspera on cloud
1421
+ If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
1249
1422
 
1250
- By specifying option: `--transfer=aoc`, WORK IN PROGRESS
1423
+ If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
1251
1424
 
1252
- ### <a name="httpgw"></a>HTTP Gateway
1425
+ ### <a id="agt_httpgw"></a>HTTP Gateway
1253
1426
 
1254
- If it possible to send using a HTTP gateway, in case FASP is not allowed.
1427
+ If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
1255
1428
 
1256
1429
  Example:
1257
1430
 
1258
1431
  ```
1259
- $ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
1432
+ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
1260
1433
  ```
1261
1434
 
1262
1435
  Note that the gateway only supports transfers authorized with a token.
1263
1436
 
1264
- ## <a name="transferspec"></a>Transfer Specification
1437
+ ### <a id="agt_trsdk"></a>Transfer SDK
1438
+
1439
+ Another possibility is to use the Transfer SDK daemon (asperatransferd).
1440
+
1441
+ By default it will listen on local port `55002` on `127.0.0.1`.
1442
+
1443
+ ## <a id="transferspec"></a>Transfer Specification
1265
1444
 
1266
1445
  Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
1267
1446
  is described in a _transfer-spec_ (Transfer Specification), such as:
@@ -1274,9 +1453,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
1274
1453
 
1275
1454
  <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
1276
1455
 
1277
- If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1456
+ If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
1278
1457
 
1279
- It is possible to specify ascp options when the `transfer` option is set to [`direct`](#direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1458
+ It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
1280
1459
 
1281
1460
  The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1282
1461
 
@@ -1285,21 +1464,22 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1285
1464
 
1286
1465
  A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
1287
1466
 
1288
- ## <a name="transferparams"></a>Transfer Parameters
1467
+ ## <a id="transferparams"></a>Transfer Parameters
1289
1468
 
1290
- All standard _transfer-spec_ parameters can be speficied.
1469
+ All standard _transfer-spec_ parameters can be specified.
1291
1470
  [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1292
1471
 
1293
1472
  References:
1294
1473
 
1295
1474
  * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1296
1475
  * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1476
+ * [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) &rarr; search `The parameters for starting a transfer.`
1297
1477
 
1298
1478
  Parameters can be displayed with commands:
1299
1479
 
1300
1480
  ```
1301
- $ <%=cmd%> config ascp spec
1302
- $ <%=cmd%> config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
1481
+ <%=cmd%> config ascp spec
1482
+ <%=cmd%> config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
1303
1483
  ```
1304
1484
 
1305
1485
  Columns:
@@ -1307,9 +1487,10 @@ Columns:
1307
1487
  * D=Direct (local `ascp` execution)
1308
1488
  * N=Node API
1309
1489
  * C=Connect Client
1310
- * arg=`ascp` argument or environment variable
1311
1490
 
1312
- Fields with EX_ prefix are extensions to transfer agent [`direct`](#direct). (only in <%=tool%>).
1491
+ `ascp` argument or environment variable is provided in description.
1492
+
1493
+ Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in <%=tool%>).
1313
1494
 
1314
1495
  <%=spec_table%>
1315
1496
 
@@ -1327,17 +1508,17 @@ The option `to_folder` provides an equivalent and convenient way to change this
1327
1508
 
1328
1509
  ### List of files for transfers
1329
1510
 
1330
- When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files (Extensed value) is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1511
+ When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1331
1512
  So, by default, the list of files to transfer will be simply specified on the command line:
1332
1513
 
1333
1514
  ```
1334
- $ <%=cmd%> server upload ~/mysample.file secondfile
1515
+ <%=cmd%> server upload ~/mysample.file secondfile
1335
1516
  ```
1336
1517
 
1337
1518
  This is equivalent to:
1338
1519
 
1339
1520
  ```
1340
- $ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1521
+ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1341
1522
  ```
1342
1523
 
1343
1524
  More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
@@ -1363,13 +1544,19 @@ For ease of use and flexibility, the list of files to transfer is specified by t
1363
1544
  --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
1364
1545
  ```
1365
1546
 
1547
+ providing a file list directly to ascp:
1548
+
1549
+ ```
1550
+ ... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
1551
+ ```
1552
+
1366
1553
  * Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
1367
1554
 
1368
1555
  ```
1369
1556
  --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
1370
1557
  ```
1371
1558
 
1372
- This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1559
+ This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1373
1560
 
1374
1561
  In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1375
1562
 
@@ -1379,10 +1566,10 @@ In case the file list is provided on the command line i.e. using `--sources=@arg
1379
1566
  Example:
1380
1567
 
1381
1568
  ```
1382
- $ <%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1569
+ <%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1383
1570
  ```
1384
1571
 
1385
- Internally, when transfer agent [`direct`](#direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pait-list` is provided in `ts` in `EX_ascp_args`.
1572
+ Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
1386
1573
 
1387
1574
  Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
1388
1575
 
@@ -1395,7 +1582,7 @@ Source files are located on "Aspera on cloud", when :
1395
1582
  * the server is Aspera on Cloud, and making a download / recv
1396
1583
  * the agent is Aspera on Cloud, and making an upload / send
1397
1584
 
1398
- ### <a name="multisession"></a>Support of multi-session
1585
+ ### <a id="multisession"></a>Support of multi-session
1399
1586
 
1400
1587
  Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1401
1588
 
@@ -1446,16 +1633,14 @@ When multi-session is used, one separate UDP port is used per session (refer to
1446
1633
  --ts=@json:'{"precalculate_job_size":true}'
1447
1634
  ```
1448
1635
 
1449
-
1450
-
1451
- ## <a name="scheduling"></a>Lock for exclusive execution
1636
+ ## <a id="scheduling"></a>Lock for exclusive execution
1452
1637
 
1453
1638
  In some conditions, it may be desirable to ensure that <%=tool%> is not executed several times in parallel.
1454
1639
 
1455
1640
  For instance when <%=tool%> is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
1456
1641
 
1457
1642
  * Executing instances may pile-up and kill the system
1458
- * The same file may be transfered by multiple instances at the same time.
1643
+ * The same file may be transferred by multiple instances at the same time.
1459
1644
  * `preview` may generate the same files in multiple instances.
1460
1645
 
1461
1646
  Usually the OS native scheduler already provides some sort of protection against parallel execution:
@@ -1475,7 +1660,7 @@ Example:
1475
1660
  Run this same command in two separate terminals within less than 30 seconds:
1476
1661
 
1477
1662
  ```
1478
- ascli config echo @ruby:'sleep(30)' --lock-port=12345
1663
+ <%=cmd%> config echo @ruby:'sleep(30)' --lock-port=12345
1479
1664
  ```
1480
1665
 
1481
1666
  The first instance will sleep 30 seconds, the second one will immediately exit like this:
@@ -1489,14 +1674,14 @@ WARN -- : Another instance is already running (Address already in use - bind(2)
1489
1674
  `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
1490
1675
  , some are provided om shared libraries and must be activated. (e.g. using `trapd`)
1491
1676
 
1492
- The list of supported *PVCL* adapters can be retried with command:
1677
+ The list of supported *PVCL* adapters can be retrieved with command:
1493
1678
 
1494
1679
  ```
1495
- $ <%=cmd%> conf ascp info
1680
+ <%=cmd%> conf ascp info
1496
1681
  +--------------------+-----------------------------------------------------------+
1497
1682
  | key | value |
1498
1683
  +--------------------+-----------------------------------------------------------+
1499
- -----8<----snip---------
1684
+ -----8<-----snip-----8<-----
1500
1685
  | product_name | IBM Aspera SDK |
1501
1686
  | product_version | 4.0.1.182389 |
1502
1687
  | process | pvcl |
@@ -1521,20 +1706,26 @@ The simplified format is:
1521
1706
 
1522
1707
  One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
1523
1708
 
1524
- ## <a name="faux_testing"></a>`faux:` for testing
1709
+ ## <a id="faux_testing"></a>`faux:` for testing
1525
1710
 
1526
- This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>
1711
+ This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>.
1527
1712
 
1528
1713
  This adapter can be used to simulate a file or a directory.
1529
1714
 
1530
- To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form `faux:///fname?fsize` where:
1715
+ To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
1716
+
1717
+ ```
1718
+ faux:///filename?filesize
1719
+ ```
1720
+
1721
+ where:
1531
1722
 
1532
- * `fname` is the name that will be assigned to the file on the destination
1533
- * `fsize` is the number of bytes that will be sent (in decimal).
1723
+ * `filename` is the name that will be assigned to the file on the destination
1724
+ * `filesize` is the number of bytes that will be sent (in decimal).
1534
1725
 
1535
- Note that the character `?` is a special shell character (wildcard), so `faux` file specification on command line shall be protected (using `\?` and `\&` or using quotes). If not, the shell may give error: `no matches found` or equivalent.
1726
+ Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
1536
1727
 
1537
- For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2^20, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2^60. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of system memory unless `faux://`.
1728
+ For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
1538
1729
 
1539
1730
  To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
1540
1731
 
@@ -1542,21 +1733,21 @@ To send uninitialized data in place of a source directory, the source argument i
1542
1733
  faux:///dirname?<arg1>=<val1>&...
1543
1734
  ```
1544
1735
 
1545
- `dirname` is the folder name and can contain `/` to specify a subfolder.
1736
+ where:
1546
1737
 
1547
- Supported arguments are:
1738
+ * `dirname` is the folder name and can contain `/` to specify a subfolder.
1739
+ * supported arguments are:
1548
1740
 
1549
1741
  <table>
1550
1742
  <tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
1551
- <tr><td>count</td><td>int</td><td>mandatory</td><td>number of files</td></tr>
1552
- <tr><td>file</td><td>string</td><td>file</td><td>basename for files</td></tr>
1553
- <tr><td>size</td><td>int</td><td>0</td><td>size of first file.</td></tr>
1554
- <tr><td>inc</td><td>int</td><td>0</td><td>increment applied to determine next file size</td></tr>
1555
- <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>sequence in determining next file size</td></tr>
1556
- <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>how source data initialized.<br/>Option 'none' is not allowed for downloads.</td></tr>
1743
+ <tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
1744
+ <tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
1745
+ <tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
1746
+ <tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
1747
+ <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
1748
+ <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
1557
1749
  </table>
1558
1750
 
1559
-
1560
1751
  The sequence parameter is applied as follows:
1561
1752
 
1562
1753
  * If `seq` is `random` then each file size is:
@@ -1564,40 +1755,40 @@ The sequence parameter is applied as follows:
1564
1755
  * size +/- (inc * rand())
1565
1756
  * Where rand is a random number between 0 and 1
1566
1757
  * Note that file size must not be negative, inc will be set to size if it is greater than size
1567
- * Similarly, overall file size must be less than 8 * 2^60. If size + inc is greater, inc will be reduced to limit size + inc to 7 * 2^60.
1758
+ * Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
1568
1759
 
1569
1760
  * If `seq` is `sequential` then each file size is:
1570
1761
 
1571
- * size + ((fileindex - 1) * inc)
1762
+ * `size + ((fileindex - 1) * inc)`
1572
1763
  * Where first file is index 1
1573
- * So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
1574
- * As with random, inc will be adjusted if size + (count * inc) is not less then 8 ^ 2^60.
1764
+ * So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
1765
+ * As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
1575
1766
 
1576
- Filenames generated are of the form: `<file>_<00000 . . . count>_<filesize>`
1767
+ Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
1577
1768
 
1578
1769
  To discard data at the destination, the destination argument is set to `faux://` .
1579
1770
 
1580
1771
  Examples:
1581
1772
 
1582
- * Upload 20 gigabytes of random data to file myfile to directory /Upload
1773
+ * Upload 20 gibibytes of random data to file myfile to directory /Upload
1583
1774
 
1584
1775
  ```
1585
- $ <%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
1776
+ <%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
1586
1777
  ```
1587
1778
 
1588
1779
  * Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
1589
1780
 
1590
1781
  ```
1591
- $ <%=cmd%> server upload /tmp/sample --to-folder=faux://
1782
+ <%=cmd%> server upload /tmp/sample --to-folder=faux://
1592
1783
  ```
1593
1784
 
1594
- * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 M - 2 bytes, with the basename of each file being `testfile` to /Upload
1785
+ * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
1595
1786
 
1596
1787
  ```
1597
- $ <%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1788
+ <%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1598
1789
  ```
1599
1790
 
1600
- ## <a name="commands"></a>Sample Commands
1791
+ ## <a id="commands"></a>Sample Commands
1601
1792
 
1602
1793
  A non complete list of commands used in unit tests:
1603
1794
 
@@ -1606,44 +1797,28 @@ A non complete list of commands used in unit tests:
1606
1797
  ...and more
1607
1798
  ```
1608
1799
 
1609
- ## <a name="usage"></a>Usage
1800
+ ## <a id="usage"></a>Usage
1610
1801
 
1611
1802
  ```
1612
- $ <%=cmd%> -h
1803
+ <%=cmd%> -h
1613
1804
  <%=File.read(ENV["INCL_USAGE"])%>
1614
1805
 
1615
1806
  ```
1616
1807
 
1617
1808
  Note that actions and parameter values can be written in short form.
1618
1809
 
1619
- # <a name="plugins"></a>Plugins: Application URL and Authentication
1620
-
1621
- <%=tool%> comes with several Aspera application plugins.
1622
-
1623
- REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
1624
-
1625
- Those are using options:
1626
-
1627
- * url
1628
- * username
1629
- * password
1630
-
1631
- Those can be provided using command line, parameter set, env var, see section above.
1632
-
1633
- Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
1634
-
1635
- # <a name="aoc"></a>Plugin: Aspera on Cloud
1810
+ # <a id="aoc"></a>Plugin: Aspera on Cloud
1636
1811
 
1637
1812
  Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
1638
1813
 
1639
1814
  It is recommended to use the wizard to set it up, but manual configuration is also possible.
1640
1815
 
1641
- ## <a name="aocwizard"></a>Configuration: using Wizard
1816
+ ## <a id="aocwizard"></a>Configuration: using Wizard
1642
1817
 
1643
1818
  <%=tool%> provides a configuration wizard. Here is a sample invocation :
1644
1819
 
1645
1820
  ```
1646
- $ <%=cmd%> config wizard
1821
+ <%=cmd%> config wizard
1647
1822
  option: url> https://myorg.ibmaspera.com
1648
1823
  Detected: Aspera on Cloud
1649
1824
  Preparing preset: aoc_myorg
@@ -1659,7 +1834,7 @@ Setting config preset as default for aspera
1659
1834
  saving config file
1660
1835
  Done.
1661
1836
  You can test with:
1662
- $ <%=cmd%> aoc user info show
1837
+ <%=cmd%> aoc user info show
1663
1838
  ```
1664
1839
 
1665
1840
  Optionally, it is possible to create a new organization-specific "integration".
@@ -1667,7 +1842,7 @@ For this, specify the option: `--use-generic-client=no`.
1667
1842
 
1668
1843
  This will guide you through the steps to create.
1669
1844
 
1670
- ## <a name="aocmanual"></a>Configuration: using manual setup
1845
+ ## <a id="aocmanual"></a>Configuration: using manual setup
1671
1846
 
1672
1847
  If you used the wizard (recommended): skip this section.
1673
1848
 
@@ -1679,7 +1854,7 @@ Several types of OAuth authentication are supported:
1679
1854
  * Web based authentication : authentication is made by user using a browser
1680
1855
  * URL Token : external users authentication with url tokens (public links)
1681
1856
 
1682
- The authentication method is controled by option `auth`.
1857
+ The authentication method is controlled by option `auth`.
1683
1858
 
1684
1859
  For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [<%=prst%> for Aspera on Cloud](#aocpreset).
1685
1860
 
@@ -1687,7 +1862,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
1687
1862
 
1688
1863
  In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.<%=tool%> saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
1689
1864
 
1690
- ### <a name="clientreg"></a>Optional: API Client Registration
1865
+ ### <a id="clientreg"></a>Optional: API Client Registration
1691
1866
 
1692
1867
  If you use the built-in client_id and client_secret, skip this and do not set them in next section.
1693
1868
 
@@ -1711,31 +1886,31 @@ Note: for web based authentication, <%=tool%> listens on a local port (e.g. spec
1711
1886
 
1712
1887
  Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
1713
1888
 
1714
- ### <a name="aocpreset"></a><%=prst%> for Aspera on Cloud
1889
+ ### <a id="aocpreset"></a><%=prst%> for Aspera on Cloud
1715
1890
 
1716
1891
  If you did not use the wizard, you can also manually create a <%=prst%> for <%=tool%> in its configuration file.
1717
1892
 
1718
1893
  Lets create an <%=prst%> called: `my_aoc_org` using `ask` interactive input (client info from previous step):
1719
1894
 
1720
1895
  ```
1721
- $ <%=cmd%> config id my_aoc_org ask url client_id client_secret
1896
+ <%=cmd%> config preset ask my_aoc_org url client_id client_secret
1722
1897
  option: url> https://myorg.ibmaspera.com/
1723
1898
  option: client_id> BJLPObQiFw
1724
1899
  option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
1725
1900
  updated: my_aoc_org
1726
1901
  ```
1727
1902
 
1728
- (This can also be done in one line using the command `config id my_aoc_org update --url=...`)
1903
+ (This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
1729
1904
 
1730
1905
  Define this <%=prst%> as default configuration for the `aspera` plugin:
1731
1906
 
1732
1907
  ```
1733
- $ <%=cmd%> config id default set aoc my_aoc_org
1908
+ <%=cmd%> config preset set default aoc my_aoc_org
1734
1909
  ```
1735
1910
 
1736
1911
  Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
1737
1912
 
1738
- ### <a name="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
1913
+ ### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
1739
1914
 
1740
1915
  For a Browser-less, Private Key-based authentication, use the following steps.
1741
1916
 
@@ -1750,13 +1925,13 @@ This can be done using any of the following method:
1750
1925
  * using the CLI:
1751
1926
 
1752
1927
  ```
1753
- $ <%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
1928
+ <%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
1754
1929
  ```
1755
1930
 
1756
1931
  * `ssh-keygen`:
1757
1932
 
1758
1933
  ```
1759
- $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1934
+ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1760
1935
  ```
1761
1936
 
1762
1937
  * `openssl`
@@ -1764,11 +1939,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1764
1939
  (on some openssl implementation (mac) there is option: -nodes (no DES))
1765
1940
 
1766
1941
  ```
1767
- $ APIKEY=~/.aspera/<%=cmd%>/aocapikey
1768
- $ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
1769
- $ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
1770
- $ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
1771
- $ rm -f ${APIKEY}.protected
1942
+ APIKEY=~/.aspera/<%=cmd%>/aocapikey
1943
+ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
1944
+ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
1945
+ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
1946
+ rm -f ${APIKEY}.protected
1772
1947
  ```
1773
1948
 
1774
1949
  #### API Client JWT activation
@@ -1787,13 +1962,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
1787
1962
  * Using command line
1788
1963
 
1789
1964
  ```
1790
- $ <%=cmd%> aoc admin res client list
1965
+ <%=cmd%> aoc admin res client list
1791
1966
  :............:.........:
1792
1967
  : id : name :
1793
1968
  :............:.........:
1794
1969
  : BJLPObQiFw : <%=cmd%> :
1795
1970
  :............:.........:
1796
- $ <%=cmd%> aoc admin res client --id=BJLPObQiFw modify @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
1971
+ <%=cmd%> aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
1797
1972
  modified
1798
1973
  ```
1799
1974
 
@@ -1814,14 +1989,14 @@ open the previously generated public key located here: `$HOME/.aspera/<%=cmd%>/a
1814
1989
  * Using command line
1815
1990
 
1816
1991
  ```
1817
- $ <%=cmd%> aoc admin res user list
1992
+ <%=cmd%> aoc admin res user list
1818
1993
  :........:................:
1819
1994
  : id : name :
1820
1995
  :........:................:
1821
1996
  : 109952 : Tech Support :
1822
1997
  : 109951 : LAURENT MARTIN :
1823
1998
  :........:................:
1824
- $ <%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
1999
+ <%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
1825
2000
  modified
1826
2001
  ```
1827
2002
 
@@ -1829,7 +2004,7 @@ Note: the `aspera user info show` command can be used to verify modifications.
1829
2004
 
1830
2005
  ### <%=prst%> modification for JWT
1831
2006
 
1832
- To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the folowing:
2007
+ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the following:
1833
2008
 
1834
2009
  * change auth method to JWT
1835
2010
  * provide location of private key
@@ -1838,7 +2013,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
1838
2013
  Execute:
1839
2014
 
1840
2015
  ```
1841
- $ <%=cmd%> config id my_aoc_org update --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
2016
+ <%=cmd%> config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
1842
2017
  ```
1843
2018
 
1844
2019
  Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
@@ -1846,26 +2021,29 @@ Note: the private key argument represents the actual PEM string. In order to rea
1846
2021
  After this last step, commands do not require web login anymore.
1847
2022
 
1848
2023
 
1849
- ### <a name="aocfirst"></a>First Use
2024
+ ### <a id="aocfirst"></a>First Use
1850
2025
 
1851
2026
  Once client has been registered and <%=prst%> created: <%=tool%> can be used:
1852
2027
 
1853
2028
  ```
1854
- $ <%=cmd%> aoc files br /
2029
+ <%=cmd%> aoc files br /
1855
2030
  Current Workspace: Default Workspace (default)
1856
2031
  empty
1857
2032
  ```
1858
2033
 
1859
2034
 
1860
- ### Administration
2035
+ ## Administration
1861
2036
 
1862
2037
  The `admin` command allows several administrative tasks (and require admin privilege).
1863
2038
 
1864
2039
  It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
1865
2040
 
1866
- Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2041
+ ### Bulk creation and deletion of resource
2042
+
2043
+ Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
2044
+ In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
1867
2045
 
1868
- #### Listing resources
2046
+ ### Listing resources
1869
2047
 
1870
2048
  The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
1871
2049
 
@@ -1887,7 +2065,7 @@ Other parameters are directly sent as parameters to the GET request on API.
1887
2065
 
1888
2066
  `page` and `per_page` are normally added by <%=tool%> to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
1889
2067
 
1890
- `q` and `sort` are available on most resrouce types.
2068
+ `q` and `sort` are available on most resource types.
1891
2069
 
1892
2070
  Other parameters depend on the type of entity (refer to AoC API).
1893
2071
 
@@ -1899,7 +2077,7 @@ Examples:
1899
2077
  <%=cmd%> aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
1900
2078
  ```
1901
2079
 
1902
- * List users who logded-in before a date:
2080
+ * List users who logged-in before a date:
1903
2081
 
1904
2082
  ```
1905
2083
  <%=cmd%> aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
@@ -1915,40 +2093,85 @@ Refer to the AoC API for full list of query parameters, or use the browser in de
1915
2093
 
1916
2094
  Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
1917
2095
 
1918
- #### Access Key secrets
2096
+ ### <a id="res_select"></a>Selecting a resource
2097
+
2098
+ Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
1919
2099
 
1920
- In order to access some administrative actions on "nodes" (in fact, access keys), the associated
1921
- secret is required, it is usually provided using the `secret` option. For example in a command like:
2100
+ To execute an action on a specific resource, select it using one of those methods:
2101
+
2102
+ * *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
2103
+ * give name on command line *after the action*: `aoc admin res node show name abc`
2104
+ * provide option `id` : `aoc admin res node show --id=123`
2105
+ * provide option `name` : `aoc admin res node show --name=abc`
2106
+
2107
+ ### Access Key secrets
2108
+
2109
+ In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
2110
+ It is usually provided using the `secret` option.
2111
+ For example in a command like:
1922
2112
 
1923
2113
  ```
1924
- $ <%=cmd%> aoc admin res node --id="access_key1" --secret="secret1" v3 info
2114
+ <%=cmd%> aoc admin res node --id=123 --secret="secret1" v3 info
1925
2115
  ```
1926
2116
 
1927
- It is also possible to provide a set of secrets used on a regular basis. This can be done using the `secrets` option. The value provided shall be a Hash, where keys are access key ids, and values are the associated secrets.
2117
+ It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
1928
2118
 
1929
- First choose a repository name, for example `my_secrets`, and populate it like this:
2119
+ ### Activity
2120
+
2121
+ The activity app can be queried with:
1930
2122
 
1931
2123
  ```
1932
- $ <%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
1933
- $ <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
1934
- $ <%=cmd%> conf id default get config
1935
- "cli_default"
2124
+ <%=cmd%> aoc admin analytics transfers
1936
2125
  ```
1937
2126
 
1938
- Here above, one already has set a `config` global preset to preset `cli_default` (refer to earlier in documentation), then the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
2127
+ It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2128
+
2129
+ `mytemplate.erb`:
1939
2130
 
1940
2131
  ```
1941
- $ <%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
2132
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2133
+ To: <<%='<'%>%=ev['user_email']%>>
2134
+ Subject: <%='<'%>%=ev['files_completed']%> files received
2135
+
2136
+ Dear <%='<'%>%=ev[:user_email.to_s]%>,
2137
+ We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2138
+ <%='<'%>%=ev['content']%>
2139
+
2140
+ Thank you.
1942
2141
  ```
2142
+ The environment provided contains the following additional variable:
1943
2143
 
1944
- A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
2144
+ * ev : all details on the transfer event
2145
+
2146
+ Example:
2147
+
2148
+ ```
2149
+ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2150
+ --query=@json:'{"status":"completed","direction":"receive"}' \
2151
+ --notif-to=active --notif-template=@file:mytemplate.erb
2152
+ ```
1945
2153
 
1946
- #### Examples
2154
+ Options:
2155
+
2156
+ * `once_only` keep track of last date it was called, so next call will get only new events
2157
+ * `query` filter (on API call)
2158
+ * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2159
+
2160
+ Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
1947
2161
 
1948
- * Bulk creation
2162
+ ### Transfer: Using specific transfer ports
2163
+
2164
+ By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
2165
+ The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2166
+
2167
+ ### Using ATS
2168
+
2169
+ Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
2170
+
2171
+ ### Example: Bulk creation of users
1949
2172
 
1950
2173
  ```
1951
- $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
2174
+ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
1952
2175
  :.......:.........:
1953
2176
  : id : status :
1954
2177
  :.......:.........:
@@ -1957,20 +2180,20 @@ $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exa
1957
2180
  :.......:.........:
1958
2181
  ```
1959
2182
 
1960
- * Find with filter and delete
2183
+ ### Example: Find with filter and delete
1961
2184
 
1962
2185
  ```
1963
- $ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
2186
+ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
1964
2187
  :.......:........................:
1965
2188
  : id : email :
1966
2189
  :.......:........................:
1967
2190
  : 98398 : dummyuser1@example.com :
1968
2191
  : 98399 : dummyuser2@example.com :
1969
2192
  :.......:........................:
1970
- $ thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
1971
- $ echo $thelist
2193
+ thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
2194
+ echo $thelist
1972
2195
  ["113501","354061"]
1973
- $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
2196
+ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
1974
2197
  :.......:.........:
1975
2198
  : id : status :
1976
2199
  :.......:.........:
@@ -1979,18 +2202,18 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
1979
2202
  :.......:.........:
1980
2203
  ```
1981
2204
 
1982
- * <a name="deactuser"></a>Find deactivated users since more than 2 years
2205
+ ### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
1983
2206
 
1984
2207
  ```
1985
- ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
2208
+ <%=cmd%> aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
1986
2209
  ```
1987
2210
 
1988
2211
  To delete them use the same method as before
1989
2212
 
1990
- * Display current user's workspaces
2213
+ ### Example: Display current user's workspaces
1991
2214
 
1992
2215
  ```
1993
- $ <%=cmd%> aoc user workspaces
2216
+ <%=cmd%> aoc user workspaces
1994
2217
  :......:............................:
1995
2218
  : id : name :
1996
2219
  :......:............................:
@@ -2000,18 +2223,18 @@ $ <%=cmd%> aoc user workspaces
2000
2223
  :......:............................:
2001
2224
  ```
2002
2225
 
2003
- * Create a sub access key in a "node"
2226
+ ### Example: Create a sub access key in a "node"
2004
2227
 
2005
2228
  Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
2006
2229
 
2007
2230
  ```
2008
- $ <%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
2231
+ <%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
2009
2232
  ```
2010
2233
 
2011
- * Display transfer events (ops/transfer)
2234
+ ### Example: Display transfer events (ops/transfer)
2012
2235
 
2013
2236
  ```
2014
- $ <%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2237
+ <%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2015
2238
  ```
2016
2239
 
2017
2240
  Examples of query (TODO: cleanup):
@@ -2029,16 +2252,16 @@ Examples of query (TODO: cleanup):
2029
2252
  # active_only=true|false
2030
2253
  ```
2031
2254
 
2032
- * Display node events (events)
2255
+ ### Example: Display node events (events)
2033
2256
 
2034
2257
  ```
2035
- $ <%=cmd%> aoc admin res node --secret=_secret_ v3 events
2258
+ <%=cmd%> aoc admin res node --secret=_secret_ v3 events
2036
2259
  ```
2037
2260
 
2038
- * display members of a workspace
2261
+ ### Example: Display members of a workspace
2039
2262
 
2040
2263
  ```
2041
- $ <%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2264
+ <%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2042
2265
  :.............:.........:..................................:
2043
2266
  : member_type : manager : member.email :
2044
2267
  :.............:.........:..................................:
@@ -2057,7 +2280,7 @@ other query parameters:
2057
2280
  {"workspace_membership_through":true,"include_indirect":true}
2058
2281
  ```
2059
2282
 
2060
- * <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
2283
+ ### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
2061
2284
 
2062
2285
  a- Get id of first workspace
2063
2286
 
@@ -2076,7 +2299,7 @@ WS2ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' -
2076
2299
  c- Extract membership information
2077
2300
 
2078
2301
  ```
2079
- $ <%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2302
+ <%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2080
2303
  ```
2081
2304
 
2082
2305
  d- Convert to creation data for second workspace:
@@ -2094,13 +2317,13 @@ jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"
2094
2317
  e- Add members to second workspace
2095
2318
 
2096
2319
  ```
2097
- $ <%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2320
+ <%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2098
2321
  ```
2099
2322
 
2100
- * Get users who did not log since a date
2323
+ ### Example: Get users who did not log since a date
2101
2324
 
2102
2325
  ```
2103
- $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2326
+ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2104
2327
  :...............................:
2105
2328
  : email :
2106
2329
  :...............................:
@@ -2109,20 +2332,20 @@ $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_logi
2109
2332
  :...............................:
2110
2333
  ```
2111
2334
 
2112
- * List "Limited" users
2335
+ ### Example: List "Limited" users
2113
2336
 
2114
2337
  ```
2115
- $ <%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
2338
+ <%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
2116
2339
  ```
2117
2340
 
2118
- * Perform a multi Gbps transfer between two remote shared folders
2341
+ ### Example: Perform a multi Gbps transfer between two remote shared folders
2119
2342
 
2120
- In this example, a user has access to a workspace where two shared folders are located on differente sites, e.g. different cloud regions.
2343
+ In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
2121
2344
 
2122
2345
  First, setup the environment (skip if already done)
2123
2346
 
2124
2347
  ```
2125
- $ <%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
2348
+ <%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
2126
2349
  Detected: Aspera on Cloud
2127
2350
  Preparing preset: aoc_sedemo
2128
2351
  Using existing key:
@@ -2141,7 +2364,7 @@ Setting config preset as default for aspera
2141
2364
  saving config file
2142
2365
  Done.
2143
2366
  You can test with:
2144
- $ <%=cmd%> aoc user info show
2367
+ <%=cmd%> aoc user info show
2145
2368
  ```
2146
2369
 
2147
2370
  This creates the option preset "aoc_&lt;org name&gt;" to allow seamless command line access and sets it as default for aspera on cloud.
@@ -2151,20 +2374,20 @@ Then, create two shared folders located in two regions, in your files home, in a
2151
2374
  Then, transfer between those:
2152
2375
 
2153
2376
  ```
2154
- $ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2377
+ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2155
2378
  ```
2156
2379
 
2157
- * create registration key to register a node
2380
+ ### Example: create registration key to register a node
2158
2381
 
2159
2382
  ```
2160
- $ <%=cmd%> aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2383
+ <%=cmd%> aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2161
2384
  jfqslfdjlfdjfhdjklqfhdkl
2162
2385
  ```
2163
2386
 
2164
- * delete all registration keys
2387
+ ### Example: delete all registration keys
2165
2388
 
2166
2389
  ```
2167
- $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc admin res admin/client delete --bulk=yes --id=@lines:@stdin:
2390
+ <%=cmd%> aoc admin res client list --fields=id --format=csv|<%=cmd%> aoc admin res client delete --bulk=yes --id=@lines:@stdin:
2168
2391
  +-----+---------+
2169
2392
  | id | status |
2170
2393
  +-----+---------+
@@ -2175,81 +2398,107 @@ $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc
2175
2398
  +-----+---------+
2176
2399
  ```
2177
2400
 
2178
- * List packages in a given shared inbox
2401
+ ### Example: Create a node
2179
2402
 
2180
- First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2181
- (To find out available filters, consult the API definition, or use the web interface in developer mode).
2403
+ AoC nodes as actually composed with two related entities:
2182
2404
 
2183
- Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
2405
+ * An access key created on the Transfer Server (HSTS/ATS)
2406
+ * a `node` resource in the AoC application.
2184
2407
 
2185
- ```
2186
- shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
2408
+ The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
2409
+ Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
2410
+
2411
+ So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
2187
2412
 
2188
- ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2413
+ * create the access key on ATS
2414
+
2415
+ ```
2416
+ <%=cmd%> aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
2189
2417
  ```
2190
2418
 
2191
- ## Shared folders
2419
+ Take a note of the randomly generated `id` and `secret`.
2192
2420
 
2193
- * list shared folders in node
2421
+ * Retrieve the ATS node address
2194
2422
 
2195
2423
  ```
2196
- $ <%=cmd%> aoc admin res node --id=8669 shared_folders
2424
+ <%=cmd%> aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
2197
2425
  ```
2198
2426
 
2199
- * list shared folders in workspace
2427
+ * Create the node entity
2200
2428
 
2201
2429
  ```
2202
- $ <%=cmd%> aoc admin res workspace --id=10818 shared_folders
2430
+ <%=cmd%> aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
2203
2431
  ```
2204
2432
 
2205
- * list members of shared folder
2433
+ Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
2434
+
2435
+ ### Example: List packages in a given shared inbox
2436
+
2437
+ First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2438
+ (To find out available filters, consult the API definition, or use the web interface in developer mode).
2439
+
2440
+ Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
2206
2441
 
2207
2442
  ```
2208
- $ <%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
2443
+ shbxid=$(<%=cmd%> aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
2444
+
2445
+ <%=cmd%> aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2209
2446
  ```
2210
2447
 
2211
- ## Send a Package
2448
+ ## Packages
2449
+
2450
+ The webmail-like application.
2451
+
2452
+ ### Send a Package
2212
2453
 
2213
2454
  Send a package:
2214
2455
 
2215
2456
  ```
2216
- $ <%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
2457
+ <%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
2217
2458
  ```
2218
2459
 
2219
2460
  Notes:
2220
2461
 
2221
- * the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
2222
- * to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". <%=cmd%> will resolve the list of email addresses to expected user ids.
2223
- * a recipient can be a shared inbox, in this case just use the name of the shared inbox as recipient.
2224
- * If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
2225
- * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
2462
+ * The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
2463
+ * List allowed shared inbox destinations with: `<%=cmd%> aoc user shared_inboxes`
2464
+ * Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
2465
+ * Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
2466
+ * or just names: `"recipients":[{"The Dest"}]` . <%=cmd%> will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
2467
+ * If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
2468
+ * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
2226
2469
  * if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
2227
2470
 
2228
2471
  Examples:
2229
2472
 
2473
+ * Send a package with one file to two users, using their email
2474
+
2230
2475
  ```
2231
- $ <%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
2476
+ <%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
2232
2477
  ```
2233
2478
 
2479
+ * Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
2480
+
2234
2481
  ```
2235
- $ <%=cmd%> aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
2482
+ <%=cmd%> aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
2236
2483
  ```
2237
2484
 
2485
+ * Send a package with one file to a shared inbox (by name) with metadata
2486
+
2238
2487
  ```
2239
- $ <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2488
+ <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2240
2489
  ```
2241
2490
 
2242
- ## <a name="aoccargo"></a>Receive new packages only
2491
+ ### <a id="aoccargo"></a>Receive new packages only (Cargo)
2243
2492
 
2244
2493
  It is possible to automatically download new packages, like using Aspera Cargo:
2245
2494
 
2246
2495
  ```
2247
- $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2496
+ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2248
2497
  ```
2249
2498
 
2250
2499
  * `--id=ALL` (case sensitive) will download all packages
2251
- * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
2252
- * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
2500
+ * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
2501
+ * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
2253
2502
 
2254
2503
  Typically, one would execute this command on a regular basis, using the method of your choice:
2255
2504
 
@@ -2257,101 +2506,107 @@ Typically, one would execute this command on a regular basis, using the method o
2257
2506
  * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
2258
2507
  * etc...
2259
2508
 
2260
- ## Download Files
2509
+ ## Files
2510
+
2511
+ Folder sharing app.
2512
+
2513
+ ### Download Files
2261
2514
 
2262
2515
  Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2263
2516
 
2264
2517
  * if only one source is provided, it is downloaded
2265
2518
  * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2266
2519
 
2267
- ## Find Files
2268
-
2269
- The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2520
+ ### Shared folders
2270
2521
 
2271
- The expression can be of 3 formats:
2272
-
2273
- * empty (default) : all files, equivalent to value: `exec:true`
2274
- * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2275
-
2276
- For instance, to find files with a special extension, use `--value='\.myext$'`
2277
-
2278
- * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2279
-
2280
- Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2281
-
2282
- * Find files more recent than 100 days
2522
+ * list shared folders in node
2283
2523
 
2284
2524
  ```
2285
- f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2525
+ <%=cmd%> aoc admin res node --id=8669 shared_folders
2286
2526
  ```
2287
2527
 
2288
- * Find files older than 1 year on a given node and store in file list
2528
+ * list shared folders in workspace
2289
2529
 
2290
2530
  ```
2291
- $ <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2531
+ <%=cmd%> aoc admin res workspace --id=10818 shared_folders
2292
2532
  ```
2293
2533
 
2294
- * Delete the files, one by one
2534
+ * list members of shared folder
2295
2535
 
2296
2536
  ```
2297
- $ cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2537
+ <%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
2298
2538
  ```
2299
2539
 
2300
- * Delete the files in bulk
2540
+ ### Cross Organization transfers
2301
2541
 
2302
- ```
2303
- cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
2304
- ```
2542
+ It is possible to transfer files directly between organizations without having to first download locally and then upload...
2305
2543
 
2306
- ## Activity
2544
+ Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
2307
2545
 
2308
- The activity app can be queried with:
2546
+ Procedure to send a file from org1 to org2:
2547
+
2548
+ * Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2549
+ * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2550
+ * Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
2551
+ * Check that access works and locate the destination folder `mydestfolder`
2552
+ * execute the following:
2309
2553
 
2310
2554
  ```
2311
- $ <%=cmd%> aoc admin analytics transfers
2555
+ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2312
2556
  ```
2313
2557
 
2314
- It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2558
+ Explanation:
2315
2559
 
2316
- `mytemplate.erb`:
2560
+ * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2561
+ * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2562
+ * `--format=json` format the output in JSON (instead of default text table)
2563
+ * `--display=data` display only the result, and remove other information, such as workspace name
2564
+ * `|` the standard output of the first command is fed into the second one
2565
+ * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2566
+ * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2567
+ * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
2568
+ * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2317
2569
 
2318
- ```
2319
- From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2320
- To: <<%='<'%>%=ev['user_email']%>>
2321
- Subject: <%='<'%>%=ev['files_completed']%> files received
2570
+ ### Find Files
2322
2571
 
2323
- Dear <%='<'%>%=ev[:user_email.to_s]%>,
2324
- We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2325
- <%='<'%>%=ev['content']%>
2572
+ The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2326
2573
 
2327
- Thank you.
2328
- ```
2329
- The environment provided contains the following additional variable:
2574
+ The expression can be of 3 formats:
2330
2575
 
2331
- * ev : all details on the transfer event
2576
+ * empty (default) : all files, equivalent to value: `exec:true`
2577
+ * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2332
2578
 
2333
- Example:
2579
+ For instance, to find files with a special extension, use `--value='\.myext$'`
2580
+
2581
+ * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2582
+
2583
+ Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2584
+
2585
+ * Find files more recent than 100 days
2334
2586
 
2335
2587
  ```
2336
- $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2337
- --query=@json:'{"status":"completed","direction":"receive"}' \
2338
- --notif-to=active --notif-template=@file:mytemplate.erb
2588
+ f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2339
2589
  ```
2340
2590
 
2341
- Options:
2591
+ * Find files older than 1 year on a given node and store in file list
2342
2592
 
2343
- * `once_only` keep track of last date it was called, so next call will get only new events
2344
- * `query` filter (on API call)
2345
- * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2593
+ ```
2594
+ <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2595
+ ```
2346
2596
 
2347
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
2597
+ * Delete the files, one by one
2348
2598
 
2349
- ## Using specific transfer ports
2599
+ ```
2600
+ cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2601
+ ```
2350
2602
 
2351
- By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that. The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2603
+ * Delete the files in bulk
2352
2604
 
2605
+ ```
2606
+ cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
2607
+ ```
2353
2608
 
2354
- # Plugin: Aspera Transfer Service
2609
+ # <a id="ats"></a>Plugin: Aspera Transfer Service
2355
2610
 
2356
2611
  ATS is usable either :
2357
2612
 
@@ -2361,10 +2616,13 @@ ATS is usable either :
2361
2616
 
2362
2617
  ## IBM Cloud ATS : creation of api key
2363
2618
 
2619
+ This section is about using ATS with an IBM cloud subscription.
2620
+ If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
2621
+
2364
2622
  First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
2365
2623
 
2366
2624
  ```
2367
- $ ibmcloud iam api-key-create mykeyname -d 'my sample key'
2625
+ ibmcloud iam api-key-create mykeyname -d 'my sample key'
2368
2626
  OK
2369
2627
  API key mykeyname was created
2370
2628
 
@@ -2383,78 +2641,47 @@ References:
2383
2641
  * [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
2384
2642
  * [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
2385
2643
 
2386
-
2387
2644
  Then, to register the key by default for the ats plugin, create a preset. Execute:
2388
2645
 
2389
2646
  ```
2390
- $ <%=cmd%> config id my_ibm_ats update --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
2391
- $ <%=cmd%> config id default set ats my_ibm_ats
2392
- $ <%=cmd%> ats api_key instances
2647
+ <%=cmd%> config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
2648
+ <%=cmd%> config preset set default ats my_ibm_ats
2649
+ <%=cmd%> ats api_key instances
2393
2650
  +--------------------------------------+
2394
2651
  | instance |
2395
2652
  +--------------------------------------+
2396
2653
  | aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
2397
2654
  +--------------------------------------+
2398
- $ <%=cmd%> config id my_ibm_ats update --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
2399
- $ <%=cmd%> ats api_key create
2655
+ <%=cmd%> config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
2656
+ <%=cmd%> ats api_key create
2400
2657
  +--------+----------------------------------------------+
2401
2658
  | key | value |
2402
2659
  +--------+----------------------------------------------+
2403
2660
  | id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
2404
2661
  | secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
2405
2662
  +--------+----------------------------------------------+
2406
- $ <%=cmd%> config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2663
+ <%=cmd%> config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2407
2664
  ```
2408
- ## Cross Organization transfers
2409
-
2410
- It is possible to transfer files directly between organizations without having to first download locally and then upload...
2411
-
2412
- Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
2413
-
2414
- Procedure to send a file from org1 to org2:
2415
-
2416
- * Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2417
- * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2418
- * Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
2419
- * Check that access works and locate the destination folder `mydestfolder`
2420
- * execute the following:
2421
2665
 
2422
- ```
2423
- $ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2424
- ```
2425
-
2426
- Explanation:
2427
-
2428
- * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2429
- * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2430
- * `--format=json` format the output in JSON (instead of default text table)
2431
- * `--display=data` display only the result, and remove other information, such as workspace name
2432
- * `|` the standard output of the first command is fed into the second one
2433
- * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2434
- * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2435
- * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
2436
- * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2437
-
2438
- Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
2439
2666
  ## Examples
2440
2667
 
2441
- Example: create access key on softlayer:
2668
+ Example: create access key on IBM Cloud (softlayer):
2442
2669
 
2443
2670
  ```
2444
- $ <%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
2671
+ <%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
2445
2672
  ```
2446
2673
 
2447
2674
  Example: create access key on AWS:
2448
2675
 
2449
2676
  ```
2450
- $ <%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
2677
+ <%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
2451
2678
 
2452
2679
  ```
2453
2680
 
2454
2681
  Example: create access key on Azure SAS:
2455
2682
 
2456
2683
  ```
2457
- $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
2684
+ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
2458
2685
 
2459
2686
  ```
2460
2687
 
@@ -2463,7 +2690,7 @@ $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{
2463
2690
  Example: create access key on Azure:
2464
2691
 
2465
2692
  ```
2466
- $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
2693
+ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
2467
2694
 
2468
2695
  ```
2469
2696
 
@@ -2473,74 +2700,77 @@ delete all my access keys:
2473
2700
  for k in $(<%=cmd%> ats access_key list --field=id --format=csv);do <%=cmd%> ats access_key id $k delete;done
2474
2701
  ```
2475
2702
 
2703
+ The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
2704
+
2476
2705
  # Plugin: IBM Aspera High Speed Transfer Server (transfer)
2477
2706
 
2478
- This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
2707
+ This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
2708
+ It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
2709
+ Modern mode is to use the node API and transfer tokens.
2479
2710
 
2480
2711
  ## Authentication
2481
2712
 
2482
2713
  Both password and SSH keys auth are supported.
2483
2714
 
2484
- If not username is provided, the default transfer user `xfer` is used.
2715
+ If username is not provided, the default transfer user `xfer` is used.
2485
2716
 
2486
- If no ssh password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used.
2717
+ If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
2487
2718
 
2488
2719
  ```
2489
- $ <%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2720
+ <%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2490
2721
  ```
2491
2722
 
2492
- Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
2723
+ Multiple SSH key paths can be provided.
2724
+ The value of the parameter `ssh_keys` can be a single value or an array.
2725
+ Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
2493
2726
 
2494
2727
  Examples:
2495
2728
 
2496
2729
  ```
2497
- $ <%=cmd%> server --ssh-keys=~/.ssh/id_rsa
2498
- $ <%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
2499
- $ <%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
2730
+ <%=cmd%> server --ssh-keys=~/.ssh/id_rsa
2731
+ <%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
2732
+ <%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
2500
2733
  ```
2501
2734
 
2502
- The underlying ssh library `net::ssh` provides several options that may be used depending on environment. By default the ssh library expect that an ssh-agent is running.
2735
+ The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
2736
+ By default the ssh library expect that an ssh-agent is running.
2503
2737
 
2504
- If you get an error message such as:
2738
+ On Linux, if you get an error message such as:
2505
2739
 
2506
2740
  ```
2507
- [Linux]
2508
2741
  ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
2509
2742
  ```
2510
2743
 
2511
- or
2744
+ or on Windows:
2512
2745
 
2513
2746
  ```
2514
- [Windows]
2515
2747
  ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
2516
2748
  ```
2517
2749
 
2518
- This means that you dont have such an ssh agent running:
2750
+ This means that you don't have such an ssh agent running, then:
2519
2751
 
2520
2752
  * check env var: `SSH_AGENT_SOCK`
2521
- * check if the key is protected with a passphrase
2753
+ * check if the ssh key is protected with a passphrase
2522
2754
  * [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
2523
- * To diable use of `ssh-agent`, use the option `ssh_option` like this (or set in preset):
2755
+ * To disable use of `ssh-agent`, use the option `ssh_option` like this:
2524
2756
 
2525
2757
  ```
2526
- $ <%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
2758
+ <%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
2527
2759
  ```
2528
2760
 
2529
- This can also be set as default using a preset.
2761
+ This can also be set as default using a global preset.
2530
2762
 
2531
2763
  ## Example
2532
2764
 
2533
- One can test the "server" application using the well known demo server:
2765
+ One can test the `server` application using the well known demo server:
2534
2766
 
2535
2767
  ```
2536
- $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
2537
- $ <%=cmd%> config id default set server aspera_demo_server
2538
- $ <%=cmd%> server browse /aspera-test-dir-large
2539
- $ <%=cmd%> server download /aspera-test-dir-large/200MB
2768
+ <%=cmd%> config initdemo
2769
+ <%=cmd%> server browse /aspera-test-dir-large
2770
+ <%=cmd%> server download /aspera-test-dir-large/200MB
2540
2771
  ```
2541
2772
 
2542
- This creates a <%=prst%> "aspera_demo_server" and set it as default for application "server"
2543
-
2773
+ `initdemo` creates a <%=prst%> `demoserver` and set it as default for plugin `server`.
2544
2774
 
2545
2775
  # Plugin: IBM Aspera High Speed Transfer Server (node)
2546
2776
 
@@ -2572,12 +2802,12 @@ For transfers, it is possible to control how transfer is authorized using option
2572
2802
 
2573
2803
  ## Central
2574
2804
 
2575
- The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
2805
+ The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
2576
2806
 
2577
2807
  Filtering can be applied:
2578
2808
 
2579
2809
  ```
2580
- $ <%=cmd%> node central file list
2810
+ <%=cmd%> node central file list
2581
2811
  ```
2582
2812
 
2583
2813
  by providing the `validator` option, offline transfer validation can be done.
@@ -2589,7 +2819,7 @@ It is possible to start a FASPStream session using the node API:
2589
2819
  Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
2590
2820
 
2591
2821
  ```
2592
- $ <%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
2822
+ <%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
2593
2823
  ```
2594
2824
 
2595
2825
  ## Watchfolder
@@ -2602,9 +2832,9 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
2602
2832
  * configure a watchfolder to define automated transfers
2603
2833
 
2604
2834
  ```
2605
- $ <%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2606
- $ <%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
2607
- $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
2835
+ <%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2836
+ <%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
2837
+ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
2608
2838
  ```
2609
2839
 
2610
2840
  ## Out of Transfer File Validation
@@ -2612,13 +2842,13 @@ $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watc
2612
2842
  Follow the Aspera Transfer Server configuration to activate this feature.
2613
2843
 
2614
2844
  ```
2615
- $ <%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
2845
+ <%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
2616
2846
  :..............:..............:............:......................................:
2617
2847
  : session_uuid : file_id : status : path :
2618
2848
  :..............:..............:............:......................................:
2619
2849
  : 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
2620
2850
  :..............:..............:............:......................................:
2621
- $ <%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
2851
+ <%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
2622
2852
  updated
2623
2853
  ```
2624
2854
 
@@ -2634,7 +2864,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
2634
2864
  Then execute the following command:
2635
2865
 
2636
2866
  ```
2637
- $ <%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
2867
+ <%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
2638
2868
  ```
2639
2869
 
2640
2870
  This will get transfer information from the SHOD instance and tell the Azure ATS instance
@@ -2643,7 +2873,7 @@ to download files.
2643
2873
  ## Create access key
2644
2874
 
2645
2875
  ```
2646
- $ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2876
+ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2647
2877
  ```
2648
2878
 
2649
2879
  # Plugin: IBM Aspera Faspex5
@@ -2668,7 +2898,7 @@ For boot method: (will be removed in future)
2668
2898
  Use it as password and use `--auth=boot`.
2669
2899
 
2670
2900
  ```
2671
- $ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2901
+ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2672
2902
  ```
2673
2903
 
2674
2904
  Ready to use Faspex5 with CLI.
@@ -2714,7 +2944,7 @@ If no parameter `max` or `pmax` is provided, then all packages will be listed in
2714
2944
  ### Example
2715
2945
 
2716
2946
  ```
2717
- $ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2947
+ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2718
2948
  ```
2719
2949
 
2720
2950
  List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
@@ -2728,14 +2958,14 @@ The command is `package recv`, possible methods are:
2728
2958
  * provide a `faspe:` URI with option `link`
2729
2959
 
2730
2960
  ```
2731
- $ <%=cmd%> faspex package recv --id=12345
2732
- $ <%=cmd%> faspex package recv --link=faspe://...
2961
+ <%=cmd%> faspex package recv --id=12345
2962
+ <%=cmd%> faspex package recv --link=faspe://...
2733
2963
  ```
2734
2964
 
2735
2965
  If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
2736
2966
 
2737
2967
  ```
2738
- $ <%=cmd%> faspex package list --recipient='*thedropboxname'
2968
+ <%=cmd%> faspex package list --recipient='*thedropboxname'
2739
2969
  ```
2740
2970
 
2741
2971
  if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
@@ -2747,7 +2977,7 @@ The command is `faspex package send`. Package information (title, note, metadata
2747
2977
  Example:
2748
2978
 
2749
2979
  ```
2750
- $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2980
+ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2751
2981
  ```
2752
2982
 
2753
2983
  If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
@@ -2764,7 +2994,7 @@ Like for any transfer, a notification can be sent by email using parameters: `no
2764
2994
  Example:
2765
2995
 
2766
2996
  ```
2767
- $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
2997
+ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
2768
2998
  ```
2769
2999
 
2770
3000
  In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
@@ -2774,9 +3004,9 @@ In this example the notification template is directly provided on command line.
2774
3004
  Example:
2775
3005
 
2776
3006
  ```
2777
- $ <%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
2778
- $ <%=cmd%> faspex v4 dropbox list
2779
- $ <%=cmd%> faspex v4 dropbox delete --id=36
3007
+ <%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
3008
+ <%=cmd%> faspex v4 dropbox list
3009
+ <%=cmd%> faspex v4 dropbox delete --id=36
2780
3010
  ```
2781
3011
 
2782
3012
  ## Remote sources
@@ -2814,7 +3044,7 @@ It is possible to tell <%=tool%> to download newly received packages, much like
2814
3044
  cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
2815
3045
 
2816
3046
  ```
2817
- $ <%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
3047
+ <%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
2818
3048
  ```
2819
3049
 
2820
3050
  # Plugin: IBM Aspera Shares
@@ -2824,7 +3054,7 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
2824
3054
  In Shares2, users, groups listing are paged, to display sequential pages:
2825
3055
 
2826
3056
  ```
2827
- $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
3057
+ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
2828
3058
  ```
2829
3059
 
2830
3060
  # Plugin: IBM Cloud Object Storage
@@ -2833,9 +3063,9 @@ The IBM Cloud Object Storage provides the possibility to execute transfers using
2833
3063
  It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
2834
3064
  Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
2835
3065
 
2836
- There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the forst method. If you dont have credentials but have access to the IBM Cloud console, then use the second method.
3066
+ There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
2837
3067
 
2838
- ## Using endpoint, apikey and Ressource Instance ID (CRN)
3068
+ ## Using endpoint, apikey and Resource Instance ID (CRN)
2839
3069
 
2840
3070
  If you have those parameters already, then following options shall be provided:
2841
3071
 
@@ -2847,15 +3077,15 @@ If you have those parameters already, then following options shall be provided:
2847
3077
  For example, let us create a default configuration:
2848
3078
 
2849
3079
  ```
2850
- $ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2851
- $ <%=cmd%> conf id default set cos mycos
3080
+ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
3081
+ <%=cmd%> conf id default set cos mycos
2852
3082
  ```
2853
3083
 
2854
3084
  Then, jump to the transfer example.
2855
3085
 
2856
3086
  ## Using service credential file
2857
3087
 
2858
- If you are the COS administrator and dont have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
3088
+ If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
2859
3089
 
2860
3090
  Navigation Menu &rarr; Resource List &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
2861
3091
 
@@ -2864,11 +3094,11 @@ Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
2864
3094
  or using the IBM Cloud CLI:
2865
3095
 
2866
3096
  ```
2867
- $ ibmcloud resource service-keys
2868
- $ ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
3097
+ ibmcloud resource service-keys
3098
+ ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
2869
3099
  ```
2870
3100
 
2871
- (if you dont have `jq` installed, extract the structure as follows)
3101
+ (if you don't have `jq` installed, extract the structure as follows)
2872
3102
 
2873
3103
  It consists in the following structure:
2874
3104
 
@@ -2903,8 +3133,8 @@ The required options for this method are:
2903
3133
  For example, let us create a default configuration:
2904
3134
 
2905
3135
  ```
2906
- $ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
2907
- $ <%=cmd%> conf id default set cos mycos
3136
+ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
3137
+ <%=cmd%> conf id default set cos mycos
2908
3138
  ```
2909
3139
 
2910
3140
  ## Operations, transfers
@@ -2914,8 +3144,8 @@ Let's assume you created a default configuration from once of the two previous s
2914
3144
  A subset of `node` plugin operations are supported, basically node API:
2915
3145
 
2916
3146
  ```
2917
- $ <%=cmd%> cos node info
2918
- $ <%=cmd%> cos node upload 'faux:///sample1G?1g'
3147
+ <%=cmd%> cos node info
3148
+ <%=cmd%> cos node upload 'faux:///sample1G?1g'
2919
3149
  ```
2920
3150
 
2921
3151
  Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
@@ -2944,8 +3174,10 @@ Specify the previews folder as shown in:
2944
3174
  By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
2945
3175
 
2946
3176
  ```
2947
- # /opt/aspera/bin/asconfigurator -x "server;preview_dir,previews"
2948
- # /opt/aspera/bin/asnodeadmin --reload
3177
+ PATH=/opt/aspera/bin:$PATH
3178
+
3179
+ asconfigurator -x "server;preview_dir,previews"
3180
+ asnodeadmin --reload
2949
3181
  ```
2950
3182
 
2951
3183
  Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
@@ -2958,16 +3190,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
2958
3190
  To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
2959
3191
 
2960
3192
  ```
2961
- # /opt/aspera/bin/asuserdata -a | grep max_request_file_create_size_kb
3193
+ asuserdata -a | grep max_request_file_create_size_kb
3194
+
2962
3195
  max_request_file_create_size_kb: "1024"
2963
- # /opt/aspera/bin/asconfigurator -x "server; max_request_file_create_size_kb,16384"
3196
+
3197
+ asconfigurator -x "server; max_request_file_create_size_kb,16384"
2964
3198
  ```
2965
3199
 
2966
3200
  If you use a value different than 16777216, then specify it using option `max_size`.
2967
3201
 
2968
3202
  Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
2969
3203
 
2970
- ## <a name="prev_ext"></a>External tools: Linux
3204
+ ## <a id="prev_ext"></a>External tools: Linux
2971
3205
 
2972
3206
  The tool requires the following external tools available in the `PATH`:
2973
3207
 
@@ -2984,7 +3218,7 @@ Other OSes should work as well, but are note tested.
2984
3218
  To check if all tools are found properly, execute:
2985
3219
 
2986
3220
  ```
2987
- $ <%=cmd%> preview check
3221
+ <%=cmd%> preview check
2988
3222
  ```
2989
3223
 
2990
3224
  ### mimemagic
@@ -2992,22 +3226,22 @@ $ <%=cmd%> preview check
2992
3226
  To benefit from extra mime type detection install gem mimemagic:
2993
3227
 
2994
3228
  ```
2995
- # gem install mimemagic
3229
+ gem install mimemagic
2996
3230
  ```
2997
3231
 
2998
3232
  or to install an earlier version if any problem:
2999
3233
 
3000
3234
  ```
3001
- # gem install mimemagic -v '~> 0.3.0'
3235
+ gem install mimemagic -v '~> 0.3.0'
3002
3236
  ```
3003
3237
 
3004
3238
  To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
3005
3239
 
3006
3240
  If not used, Mime type used for conversion is the one provided by the node API.
3007
3241
 
3008
- If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
3242
+ If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
3009
3243
 
3010
- ### Image: Imagemagick and optipng
3244
+ ### Image: ImageMagick and optipng
3011
3245
 
3012
3246
  ```
3013
3247
  yum install -y ImageMagick optipng
@@ -3023,25 +3257,25 @@ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.ta
3023
3257
 
3024
3258
  ### Office: Unoconv and Libreoffice
3025
3259
 
3026
- If you dont want to have preview for office dpcuments or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
3260
+ If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
3027
3261
 
3028
3262
  The generation of preview in based on the use of `unoconv` and `libreoffice`
3029
3263
 
3030
- * Centos 8
3264
+ * CentOS 8
3031
3265
 
3032
3266
  ```
3033
- # dnf install unoconv
3267
+ dnf install unoconv
3034
3268
  ```
3035
3269
 
3036
3270
  * Amazon Linux
3037
3271
 
3038
3272
  ```
3039
- # amazon-linux-extras enable libreoffice
3040
- # yum clean metadata
3041
- # yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
3042
- # wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
3043
- # mv unoconv /usr/bin
3044
- # chmod a+x /usr/bin/unoconv
3273
+ amazon-linux-extras enable libreoffice
3274
+ yum clean metadata
3275
+ yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
3276
+ wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
3277
+ mv unoconv /usr/bin
3278
+ chmod a+x /usr/bin/unoconv
3045
3279
  ```
3046
3280
 
3047
3281
  ## Configuration
@@ -3053,9 +3287,11 @@ Like any <%=tool%> commands, parameters can be passed on command line or using a
3053
3287
  Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
3054
3288
 
3055
3289
  ```
3056
- # su -s /bin/bash - xfer
3057
- $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
3058
- $ <%=cmd%> config id default set preview previewconf
3290
+ su -s /bin/bash - xfer
3291
+
3292
+ <%=cmd%> config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
3293
+
3294
+ <%=cmd%> config preset set default preview previewconf
3059
3295
  ```
3060
3296
 
3061
3297
  Here we assume that Office file generation is disabled, else remove this option.
@@ -3064,7 +3300,7 @@ Here we assume that Office file generation is disabled, else remove this option.
3064
3300
  One can check if the access key is well configured using:
3065
3301
 
3066
3302
  ```
3067
- $ <%=cmd%> -Ppreviewconf node browse /
3303
+ <%=cmd%> -Ppreviewconf node browse /
3068
3304
  ```
3069
3305
 
3070
3306
  This shall list the contents of the storage root of the access key.
@@ -3080,8 +3316,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
3080
3316
  Lets do a one shot test, using the configuration previously created:
3081
3317
 
3082
3318
  ```
3083
- # su -s /bin/bash - xfer
3084
- xfer$ <%=cmd%> preview scan --overwrite=always
3319
+ su -s /bin/bash - xfer
3320
+
3321
+ <%=cmd%> preview scan --overwrite=always
3085
3322
  ```
3086
3323
 
3087
3324
  When the preview generator is first executed it will create a file: `.aspera_access_key`
@@ -3095,13 +3332,13 @@ Adapt the scripts to your own preference.
3095
3332
 
3096
3333
  We assume here that a configuration preset was created as shown previously.
3097
3334
 
3098
- Lets first setup a script that will be used in the sceduler and sets up the environment.
3335
+ Lets first setup a script that will be used in the scheduler and sets up the environment.
3099
3336
 
3100
3337
  Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
3101
3338
 
3102
3339
  ```
3103
- #!/bin/bash
3104
- # set a timeout protection, just in case
3340
+ #!/bin/bash
3341
+ # set a timeout protection, just in case
3105
3342
  case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
3106
3343
  . /etc/profile.d/rvm.sh
3107
3344
  rvm use 2.6 --quiet
@@ -3111,13 +3348,13 @@ exec timeout ${tmout} <%=cmd%> "${@}"
3111
3348
  Here the cronjob is created for user `xfer`.
3112
3349
 
3113
3350
  ```
3114
- xfer$ crontab<<EOF
3351
+ crontab<<EOF
3115
3352
  0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
3116
3353
  2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
3117
3354
  EOF
3118
3355
  ```
3119
3356
 
3120
- Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
3357
+ Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
3121
3358
 
3122
3359
  ## Candidate detection for creation or update (or deletion)
3123
3360
 
@@ -3142,12 +3379,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
3142
3379
  If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
3143
3380
 
3144
3381
  ```
3145
- $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
3382
+ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
3146
3383
  ```
3147
3384
 
3148
3385
  The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
3149
3386
 
3150
- When scanning the option `value` has the same behaviour as for the `node find` command.
3387
+ When scanning the option `value` has the same behavior as for the `node find` command.
3151
3388
 
3152
3389
  For instance to filter out files beginning with `._` do:
3153
3390
 
@@ -3166,7 +3403,7 @@ Use option `skip_format` to skip generation of a format.
3166
3403
 
3167
3404
  ## Supported input Files types
3168
3405
 
3169
- The preview generator supports redering of those file categories:
3406
+ The preview generator supports rendering of those file categories:
3170
3407
 
3171
3408
  * image
3172
3409
  * pdf
@@ -3186,7 +3423,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
3186
3423
 
3187
3424
  ## Access to original files and preview creation
3188
3425
 
3189
- Standard open source tools are used to create thumnails and video previews.
3426
+ Standard open source tools are used to create thumbnails and video previews.
3190
3427
  Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
3191
3428
  The tool provides 2 ways to read and write files with the option: `file_access`
3192
3429
 
@@ -3216,28 +3453,28 @@ The `smtp` option is a hash table (extended value) with the following fields:
3216
3453
  ## Example of configuration:
3217
3454
 
3218
3455
  ```
3219
- $ <%=cmd%> config id smtp_google set server smtp.google.com
3220
- $ <%=cmd%> config id smtp_google set username john@gmail.com
3221
- $ <%=cmd%> config id smtp_google set password P@ssw0rd
3456
+ <%=cmd%> config preset set smtp_google server smtp.google.com
3457
+ <%=cmd%> config preset set smtp_google username john@gmail.com
3458
+ <%=cmd%> config preset set smtp_google password P@ssw0rd
3222
3459
  ```
3223
3460
 
3224
3461
  or
3225
3462
 
3226
3463
  ```
3227
- $ <%=cmd%> config id smtp_google init @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
3464
+ <%=cmd%> config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
3228
3465
  ```
3229
3466
 
3230
3467
  or
3231
3468
 
3232
3469
  ```
3233
- $ <%=cmd%> config id smtp_google update --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
3470
+ <%=cmd%> config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
3234
3471
  ```
3235
3472
 
3236
- Set this configation as global default, for instance:
3473
+ Set this configuration as global default, for instance:
3237
3474
 
3238
3475
  ```
3239
- $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
3240
- $ <%=cmd%> config id default set config cli_default
3476
+ <%=cmd%> config preset set cli_default smtp @val:@preset:smtp_google
3477
+ <%=cmd%> config preset set default config cli_default
3241
3478
  ```
3242
3479
 
3243
3480
  ## Email templates
@@ -3259,8 +3496,8 @@ Other variables are defined depending on context.
3259
3496
  Check settings with `smtp_settings` command. Send test email with `email_test`.
3260
3497
 
3261
3498
  ```
3262
- $ <%=cmd%> config --smtp=@preset:smtp_google smtp
3263
- $ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3499
+ <%=cmd%> config --smtp=@preset:smtp_google smtp
3500
+ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3264
3501
  ```
3265
3502
 
3266
3503
  ## Notifications for transfer status
@@ -3269,7 +3506,7 @@ An e-mail notification can be sent upon transfer success and failure (one email
3269
3506
 
3270
3507
  To activate, use option `notif_to`.
3271
3508
 
3272
- A default e-mail template is used, but it can be overriden with option `notif_template`.
3509
+ A default e-mail template is used, but it can be overridden with option `notif_template`.
3273
3510
 
3274
3511
  The environment provided contains the following additional variables:
3275
3512
 
@@ -3293,7 +3530,7 @@ Transfer is: <%='<'%>%=global_transfer_status%>
3293
3530
  This gem comes with a second executable tool providing a simplified standardized interface
3294
3531
  to start a FASP session: `asession`.
3295
3532
 
3296
- It aims at simplifying the startup of a FASP session from a programmatic stand point as formating a [_transfer-spec_](#transferspec) is:
3533
+ It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
3297
3534
 
3298
3535
  * common to Aspera Node API (HTTP POST /ops/transfer)
3299
3536
  * common to Aspera Connect API (browser javascript startTransfer)
@@ -3305,7 +3542,7 @@ This makes it easy to integrate with any language provided that one can spawn a
3305
3542
 
3306
3543
  The tool expect one single argument: a [_transfer-spec_](#transferspec).
3307
3544
 
3308
- If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formated [_transfer-spec_](#transferspec) on stdin.
3545
+ If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
3309
3546
 
3310
3547
  Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
3311
3548
 
@@ -3316,7 +3553,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
3316
3553
  * `EX_loglevel` to change log level of the tool
3317
3554
  * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
3318
3555
 
3319
- Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3556
+ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3320
3557
 
3321
3558
  ## Comparison of interfaces
3322
3559
 
@@ -3343,7 +3580,7 @@ echo "${MY_TSPEC}"|asession
3343
3580
  This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
3344
3581
 
3345
3582
  ```
3346
- $ asession
3583
+ asession
3347
3584
  {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
3348
3585
  {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
3349
3586
  {"type":"DONE"}
@@ -3358,7 +3595,7 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
3358
3595
  ## Help
3359
3596
 
3360
3597
  ```
3361
- $ asession -h
3598
+ asession -h
3362
3599
  <%=File.read(ENV["INCL_ASESSION"])%>
3363
3600
  ```
3364
3601
 
@@ -3397,7 +3634,7 @@ Note that:
3397
3634
 
3398
3635
  * <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
3399
3636
  * most, but not all native ascp arguments are available as standard transfer_spec parameters
3400
- * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#direct) transfer agent (not connect or node)
3637
+ * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
3401
3638
 
3402
3639
  ### server side and configuration
3403
3640
 
@@ -3412,18 +3649,18 @@ Once <%=tool%> parameters are defined, run the command using the OS native sched
3412
3649
  ## Example
3413
3650
 
3414
3651
  ```
3415
- $ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
3652
+ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
3416
3653
 
3417
3654
  ```
3418
3655
 
3419
- The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3656
+ The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3420
3657
 
3421
- # Aspera Health check and Nagios
3658
+ # Health check and Nagios
3422
3659
 
3423
- Each plugin provide a `health` command that will check the health status of the application. Example:
3660
+ Most plugin provide a `health` command that will check the health status of the application. Example:
3424
3661
 
3425
3662
  ```
3426
- $ <%=cmd%> console health
3663
+ <%=cmd%> console health
3427
3664
  +--------+-------------+------------+
3428
3665
  | status | component | message |
3429
3666
  +--------+-------------+------------+
@@ -3440,13 +3677,13 @@ Typically, the health check uses the REST API of the application with the follow
3440
3677
  <%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
3441
3678
 
3442
3679
  ```
3443
- $ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3680
+ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3444
3681
  OK - [transfer:ok]
3445
- $ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3682
+ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3446
3683
  OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
3447
3684
  ```
3448
3685
 
3449
- # Module: `Aspera`
3686
+ # Ruby Module: `Aspera`
3450
3687
 
3451
3688
  Main components:
3452
3689
 
@@ -3457,8 +3694,8 @@ Main components:
3457
3694
  A working example can be found in the gem, example:
3458
3695
 
3459
3696
  ```
3460
- $ <%=cmd%> config gem_path
3461
- $ cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
3697
+ <%=cmd%> config gem_path
3698
+ cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
3462
3699
  ```
3463
3700
 
3464
3701
  This sample code shows some example of use of the API as well as
@@ -3495,130 +3732,141 @@ So, it evolved into <%=tool%>:
3495
3732
 
3496
3733
  * <%=gemspec.version.to_s%>
3497
3734
 
3498
- * new: `aoc packages list` add possibility to add filter with option `query`
3499
- * new: `aoc admin res xxx list` now get all items by default #50
3500
- * new: `preset` option can specify name or hash value
3501
- * new: `node` plugin accepts bearer token and access key as credential
3502
- * new: `node` option `token_type` allows using basic token in addition to aspera type.
3503
- * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
3504
- * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
3735
+ * new: support transfer agent: [Transfer SDK](#agt_trsdk)
3736
+ * new: support [http socket options](#http_options)
3737
+ * new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
3738
+ * new: `config vault` supports encrypted passwords, also macos keychain
3739
+ * new: `config preset` command for consistency with id
3740
+ * new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
3741
+ * change: when using wss, use [ruby's CA certs](#certificates)
3742
+ * change: unexpected parameter makes exit code not zero
3743
+ * change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
3744
+ * change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
3745
+ * change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
3746
+ * fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
3747
+
3748
+ * 4.4.0
3749
+
3750
+ * new: `aoc packages list` add possibility to add filter with option `query`
3751
+ * new: `aoc admin res xxx list` now get all items by default #50
3752
+ * new: `preset` option can specify name or hash value
3753
+ * new: `node` plugin accepts bearer token and access key as credential
3754
+ * new: `node` option `token_type` allows using basic token in addition to aspera type.
3755
+ * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
3756
+ * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
3505
3757
 
3506
3758
  * 4.3.0
3507
3759
 
3508
- * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#direct) transfer agent.
3509
- * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3760
+ * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
3761
+ * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3510
3762
 
3511
3763
  * 4.2.2
3512
3764
 
3513
- * new: `faspex package list` retrieves the whole list, not just first page
3514
- * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3515
- * new: the error "Remote host is not who we expected" displays a special remediation message
3516
- * new: `conf ascp spec` displays supported transfer spec
3517
- * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3518
- * fix: space character in `faspe:` url are precent encoded if needed
3519
- * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3520
- * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3521
- * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3765
+ * new: `faspex package list` retrieves the whole list, not just first page
3766
+ * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3767
+ * new: the error "Remote host is not who we expected" displays a special remediation message
3768
+ * new: `conf ascp spec` displays supported transfer spec
3769
+ * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3770
+ * fix: space character in `faspe:` url are precent encoded if needed
3771
+ * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3772
+ * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3773
+ * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3522
3774
 
3523
3775
  * 4.2.1
3524
3776
 
3525
- * new: command `faspex package recv` supports link of type: `faspe:`
3526
- * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3777
+ * new: command `faspex package recv` supports link of type: `faspe:`
3778
+ * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3527
3779
 
3528
3780
  * 4.2.0
3529
3781
 
3530
- * new: command `aoc remind` to receive organization membership by email
3531
- * new: in `preview` option `value` to filter out on file name
3532
- * new: `initdemo` to initialize for demo server
3533
- * new: [`direct`](#direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3534
- * fix: on Windows `conf ascp use` expects ascp.exe
3535
- * fix: (break) multi_session_threshold is Integer, not String
3536
- * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3537
- * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3538
- * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3539
- * change: (break) parameters for resume in `transfer-info` for [`direct`](#direct) are now in sub-key `"resume"`
3782
+ * new: command `aoc remind` to receive organization membership by email
3783
+ * new: in `preview` option `value` to filter out on file name
3784
+ * new: `initdemo` to initialize for demo server
3785
+ * new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3786
+ * fix: on Windows `conf ascp use` expects ascp.exe
3787
+ * fix: (break) multi_session_threshold is Integer, not String
3788
+ * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3789
+ * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3790
+ * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3791
+ * change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
3540
3792
 
3541
3793
  * 4.1.0
3542
3794
 
3543
- * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3544
- * new: update documentation with regard to offline and docker installation
3545
- * new: renamed command `nagios_check` to `health`
3546
- * new: agent `http_gw` now supports upload
3547
- * new: added option `sdk_url` to install SDK from local file for offline install
3548
- * new: check new gem version periodically
3549
- * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3550
- * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3551
- * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3552
- * new: AoC a password can be provided for a public link
3553
- * new: `conf doc` take an optional parameter to go to a section
3554
- * new: initial support for Faspex 5 Beta 1
3795
+ * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3796
+ * new: update documentation with regard to offline and docker installation
3797
+ * new: renamed command `nagios_check` to `health`
3798
+ * new: agent `http_gw` now supports upload
3799
+ * new: added option `sdk_url` to install SDK from local file for offline install
3800
+ * new: check new gem version periodically
3801
+ * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3802
+ * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3803
+ * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3804
+ * new: AoC a password can be provided for a public link
3805
+ * new: `conf doc` take an optional parameter to go to a section
3806
+ * new: initial support for Faspex 5 Beta 1
3555
3807
 
3556
3808
  * 4.0.0
3557
3809
 
3558
- * now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
3559
- * changed default tool name from `mlia` to `ascli`
3560
- * changed `aspera` command to `aoc`
3561
- * changed gem name from `asperalm` to `aspera-cli`
3562
- * changed module name from `Asperalm` to `Aspera`
3563
- * removed command `folder` in `preview`, merged to `scan`
3564
- * persistency files go to sub folder instead of main folder
3565
- * added possibility to install SDK: `config ascp install`
3810
+ * now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
3811
+ * changed default tool name from `mlia` to `ascli`
3812
+ * changed `aspera` command to `aoc`
3813
+ * changed gem name from `asperalm` to `aspera-cli`
3814
+ * changed module name from `Asperalm` to `Aspera`
3815
+ * removed command `folder` in `preview`, merged to `scan`
3816
+ * persistency files go to sub folder instead of main folder
3817
+ * added possibility to install SDK: `config ascp install`
3566
3818
 
3567
3819
  * 0.11.8
3568
3820
 
3569
- * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
3821
+ * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
3570
3822
 
3571
3823
  * 0.11.7
3572
3824
 
3573
- * rework on rest call error handling
3574
- * use option `display` with value `data` to remove out of extraneous information
3575
- * fixed option `lock_port` not working
3576
- * generate special icon if preview failed
3577
- * possibility to choose transfer progress bar type with option `progress`
3578
- * AoC package creation now output package id
3825
+ * rework on rest call error handling
3826
+ * use option `display` with value `data` to remove out of extraneous information
3827
+ * fixed option `lock_port` not working
3828
+ * generate special icon if preview failed
3829
+ * possibility to choose transfer progress bar type with option `progress`
3830
+ * AoC package creation now output package id
3579
3831
 
3580
3832
  * 0.11.6
3581
3833
 
3582
- * orchestrator : added more choice in auth type
3583
- * preview: cleanup in generator (removed and renamed parameters)
3584
- * preview: better documentation
3585
- * preview: animated thumbnails for video (option: `video_png_conv=animated`)
3586
- * preview: new event trigger: `trevents` (`events` seems broken)
3587
- * preview: unique tmp folder to avoid clash of multiple instances
3588
- * repo: added template for secrets used for testing
3834
+ * orchestrator : added more choice in auth type
3835
+ * preview: cleanup in generator (removed and renamed parameters)
3836
+ * preview: better documentation
3837
+ * preview: animated thumbnails for video (option: `video_png_conv=animated`)
3838
+ * preview: new event trigger: `trevents` (`events` seems broken)
3839
+ * preview: unique tmp folder to avoid clash of multiple instances
3840
+ * repo: added template for secrets used for testing
3589
3841
 
3590
3842
  * 0.11.5
3591
3843
 
3592
- * added option `default_ports` for AoC (see manual)
3593
- * allow bulk delete in `aspera files` with option `bulk=yes`
3594
- * fix getting connect versions
3595
- * added section for Aix
3596
- * support all ciphers for [`direct`](#direct) agent (including gcm, etc..)
3597
- * added transfer spec param `apply_local_docroot` for [`direct`](#direct)
3844
+ * added option `default_ports` for AoC (see manual)
3845
+ * allow bulk delete in `aspera files` with option `bulk=yes`
3846
+ * fix getting connect versions
3847
+ * added section for Aix
3848
+ * support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
3849
+ * added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
3598
3850
 
3599
3851
  * 0.11.4
3600
3852
 
3601
- * possibility to give shared inbox name when sending a package (else use id and type)
3853
+ * possibility to give shared inbox name when sending a package (else use id and type)
3602
3854
 
3603
3855
  * 0.11.3
3604
3856
 
3605
- * minor fixes on multi-session: avoid exception on progress bar
3857
+ * minor fixes on multi-session: avoid exception on progress bar
3606
3858
 
3607
3859
  * 0.11.2
3608
3860
 
3609
- * fixes on multi-session: progress bat and transfer spec param for "direct"
3861
+ * fixes on multi-session: progress bat and transfer spec param for "direct"
3610
3862
 
3611
3863
  * 0.11.1
3612
3864
 
3613
- * enhanced short_link creation commands (see examples)
3865
+ * enhanced short_link creation commands (see examples)
3614
3866
 
3615
3867
  * 0.11
3616
3868
 
3617
- * add option to provide file list directly to ascp like this (only for direct transfer agent):
3618
-
3619
- ```
3620
- ... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
3621
- ```
3869
+ * add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
3622
3870
 
3623
3871
  * 0.10.18
3624
3872
 
@@ -3627,7 +3875,7 @@ So, it evolved into <%=tool%>:
3627
3875
  * 0.10.17
3628
3876
 
3629
3877
  * fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
3630
- * new modifier: `@list:<saparator>val1<separator>...`
3878
+ * new modifier: `@list:<separator>val1<separator>...`
3631
3879
 
3632
3880
  * 0.10.16
3633
3881
 
@@ -3649,7 +3897,7 @@ So, it evolved into <%=tool%>:
3649
3897
  * 0.10.12
3650
3898
 
3651
3899
  * added support for AoC node registration keys
3652
- * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#direct)
3900
+ * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
3653
3901
  * Transfer agent is no more a Singleton instance, but only one is used in CLI
3654
3902
  * `@incps` : new extended value modifier
3655
3903
  * ATS: no more provides access keys secrets: now user must provide it
@@ -3681,7 +3929,7 @@ So, it evolved into <%=tool%>:
3681
3929
 
3682
3930
  * 0.10.6
3683
3931
 
3684
- * FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
3932
+ * FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
3685
3933
  * gem version requirements made more open
3686
3934
 
3687
3935
  * 0.10.5
@@ -3831,10 +4079,10 @@ So, it evolved into <%=tool%>:
3831
4079
 
3832
4080
  * 0.9.7
3833
4081
 
3834
- * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#direct) transfer agents
4082
+ * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
3835
4083
  * preview persistency goes to unique file by default
3836
4084
  * catch mxf extension in preview as video
3837
- * Faspex: possibility to download all paclages by specifying id=ALL
4085
+ * Faspex: possibility to download all packages by specifying id=ALL
3838
4086
  * Faspex: to come: cargo-like function to download only new packages with id=NEW
3839
4087
 
3840
4088
  * 0.9.6
@@ -3914,16 +4162,16 @@ For issues or feature requests use the Github repository and issues.
3914
4162
 
3915
4163
  You can also contribute to this open source project.
3916
4164
 
3917
- One can also create one's own command nplugin.
4165
+ One can also [create one's own plugin](#createownplugin).
3918
4166
 
3919
4167
  ## Only one value for any option
3920
4168
 
3921
4169
  Some commands and sub commands may ask for the same option name.
3922
- Currently, since option definition is position independant (last one wins), it is not possible
4170
+ Currently, since option definition is position independent (last one wins), it is not possible
3923
4171
  to give an option to a command and the same option with different value to a sub command.
3924
4172
 
3925
- For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
3926
- As a workaround use another option, if available, to identify the entity.
4173
+ For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
4174
+ As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
3927
4175
 
3928
4176
  This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
3929
4177
 
@@ -3955,7 +4203,7 @@ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or
3955
4203
 
3956
4204
  References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
3957
4205
 
3958
- ## Miscelaneous
4206
+ ## Miscellaneous
3959
4207
 
3960
4208
  * remove rest and oauth classes and use ruby standard gems:
3961
4209