aspera-cli 4.2.1 → 4.5.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (59) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +1580 -946
  3. data/bin/ascli +1 -1
  4. data/bin/asession +3 -5
  5. data/docs/Makefile +8 -11
  6. data/docs/README.erb.md +1521 -829
  7. data/docs/doc_tools.rb +58 -0
  8. data/docs/test_env.conf +3 -1
  9. data/examples/faspex4.rb +28 -19
  10. data/examples/transfer.rb +2 -2
  11. data/lib/aspera/aoc.rb +157 -134
  12. data/lib/aspera/cli/listener/progress_multi.rb +5 -5
  13. data/lib/aspera/cli/main.rb +106 -48
  14. data/lib/aspera/cli/manager.rb +19 -20
  15. data/lib/aspera/cli/plugin.rb +22 -7
  16. data/lib/aspera/cli/plugins/aoc.rb +260 -208
  17. data/lib/aspera/cli/plugins/ats.rb +11 -10
  18. data/lib/aspera/cli/plugins/bss.rb +2 -2
  19. data/lib/aspera/cli/plugins/config.rb +360 -189
  20. data/lib/aspera/cli/plugins/faspex.rb +119 -56
  21. data/lib/aspera/cli/plugins/faspex5.rb +32 -17
  22. data/lib/aspera/cli/plugins/node.rb +72 -31
  23. data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
  24. data/lib/aspera/cli/plugins/preview.rb +94 -68
  25. data/lib/aspera/cli/plugins/server.rb +16 -5
  26. data/lib/aspera/cli/plugins/shares.rb +17 -0
  27. data/lib/aspera/cli/transfer_agent.rb +64 -82
  28. data/lib/aspera/cli/version.rb +1 -1
  29. data/lib/aspera/command_line_builder.rb +48 -31
  30. data/lib/aspera/cos_node.rb +4 -3
  31. data/lib/aspera/environment.rb +4 -4
  32. data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
  33. data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
  34. data/lib/aspera/fasp/{local.rb → agent_direct.rb} +42 -38
  35. data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +50 -29
  36. data/lib/aspera/fasp/{node.rb → agent_node.rb} +43 -4
  37. data/lib/aspera/fasp/agent_trsdk.rb +106 -0
  38. data/lib/aspera/fasp/default.rb +17 -0
  39. data/lib/aspera/fasp/installation.rb +64 -48
  40. data/lib/aspera/fasp/parameters.rb +78 -91
  41. data/lib/aspera/fasp/parameters.yaml +531 -0
  42. data/lib/aspera/fasp/uri.rb +1 -1
  43. data/lib/aspera/faspex_gw.rb +12 -11
  44. data/lib/aspera/id_generator.rb +22 -0
  45. data/lib/aspera/keychain/encrypted_hash.rb +120 -0
  46. data/lib/aspera/keychain/macos_security.rb +94 -0
  47. data/lib/aspera/log.rb +45 -32
  48. data/lib/aspera/node.rb +9 -4
  49. data/lib/aspera/oauth.rb +116 -100
  50. data/lib/aspera/persistency_action_once.rb +11 -7
  51. data/lib/aspera/persistency_folder.rb +6 -26
  52. data/lib/aspera/rest.rb +66 -50
  53. data/lib/aspera/sync.rb +40 -35
  54. data/lib/aspera/timer_limiter.rb +22 -0
  55. metadata +86 -29
  56. data/docs/transfer_spec.html +0 -99
  57. data/lib/aspera/api_detector.rb +0 -60
  58. data/lib/aspera/fasp/aoc.rb +0 -24
  59. data/lib/aspera/secrets.rb +0 -20
data/docs/README.erb.md CHANGED
@@ -1,37 +1,24 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
- <%
3
- # check that required env vars exist, and files
4
- %w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_TRSPEC}.each do |e|
5
- raise "missing env var #{e}" unless ENV.has_key?(e)
6
- raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
7
- end
8
- cmd=ENV["EXENAME"] # just command name
9
- tool='`'+cmd+'`' # used in text with formatting of command
10
- evp=cmd.upcase+'_' # prefix for env vars
11
- opprst='option preset' # just the name for "option preset"
12
- prst='['+opprst+'](#lprt)'
13
- prsts='['+opprst+'s](#lprt)'
14
- prstt=opprst.capitalize # in title
15
- gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
16
- geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
17
- -%>
18
- # <%=tool%> : Command Line Interface for IBM Aspera products
19
-
20
- Version : <%= gemspec.version.to_s %>
2
+ <%load File.join(File.dirname(__FILE__),'doc_tools.rb')-%>
3
+ <font size="+12"><center><%=tool%> : Command Line Interface for IBM Aspera products</center></font>
4
+
5
+ Version : <%=gemspec.version.to_s%>
21
6
 
22
7
  _Laurent/2016-<%=Time.new.year%>_
23
8
 
24
- This gem provides <%=tool%>: a command line interface to Aspera Applications.
9
+ This gem provides the <%=tool%> Command Line Interface to IBM Aspera software.
25
10
 
26
11
  <%=tool%> is a also great tool to learn Aspera APIs.
27
12
 
28
- Ruby Gem: [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubygems_uri'] %>)
13
+ Ruby Gem: [<%=gemspec.metadata['rubygems_uri']%>](<%=gemspec.metadata['rubygems_uri']%>)
14
+
15
+ Ruby Doc: [<%=gemspec.metadata['documentation_uri']%>](<%=gemspec.metadata['documentation_uri']%>)
29
16
 
30
- Ruby Doc: [<%= gemspec.metadata['documentation_uri'] %>](<%= gemspec.metadata['documentation_uri'] %>)
17
+ Required Ruby version: <%=gemspec.required_ruby_version%>
31
18
 
32
- Ruby version must be >= <%= gemspec.required_ruby_version %>
19
+ [Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
33
20
 
34
- # <a name="when_to_use"></a>When to use and when not to use
21
+ # <a id="when_to_use"></a>When to use and when not to use
35
22
 
36
23
  <%=tool%> is designed to be used as a command line tool to:
37
24
 
@@ -45,7 +32,8 @@ So it is designed for:
45
32
 
46
33
  <%=tool%> can be seen as a command line tool integrating:
47
34
 
48
- * a configuration file (config.yaml) and advanced command line options
35
+ * a configuration file (config.yaml)
36
+ * advanced command line options
49
37
  * cURL (for REST calls)
50
38
  * Aspera transfer (ascp)
51
39
 
@@ -53,26 +41,23 @@ One might be tempted to use it as an integration element, e.g. by building a com
53
41
  For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
54
42
 
55
43
  * Product APIs (REST) : e.g. AoC, Faspex, node
56
- * Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
44
+ * Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
57
45
 
58
46
  Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
59
47
 
60
48
  For scripting and ad'hoc command line operations, <%=tool%> is perfect.
61
49
 
62
- # Notations
50
+ # <a id="parsing"></a>Notations, Shell and Command line parsing
63
51
 
64
- In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
65
- Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
52
+ In examples, command line operations are shown using a shell such: `bash` or `zsh`.
66
53
 
67
54
  Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
68
55
 
69
- # <a name="parsing"></a>Shell and Command line parsing
70
-
71
56
  <%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
72
57
 
73
58
  On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
74
59
 
75
- On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
60
+ On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
76
61
 
77
62
  * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
78
63
  * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
@@ -80,12 +65,15 @@ On Windows, `cmd` is typically used. Windows process creation does not receive t
80
65
  In case of doubt of argument values after parsing test like this:
81
66
 
82
67
  ```
83
- $ <%=cmd%> conf echo "Hello World" arg2 3
68
+ <%=cmd%> conf echo "Hello World" arg2 3
69
+ ```
70
+
71
+ ```
84
72
  "Hello World"
85
73
  ERROR: Argument: unprocessed values: ["arg2", "3"]
86
74
  ```
87
75
 
88
- `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
76
+ `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (<%=tool%>), next command line arguments are shown in the error message.
89
77
 
90
78
  # Quick Start
91
79
 
@@ -96,8 +84,11 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
96
84
  Once the gem is installed, <%=tool%> shall be accessible:
97
85
 
98
86
  ```
99
- $ <%=cmd%> --version
100
- <%= gemspec.version.to_s %>
87
+ <%=cmd%> --version
88
+ ```
89
+
90
+ ```
91
+ <%=gemspec.version.to_s%>
101
92
  ```
102
93
 
103
94
  ## First use
@@ -109,8 +100,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
109
100
  To test with Aspera demo transfer server, setup the environment and then test:
110
101
 
111
102
  ```
112
- $ <%=cmd%> config initdemo
113
- $ <%=cmd%> server browse /
103
+ <%=cmd%> config initdemo
104
+ ```
105
+
106
+ ```
107
+ <%=cmd%> server browse /
108
+ ```
109
+
110
+ ```
114
111
  :............:...........:......:........:...........................:.......................:
115
112
  : zmode : zuid : zgid : size : mtime : name :
116
113
  :............:...........:......:........:...........................:.......................:
@@ -124,16 +121,31 @@ $ <%=cmd%> server browse /
124
121
  If you want to use <%=tool%> with another server, and in order to make further calls more convenient, it is advised to define a <%=prst%> for the server's authentication options. The following example will:
125
122
 
126
123
  * create a <%=prst%>
127
- * define it as default for "server" plugin
124
+ * define it as default for `server` plugin
128
125
  * list files in a folder
129
126
  * download a file
130
127
 
131
128
  ```
132
- $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
129
+ <%=cmd%> config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
130
+ ```
131
+
132
+ ```
133
133
  updated: myserver
134
- $ <%=cmd%> config id default set server myserver
134
+ ```
135
+
136
+ ```
137
+ <%=cmd%> config preset set default server myserver
138
+ ```
139
+
140
+ ```
135
141
  updated: default&rarr;server to myserver
136
- $ <%=cmd%> server browse /aspera-test-dir-large
142
+ ```
143
+
144
+ ```
145
+ <%=cmd%> server browse /aspera-test-dir-large
146
+ ```
147
+
148
+ ```
137
149
  :............:...........:......:..............:...........................:............................:
138
150
  : zmode : zuid : zgid : size : mtime : name :
139
151
  :............:...........:......:..............:...........................:............................:
@@ -150,7 +162,13 @@ $ <%=cmd%> server browse /aspera-test-dir-large
150
162
  : -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
151
163
  : -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
152
164
  :............:...........:......:..............:...........................:............................:
153
- $ <%=cmd%> server download /aspera-test-dir-large/200MB
165
+ ```
166
+
167
+ ```
168
+ <%=cmd%> server download /aspera-test-dir-large/200MB
169
+ ```
170
+
171
+ ```
154
172
  Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
155
173
  complete
156
174
  ```
@@ -161,43 +179,48 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
161
179
 
162
180
  Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
163
181
 
164
- # <a name="installation"></a>Installation
182
+ # <a id="installation"></a>Installation
165
183
 
166
- It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
184
+ It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
167
185
 
168
186
  The direct installation is recommended and consists in installing:
169
187
 
170
- * [Ruby](#ruby) version >= <%= gemspec.required_ruby_version %>
171
- * [<%= gemspec.name %>](#the_gem)
188
+ * [Ruby](#ruby) version <%=gemspec.required_ruby_version%>
189
+ * [<%=gemspec.name%>](#the_gem)
172
190
  * [Aspera SDK (ascp)](#fasp_prot)
173
191
 
174
192
  The following sections provide information on the various installation methods.
175
193
 
176
- An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
194
+ An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
177
195
 
178
196
  ## Docker container
179
197
 
180
198
  Use this method only if you know what you do, else use the standard recommended method as described here above.
181
199
 
182
- This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
200
+ This method installs a docker image that contains: Ruby, <%=tool%> and the FASP sdk.
201
+
202
+ The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
183
203
 
184
204
  Ensure that you have Docker installed.
185
205
 
186
206
  ```
187
- $ docker --version
207
+ docker --version
188
208
  ```
189
209
 
190
210
  Download the wrapping script:
191
211
 
192
212
  ```
193
- $ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
194
- $ chmod a+x <%=cmd%>
213
+ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
214
+ ```
215
+
216
+ ```
217
+ chmod a+x <%=cmd%>
195
218
  ```
196
219
 
197
220
  Install the container image:
198
221
 
199
222
  ```
200
- $ ./<%=cmd%> install
223
+ ./<%=cmd%> install
201
224
  ```
202
225
 
203
226
  Start using it !
@@ -207,23 +230,23 @@ Note that the tool is run in the container, so transfers are also executed in th
207
230
  The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
208
231
 
209
232
  To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
210
- To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
233
+ To add local storage as a volume edit the script: <%=tool%> and add a `--volume` stanza.
211
234
 
212
- ## <a name="ruby"></a>Ruby
235
+ ## <a id="ruby"></a>Ruby
213
236
 
214
237
  Use this method to install on the native host.
215
238
 
216
239
  A ruby interpreter is required to run the tool or to use the gem and tool.
217
240
 
218
- Ruby minimum version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
241
+ Required Ruby version: <%=gemspec.required_ruby_version%>. Ruby version 3 is also supported.
219
242
 
220
243
  *Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
221
244
 
222
245
  Refer to the following sections for a proposed method for specific operating systems.
223
246
 
224
- The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
247
+ The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
225
248
  If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
226
- If you have a simpler better way to install Ruby version >= <%= gemspec.required_ruby_version %> : use it !
249
+ If you have a simpler better way to install Ruby version <%=gemspec.required_ruby_version%> : use it !
227
250
 
228
251
  ### Generic: RVM: single user installation (not root)
229
252
 
@@ -234,31 +257,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
234
257
  Install the 2 keys
235
258
 
236
259
  ```
237
- $ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
260
+ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
238
261
  ```
239
262
 
240
263
  Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
241
264
 
242
265
  ```
243
- $ \curl -sSL https://get.rvm.io | bash -s stable
266
+ \curl -sSL https://get.rvm.io | bash -s stable
244
267
  ```
245
268
 
246
- If you keep the same terminal (ont needed if re-login):
269
+ If you keep the same terminal (not needed if re-login):
247
270
 
248
271
  ```
249
- $ source ~/.rvm/scripts/rvm
272
+ source ~/.rvm/scripts/rvm
250
273
  ```
251
274
 
252
- It is advised to get one of the pre-compiled ruby version, you can list with:
275
+ It is advised to get one of the pre-compiled ruby version, you can list with:
253
276
 
254
277
  ```
255
- $ rvm list --remote
278
+ rvm list --remote
256
279
  ```
257
280
 
258
281
  Install the chosen pre-compiled Ruby version:
259
282
 
260
283
  ```
261
- $ rvm install 2.7.2 --binary
284
+ rvm install 2.7.2 --binary
262
285
  ```
263
286
 
264
287
  Ruby is now installed for the user, go on to Gem installation.
@@ -271,7 +294,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
271
294
  One can install in another location with :
272
295
 
273
296
  ```
274
- # curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
297
+ curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
275
298
  ```
276
299
 
277
300
  As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
@@ -279,8 +302,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
279
302
  To activate ruby (and <%=cmd%>) later, source it:
280
303
 
281
304
  ```
282
- # source /etc/profile.d/rvm.sh.ok
283
- # rvm version
305
+ source /etc/profile.d/rvm.sh.ok
306
+ ```
307
+
308
+ ```
309
+ rvm version
284
310
  ```
285
311
 
286
312
  ### Windows: Installer
@@ -293,16 +319,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
293
319
 
294
320
  ### macOS: pre-installed or `brew`
295
321
 
296
- MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%= gemspec.name %> using `sudo` :
322
+ macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
297
323
 
298
324
  ```
299
- $ sudo gem install <%= gemspec.name %><%=geminstadd%>
325
+ sudo gem install <%=gemspec.name%><%=geminstadd%>
300
326
  ```
301
327
 
302
328
  Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
303
329
 
304
330
  ```
305
- $ brew install ruby
331
+ brew install ruby
306
332
  ```
307
333
 
308
334
  ### Linux: package
@@ -312,90 +338,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
312
338
  Example:
313
339
 
314
340
  ```
315
- # yum install -y ruby rubygems ruby-json
341
+ yum install -y ruby rubygems ruby-json
316
342
  ```
317
343
 
318
344
  One can cleanup the whole yum-installed ruby environment like this to uninstall:
319
345
 
320
346
  ```
321
347
  gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
348
+ ```
349
+
350
+ ```
322
351
  yum remove -y ruby ruby-libs
323
352
  ```
324
353
 
325
354
  ### Other Unixes: Aix, etc...
326
355
 
327
- If your unix do not provide a pre-built ruby, you can get it using one of those
356
+ If your Unix does not provide a pre-built ruby, you can get it using one of those
328
357
  [methods](https://www.ruby-lang.org/en/documentation/installation/)
329
358
 
330
359
  For instance to build from source, and install in `/opt/ruby` :
331
360
 
332
361
  ```
333
- # wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
334
- # gzip -d ruby-2.7.2.tar.gz
335
- # tar xvf ruby-2.7.2.tar
336
- # cd ruby-2.7.2
337
- # ./configure --prefix=/opt/ruby
338
- # make ruby.imp
339
- # make
340
- # make install
341
- ```
342
-
343
- ### <a name="offline_install"></a>Installation without internet access
344
-
345
- Note that currently no pre-packaged version exist yet.
346
- A method to build one provided here:
347
-
348
- On a server with the same OS version and with internet access follow the "Generic single user installation" method.
349
-
350
- Then create an archive:
351
-
352
- ```
353
- $ cd
354
- $ tar zcvf rvm-<%=cmd%>.tgz .rvm
355
- ```
356
-
357
- Get the Aspera SDK. Execute:
358
-
359
- ```
360
- $ <%=cmd%> conf --show-config|grep sdk_url
361
- ```
362
+ wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
362
363
 
363
- Then download the SDK archive from that URL.
364
+ gzip -d ruby-2.7.2.tar.gz
364
365
 
365
- Another method for the SDK is to install the SDK (`<%=cmd%> conf ascp install`) on the first system, and archive `$HOME/.aspera`.
366
+ tar xvf ruby-2.7.2.tar
366
367
 
367
- Transfer those 2 archives to the target system without internet access.
368
+ cd ruby-2.7.2
368
369
 
369
- On the target system:
370
+ ./configure --prefix=/opt/ruby
370
371
 
371
- * Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
372
- * in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
372
+ make ruby.imp
373
373
 
374
- ```
375
- source path_to_rvm_root/scripts/rvm
376
- rvm use 2.7.2
377
- ```
374
+ make
378
375
 
379
- For the SDK, either install from archive:
380
-
381
- ```
382
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
376
+ make install
383
377
  ```
384
378
 
385
- or restore the `$HOME/.aspera` folder for the user.
386
-
387
- ## <a name="the_gem"></a>`<%= gemspec.name %>` gem
379
+ ## <a id="the_gem"></a>`<%=gemspec.name%>` gem
388
380
 
389
381
  Once you have Ruby and rights to install gems: Install the gem and its dependencies:
390
382
 
391
383
  ```
392
- # gem install <%= gemspec.name %><%=geminstadd%>
384
+ gem install <%=gemspec.name%><%=geminstadd%>
393
385
  ```
394
386
 
395
387
  To upgrade to the latest version:
396
388
 
397
389
  ```
398
- # gem update <%= gemspec.name %>
390
+ gem update <%=gemspec.name%>
399
391
  ```
400
392
 
401
393
  <%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
@@ -403,12 +395,10 @@ To upgrade to the latest version:
403
395
  To check manually:
404
396
 
405
397
  ```
406
- # <%=cmd%> conf check_update
398
+ <%=cmd%> conf check_update
407
399
  ```
408
400
 
409
-
410
-
411
- ## <a name="fasp_prot"></a>FASP Protocol
401
+ ## <a id="fasp_prot"></a>FASP Protocol
412
402
 
413
403
  Most file transfers will be done using the FASP protocol, using `ascp`.
414
404
  Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
@@ -416,17 +406,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
416
406
  * ascp
417
407
  * aspera-license (in same folder, or ../etc)
418
408
 
419
- This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
409
+ This can be installed either be installing an Aspera transfer software, or using an embedded command:
420
410
 
421
411
  ```
422
- $ <%=cmd%> conf ascp install
412
+ <%=cmd%> conf ascp install
423
413
  ```
424
414
 
425
- If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
415
+ If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
426
416
 
427
417
  ```
428
- $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
429
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
418
+ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
419
+ ```
420
+
421
+ ```
422
+ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
430
423
  ```
431
424
 
432
425
  The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
@@ -445,39 +438,58 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
445
438
  <%=tool%> will detect most of Aspera transfer products in standard locations and use the first one found.
446
439
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
447
440
 
448
- Several methods are provided on how to start a transfer. Use of a local client is one of them, but
449
- other methods are available. Refer to section: [Transfer Agents](#agents)
441
+ Several methods are provided to start a transfer.
442
+ Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
443
+
444
+ ## <a id="offline_install"></a>Installation in air gapped environment
450
445
 
451
- ## <a name="offline_install"></a>Offline Installation (without internet)
446
+ Note that currently no pre-packaged version exist yet.
447
+ A method to build one is provided here:
452
448
 
453
- The procedure consists in:
449
+ The procedure:
454
450
 
455
451
  * Follow the non-root installation procedure with RVM, including gem
456
- * archive (zip, tar) the main RVM folder (includes <%=cmd%>):
452
+ * Archive (zip, tar) the main RVM folder (includes <%=cmd%>):
457
453
 
458
454
  ```
459
- $ cd ~
460
- $ tar zcvf rvm_<%=cmd%>.tgz .rvm
455
+ cd $HOME && tar zcvf rvm-<%=cmd%>.tgz .rvm
461
456
  ```
462
457
 
463
- * retrieve the SDK:
458
+ * Get the Aspera SDK.
464
459
 
465
460
  ```
466
- $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
461
+ <%=cmd%> conf --show-config --fields=sdk_url
467
462
  ```
468
463
 
469
- * on the system without internet access:
464
+ * Download the SDK archive from that URL.
470
465
 
471
466
  ```
472
- $ cd ~
473
- $ tar zxvf rvm_<%=cmd%>.tgz
474
- $ source ~/.rvm/scripts/rvm
475
- $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
467
+ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
476
468
  ```
477
469
 
478
- # <a name="cli"></a>Command Line Interface: <%=tool%>
470
+ * Transfer those 2 files to the target system
479
471
 
480
- The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
472
+ * On target system
473
+
474
+ ```
475
+ cd $HOME
476
+
477
+ tar zxvf rvm-<%=cmd%>.tgz
478
+
479
+ source ~/.rvm/scripts/rvm
480
+
481
+ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
482
+ ```
483
+
484
+ * Add those lines to shell init (`.profile`)
485
+
486
+ ```
487
+ source ~/.rvm/scripts/rvm
488
+ ```
489
+
490
+ # <a id="cli"></a>Command Line Interface: <%=tool%>
491
+
492
+ The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
481
493
 
482
494
  * IBM Aspera High Speed Transfer Server (FASP and Node)
483
495
  * IBM Aspera on Cloud (including ATS)
@@ -492,7 +504,7 @@ The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which inte
492
504
  * Supports most Aspera server products (on-premise and SaaS)
493
505
  * Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
494
506
  * Supports Commands, Option values and Parameters shortcuts
495
- * FASP [Transfer Agents](#agents) can be: FaspManager (local ascp), or Connect Client, or any transfer node
507
+ * FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
496
508
  * Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
497
509
  * Allows transfers from products to products, essentially at node level (using the node transfer agent)
498
510
  * Supports FaspStream creation (using Node API)
@@ -504,7 +516,7 @@ The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which inte
504
516
  Basic usage is displayed by executing:
505
517
 
506
518
  ```
507
- $ <%=cmd%> -h
519
+ <%=cmd%> -h
508
520
  ```
509
521
 
510
522
  Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
@@ -518,7 +530,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
518
530
  There are two types of arguments: Commands and Options. Example :
519
531
 
520
532
  ```
521
- $ <%=cmd%> command --option-name=VAL1 VAL2
533
+ <%=cmd%> command --option-name=VAL1 VAL2
522
534
  ```
523
535
 
524
536
  * executes _command_: `command`
@@ -546,13 +558,16 @@ Exceptions:
546
558
  * the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
547
559
 
548
560
  ```
549
- $ <%=cmd%> config echo -- --sample
561
+ <%=cmd%> config echo -- --sample
562
+ ```
563
+
564
+ ```
550
565
  "--sample"
551
566
  ```
552
567
 
553
568
  Note that `--sample` is taken as an argument, and not option.
554
569
 
555
- Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on comand line and evaluated in order.
570
+ Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
556
571
 
557
572
  The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
558
573
 
@@ -573,7 +588,7 @@ Command line arguments that are not options are either commands or arguments. If
573
588
 
574
589
  Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
575
590
 
576
- The behaviour can be controlled with:
591
+ The behavior can be controlled with:
577
592
 
578
593
  * --interactive=&lt;yes|no&gt; (default=yes if STDIN is a terminal, else no)
579
594
  * yes : missing mandatory parameters/options are asked to the user
@@ -590,8 +605,8 @@ The information displayed depends on the action.
590
605
 
591
606
  Depending on action, the output will contain:
592
607
 
593
- * `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is atteribute value. Nested hashes are collapsed.
594
- * `object_list` : displayed as a 2 dimensional table: one line per item, one colum per attribute.
608
+ * `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
609
+ * `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
595
610
  * `value_list` : a table with one column.
596
611
  * `empty` : nothing
597
612
  * `status` : a message
@@ -616,12 +631,15 @@ The style of output can be set using the `format` parameter, supporting:
616
631
  * `yaml` : YAML
617
632
  * `csv` : Comma Separated Values
618
633
 
619
- ### Filtering columns for `object_list`
634
+ ### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
620
635
 
621
636
  Table output can be filtered using the `select` parameter. Example:
622
637
 
623
638
  ```
624
- $ <%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"per_page":1000,"page":1,"sort":"name"}' --select=@json:'{"ats_admin":true}'
639
+ <%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
640
+ ```
641
+
642
+ ```
625
643
  :...............................:..................................:...........:
626
644
  : name : email : ats_admin :
627
645
  :...............................:..................................:...........:
@@ -634,7 +652,7 @@ Note that `select` filters selected elements from the result of API calls, while
634
652
 
635
653
  ### Verbosity of output
636
654
 
637
- Outpout messages are categorized in 3 types:
655
+ Output messages are categorized in 3 types:
638
656
 
639
657
  * `info` output contain additional information, such as number of elements in a table
640
658
  * `data` output contain the actual output of the command (object, or list of objects)
@@ -648,7 +666,7 @@ The option `display` controls the level of output:
648
666
 
649
667
  ### Selection of output object properties
650
668
 
651
- By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific colums to be displayed, by setting the `fields` option to one of the following value:
669
+ By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
652
670
 
653
671
  * DEF : default display of columns (that's the default, when not set)
654
672
  * ALL : all columns available
@@ -657,7 +675,7 @@ By default, a table output will display one line per entry, and columns for each
657
675
  * +a,b,c : add selected properties to the default selection.
658
676
  * -a,b,c : remove selected properties from the default selection.
659
677
 
660
- ## <a name="extended"></a>Extended Value Syntax
678
+ ## <a id="extended"></a>Extended Value Syntax
661
679
 
662
680
  Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
663
681
 
@@ -694,23 +712,32 @@ To display the result of an extended value, use the `config echo` command.
694
712
  Example: read the content of the specified file, then, base64 decode, then unzip:
695
713
 
696
714
  ```
697
- $ <%=cmd%> config echo @zlib:@base64:@file:myfile.dat
715
+ <%=cmd%> config echo @zlib:@base64:@file:myfile.dat
698
716
  ```
699
717
 
700
718
  Example: create a value as a hash, with one key and the value is read from a file:
701
719
 
702
720
  ```
703
- $ <%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
721
+ <%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
704
722
  ```
705
723
 
706
724
  Example: read a csv file and create a list of hash for bulk provisioning:
707
725
 
708
726
  ```
709
- $ cat test.csv
727
+ cat test.csv
728
+ ```
729
+
730
+ ```
710
731
  name,email
711
732
  lolo,laurent@example.com
712
733
  toto,titi@tutu.tata
713
- $ <%=cmd%> config echo @csvt:@file:test.csv
734
+ ```
735
+
736
+ ```
737
+ <%=cmd%> config echo @csvt:@file:test.csv
738
+ ```
739
+
740
+ ```
714
741
  :......:.....................:
715
742
  : name : email :
716
743
  :......:.....................:
@@ -722,13 +749,16 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
722
749
  Example: create a hash and include values from preset named "config" of config file in this hash
723
750
 
724
751
  ```
725
- $ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
752
+ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
753
+ ```
754
+
755
+ ```
726
756
  {"version"=>"0.9", "hello"=>true}
727
757
  ```
728
758
 
729
759
  Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
730
760
 
731
- ## <a name="native"></a>Structured Value
761
+ ## <a id="native"></a>Structured Value
732
762
 
733
763
  Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
734
764
 
@@ -739,7 +769,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
739
769
 
740
770
  It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
741
771
 
742
- ## <a name="conffolder"></a>Configuration and Persistency Folder
772
+ ## <a id="conffolder"></a>Configuration and Persistency Folder
743
773
 
744
774
  <%=tool%> configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/<%=cmd%>`.
745
775
 
@@ -749,53 +779,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
749
779
  The main folder can be displayed using :
750
780
 
751
781
  ```
752
- $ <%=cmd%> config folder
782
+ <%=cmd%> config folder
783
+ ```
784
+
785
+ ```
753
786
  /Users/kenji/.aspera/<%=cmd%>
754
787
  ```
755
788
 
756
- It can be overriden using the envinonment variable `<%=evp%>HOME`.
789
+ It can be overridden using the environment variable `<%=evp%>HOME`.
757
790
 
758
791
  Example (Windows):
759
792
 
760
793
  ```
761
- $ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
762
- $ <%=cmd%> config folder
794
+ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
795
+ <%=cmd%> config folder
763
796
  C:\Users\Kenji\.aspera\<%=cmd%>
764
797
  ```
765
798
 
766
- ## <a name="configfile"></a>Configuration file
799
+ ## <a id="configfile"></a>Configuration file
767
800
 
768
801
  On the first execution of <%=tool%>, an empty configuration file is created in the configuration folder.
769
802
  Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
770
803
 
771
- Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it
772
- using the `config` command.
804
+ Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it using the `config` command.
773
805
 
774
- All options for <%=tool%> commands can be set on command line, or by env vars, or using <%=prsts%> in the configuratin file.
806
+ All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuration file.
775
807
 
776
- A configuration file provides a way to define default values, especially
777
- for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
808
+ A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
778
809
 
779
- The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml`
780
- (this can be overriden with option `--config-file=path` or equivalent env var).
810
+ The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
781
811
 
782
- So, finally, the configuration file is simply a catalog of pre-defined lists of options,
783
- called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
812
+ The configuration file is simply a catalog of pre-defined lists of options, called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
784
813
 
785
- ### <a name="lprt"></a><%=prstt%>
814
+ ### <a id="lprt"></a><%=prstt%>
786
815
 
787
816
  A <%=prst%> is simply a collection of parameters and their associated values in a named section in the configuration file.
788
817
 
789
818
  A named <%=prst%> can be modified directly using <%=tool%>, which will update the configuration file :
790
819
 
791
820
  ```
792
- $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
821
+ <%=cmd%> config preset set|delete|show|initialize|update <<%=opprst%>>
793
822
  ```
794
823
 
795
824
  The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
796
825
 
797
826
  ```
798
- $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
827
+ <%=cmd%> config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
799
828
  ```
800
829
 
801
830
  * This creates a <%=prst%> `demo_server` with all provided options.
@@ -803,39 +832,48 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
803
832
  The command `set` allows setting individual options in a <%=prst%>.
804
833
 
805
834
  ```
806
- $ <%=cmd%> config id demo_server set password _demo_pass_
835
+ <%=cmd%> config preset set demo_server password _demo_pass_
807
836
  ```
808
837
 
809
838
  The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
810
839
 
811
840
  ```
812
- $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
841
+ <%=cmd%> config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
813
842
  ```
814
843
 
815
- A good practice is to not manually edit the configuration file and use modification commands instead.
816
- If necessary, the configuration file can be edited (or simply consulted) with:
844
+ A full terminal based overview of the configuration can be displayed using:
817
845
 
818
846
  ```
819
- $ <%=cmd%> config open
847
+ <%=cmd%> config preset over
820
848
  ```
821
849
 
822
- A full terminal based overview of the configuration can be displayed using:
850
+ A list of <%=prst%> can be displayed using:
823
851
 
824
852
  ```
825
- $ <%=cmd%> config over
853
+ <%=cmd%> config preset list
826
854
  ```
827
855
 
828
- A list of <%=prst%> can be displayed using:
856
+ A good practice is to not manually edit the configuration file and use modification commands instead.
857
+ If necessary, the configuration file can opened in a text editor with:
829
858
 
830
859
  ```
831
- $ <%=cmd%> config list
860
+ <%=cmd%> config open
832
861
  ```
833
862
 
834
- ### <a name="lprtconf"></a>Special <%=prstt%>: config
863
+ Older format for commands are still supported:
864
+
865
+ ```
866
+ <%=cmd%> config id <name> set|delete|show|initialize|update
867
+ <%=cmd%> config over
868
+ <%=cmd%> config list
869
+ ```
870
+
871
+
872
+ ### <a id="lprtconf"></a>Special <%=prstt%>: config
835
873
 
836
874
  This preset name is reserved and contains a single key: `version`. This is the version of <%=tool%> which created the file.
837
875
 
838
- ### <a name="lprtdef"></a>Special <%=prstt%>: default
876
+ ### <a id="lprtdef"></a>Special <%=prstt%>: default
839
877
 
840
878
  This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
841
879
 
@@ -846,16 +884,16 @@ Note that special plugin name: `config` can be associated with a preset that is
846
884
  Operations on this preset are done using regular `config` operations:
847
885
 
848
886
  ```
849
- $ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
850
- $ <%=cmd%> config id default get _plugin_name_
887
+ <%=cmd%> config preset set default _plugin_name_ _default_preset_for_plugin_
888
+ <%=cmd%> config preset get default _plugin_name_
851
889
  "_default_preset_for_plugin_"
852
890
  ```
853
891
 
854
- ### <a name="lprtdef"></a>Special Plugin: config
892
+ ### <a id="lplugconf"></a>Special Plugin: config
855
893
 
856
894
  Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
857
895
 
858
- When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
896
+ When <%=tool%> starts, it looks for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
859
897
 
860
898
  If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
861
899
 
@@ -880,7 +918,7 @@ demo_server:
880
918
  We can see here:
881
919
 
882
920
  * The configuration was created with CLI version 0.3.7
883
- * the default <%=prst%> to load for plugin "server" is : `demo_server`
921
+ * the default <%=prst%> to load for `server` plugin is : `demo_server`
884
922
  * the <%=prst%> `demo_server` defines some parameters: the URL and credentials
885
923
  * the default <%=prst%> to load in any case is : `cli_default`
886
924
 
@@ -894,10 +932,10 @@ The user may create as many <%=prsts%> as needed. For instance, a particular <%=
894
932
 
895
933
  Values in the configuration also follow the [Extended Value Syntax](#extended).
896
934
 
897
- Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config id update` command, the user shall use the `@val:` prefix. Example:
935
+ Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
898
936
 
899
937
  ```
900
- $ <%=cmd%> config id my_aoc_org set private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
938
+ <%=cmd%> config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
901
939
  ```
902
940
 
903
941
  This creates the <%=prst%>:
@@ -911,36 +949,51 @@ my_aoc_org:
911
949
 
912
950
  So, the key file will be read only at execution time, but not be embedded in the configuration file.
913
951
 
952
+ ### Options evaluation order
953
+
954
+ Some options are global, some options are available only for some plugins. (the plugin is the first level command).
955
+
914
956
  Options are loaded using this algorithm:
915
957
 
916
- * if option '--preset=xxxx' is specified (or -Pxxxx), this reads the <%=prst%> specified from the configuration file.
917
- * else if option --no-default (or -N) is specified, then dont load default
918
- * else it looks for the name of the default <%=prst%> in section "default" and loads it
919
- * environment variables are evaluated
920
- * command line options are evaluated
958
+ * If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
959
+ * else it looks for the name of the plugin as key in section `default`, the value is the name of the default <%=prst%> for it, and loads it.
960
+ * If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the <%=prst%> specified from the configuration file, or of the value is a Hash, it uses it as options values.
961
+ * Environment variables are evaluated
962
+ * Command line options are evaluated
921
963
 
922
964
  Parameters are evaluated in the order of command line.
923
965
 
924
- To avoid loading the default <%=prst%> for a plugin, just specify a non existing configuration: `-Pnone`
966
+ To avoid loading the default <%=prst%> for a plugin, use: `-N`
925
967
 
926
968
  On command line, words in parameter names are separated by a dash, in configuration file, separator
927
969
  is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
928
970
 
929
- Note: before version 0.4.5, some keys could be ruby symbols, from 0.4.5 all keys are strings. To
930
- convert olver versions, remove the leading ":" in front of keys.
971
+ The main plugin name is `config`, so it is possible to define a default <%=prst%> for the main plugin with:
931
972
 
932
- The main plugin name is *config*, so it is possible to define a default <%=prst%> for
933
- the main plugin with:
973
+ ```
974
+ <%=cmd%> config preset set cli_default interactive no
975
+ ```
934
976
 
935
977
  ```
936
- $ <%=cmd%> config id cli_default set interactive no
937
- $ <%=cmd%> config id default set config cli_default
978
+ <%=cmd%> config preset set default config cli_default
938
979
  ```
939
980
 
940
981
  A <%=prst%> value can be removed with `unset`:
941
982
 
942
983
  ```
943
- $ <%=cmd%> config id cli_default unset interactive
984
+ <%=cmd%> config preset unset cli_default interactive
985
+ ```
986
+
987
+ Example: Define options using command line:
988
+
989
+ ```
990
+ <%=cmd%> -N --url=x --password=y --username=y node --show-config
991
+ ```
992
+
993
+ Example: Define options using a hash:
994
+
995
+ ```
996
+ <%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
944
997
  ```
945
998
 
946
999
  ### Examples
@@ -950,44 +1003,120 @@ only username/password and url are required (either on command line, or from con
950
1003
  Those can usually be provided on the command line:
951
1004
 
952
1005
  ```
953
- $ <%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
1006
+ <%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
954
1007
  ```
955
1008
 
956
1009
  This can also be provisioned in a config file:
957
1010
 
1011
+ * Build <%=prst%>
1012
+
958
1013
  ```
959
- 1$ <%=cmd%> config id shares06 set url https://10.25.0.6
960
- 2$ <%=cmd%> config id shares06 set username john
961
- 3$ <%=cmd%> config id shares06 set password 4sp3ra
962
- 4$ <%=cmd%> config id default set shares shares06
963
- 5$ <%=cmd%> config overview
964
- 6$ <%=cmd%> shares repo browse /
1014
+ <%=cmd%> config preset set shares06 url https://10.25.0.6
1015
+ <%=cmd%> config preset set shares06 username john
1016
+ <%=cmd%> config preset set shares06 password 4sp3ra
965
1017
  ```
966
1018
 
967
- The three first commands build a <%=prst%>.
968
1019
  Note that this can also be done with one single command:
969
1020
 
970
1021
  ```
971
- $ <%=cmd%> config id shares06 init @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
1022
+ <%=cmd%> config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
1023
+ ```
1024
+
1025
+ or
1026
+
1027
+ ```
1028
+ <%=cmd%> config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
1029
+ ```
1030
+
1031
+ * Define this <%=prst%> as the default <%=prst%> for the specified plugin (`shares`)
1032
+
1033
+ ```
1034
+ <%=cmd%> config preset set default shares shares06
1035
+ ```
1036
+
1037
+ * Display the content of configuration file in table format
1038
+
1039
+ ```
1040
+ <%=cmd%> config overview
1041
+ ```
1042
+
1043
+ * Execute a command on the shares application using default parameters
1044
+
1045
+ ```
1046
+ <%=cmd%> shares repo browse /
1047
+ ```
1048
+
1049
+ ## <a id="vault"></a>Secret Vault
1050
+
1051
+ When a secret or password is needed, it is possible to store in the secret vault.
1052
+
1053
+ By default the vault is defined using option `secrets`.
1054
+
1055
+ ### Using system keychain
1056
+
1057
+ Only on macOS.
1058
+
1059
+ It is possible to store secrets in macOS keychain (only read supported currently).
1060
+
1061
+ Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
1062
+
1063
+ ### Modern config file format: encrypted in config file
1064
+
1065
+ It is possible to store and use secrets encrypted.
1066
+ For this use the `config vault` command.
1067
+
1068
+ The vault can be initialized with `config vault init`
1069
+
1070
+ Then secrets can be manipulated using commands:
1071
+
1072
+ * `set`
1073
+ * `get`
1074
+ * `list`
1075
+ * `delete`
1076
+
1077
+ Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
1078
+
1079
+ ### Legacy config file format
1080
+
1081
+ The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
1082
+
1083
+ For example, choose a repository name, for example `my_secrets`, and populate it like this:
1084
+
1085
+ ```
1086
+ <%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
1087
+
1088
+ <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
1089
+
1090
+ <%=cmd%> conf id default get config
1091
+
1092
+ cli_default
972
1093
  ```
973
1094
 
974
- The fourth command defines this <%=prst%> as the default <%=prst%> for the
975
- specified application ("shares"). The 5th command displays the content of configuration file in table format.
976
- Alternative <%=prsts%> can be used with option "-P&lt;<%=prst%>&gt;"
977
- (or --preset=&lt;<%=prst%>&gt;)
1095
+ Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
1096
+ So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
1097
+
1098
+ ```
1099
+ <%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
1100
+ ```
978
1101
 
979
- Eventually, the last command shows a call to the shares application using default parameters.
1102
+ A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
980
1103
 
1104
+ To test if a secret can be found use:
1105
+
1106
+ ```
1107
+ <%=cmd%> conf vault get --username=access_key1
1108
+ ```
981
1109
 
982
1110
  ## Plugins
983
1111
 
984
1112
  The CLI tool uses a plugin mechanism. The first level command (just after <%=tool%> on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
985
1113
  For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
986
1114
 
987
- ### Create your own plugin
1115
+ ### <a id="createownplugin"></a>Create your own plugin
1116
+
988
1117
  ```
989
- $ mkdir -p ~/.aspera/<%=cmd%>/plugins
990
- $ cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
1118
+ mkdir -p ~/.aspera/<%=cmd%>/plugins
1119
+ cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
991
1120
  require 'aspera/cli/plugin'
992
1121
  module Aspera
993
1122
  module Cli
@@ -1002,39 +1131,78 @@ end # Aspera
1002
1131
  EOF
1003
1132
  ```
1004
1133
 
1005
- ## Debugging
1134
+ ### <a id="plugins"></a>Plugins: Application URL and Authentication
1135
+
1136
+ <%=tool%> comes with several Aspera application plugins.
1137
+
1138
+ REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
1006
1139
 
1007
- The gem is equipped with traces. By default logging level is "warn". To increase debug level, use parameter `log_level`, so either command line `--log-level=xx` or env var `<%=evp%>LOG_LEVEL`.
1140
+ Those are using options:
1008
1141
 
1009
- It is also possible to activate traces before initialisation using env var `AS_LOG_LEVEL`.
1142
+ * url
1143
+ * username
1144
+ * password
1145
+
1146
+ Those can be provided using command line, parameter set, env var, see section above.
1147
+
1148
+ Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
1149
+
1150
+ ## Logging, Debugging
1151
+
1152
+ The gem is equipped with traces. By default logging level is `warn`.
1153
+ To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `<%=evp%>LOG_LEVEL`, or parameter in con file).
1154
+
1155
+ It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
1156
+
1157
+ By default passwords and secrets are removed from logs.
1158
+ Use option `log_passwords` to change this behaviour.
1010
1159
 
1011
1160
  ## Learning Aspera Product APIs (REST)
1012
1161
 
1013
1162
  This CLI uses REST APIs.
1014
- To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
1015
- exact content or HTTP requests and responses.
1163
+ To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
1016
1164
 
1017
1165
  In order to get traces of execution, use argument : `--log-level=debug`
1018
1166
 
1019
- ## <a name="graphical"></a>Graphical Interactions: Browser and Text Editor
1167
+ ## <a id="http_options"></a>HTTP socket parameters
1168
+
1169
+ If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
1170
+
1171
+ Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
1172
+
1173
+ * `read_timeout` 60 sec
1174
+ * `write_timeout` 60 sec
1175
+
1176
+ Default values are the ones of Ruby.
1177
+
1178
+ Example:
1179
+
1180
+ ```
1181
+ <%=cmd%> aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
1182
+ ```
1183
+
1184
+ ## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
1020
1185
 
1021
1186
  Some actions may require the use of a graphical tool:
1022
1187
 
1023
1188
  * a browser for Aspera on Cloud authentication (web auth method)
1024
1189
  * a text editor for configuration file edition
1025
1190
 
1026
- By default the CLI will assume that a graphical environment is available on windows,
1027
- and on other systems, rely on the presence of the "DISPLAY" environment variable.
1191
+ By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
1028
1192
  It is also possible to force the graphical mode with option --ui :
1029
1193
 
1030
- * `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
1031
- a text editor for file edition.
1032
- * `--ui=text` forces a text environment, the URL or file path to open is displayed on
1033
- terminal.
1194
+ * `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
1195
+ * `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
1034
1196
 
1035
1197
  ## HTTP proxy for REST
1036
1198
 
1037
- To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honoured by Ruby when calling REST APIs.
1199
+ To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
1200
+
1201
+ ## <a id="certificates"></a>SSL CA certificate bundle
1202
+
1203
+ <%=tool%> uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
1204
+
1205
+ `ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). <%=tool%> overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
1038
1206
 
1039
1207
  ## Proxy auto config
1040
1208
 
@@ -1043,13 +1211,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
1043
1211
  The PAC file can be tested with command: `config proxy_check` , example:
1044
1212
 
1045
1213
  ```
1046
- $ <%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
1214
+ <%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
1047
1215
  PROXY proxy.example.com:8080
1048
1216
  ```
1049
1217
 
1050
1218
  This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
1051
1219
 
1052
- ## <a name="client"></a>FASP configuration
1220
+ ## <a id="client"></a>FASP configuration
1053
1221
 
1054
1222
  The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
1055
1223
 
@@ -1061,9 +1229,9 @@ The `config` plugin also allows specification for the use of a local FASP client
1061
1229
  ### Show path of currently used `ascp`
1062
1230
 
1063
1231
  ```
1064
- $ <%=cmd%> config ascp show
1232
+ <%=cmd%> config ascp show
1065
1233
  /Users/laurent/.aspera/ascli/sdk/ascp
1066
- $ <%=cmd%> config ascp info
1234
+ <%=cmd%> config ascp info
1067
1235
  +--------------------+-----------------------------------------------------------+
1068
1236
  | key | value |
1069
1237
  +--------------------+-----------------------------------------------------------+
@@ -1071,7 +1239,7 @@ $ <%=cmd%> config ascp info
1071
1239
  ...
1072
1240
  ```
1073
1241
 
1074
- ### Selection of local `ascp`
1242
+ ### Selection of `ascp` location for [`direct`](#agt_direct) agent
1075
1243
 
1076
1244
  By default, <%=tool%> uses any found local product with ascp, including SDK.
1077
1245
 
@@ -1082,7 +1250,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
1082
1250
  Using a POSIX shell:
1083
1251
 
1084
1252
  ```
1085
- $ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
1253
+ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
1086
1254
  ascp version: 4.0.0.182279
1087
1255
  Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
1088
1256
  Saved to default global preset global_common_defaults
@@ -1091,7 +1259,7 @@ Saved to default global preset global_common_defaults
1091
1259
  Windows:
1092
1260
 
1093
1261
  ```
1094
- $ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1262
+ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1095
1263
  ascp version: 4.0.0.182279
1096
1264
  Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1097
1265
  Saved to default global preset global_common_defaults
@@ -1104,7 +1272,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
1104
1272
  Locally installed Aspera products can be listed with:
1105
1273
 
1106
1274
  ```
1107
- $ <%=cmd%> config ascp products list
1275
+ <%=cmd%> config ascp products list
1108
1276
  :.........................................:................................................:
1109
1277
  : name : app_root :
1110
1278
  :.........................................:................................................:
@@ -1115,7 +1283,7 @@ $ <%=cmd%> config ascp products list
1115
1283
  :.........................................:................................................:
1116
1284
  ```
1117
1285
 
1118
- ### Selection of local client
1286
+ ### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
1119
1287
 
1120
1288
  If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
1121
1289
 
@@ -1124,14 +1292,14 @@ Using the option use_product finds the ascp binary of the selected product.
1124
1292
  To permanently use the ascp of a product:
1125
1293
 
1126
1294
  ```
1127
- $ <%=cmd%> config ascp products use 'Aspera Connect'
1295
+ <%=cmd%> config ascp products use 'Aspera Connect'
1128
1296
  saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
1129
1297
  ```
1130
1298
 
1131
1299
  ### Installation of Connect Client on command line
1132
1300
 
1133
1301
  ```
1134
- $ <%=cmd%> config ascp connect list
1302
+ <%=cmd%> config ascp connect list
1135
1303
  :...............................................:......................................:..............:
1136
1304
  : id : title : version :
1137
1305
  :...............................................:......................................:..............:
@@ -1144,7 +1312,7 @@ $ <%=cmd%> config ascp connect list
1144
1312
  : urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
1145
1313
  : urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
1146
1314
  :...............................................:......................................:..............:
1147
- $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1315
+ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1148
1316
  :.............................................:..........................:.......................................................................:..........:...............:
1149
1317
  : title : type : href : hreflang : rel :
1150
1318
  :.............................................:..........................:.......................................................................:..........:...............:
@@ -1157,11 +1325,11 @@ $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
1157
1325
  : Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
1158
1326
  : Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
1159
1327
  :.............................................:..........................:.......................................................................:..........:...............:
1160
- $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
1328
+ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
1161
1329
  downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
1162
1330
  ```
1163
1331
 
1164
- ## <a name="agents"></a>Transfer Agents
1332
+ ## <a id="agents"></a>Transfer Agents
1165
1333
 
1166
1334
  Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
1167
1335
 
@@ -1170,94 +1338,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
1170
1338
 
1171
1339
  There are currently 3 agents:
1172
1340
 
1173
- * `direct` : a local execution of `ascp`
1174
- * `connect` : use of a local Connect Client
1175
- * `node` : use of an Aspera Transfer Node (potentially _remote_).
1176
- * `httpgw` : use of an Aspera HTTP Gateway
1341
+ * [`direct`](#agt_direct) : a local execution of `ascp`
1342
+ * [`connect`](#agt_connect) : use of a local Connect Client
1343
+ * [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
1344
+ * [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
1345
+ * [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
1177
1346
 
1178
1347
  Note that all transfer operation are seen from the point of view of the agent.
1179
1348
  For instance, a node agent making an "upload", or "package send" operation,
1180
1349
  will effectively push files to the related server from the agent node.
1181
1350
 
1182
- <%=tool%> standadizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1351
+ <%=tool%> standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1183
1352
 
1184
1353
 
1185
- ### <a name="direct"></a>Direct (local ascp using FASPManager API)
1354
+ ### <a id="agt_direct"></a>Direct
1186
1355
 
1187
- By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
1188
- <%=tool%> will detect locally installed Aspera products.
1356
+ The `direct` agent directly executes a local ascp.
1357
+ This is the default for <%=tool%>.
1358
+ This is equivalent to specifying `--transfer=direct`.
1359
+ <%=tool%> will detect locally installed Aspera products, including SDK.
1189
1360
  Refer to section [FASP](#client).
1190
1361
 
1191
- To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
1192
-
1193
- * `EX_fasp_proxy_url`
1194
- * `EX_http_proxy_url` (proxy for legacy http fallback)
1195
- * `EX_ascp_args`
1196
-
1197
- The `transfer-info` accepts the following optional parameters:
1362
+ The `transfer-info` accepts the following optional parameters to control multi-session, WSS
1198
1363
 
1199
1364
  <table>
1200
- <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1201
- <tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
1202
- <tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
1203
- <tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1204
- <tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
1365
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1366
+ <tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
1367
+ <tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
1368
+ <tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
1369
+ <tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
1370
+ <tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
1371
+ <tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
1372
+ <tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
1373
+ <tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
1374
+ <tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
1205
1375
  </table>
1206
1376
 
1207
- Resume parameters:
1377
+ Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1378
+ Sleep between iterations is:
1208
1379
 
1209
- <table>
1210
- <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1211
- <tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
1212
- <tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
1213
- <tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
1214
- <tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
1215
- </table>
1380
+ ```
1381
+ max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
1382
+ ```
1383
+
1384
+ Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
1216
1385
 
1217
1386
  Examples:
1218
1387
 
1219
1388
  ```
1220
- $ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1221
- $ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
1389
+ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1390
+ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1222
1391
  ```
1223
1392
 
1224
- ### IBM Aspera Connect Client GUI
1393
+ To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
1394
+
1395
+ * `EX_fasp_proxy_url`
1396
+ * `EX_http_proxy_url` (proxy for legacy http fallback)
1397
+ * `EX_ascp_args`
1398
+
1399
+ ### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
1225
1400
 
1226
- By specifying option: `--transfer=connect`, <%=tool%> will start transfers
1227
- using the locally installed Aspera Connect Client.
1401
+ By specifying option: `--transfer=connect`, <%=tool%> will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
1228
1402
 
1229
- ### Aspera Node API : Node to node transfers
1403
+ ### <a id="agt_node"></a>Aspera Node API : Node to node transfers
1230
1404
 
1231
1405
  By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
1232
1406
  Transfer Server using the Node API, either on a local or remote node.
1407
+ Parameters provided in option `transfer_info` are:
1233
1408
 
1234
- If a default node has been configured
1235
- in the configuration file, then this node is used by default else the parameter
1236
- `--transfer-info` is required. The node specification shall be a hash table with
1237
- three keys: url, username and password, corresponding to the URL of the node API
1238
- and associated credentials (node user or access key).
1409
+ <table>
1410
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1411
+ <tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
1412
+ <tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
1413
+ <tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
1414
+ <tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
1415
+ </table>
1239
1416
 
1240
- The `--transfer-info` parameter can directly specify a pre-configured <%=prst%> :
1241
- `--transfer-info=@preset:<psetname>` or specified using the option syntax :
1417
+ Like any other option, `transfer_info` can get its value from a pre-configured <%=prst%> :
1418
+ `--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
1242
1419
  `--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
1243
1420
 
1244
- ### <a name="trinfoaoc"></a>Aspera on cloud
1421
+ If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
1245
1422
 
1246
- By specifying option: `--transfer=aoc`, WORK IN PROGRESS
1423
+ If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
1247
1424
 
1248
- ### <a name="httpgw"></a>HTTP Gateway
1425
+ ### <a id="agt_httpgw"></a>HTTP Gateway
1249
1426
 
1250
- If it possible to send using a HTTP gateway, in case FASP is not allowed.
1427
+ If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
1251
1428
 
1252
1429
  Example:
1253
1430
 
1254
1431
  ```
1255
- $ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
1432
+ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
1256
1433
  ```
1257
1434
 
1258
1435
  Note that the gateway only supports transfers authorized with a token.
1259
1436
 
1260
- ## <a name="transferspec"></a>Transfer Specification
1437
+ ### <a id="agt_trsdk"></a>Transfer SDK
1438
+
1439
+ Another possibility is to use the Transfer SDK daemon (asperatransferd).
1440
+
1441
+ By default it will listen on local port `55002` on `127.0.0.1`.
1442
+
1443
+ ## <a id="transferspec"></a>Transfer Specification
1261
1444
 
1262
1445
  Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
1263
1446
  is described in a _transfer-spec_ (Transfer Specification), such as:
@@ -1270,9 +1453,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
1270
1453
 
1271
1454
  <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
1272
1455
 
1273
- If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1456
+ If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
1274
1457
 
1275
- It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1458
+ It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
1276
1459
 
1277
1460
  The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1278
1461
 
@@ -1281,13 +1464,35 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1281
1464
 
1282
1465
  A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
1283
1466
 
1284
- ## <a name="transferparams"></a>Transfer Parameters
1467
+ ## <a id="transferparams"></a>Transfer Parameters
1285
1468
 
1286
- All standard _transfer-spec_ parameters can be overloaded. To display parameters,
1287
- run in debug mode (--log-level=debug). [_transfer-spec_](#transferspec) can
1288
- also be saved/overridden in the config file.
1469
+ All standard _transfer-spec_ parameters can be specified.
1470
+ [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1289
1471
 
1290
- <%= File.read(ENV['INCL_TRSPEC']).gsub(/.*<body>(.*)<\/body>.*/m,'\1') %>
1472
+ References:
1473
+
1474
+ * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1475
+ * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1476
+ * [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) &rarr; search `The parameters for starting a transfer.`
1477
+
1478
+ Parameters can be displayed with commands:
1479
+
1480
+ ```
1481
+ <%=cmd%> config ascp spec
1482
+ <%=cmd%> config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
1483
+ ```
1484
+
1485
+ Columns:
1486
+
1487
+ * D=Direct (local `ascp` execution)
1488
+ * N=Node API
1489
+ * C=Connect Client
1490
+
1491
+ `ascp` argument or environment variable is provided in description.
1492
+
1493
+ Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in <%=tool%>).
1494
+
1495
+ <%=spec_table%>
1291
1496
 
1292
1497
  ### Destination folder for transfers
1293
1498
 
@@ -1303,20 +1508,20 @@ The option `to_folder` provides an equivalent and convenient way to change this
1303
1508
 
1304
1509
  ### List of files for transfers
1305
1510
 
1306
- When uploading, downloading or sending files, the user must specify the list of files to transfer.
1307
- Most of the time, the list of files to transfer will be simply specified on the command line:
1511
+ When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1512
+ So, by default, the list of files to transfer will be simply specified on the command line:
1308
1513
 
1309
1514
  ```
1310
- $ <%=cmd%> server upload ~/mysample.file secondfile
1515
+ <%=cmd%> server upload ~/mysample.file secondfile
1311
1516
  ```
1312
1517
 
1313
1518
  This is equivalent to:
1314
1519
 
1315
1520
  ```
1316
- $ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1521
+ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1317
1522
  ```
1318
1523
 
1319
- More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1524
+ More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1320
1525
 
1321
1526
  Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
1322
1527
  all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
@@ -1339,13 +1544,21 @@ For ease of use and flexibility, the list of files to transfer is specified by t
1339
1544
  --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
1340
1545
  ```
1341
1546
 
1342
- * Although not recommended, because it applies *only* to the `local` transfer agent (i.e. bare ascp), it is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`. In that case, one must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided.
1547
+ providing a file list directly to ascp:
1548
+
1549
+ ```
1550
+ ... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
1551
+ ```
1552
+
1553
+ * Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
1343
1554
 
1344
1555
  ```
1345
1556
  --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
1346
1557
  ```
1347
1558
 
1348
- In case the file list is provided on the command line (i.e. using `--sources=@args` or `--sources=<Array>`, but not `--sources=@ts`), the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1559
+ This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1560
+
1561
+ In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1349
1562
 
1350
1563
  * `list` : (default) the path of destination is the same as source
1351
1564
  * `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
@@ -1353,9 +1566,11 @@ In case the file list is provided on the command line (i.e. using `--sources=@ar
1353
1566
  Example:
1354
1567
 
1355
1568
  ```
1356
- $ <%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1569
+ <%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1357
1570
  ```
1358
1571
 
1572
+ Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
1573
+
1359
1574
  Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
1360
1575
 
1361
1576
  * All files must be in the same source folder.
@@ -1367,9 +1582,9 @@ Source files are located on "Aspera on cloud", when :
1367
1582
  * the server is Aspera on Cloud, and making a download / recv
1368
1583
  * the agent is Aspera on Cloud, and making an upload / send
1369
1584
 
1370
- ### <a name="multisession"></a>Support of multi-session
1585
+ ### <a id="multisession"></a>Support of multi-session
1371
1586
 
1372
- Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
1587
+ Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1373
1588
 
1374
1589
  * when agent=node :
1375
1590
 
@@ -1390,6 +1605,7 @@ shall be preferred.
1390
1605
 
1391
1606
  Multi-session spawn is done by <%=tool%>.
1392
1607
 
1608
+ When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
1393
1609
 
1394
1610
  ### Examples
1395
1611
 
@@ -1411,71 +1627,198 @@ Multi-session spawn is done by <%=tool%>.
1411
1627
  --ts=@json:'{"http_fallback":"force"}'
1412
1628
  ```
1413
1629
 
1414
- * Activate progress when not activated by default on server
1630
+ * Activate progress when not activated by default on server
1631
+
1632
+ ```
1633
+ --ts=@json:'{"precalculate_job_size":true}'
1634
+ ```
1635
+
1636
+ ## <a id="scheduling"></a>Lock for exclusive execution
1637
+
1638
+ In some conditions, it may be desirable to ensure that <%=tool%> is not executed several times in parallel.
1639
+
1640
+ For instance when <%=tool%> is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
1641
+
1642
+ * Executing instances may pile-up and kill the system
1643
+ * The same file may be transferred by multiple instances at the same time.
1644
+ * `preview` may generate the same files in multiple instances.
1645
+
1646
+ Usually the OS native scheduler already provides some sort of protection against parallel execution:
1647
+
1648
+ * The Windows scheduler does this by default
1649
+ * Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
1650
+
1651
+ ```
1652
+ /usr/bin/flock -w 0 /var/cron.lock ascli ...
1653
+ ```
1654
+
1655
+ <%=tool%> natively supports a locking mechanism with option `lock_port`.
1656
+ (Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
1657
+
1658
+ Example:
1659
+
1660
+ Run this same command in two separate terminals within less than 30 seconds:
1661
+
1662
+ ```
1663
+ <%=cmd%> config echo @ruby:'sleep(30)' --lock-port=12345
1664
+ ```
1665
+
1666
+ The first instance will sleep 30 seconds, the second one will immediately exit like this:
1667
+
1668
+ ```
1669
+ WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
1670
+ ```
1671
+
1672
+ ## "Proven&ccedil;ale"
1673
+
1674
+ `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
1675
+ , some are provided om shared libraries and must be activated. (e.g. using `trapd`)
1676
+
1677
+ The list of supported *PVCL* adapters can be retrieved with command:
1678
+
1679
+ ```
1680
+ <%=cmd%> conf ascp info
1681
+ +--------------------+-----------------------------------------------------------+
1682
+ | key | value |
1683
+ +--------------------+-----------------------------------------------------------+
1684
+ -----8<-----snip-----8<-----
1685
+ | product_name | IBM Aspera SDK |
1686
+ | product_version | 4.0.1.182389 |
1687
+ | process | pvcl |
1688
+ | shares | pvcl |
1689
+ | noded | pvcl |
1690
+ | faux | pvcl |
1691
+ | file | pvcl |
1692
+ | stdio | pvcl |
1693
+ | stdio-tar | pvcl |
1694
+ +--------------------+-----------------------------------------------------------+
1695
+ ```
1696
+
1697
+ Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
1698
+
1699
+ Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
1700
+
1701
+ The simplified format is:
1702
+
1703
+ ```
1704
+ <adapter>:///<sub file path>?<arg1>=<val1>&...
1705
+ ```
1706
+
1707
+ One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
1708
+
1709
+ ## <a id="faux_testing"></a>`faux:` for testing
1710
+
1711
+ This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>.
1712
+
1713
+ This adapter can be used to simulate a file or a directory.
1714
+
1715
+ To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
1716
+
1717
+ ```
1718
+ faux:///filename?filesize
1719
+ ```
1720
+
1721
+ where:
1722
+
1723
+ * `filename` is the name that will be assigned to the file on the destination
1724
+ * `filesize` is the number of bytes that will be sent (in decimal).
1725
+
1726
+ Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
1727
+
1728
+ For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
1729
+
1730
+ To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
1731
+
1732
+ ```
1733
+ faux:///dirname?<arg1>=<val1>&...
1734
+ ```
1735
+
1736
+ where:
1737
+
1738
+ * `dirname` is the folder name and can contain `/` to specify a subfolder.
1739
+ * supported arguments are:
1740
+
1741
+ <table>
1742
+ <tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
1743
+ <tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
1744
+ <tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
1745
+ <tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
1746
+ <tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
1747
+ <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
1748
+ <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
1749
+ </table>
1750
+
1751
+ The sequence parameter is applied as follows:
1415
1752
 
1416
- ```
1417
- --ts=@json:'{"precalculate_job_size":true}'
1418
- ```
1753
+ * If `seq` is `random` then each file size is:
1419
1754
 
1755
+ * size +/- (inc * rand())
1756
+ * Where rand is a random number between 0 and 1
1757
+ * Note that file size must not be negative, inc will be set to size if it is greater than size
1758
+ * Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
1420
1759
 
1760
+ * If `seq` is `sequential` then each file size is:
1421
1761
 
1422
- ## <a name="scheduling"></a>Scheduling an exclusive execution
1762
+ * `size + ((fileindex - 1) * inc)`
1763
+ * Where first file is index 1
1764
+ * So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
1765
+ * As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
1423
1766
 
1424
- It is possible to ensure that a given command is only run once at a time with parameter: `--lock-port=nnnn`. This is especially usefull when scheduling a command on a regular basis, for instance involving transfers, and a transfer may last longer than the execution period.
1767
+ Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
1425
1768
 
1426
- This opens a local TCP server port, and fails if this port is already used, providing a local lock.
1769
+ To discard data at the destination, the destination argument is set to `faux://` .
1427
1770
 
1428
- This option is used when the tools is executed automatically, for instance with "preview" generation.
1771
+ Examples:
1429
1772
 
1430
- Usually the OS native scheduler shall already provide some sort of such protection (windows scheduler has it natively, linux cron can leverage `flock`).
1773
+ * Upload 20 gibibytes of random data to file myfile to directory /Upload
1431
1774
 
1432
- ## <a name="commands"></a>Sample Commands
1775
+ ```
1776
+ <%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
1777
+ ```
1433
1778
 
1434
- A non complete list of commands used in unit tests:
1779
+ * Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
1435
1780
 
1436
1781
  ```
1437
- <%= File.read(ENV["INCL_COMMANDS"]) %>
1438
- ...and more
1782
+ <%=cmd%> server upload /tmp/sample --to-folder=faux://
1439
1783
  ```
1440
1784
 
1441
- ## <a name="usage"></a>Usage
1785
+ * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
1442
1786
 
1443
1787
  ```
1444
- $ <%=cmd%> -h
1445
- <%= File.read(ENV["INCL_USAGE"]) %>
1446
-
1788
+ <%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1447
1789
  ```
1448
1790
 
1449
- Note that actions and parameter values can be written in short form.
1450
-
1451
- # <a name="plugins"></a>Plugins: Application URL and Authentication
1791
+ ## <a id="commands"></a>Sample Commands
1452
1792
 
1453
- <%=tool%> comes with several Aspera application plugins.
1793
+ A non complete list of commands used in unit tests:
1454
1794
 
1455
- REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
1795
+ ```
1796
+ <%=File.read(ENV["INCL_COMMANDS"])%>
1797
+ ...and more
1798
+ ```
1456
1799
 
1457
- Those are using options:
1800
+ ## <a id="usage"></a>Usage
1458
1801
 
1459
- * url
1460
- * username
1461
- * password
1802
+ ```
1803
+ <%=cmd%> -h
1804
+ <%=File.read(ENV["INCL_USAGE"])%>
1462
1805
 
1463
- Those can be provided using command line, parameter set, env var, see section above.
1806
+ ```
1464
1807
 
1465
- Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
1808
+ Note that actions and parameter values can be written in short form.
1466
1809
 
1467
- # <a name="aoc"></a>Plugin: Aspera on Cloud
1810
+ # <a id="aoc"></a>Plugin: Aspera on Cloud
1468
1811
 
1469
1812
  Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
1470
1813
 
1471
1814
  It is recommended to use the wizard to set it up, but manual configuration is also possible.
1472
1815
 
1473
- ## <a name="aocwizard"></a>Configuration: using Wizard
1816
+ ## <a id="aocwizard"></a>Configuration: using Wizard
1474
1817
 
1475
1818
  <%=tool%> provides a configuration wizard. Here is a sample invocation :
1476
1819
 
1477
1820
  ```
1478
- $ <%=cmd%> config wizard
1821
+ <%=cmd%> config wizard
1479
1822
  option: url> https://myorg.ibmaspera.com
1480
1823
  Detected: Aspera on Cloud
1481
1824
  Preparing preset: aoc_myorg
@@ -1491,7 +1834,7 @@ Setting config preset as default for aspera
1491
1834
  saving config file
1492
1835
  Done.
1493
1836
  You can test with:
1494
- $ <%=cmd%> aoc user info show
1837
+ <%=cmd%> aoc user info show
1495
1838
  ```
1496
1839
 
1497
1840
  Optionally, it is possible to create a new organization-specific "integration".
@@ -1499,7 +1842,7 @@ For this, specify the option: `--use-generic-client=no`.
1499
1842
 
1500
1843
  This will guide you through the steps to create.
1501
1844
 
1502
- ## <a name="aocwizard"></a>Configuration: using manual setup
1845
+ ## <a id="aocmanual"></a>Configuration: using manual setup
1503
1846
 
1504
1847
  If you used the wizard (recommended): skip this section.
1505
1848
 
@@ -1511,7 +1854,7 @@ Several types of OAuth authentication are supported:
1511
1854
  * Web based authentication : authentication is made by user using a browser
1512
1855
  * URL Token : external users authentication with url tokens (public links)
1513
1856
 
1514
- The authentication method is controled by option `auth`.
1857
+ The authentication method is controlled by option `auth`.
1515
1858
 
1516
1859
  For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [<%=prst%> for Aspera on Cloud](#aocpreset).
1517
1860
 
@@ -1519,7 +1862,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
1519
1862
 
1520
1863
  In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.<%=tool%> saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
1521
1864
 
1522
- ### <a name="clientreg"></a>Optional: API Client Registration
1865
+ ### <a id="clientreg"></a>Optional: API Client Registration
1523
1866
 
1524
1867
  If you use the built-in client_id and client_secret, skip this and do not set them in next section.
1525
1868
 
@@ -1543,31 +1886,31 @@ Note: for web based authentication, <%=tool%> listens on a local port (e.g. spec
1543
1886
 
1544
1887
  Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
1545
1888
 
1546
- ### <a name="aocpreset"></a><%=prst%> for Aspera on Cloud
1889
+ ### <a id="aocpreset"></a><%=prst%> for Aspera on Cloud
1547
1890
 
1548
1891
  If you did not use the wizard, you can also manually create a <%=prst%> for <%=tool%> in its configuration file.
1549
1892
 
1550
1893
  Lets create an <%=prst%> called: `my_aoc_org` using `ask` interactive input (client info from previous step):
1551
1894
 
1552
1895
  ```
1553
- $ <%=cmd%> config id my_aoc_org ask url client_id client_secret
1896
+ <%=cmd%> config preset ask my_aoc_org url client_id client_secret
1554
1897
  option: url> https://myorg.ibmaspera.com/
1555
1898
  option: client_id> BJLPObQiFw
1556
1899
  option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
1557
1900
  updated: my_aoc_org
1558
1901
  ```
1559
1902
 
1560
- (This can also be done in one line using the command `config id my_aoc_org update --url=...`)
1903
+ (This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
1561
1904
 
1562
1905
  Define this <%=prst%> as default configuration for the `aspera` plugin:
1563
1906
 
1564
1907
  ```
1565
- $ <%=cmd%> config id default set aoc my_aoc_org
1908
+ <%=cmd%> config preset set default aoc my_aoc_org
1566
1909
  ```
1567
1910
 
1568
1911
  Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
1569
1912
 
1570
- ### <a name="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
1913
+ ### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
1571
1914
 
1572
1915
  For a Browser-less, Private Key-based authentication, use the following steps.
1573
1916
 
@@ -1582,13 +1925,13 @@ This can be done using any of the following method:
1582
1925
  * using the CLI:
1583
1926
 
1584
1927
  ```
1585
- $ <%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
1928
+ <%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
1586
1929
  ```
1587
1930
 
1588
1931
  * `ssh-keygen`:
1589
1932
 
1590
1933
  ```
1591
- $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1934
+ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1592
1935
  ```
1593
1936
 
1594
1937
  * `openssl`
@@ -1596,11 +1939,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
1596
1939
  (on some openssl implementation (mac) there is option: -nodes (no DES))
1597
1940
 
1598
1941
  ```
1599
- $ APIKEY=~/.aspera/<%=cmd%>/aocapikey
1600
- $ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
1601
- $ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
1602
- $ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
1603
- $ rm -f ${APIKEY}.protected
1942
+ APIKEY=~/.aspera/<%=cmd%>/aocapikey
1943
+ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
1944
+ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
1945
+ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
1946
+ rm -f ${APIKEY}.protected
1604
1947
  ```
1605
1948
 
1606
1949
  #### API Client JWT activation
@@ -1619,13 +1962,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
1619
1962
  * Using command line
1620
1963
 
1621
1964
  ```
1622
- $ <%=cmd%> aoc admin res client list
1965
+ <%=cmd%> aoc admin res client list
1623
1966
  :............:.........:
1624
1967
  : id : name :
1625
1968
  :............:.........:
1626
1969
  : BJLPObQiFw : <%=cmd%> :
1627
1970
  :............:.........:
1628
- $ <%=cmd%> aoc admin res client --id=BJLPObQiFw modify @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
1971
+ <%=cmd%> aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
1629
1972
  modified
1630
1973
  ```
1631
1974
 
@@ -1646,14 +1989,14 @@ open the previously generated public key located here: `$HOME/.aspera/<%=cmd%>/a
1646
1989
  * Using command line
1647
1990
 
1648
1991
  ```
1649
- $ <%=cmd%> aoc admin res user list
1992
+ <%=cmd%> aoc admin res user list
1650
1993
  :........:................:
1651
1994
  : id : name :
1652
1995
  :........:................:
1653
1996
  : 109952 : Tech Support :
1654
1997
  : 109951 : LAURENT MARTIN :
1655
1998
  :........:................:
1656
- $ <%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
1999
+ <%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
1657
2000
  modified
1658
2001
  ```
1659
2002
 
@@ -1661,16 +2004,16 @@ Note: the `aspera user info show` command can be used to verify modifications.
1661
2004
 
1662
2005
  ### <%=prst%> modification for JWT
1663
2006
 
1664
- To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the folowing:
2007
+ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the following:
1665
2008
 
1666
2009
  * change auth method to JWT
1667
2010
  * provide location of private key
1668
- * provide username to login as (OAuthg "subject")
2011
+ * provide username to login as (OAuth "subject")
1669
2012
 
1670
2013
  Execute:
1671
2014
 
1672
2015
  ```
1673
- $ <%=cmd%> config id my_aoc_org update --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
2016
+ <%=cmd%> config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
1674
2017
  ```
1675
2018
 
1676
2019
  Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
@@ -1678,73 +2021,157 @@ Note: the private key argument represents the actual PEM string. In order to rea
1678
2021
  After this last step, commands do not require web login anymore.
1679
2022
 
1680
2023
 
1681
- ### <a name="aocfirst"></a>First Use
2024
+ ### <a id="aocfirst"></a>First Use
1682
2025
 
1683
2026
  Once client has been registered and <%=prst%> created: <%=tool%> can be used:
1684
2027
 
1685
2028
  ```
1686
- $ <%=cmd%> aoc files br /
2029
+ <%=cmd%> aoc files br /
1687
2030
  Current Workspace: Default Workspace (default)
1688
2031
  empty
1689
2032
  ```
1690
2033
 
1691
2034
 
1692
- ### Administration
2035
+ ## Administration
1693
2036
 
1694
2037
  The `admin` command allows several administrative tasks (and require admin privilege).
1695
2038
 
1696
2039
  It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
1697
2040
 
1698
- Bulk operations are possible using option `bulk` (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2041
+ ### Bulk creation and deletion of resource
2042
+
2043
+ Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
2044
+ In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2045
+
2046
+ ### Listing resources
2047
+
2048
+ The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
2049
+
2050
+ The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
2051
+
2052
+ The following parameters are supported:
1699
2053
 
1700
- To get more resources when doing request add:
2054
+ * `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
2055
+ * `sort`: name of fields to sort results, prefix with `-` for reverse order.
2056
+ * `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
2057
+ * `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
2058
+ * `page` : native api parameter, in general do not use (added by
2059
+ * `per_page` : native api parameter, number of items par api call, in general do not use
2060
+ * Other specific parameters depending on resource type.
2061
+
2062
+ Both `max` and `pmax` are processed internally in <%=tool%>, not included in actual API call and limit the number of successive pages requested to API. <%=tool%> will return all values using paging if not provided.
2063
+
2064
+ Other parameters are directly sent as parameters to the GET request on API.
2065
+
2066
+ `page` and `per_page` are normally added by <%=tool%> to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
2067
+
2068
+ `q` and `sort` are available on most resource types.
2069
+
2070
+ Other parameters depend on the type of entity (refer to AoC API).
2071
+
2072
+ Examples:
2073
+
2074
+ * List users with `laurent` in name:
2075
+
2076
+ ```
2077
+ <%=cmd%> aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
2078
+ ```
2079
+
2080
+ * List users who logged-in before a date:
2081
+
2082
+ ```
2083
+ <%=cmd%> aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2084
+ ```
2085
+
2086
+ * List external users and sort in reverse alphabetical order using name:
1701
2087
 
1702
2088
  ```
1703
- --query=@json:'{"per_page":10000}'
2089
+ <%=cmd%> aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
1704
2090
  ```
1705
2091
 
1706
- other query parameters can be used:
2092
+ Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
2093
+
2094
+ Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
2095
+
2096
+ ### <a id="res_select"></a>Selecting a resource
2097
+
2098
+ Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
2099
+
2100
+ To execute an action on a specific resource, select it using one of those methods:
2101
+
2102
+ * *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
2103
+ * give name on command line *after the action*: `aoc admin res node show name abc`
2104
+ * provide option `id` : `aoc admin res node show --id=123`
2105
+ * provide option `name` : `aoc admin res node show --name=abc`
2106
+
2107
+ ### Access Key secrets
2108
+
2109
+ In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
2110
+ It is usually provided using the `secret` option.
2111
+ For example in a command like:
2112
+
1707
2113
  ```
1708
- --query=@json:'{"member_of_any_workspace":true}'
1709
- --query=@json:'{"q":"laurent"}'
2114
+ <%=cmd%> aoc admin res node --id=123 --secret="secret1" v3 info
1710
2115
  ```
1711
2116
 
1712
- Refer to the AoC API for full list of query parameters.
2117
+ It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
1713
2118
 
1714
- #### Access Key secrets
2119
+ ### Activity
1715
2120
 
1716
- In order to access some administrative actions on "nodes" (in fact, access keys), the associated
1717
- secret is required, it is usually provided using the `secret` option. For example in a command like:
2121
+ The activity app can be queried with:
1718
2122
 
1719
2123
  ```
1720
- $ <%=cmd%> aoc admin res node --id="access_key1" --secret="secret1" v3 info
2124
+ <%=cmd%> aoc admin analytics transfers
1721
2125
  ```
1722
2126
 
1723
- It is also possible to provide a set of secrets used on a regular basis. This can be done using the `secrets` option. The value provided shall be a Hash, where keys are access key ids, and values are the associated secrets.
2127
+ It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
1724
2128
 
1725
- First choose a repository name, for example `my_secrets`, and populate it like this:
2129
+ `mytemplate.erb`:
1726
2130
 
1727
2131
  ```
1728
- $ <%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
1729
- $ <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
1730
- $ <%=cmd%> conf id default get config
1731
- "cli_default"
2132
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2133
+ To: <<%='<'%>%=ev['user_email']%>>
2134
+ Subject: <%='<'%>%=ev['files_completed']%> files received
2135
+
2136
+ Dear <%='<'%>%=ev[:user_email.to_s]%>,
2137
+ We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2138
+ <%='<'%>%=ev['content']%>
2139
+
2140
+ Thank you.
1732
2141
  ```
2142
+ The environment provided contains the following additional variable:
2143
+
2144
+ * ev : all details on the transfer event
1733
2145
 
1734
- Here above, one already has set a `config` global preset to preset `cli_default` (refer to earlier in documentation), then the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
2146
+ Example:
1735
2147
 
1736
2148
  ```
1737
- $ <%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
2149
+ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2150
+ --query=@json:'{"status":"completed","direction":"receive"}' \
2151
+ --notif-to=active --notif-template=@file:mytemplate.erb
1738
2152
  ```
1739
2153
 
1740
- A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
2154
+ Options:
2155
+
2156
+ * `once_only` keep track of last date it was called, so next call will get only new events
2157
+ * `query` filter (on API call)
2158
+ * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2159
+
2160
+ Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
1741
2161
 
1742
- #### Examples
2162
+ ### Transfer: Using specific transfer ports
1743
2163
 
1744
- * Bulk creation
2164
+ By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
2165
+ The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2166
+
2167
+ ### Using ATS
2168
+
2169
+ Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
2170
+
2171
+ ### Example: Bulk creation of users
1745
2172
 
1746
2173
  ```
1747
- $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
2174
+ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
1748
2175
  :.......:.........:
1749
2176
  : id : status :
1750
2177
  :.......:.........:
@@ -1753,20 +2180,20 @@ $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exa
1753
2180
  :.......:.........:
1754
2181
  ```
1755
2182
 
1756
- * Find with filter and delete
2183
+ ### Example: Find with filter and delete
1757
2184
 
1758
2185
  ```
1759
- $ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
2186
+ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
1760
2187
  :.......:........................:
1761
2188
  : id : email :
1762
2189
  :.......:........................:
1763
2190
  : 98398 : dummyuser1@example.com :
1764
2191
  : 98399 : dummyuser2@example.com :
1765
2192
  :.......:........................:
1766
- $ thelist=$(echo $(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email --field=id --format=csv)|tr ' ' ,)
1767
- $ echo $thelist
1768
- 98398,98399
1769
- $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
2193
+ thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
2194
+ echo $thelist
2195
+ ["113501","354061"]
2196
+ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
1770
2197
  :.......:.........:
1771
2198
  : id : status :
1772
2199
  :.......:.........:
@@ -1775,10 +2202,18 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
1775
2202
  :.......:.........:
1776
2203
  ```
1777
2204
 
1778
- * Display current user's workspaces
2205
+ ### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
2206
+
2207
+ ```
2208
+ <%=cmd%> aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
2209
+ ```
2210
+
2211
+ To delete them use the same method as before
2212
+
2213
+ ### Example: Display current user's workspaces
1779
2214
 
1780
2215
  ```
1781
- $ <%=cmd%> aoc user workspaces
2216
+ <%=cmd%> aoc user workspaces
1782
2217
  :......:............................:
1783
2218
  : id : name :
1784
2219
  :......:............................:
@@ -1788,42 +2223,45 @@ $ <%=cmd%> aoc user workspaces
1788
2223
  :......:............................:
1789
2224
  ```
1790
2225
 
1791
- * Create a sub access key in a "node"
2226
+ ### Example: Create a sub access key in a "node"
1792
2227
 
1793
2228
  Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
1794
2229
 
1795
2230
  ```
1796
- $ <%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
2231
+ <%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
1797
2232
  ```
1798
2233
 
1799
- * Display transfer events (ops/transfer)
2234
+ ### Example: Display transfer events (ops/transfer)
1800
2235
 
1801
2236
  ```
1802
- $ <%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2237
+ <%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
1803
2238
  ```
1804
2239
 
1805
- # page=1&per_page=10&q=type:(file_upload+OR+file_delete+OR+file_download+OR+file_rename+OR+folder_create+OR+folder_delete+OR+folder_share+OR+folder_share_via_public_link)&sort=-date
1806
- #events=@api_files.read('events',{'q'=>'type:(file_upload OR file_download)'})[:data]
1807
- # can add filters: tag=aspera.files.package_id%3DLA8OU3p8w
1808
- #'tag'=>'aspera.files.package_id%3DJvbl0w-5A'
2240
+ Examples of query (TODO: cleanup):
2241
+
2242
+ ```
2243
+ {"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
2244
+
2245
+ {"tag":"aspera.files.package_id=LA8OU3p8w"}
2246
+
1809
2247
  # filter= 'id', 'short_summary', or 'summary'
1810
2248
  # count=nnn
1811
2249
  # tag=x.y.z%3Dvalue
1812
2250
  # iteration_token=nnn
1813
2251
  # after_time=2016-05-01T23:53:09Z
1814
2252
  # active_only=true|false
2253
+ ```
1815
2254
 
1816
-
1817
- * Display node events (events)
2255
+ ### Example: Display node events (events)
1818
2256
 
1819
2257
  ```
1820
- $ <%=cmd%> aoc admin res node --secret=_secret_ v3 events
2258
+ <%=cmd%> aoc admin res node --secret=_secret_ v3 events
1821
2259
  ```
1822
2260
 
1823
- * display members of a workspace
2261
+ ### Example: Display members of a workspace
1824
2262
 
1825
2263
  ```
1826
- $ <%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2264
+ <%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
1827
2265
  :.............:.........:..................................:
1828
2266
  : member_type : manager : member.email :
1829
2267
  :.............:.........:..................................:
@@ -1842,29 +2280,29 @@ other query parameters:
1842
2280
  {"workspace_membership_through":true,"include_indirect":true}
1843
2281
  ```
1844
2282
 
1845
- * <a name="aoc_sample_member"></a>add all members of a workspace to another workspace
2283
+ ### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
1846
2284
 
1847
- a- get id of first workspace
2285
+ a- Get id of first workspace
1848
2286
 
1849
2287
  ```
1850
2288
  WS1='First Workspace'
1851
2289
  WS1ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
1852
2290
  ```
1853
2291
 
1854
- b- get id of second workspace
2292
+ b- Get id of second workspace
1855
2293
 
1856
2294
  ```
1857
2295
  WS2='Second Workspace'
1858
2296
  WS2ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
1859
2297
  ```
1860
2298
 
1861
- c- extract membership information and change workspace id
2299
+ c- Extract membership information
1862
2300
 
1863
2301
  ```
1864
- $ <%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"per_page":10000,"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2302
+ <%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
1865
2303
  ```
1866
2304
 
1867
- d- convert to creation data for second workspace:
2305
+ d- Convert to creation data for second workspace:
1868
2306
 
1869
2307
  ```
1870
2308
  grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
@@ -1876,16 +2314,16 @@ or, using jq:
1876
2314
  jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
1877
2315
  ```
1878
2316
 
1879
- e- add members to second workspace
2317
+ e- Add members to second workspace
1880
2318
 
1881
2319
  ```
1882
- $ <%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2320
+ <%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
1883
2321
  ```
1884
2322
 
1885
- * get users who did not log since a date
2323
+ ### Example: Get users who did not log since a date
1886
2324
 
1887
2325
  ```
1888
- $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"per_page":10000,"q":"last_login_at:<2018-05-28"}'
2326
+ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
1889
2327
  :...............................:
1890
2328
  : email :
1891
2329
  :...............................:
@@ -1894,20 +2332,20 @@ $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"per_page":100
1894
2332
  :...............................:
1895
2333
  ```
1896
2334
 
1897
- * list "Limited" users
2335
+ ### Example: List "Limited" users
1898
2336
 
1899
2337
  ```
1900
- $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"per_page":10000}' --select=@json:'{"member_of_any_workspace":false}'
2338
+ <%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
1901
2339
  ```
1902
2340
 
1903
- * Perform a multi Gbps transfer between two remote shared folders
2341
+ ### Example: Perform a multi Gbps transfer between two remote shared folders
1904
2342
 
1905
- In this example, a user has access to a workspace where two shared folders are located on differente sites, e.g. different cloud regions.
2343
+ In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
1906
2344
 
1907
2345
  First, setup the environment (skip if already done)
1908
2346
 
1909
2347
  ```
1910
- $ <%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
2348
+ <%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
1911
2349
  Detected: Aspera on Cloud
1912
2350
  Preparing preset: aoc_sedemo
1913
2351
  Using existing key:
@@ -1926,7 +2364,7 @@ Setting config preset as default for aspera
1926
2364
  saving config file
1927
2365
  Done.
1928
2366
  You can test with:
1929
- $ <%=cmd%> aoc user info show
2367
+ <%=cmd%> aoc user info show
1930
2368
  ```
1931
2369
 
1932
2370
  This creates the option preset "aoc_&lt;org name&gt;" to allow seamless command line access and sets it as default for aspera on cloud.
@@ -1936,19 +2374,20 @@ Then, create two shared folders located in two regions, in your files home, in a
1936
2374
  Then, transfer between those:
1937
2375
 
1938
2376
  ```
1939
- $ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2377
+ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
1940
2378
  ```
1941
2379
 
1942
- * create registration key to register a node
2380
+ ### Example: create registration key to register a node
2381
+
1943
2382
  ```
1944
- $ <%=cmd%> aoc admin res admin/client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2383
+ <%=cmd%> aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
1945
2384
  jfqslfdjlfdjfhdjklqfhdkl
1946
2385
  ```
1947
2386
 
1948
- * delete all registration keys
2387
+ ### Example: delete all registration keys
1949
2388
 
1950
2389
  ```
1951
- $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc admin res admin/client delete --bulk=yes --id=@lines:@stdin:
2390
+ <%=cmd%> aoc admin res client list --fields=id --format=csv|<%=cmd%> aoc admin res client delete --bulk=yes --id=@lines:@stdin:
1952
2391
  +-----+---------+
1953
2392
  | id | status |
1954
2393
  +-----+---------+
@@ -1959,143 +2398,215 @@ $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc
1959
2398
  +-----+---------+
1960
2399
  ```
1961
2400
 
1962
- ## Shared folders
2401
+ ### Example: Create a node
1963
2402
 
1964
- * list shared folders in node
2403
+ AoC nodes as actually composed with two related entities:
2404
+
2405
+ * An access key created on the Transfer Server (HSTS/ATS)
2406
+ * a `node` resource in the AoC application.
2407
+
2408
+ The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
2409
+ Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
2410
+
2411
+ So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
2412
+
2413
+ * create the access key on ATS
1965
2414
 
1966
2415
  ```
1967
- $ <%=cmd%> aoc admin res node --id=8669 shared_folders
2416
+ <%=cmd%> aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
1968
2417
  ```
1969
2418
 
1970
- * list shared folders in workspace
2419
+ Take a note of the randomly generated `id` and `secret`.
2420
+
2421
+ * Retrieve the ATS node address
1971
2422
 
1972
2423
  ```
1973
- $ <%=cmd%> aoc admin res workspace --id=10818 shared_folders
2424
+ <%=cmd%> aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
1974
2425
  ```
1975
2426
 
1976
- * list members of shared folder
2427
+ * Create the node entity
2428
+
2429
+ ```
2430
+ <%=cmd%> aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
2431
+ ```
2432
+
2433
+ Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
2434
+
2435
+ ### Example: List packages in a given shared inbox
2436
+
2437
+ First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2438
+ (To find out available filters, consult the API definition, or use the web interface in developer mode).
2439
+
2440
+ Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
1977
2441
 
1978
2442
  ```
1979
- $ <%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
2443
+ shbxid=$(<%=cmd%> aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
2444
+
2445
+ <%=cmd%> aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
1980
2446
  ```
1981
2447
 
1982
- ## Send a Package
2448
+ ## Packages
2449
+
2450
+ The webmail-like application.
2451
+
2452
+ ### Send a Package
1983
2453
 
1984
2454
  Send a package:
1985
2455
 
1986
2456
  ```
1987
- $ <%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
2457
+ <%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
1988
2458
  ```
1989
2459
 
1990
2460
  Notes:
1991
2461
 
1992
- * the `value` parameter can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package to find attributes.
1993
- * to provide the list of recipients, use fields: "recipients" and/or "bcc_recipients". <%=cmd%> will resolve the list of email addresses to expected user ids.
1994
- * a recipîent can be a shared inbox, in this case just use the name of the shared inbox as recipient.
1995
- * If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
1996
- * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account.
2462
+ * The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
2463
+ * List allowed shared inbox destinations with: `<%=cmd%> aoc user shared_inboxes`
2464
+ * Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
2465
+ * Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
2466
+ * or just names: `"recipients":[{"The Dest"}]` . <%=cmd%> will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
2467
+ * If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
2468
+ * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
1997
2469
  * if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
1998
2470
 
1999
2471
  Examples:
2000
2472
 
2473
+ * Send a package with one file to two users, using their email
2474
+
2475
+ ```
2476
+ <%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
2477
+ ```
2478
+
2479
+ * Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
2480
+
2481
+ ```
2482
+ <%=cmd%> aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
2483
+ ```
2484
+
2485
+ * Send a package with one file to a shared inbox (by name) with metadata
2486
+
2001
2487
  ```
2002
- $ <%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' --sources=@args my_file.dat
2003
- $ <%=cmd%> aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
2004
- $ <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2488
+ <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2005
2489
  ```
2006
2490
 
2007
- ## <a name="aoccargo"></a>Receive new packages only
2491
+ ### <a id="aoccargo"></a>Receive new packages only (Cargo)
2008
2492
 
2009
2493
  It is possible to automatically download new packages, like using Aspera Cargo:
2010
2494
 
2011
2495
  ```
2012
- $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2496
+ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2013
2497
  ```
2014
2498
 
2015
2499
  * `--id=ALL` (case sensitive) will download all packages
2016
- * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
2017
- * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
2500
+ * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
2501
+ * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
2018
2502
 
2019
- Typically, one would regularly execute this command on a regular basis, using the method of your choice:
2503
+ Typically, one would execute this command on a regular basis, using the method of your choice:
2020
2504
 
2021
2505
  * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2022
2506
  * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
2023
2507
  * etc...
2024
2508
 
2025
- ## Download Files
2026
-
2027
- Download of files is straightforward with a specific syntax for the `aspera files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2028
-
2029
- * if only one source is provided, it is downloaded
2030
- * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2031
-
2032
- ## Find Files
2033
-
2034
- The command `aspera files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2509
+ ## Files
2035
2510
 
2036
- The expression can be of 3 formats:
2511
+ Folder sharing app.
2037
2512
 
2038
- * empty (default) : all files, equivalent to: `exec:true`
2039
- * not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
2513
+ ### Download Files
2040
2514
 
2041
- For instance, to find files with a special extension, use `--value='\.myext$'`
2515
+ Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2042
2516
 
2043
- * starting with `exec:` : the ruby code after the prefix is executed for each entry found. the entry variable name is `f`. the file is displayed if the result is true;
2517
+ * if only one source is provided, it is downloaded
2518
+ * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2044
2519
 
2045
- Examples of expressions: (think to prefix with `exec:` and put in single quotes using bash)
2520
+ ### Shared folders
2046
2521
 
2047
- * find files more recent than 100 days
2522
+ * list shared folders in node
2048
2523
 
2049
2524
  ```
2050
- f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2525
+ <%=cmd%> aoc admin res node --id=8669 shared_folders
2051
2526
  ```
2052
2527
 
2053
- * expression to find files older than 1 year on a given node and store in file list
2528
+ * list shared folders in workspace
2054
2529
 
2055
2530
  ```
2056
- $ <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2531
+ <%=cmd%> aoc admin res workspace --id=10818 shared_folders
2057
2532
  ```
2058
2533
 
2059
- * delete the files, one by one
2534
+ * list members of shared folder
2060
2535
 
2061
2536
  ```
2062
- $ cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2537
+ <%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
2063
2538
  ```
2064
2539
 
2065
- * delete the files in bulk
2540
+ ### Cross Organization transfers
2541
+
2542
+ It is possible to transfer files directly between organizations without having to first download locally and then upload...
2543
+
2544
+ Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
2545
+
2546
+ Procedure to send a file from org1 to org2:
2547
+
2548
+ * Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2549
+ * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2550
+ * Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
2551
+ * Check that access works and locate the destination folder `mydestfolder`
2552
+ * execute the following:
2066
2553
 
2067
2554
  ```
2068
- cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
2555
+ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2069
2556
  ```
2070
2557
 
2071
- ## Activity
2558
+ Explanation:
2072
2559
 
2073
- The activity app can be queried with:
2560
+ * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2561
+ * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2562
+ * `--format=json` format the output in JSON (instead of default text table)
2563
+ * `--display=data` display only the result, and remove other information, such as workspace name
2564
+ * `|` the standard output of the first command is fed into the second one
2565
+ * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2566
+ * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2567
+ * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
2568
+ * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2569
+
2570
+ ### Find Files
2571
+
2572
+ The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2573
+
2574
+ The expression can be of 3 formats:
2575
+
2576
+ * empty (default) : all files, equivalent to value: `exec:true`
2577
+ * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2578
+
2579
+ For instance, to find files with a special extension, use `--value='\.myext$'`
2580
+
2581
+ * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2582
+
2583
+ Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2584
+
2585
+ * Find files more recent than 100 days
2074
2586
 
2075
2587
  ```
2076
- $ <%=cmd%> aoc admin analytics transfers
2588
+ f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2077
2589
  ```
2078
2590
 
2079
- It can also support filters and send notification email with a template:
2591
+ * Find files older than 1 year on a given node and store in file list
2080
2592
 
2081
2593
  ```
2082
- $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=123455 \
2083
- --query=@json:'{"status":"completed","direction":"receive"}' \
2084
- --notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
2594
+ <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2085
2595
  ```
2086
2596
 
2087
- * `once_only` keep track of last date it was called, so next call will get only new events
2088
- * `query` filter (on API call)
2089
- * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2090
-
2091
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
2597
+ * Delete the files, one by one
2092
2598
 
2093
- ## Using specific transfer ports
2599
+ ```
2600
+ cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2601
+ ```
2094
2602
 
2095
- By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that. The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2603
+ * Delete the files in bulk
2096
2604
 
2605
+ ```
2606
+ cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
2607
+ ```
2097
2608
 
2098
- # Plugin: Aspera Transfer Service
2609
+ # <a id="ats"></a>Plugin: Aspera Transfer Service
2099
2610
 
2100
2611
  ATS is usable either :
2101
2612
 
@@ -2105,10 +2616,13 @@ ATS is usable either :
2105
2616
 
2106
2617
  ## IBM Cloud ATS : creation of api key
2107
2618
 
2619
+ This section is about using ATS with an IBM cloud subscription.
2620
+ If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
2621
+
2108
2622
  First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
2109
2623
 
2110
2624
  ```
2111
- $ ibmcloud iam api-key-create mykeyname -d 'my sample key'
2625
+ ibmcloud iam api-key-create mykeyname -d 'my sample key'
2112
2626
  OK
2113
2627
  API key mykeyname was created
2114
2628
 
@@ -2127,48 +2641,47 @@ References:
2127
2641
  * [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
2128
2642
  * [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
2129
2643
 
2130
-
2131
2644
  Then, to register the key by default for the ats plugin, create a preset. Execute:
2132
2645
 
2133
2646
  ```
2134
- $ <%=cmd%> config id my_ibm_ats update --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
2135
- $ <%=cmd%> config id default set ats my_ibm_ats
2136
- $ <%=cmd%> ats api_key instances
2647
+ <%=cmd%> config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
2648
+ <%=cmd%> config preset set default ats my_ibm_ats
2649
+ <%=cmd%> ats api_key instances
2137
2650
  +--------------------------------------+
2138
2651
  | instance |
2139
2652
  +--------------------------------------+
2140
2653
  | aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
2141
2654
  +--------------------------------------+
2142
- $ <%=cmd%> config id my_ibm_ats update --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
2143
- $ <%=cmd%> ats api_key create
2655
+ <%=cmd%> config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
2656
+ <%=cmd%> ats api_key create
2144
2657
  +--------+----------------------------------------------+
2145
2658
  | key | value |
2146
2659
  +--------+----------------------------------------------+
2147
2660
  | id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
2148
2661
  | secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
2149
2662
  +--------+----------------------------------------------+
2150
- $ <%=cmd%> config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2663
+ <%=cmd%> config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2151
2664
  ```
2152
2665
 
2153
2666
  ## Examples
2154
2667
 
2155
- Example: create access key on softlayer:
2668
+ Example: create access key on IBM Cloud (softlayer):
2156
2669
 
2157
2670
  ```
2158
- $ <%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
2671
+ <%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
2159
2672
  ```
2160
2673
 
2161
2674
  Example: create access key on AWS:
2162
2675
 
2163
2676
  ```
2164
- $ <%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
2677
+ <%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
2165
2678
 
2166
2679
  ```
2167
2680
 
2168
2681
  Example: create access key on Azure SAS:
2169
2682
 
2170
2683
  ```
2171
- $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
2684
+ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
2172
2685
 
2173
2686
  ```
2174
2687
 
@@ -2177,7 +2690,7 @@ $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{
2177
2690
  Example: create access key on Azure:
2178
2691
 
2179
2692
  ```
2180
- $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
2693
+ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
2181
2694
 
2182
2695
  ```
2183
2696
 
@@ -2187,85 +2700,114 @@ delete all my access keys:
2187
2700
  for k in $(<%=cmd%> ats access_key list --field=id --format=csv);do <%=cmd%> ats access_key id $k delete;done
2188
2701
  ```
2189
2702
 
2703
+ The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
2704
+
2190
2705
  # Plugin: IBM Aspera High Speed Transfer Server (transfer)
2191
2706
 
2192
- This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
2707
+ This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
2708
+ It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
2709
+ Modern mode is to use the node API and transfer tokens.
2193
2710
 
2194
2711
  ## Authentication
2195
2712
 
2196
2713
  Both password and SSH keys auth are supported.
2197
2714
 
2198
- Multiple SSH key paths can be provided. The value of the parameter `ssh_keys` can be a single value or an array. Each value is a path to a private key and is expanded ("~" is replaced with the user's home folder).
2715
+ If username is not provided, the default transfer user `xfer` is used.
2716
+
2717
+ If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
2718
+
2719
+ ```
2720
+ <%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2721
+ ```
2722
+
2723
+ Multiple SSH key paths can be provided.
2724
+ The value of the parameter `ssh_keys` can be a single value or an array.
2725
+ Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
2199
2726
 
2200
2727
  Examples:
2201
2728
 
2202
2729
  ```
2203
- $ <%=cmd%> server --ssh-keys=~/.ssh/id_rsa
2204
- $ <%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
2205
- $ <%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
2730
+ <%=cmd%> server --ssh-keys=~/.ssh/id_rsa
2731
+ <%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
2732
+ <%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
2206
2733
  ```
2207
2734
 
2208
- The underlying ssh library `net::ssh` provides several options that may be used depending on environment. By default the ssh library expect that an ssh-agent is running.
2735
+ The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
2736
+ By default the ssh library expect that an ssh-agent is running.
2209
2737
 
2210
- If you get an error message such as:
2738
+ On Linux, if you get an error message such as:
2211
2739
 
2212
2740
  ```
2213
- [Linux]
2214
2741
  ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
2215
2742
  ```
2216
2743
 
2217
- or
2744
+ or on Windows:
2218
2745
 
2219
2746
  ```
2220
- [Windows]
2221
2747
  ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
2222
2748
  ```
2223
2749
 
2224
- This means that you dont have such an ssh agent running:
2750
+ This means that you don't have such an ssh agent running, then:
2225
2751
 
2226
2752
  * check env var: `SSH_AGENT_SOCK`
2227
- * check if the key is protected with a passphrase
2753
+ * check if the ssh key is protected with a passphrase
2228
2754
  * [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
2229
- * To diable use of `ssh-agent`, use the option `ssh_option` like this (or set in preset):
2755
+ * To disable use of `ssh-agent`, use the option `ssh_option` like this:
2230
2756
 
2231
2757
  ```
2232
- $ <%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
2758
+ <%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
2233
2759
  ```
2234
2760
 
2235
- This can also be set as default using a preset
2761
+ This can also be set as default using a global preset.
2236
2762
 
2237
2763
  ## Example
2238
2764
 
2239
- One can test the "server" application using the well known demo server:
2765
+ One can test the `server` application using the well known demo server:
2240
2766
 
2241
2767
  ```
2242
- $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
2243
- $ <%=cmd%> config id default set server aspera_demo_server
2244
- $ <%=cmd%> server browse /aspera-test-dir-large
2245
- $ <%=cmd%> server download /aspera-test-dir-large/200MB
2768
+ <%=cmd%> config initdemo
2769
+ <%=cmd%> server browse /aspera-test-dir-large
2770
+ <%=cmd%> server download /aspera-test-dir-large/200MB
2246
2771
  ```
2247
2772
 
2248
- This creates a <%=prst%> "aspera_demo_server" and set it as default for application "server"
2249
-
2773
+ `initdemo` creates a <%=prst%> `demoserver` and set it as default for plugin `server`.
2250
2774
 
2251
2775
  # Plugin: IBM Aspera High Speed Transfer Server (node)
2252
2776
 
2253
2777
  This plugin gives access to capabilities provided by HSTS node API.
2254
2778
 
2255
- ## Simple Operations
2779
+ ## File Operations
2256
2780
 
2257
2781
  It is possible to:
2258
2782
  * browse
2259
2783
  * transfer (upload / download)
2260
2784
  * ...
2261
2785
 
2786
+ For transfers, it is possible to control how transfer is authorized using option: `token_type`:
2787
+
2788
+ * `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
2789
+ * `basic` : transfer spec is created like this:
2790
+
2791
+ ```
2792
+ {
2793
+ "remote_host": address of node url,
2794
+ "remote_user": "xfer",
2795
+ "ssh_port": 33001,
2796
+ "token": "Basic <base 64 encoded user/pass>",
2797
+ "direction": send/recv
2798
+ }
2799
+ ```
2800
+
2801
+ * `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
2802
+
2262
2803
  ## Central
2263
2804
 
2264
- The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.
2805
+ The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
2265
2806
 
2266
2807
  Filtering can be applied:
2808
+
2267
2809
  ```
2268
- $ <%=cmd%> node central file list
2810
+ <%=cmd%> node central file list
2269
2811
  ```
2270
2812
 
2271
2813
  by providing the `validator` option, offline transfer validation can be done.
@@ -2277,7 +2819,7 @@ It is possible to start a FASPStream session using the node API:
2277
2819
  Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
2278
2820
 
2279
2821
  ```
2280
- $ <%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
2822
+ <%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
2281
2823
  ```
2282
2824
 
2283
2825
  ## Watchfolder
@@ -2289,11 +2831,10 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
2289
2831
  * Start watchd and watchfolderd services running as a system user having access to files
2290
2832
  * configure a watchfolder to define automated transfers
2291
2833
 
2292
-
2293
2834
  ```
2294
- $ <%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2295
- $ <%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
2296
- $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
2835
+ <%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
2836
+ <%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
2837
+ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
2297
2838
  ```
2298
2839
 
2299
2840
  ## Out of Transfer File Validation
@@ -2301,13 +2842,13 @@ $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watc
2301
2842
  Follow the Aspera Transfer Server configuration to activate this feature.
2302
2843
 
2303
2844
  ```
2304
- $ <%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
2845
+ <%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
2305
2846
  :..............:..............:............:......................................:
2306
2847
  : session_uuid : file_id : status : path :
2307
2848
  :..............:..............:............:......................................:
2308
2849
  : 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
2309
2850
  :..............:..............:............:......................................:
2310
- $ <%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
2851
+ <%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
2311
2852
  updated
2312
2853
  ```
2313
2854
 
@@ -2323,7 +2864,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
2323
2864
  Then execute the following command:
2324
2865
 
2325
2866
  ```
2326
- $ <%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
2867
+ <%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
2327
2868
  ```
2328
2869
 
2329
2870
  This will get transfer information from the SHOD instance and tell the Azure ATS instance
@@ -2332,18 +2873,22 @@ to download files.
2332
2873
  ## Create access key
2333
2874
 
2334
2875
  ```
2335
- $ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2876
+ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2336
2877
  ```
2337
2878
 
2338
2879
  # Plugin: IBM Aspera Faspex5
2339
2880
 
2340
2881
  3 authentication methods are supported:
2341
2882
 
2342
- * boot
2343
- * web
2344
2883
  * jwt
2884
+ * web
2885
+ * boot
2886
+
2887
+ For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
2345
2888
 
2346
- For boot method:
2889
+ For web method, create an API client in Faspex, and use: --auth=web
2890
+
2891
+ For boot method: (will be removed in future)
2347
2892
 
2348
2893
  * open a browser
2349
2894
  * start developer mode
@@ -2353,45 +2898,77 @@ For boot method:
2353
2898
  Use it as password and use `--auth=boot`.
2354
2899
 
2355
2900
  ```
2356
- $ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2901
+ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2357
2902
  ```
2358
2903
 
2359
- For web method, create an API client in Faspex, and use: --auth=web
2360
-
2361
- For JWT, create an API client in Faspex with jwt supporot, and use: --auth=jwt
2362
- as of beta£3 this does not allow regular users.
2363
-
2364
2904
  Ready to use Faspex5 with CLI.
2365
2905
 
2366
- Once the graphical registration form exist, ther bootstrap method can be removed.
2367
-
2368
2906
  # Plugin: IBM Aspera Faspex (4.x)
2369
2907
 
2370
2908
  Notes:
2371
2909
 
2372
- * the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2373
- * for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
2910
+ * The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2911
+ * For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
2912
+
2913
+ ## Listing Packages
2914
+
2915
+ Command: `faspex package list`
2916
+
2917
+ ### Option `box`
2918
+
2919
+ By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
2920
+
2921
+ ### Option `recipient`
2922
+
2923
+ A user can receive a package because the recipient is:
2924
+
2925
+ * the user himself (default)
2926
+ * the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
2927
+
2928
+ ### Option `query`
2929
+
2930
+ As inboxes may be large, it is possible to use the following query parameters:
2931
+
2932
+ * `count` : (native) number items in one API call (default=0, equivalent to 10)
2933
+ * `page` : (native) id of page in call (default=0)
2934
+ * `startIndex` : (native) index of item to start, default=0, oldest index=0
2935
+ * `max` : maximum number of items
2936
+ * `pmax` : maximum number of pages
2937
+
2938
+ (SQL query is `LIMIT <startIndex>, <count>`)
2939
+
2940
+ The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
2941
+
2942
+ If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
2943
+
2944
+ ### Example
2945
+
2946
+ ```
2947
+ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2948
+ ```
2949
+
2950
+ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
2374
2951
 
2375
2952
  ## Receiving a Package
2376
2953
 
2377
- The command is `package recv`, possible methosd are:
2954
+ The command is `package recv`, possible methods are:
2378
2955
 
2379
2956
  * provide a package id with option `id`
2380
2957
  * provide a public link with option `link`
2381
2958
  * provide a `faspe:` URI with option `link`
2382
2959
 
2383
2960
  ```
2384
- $ <%=cmd%> faspex package recv --id=12345
2385
- $ <%=cmd%> faspex package recv --link=faspe://...
2961
+ <%=cmd%> faspex package recv --id=12345
2962
+ <%=cmd%> faspex package recv --link=faspe://...
2386
2963
  ```
2387
2964
 
2388
2965
  If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
2389
2966
 
2390
2967
  ```
2391
- $ <%=cmd%> faspex package list --recipient='*thedropboxname'
2968
+ <%=cmd%> faspex package list --recipient='*thedropboxname'
2392
2969
  ```
2393
2970
 
2394
-
2971
+ if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
2395
2972
 
2396
2973
  ## Sending a Package
2397
2974
 
@@ -2400,7 +2977,7 @@ The command is `faspex package send`. Package information (title, note, metadata
2400
2977
  Example:
2401
2978
 
2402
2979
  ```
2403
- $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2980
+ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2404
2981
  ```
2405
2982
 
2406
2983
  If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
@@ -2410,17 +2987,29 @@ Additional optional parameters in `delivery_info`:
2410
2987
  * Package Note: : `"note":"note this and that"`
2411
2988
  * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2412
2989
 
2413
- ## operation on dropboxes
2990
+ ## Email notification on transfer
2991
+
2992
+ Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
2993
+
2994
+ Example:
2995
+
2996
+ ```
2997
+ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
2998
+ ```
2999
+
3000
+ In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
3001
+
3002
+ ## Operation on dropboxes
2414
3003
 
2415
3004
  Example:
2416
3005
 
2417
3006
  ```
2418
- $ <%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
2419
- $ <%=cmd%> faspex v4 dropbox list
2420
- $ <%=cmd%> faspex v4 dropbox delete --id=36
3007
+ <%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
3008
+ <%=cmd%> faspex v4 dropbox list
3009
+ <%=cmd%> faspex v4 dropbox delete --id=36
2421
3010
  ```
2422
3011
 
2423
- ## remote sources
3012
+ ## Remote sources
2424
3013
 
2425
3014
  Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
2426
3015
  the node API is used, for this it is required to add a section ":storage" that links
@@ -2455,7 +3044,7 @@ It is possible to tell <%=tool%> to download newly received packages, much like
2455
3044
  cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
2456
3045
 
2457
3046
  ```
2458
- $ <%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
3047
+ <%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
2459
3048
  ```
2460
3049
 
2461
3050
  # Plugin: IBM Aspera Shares
@@ -2465,42 +3054,51 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
2465
3054
  In Shares2, users, groups listing are paged, to display sequential pages:
2466
3055
 
2467
3056
  ```
2468
- $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
3057
+ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
2469
3058
  ```
2470
3059
 
2471
3060
  # Plugin: IBM Cloud Object Storage
2472
3061
 
2473
3062
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2474
- It uses the same transfer service as Aspera on Cloud.
2475
- see [https://status.aspera.io](https://status.aspera.io)
3063
+ It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
3064
+ Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
3065
+
3066
+ There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
2476
3067
 
2477
- Required options are either:
3068
+ ## Using endpoint, apikey and Resource Instance ID (CRN)
3069
+
3070
+ If you have those parameters already, then following options shall be provided:
2478
3071
 
2479
3072
  * `bucket` bucket name
2480
3073
  * `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
2481
3074
  * `apikey` API Key
2482
3075
  * `crn` resource instance id
2483
3076
 
2484
- or:
3077
+ For example, let us create a default configuration:
2485
3078
 
2486
- * `bucket` bucket name
2487
- * `region` bucket region, e.g. eu-de
2488
- * `service_credentials` see below
3079
+ ```
3080
+ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
3081
+ <%=cmd%> conf id default set cos mycos
3082
+ ```
2489
3083
 
2490
- Service credentials are directly created using the IBM cloud web ui. Navigate to:
3084
+ Then, jump to the transfer example.
3085
+
3086
+ ## Using service credential file
3087
+
3088
+ If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
2491
3089
 
2492
3090
  Navigation Menu &rarr; Resource List &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
2493
3091
 
2494
3092
  Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
2495
3093
 
2496
- or using the CLI:
3094
+ or using the IBM Cloud CLI:
2497
3095
 
2498
3096
  ```
2499
- $ ibmcloud resource service-keys
2500
- $ ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
3097
+ ibmcloud resource service-keys
3098
+ ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
2501
3099
  ```
2502
3100
 
2503
- (if you dont have `jq` installed, extract the structure as follows)
3101
+ (if you don't have `jq` installed, extract the structure as follows)
2504
3102
 
2505
3103
  It consists in the following structure:
2506
3104
 
@@ -2524,33 +3122,38 @@ The field `resource_instance_id` is for option `crn`
2524
3122
 
2525
3123
  The field `apikey` is for option `apikey`
2526
3124
 
2527
- Endpoints for regions can be found by querying the `endpoints` URL.
3125
+ (If needed: endpoints for regions can be found by querying the `endpoints` URL.)
2528
3126
 
2529
- For convenience, let us create a default configuration, for example:
3127
+ The required options for this method are:
2530
3128
 
2531
- ```
2532
- $ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
2533
- $ <%=cmd%> conf id default set cos mycos
2534
- ```
3129
+ * `bucket` bucket name
3130
+ * `region` bucket region, e.g. eu-de
3131
+ * `service_credentials` see below
2535
3132
 
2536
- or using direct parameters:
3133
+ For example, let us create a default configuration:
2537
3134
 
2538
3135
  ```
2539
- $ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2540
- $ <%=cmd%> conf id default set cos mycos
3136
+ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
3137
+ <%=cmd%> conf id default set cos mycos
2541
3138
  ```
2542
3139
 
2543
- Now, Ready to do operations, a subset of "node" plugin operations are supported, basically node API:
3140
+ ## Operations, transfers
3141
+
3142
+ Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
3143
+
3144
+ A subset of `node` plugin operations are supported, basically node API:
2544
3145
 
2545
3146
  ```
2546
- $ <%=cmd%> cos node browse /
2547
- $ <%=cmd%> cos node upload myfile.txt
3147
+ <%=cmd%> cos node info
3148
+ <%=cmd%> cos node upload 'faux:///sample1G?1g'
2548
3149
  ```
2549
3150
 
3151
+ Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
3152
+
2550
3153
  # Plugin: IBM Aspera Sync
2551
3154
 
2552
- A basic plugin to start an "async" using <%=tool%>. The main advantage is the possibility
2553
- to start from ma configuration file, using <%=tool%> standard options.
3155
+ A basic plugin to start an "async" using <%=tool%>.
3156
+ The main advantage is the possibility to start from ma configuration file, using <%=tool%> standard options.
2554
3157
 
2555
3158
  # Plugin: Preview
2556
3159
 
@@ -2571,8 +3174,10 @@ Specify the previews folder as shown in:
2571
3174
  By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
2572
3175
 
2573
3176
  ```
2574
- # /opt/aspera/bin/asconfigurator -x "server;preview_dir,previews"
2575
- # /opt/aspera/bin/asnodeadmin --reload
3177
+ PATH=/opt/aspera/bin:$PATH
3178
+
3179
+ asconfigurator -x "server;preview_dir,previews"
3180
+ asnodeadmin --reload
2576
3181
  ```
2577
3182
 
2578
3183
  Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
@@ -2585,16 +3190,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
2585
3190
  To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
2586
3191
 
2587
3192
  ```
2588
- # /opt/aspera/bin/asuserdata -a | grep max_request_file_create_size_kb
3193
+ asuserdata -a | grep max_request_file_create_size_kb
3194
+
2589
3195
  max_request_file_create_size_kb: "1024"
2590
- # /opt/aspera/bin/asconfigurator -x "server; max_request_file_create_size_kb,16384"
3196
+
3197
+ asconfigurator -x "server; max_request_file_create_size_kb,16384"
2591
3198
  ```
2592
3199
 
2593
3200
  If you use a value different than 16777216, then specify it using option `max_size`.
2594
3201
 
2595
3202
  Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
2596
3203
 
2597
- ## <a name="prev_ext"></a>External tools: Linux
3204
+ ## <a id="prev_ext"></a>External tools: Linux
2598
3205
 
2599
3206
  The tool requires the following external tools available in the `PATH`:
2600
3207
 
@@ -2611,7 +3218,7 @@ Other OSes should work as well, but are note tested.
2611
3218
  To check if all tools are found properly, execute:
2612
3219
 
2613
3220
  ```
2614
- $ <%=cmd%> preview check
3221
+ <%=cmd%> preview check
2615
3222
  ```
2616
3223
 
2617
3224
  ### mimemagic
@@ -2619,22 +3226,22 @@ $ <%=cmd%> preview check
2619
3226
  To benefit from extra mime type detection install gem mimemagic:
2620
3227
 
2621
3228
  ```
2622
- # gem install mimemagic
3229
+ gem install mimemagic
2623
3230
  ```
2624
3231
 
2625
3232
  or to install an earlier version if any problem:
2626
3233
 
2627
3234
  ```
2628
- # gem install mimemagic -v '~> 0.3.0'
3235
+ gem install mimemagic -v '~> 0.3.0'
2629
3236
  ```
2630
3237
 
2631
3238
  To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
2632
3239
 
2633
3240
  If not used, Mime type used for conversion is the one provided by the node API.
2634
3241
 
2635
- If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
3242
+ If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
2636
3243
 
2637
- ### Image: Imagemagick and optipng
3244
+ ### Image: ImageMagick and optipng
2638
3245
 
2639
3246
  ```
2640
3247
  yum install -y ImageMagick optipng
@@ -2642,31 +3249,33 @@ yum install -y ImageMagick optipng
2642
3249
 
2643
3250
  ### Video: FFmpeg
2644
3251
 
3252
+ The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
3253
+
2645
3254
  ```
2646
- curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && tar xJvf - && rm -f /opt/ffmpeg /usr/bin/{ffmpeg,ffprobe} && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3255
+ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
2647
3256
  ```
2648
3257
 
2649
3258
  ### Office: Unoconv and Libreoffice
2650
3259
 
2651
- If you dont want to have preview for office dpcuments or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
3260
+ If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
2652
3261
 
2653
3262
  The generation of preview in based on the use of `unoconv` and `libreoffice`
2654
3263
 
2655
- * Centos 8
3264
+ * CentOS 8
2656
3265
 
2657
3266
  ```
2658
- # dnf install unoconv
3267
+ dnf install unoconv
2659
3268
  ```
2660
3269
 
2661
3270
  * Amazon Linux
2662
3271
 
2663
3272
  ```
2664
- # amazon-linux-extras enable libreoffice
2665
- # yum clean metadata
2666
- # yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
2667
- # wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
2668
- # mv unoconv /usr/bin
2669
- # chmod a+x /usr/bin/unoconv
3273
+ amazon-linux-extras enable libreoffice
3274
+ yum clean metadata
3275
+ yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
3276
+ wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
3277
+ mv unoconv /usr/bin
3278
+ chmod a+x /usr/bin/unoconv
2670
3279
  ```
2671
3280
 
2672
3281
  ## Configuration
@@ -2678,9 +3287,11 @@ Like any <%=tool%> commands, parameters can be passed on command line or using a
2678
3287
  Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
2679
3288
 
2680
3289
  ```
2681
- # su -s /bin/bash - xfer
2682
- $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
2683
- $ <%=cmd%> config id default set preview previewconf
3290
+ su -s /bin/bash - xfer
3291
+
3292
+ <%=cmd%> config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
3293
+
3294
+ <%=cmd%> config preset set default preview previewconf
2684
3295
  ```
2685
3296
 
2686
3297
  Here we assume that Office file generation is disabled, else remove this option.
@@ -2689,7 +3300,7 @@ Here we assume that Office file generation is disabled, else remove this option.
2689
3300
  One can check if the access key is well configured using:
2690
3301
 
2691
3302
  ```
2692
- $ <%=cmd%> -Ppreviewconf node browse /
3303
+ <%=cmd%> -Ppreviewconf node browse /
2693
3304
  ```
2694
3305
 
2695
3306
  This shall list the contents of the storage root of the access key.
@@ -2705,8 +3316,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
2705
3316
  Lets do a one shot test, using the configuration previously created:
2706
3317
 
2707
3318
  ```
2708
- # su -s /bin/bash - xfer
2709
- xfer$ <%=cmd%> preview scan --overwrite=always
3319
+ su -s /bin/bash - xfer
3320
+
3321
+ <%=cmd%> preview scan --overwrite=always
2710
3322
  ```
2711
3323
 
2712
3324
  When the preview generator is first executed it will create a file: `.aspera_access_key`
@@ -2720,13 +3332,13 @@ Adapt the scripts to your own preference.
2720
3332
 
2721
3333
  We assume here that a configuration preset was created as shown previously.
2722
3334
 
2723
- Lets first setup a script that will be used in the sceduler and sets up the environment.
3335
+ Lets first setup a script that will be used in the scheduler and sets up the environment.
2724
3336
 
2725
3337
  Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
2726
3338
 
2727
3339
  ```
2728
- #!/bin/bash
2729
- # set a timeout protection, just in case
3340
+ #!/bin/bash
3341
+ # set a timeout protection, just in case
2730
3342
  case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
2731
3343
  . /etc/profile.d/rvm.sh
2732
3344
  rvm use 2.6 --quiet
@@ -2736,13 +3348,13 @@ exec timeout ${tmout} <%=cmd%> "${@}"
2736
3348
  Here the cronjob is created for user `xfer`.
2737
3349
 
2738
3350
  ```
2739
- xfer$ crontab<<EOF
3351
+ crontab<<EOF
2740
3352
  0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
2741
3353
  2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
2742
3354
  EOF
2743
3355
  ```
2744
3356
 
2745
- Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
3357
+ Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
2746
3358
 
2747
3359
  ## Candidate detection for creation or update (or deletion)
2748
3360
 
@@ -2767,12 +3379,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
2767
3379
  If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
2768
3380
 
2769
3381
  ```
2770
- $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
3382
+ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
2771
3383
  ```
2772
3384
 
2773
3385
  The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
2774
3386
 
2775
- When scanning the option `value` has the same behaviour as for the `node find` command.
3387
+ When scanning the option `value` has the same behavior as for the `node find` command.
2776
3388
 
2777
3389
  For instance to filter out files beginning with `._` do:
2778
3390
 
@@ -2791,7 +3403,7 @@ Use option `skip_format` to skip generation of a format.
2791
3403
 
2792
3404
  ## Supported input Files types
2793
3405
 
2794
- The preview generator supports redering of those file categories:
3406
+ The preview generator supports rendering of those file categories:
2795
3407
 
2796
3408
  * image
2797
3409
  * pdf
@@ -2811,7 +3423,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
2811
3423
 
2812
3424
  ## Access to original files and preview creation
2813
3425
 
2814
- Standard open source tools are used to create thumnails and video previews.
3426
+ Standard open source tools are used to create thumbnails and video previews.
2815
3427
  Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
2816
3428
  The tool provides 2 ways to read and write files with the option: `file_access`
2817
3429
 
@@ -2828,50 +3440,89 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
2828
3440
  The `smtp` option is a hash table (extended value) with the following fields:
2829
3441
  <table>
2830
3442
  <tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
2831
- <tr><td>server</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
2832
- <tr><td>tls</td><td>true</td><td>false</td><td>use of TLS</td></tr>
2833
- <tr><td>port</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
2834
- <tr><td>domain</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
2835
- <tr><td>username</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
2836
- <tr><td>password</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
2837
- <tr><td>from\_email</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
2838
- <tr><td>from\_name</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
3443
+ <tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
3444
+ <tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
3445
+ <tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
3446
+ <tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
3447
+ <tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
3448
+ <tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
3449
+ <tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
3450
+ <tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
2839
3451
  </table>
2840
3452
 
2841
3453
  ## Example of configuration:
2842
3454
 
2843
3455
  ```
2844
- $ <%=cmd%> config id smtp_google set server smtp.google.com
2845
- $ <%=cmd%> config id smtp_google set username john@gmail.com
2846
- $ <%=cmd%> config id smtp_google set password P@ssw0rd
3456
+ <%=cmd%> config preset set smtp_google server smtp.google.com
3457
+ <%=cmd%> config preset set smtp_google username john@gmail.com
3458
+ <%=cmd%> config preset set smtp_google password P@ssw0rd
2847
3459
  ```
2848
3460
 
2849
3461
  or
2850
3462
 
2851
3463
  ```
2852
- $ <%=cmd%> config id smtp_google init @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
3464
+ <%=cmd%> config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
2853
3465
  ```
2854
3466
 
2855
3467
  or
2856
3468
 
2857
3469
  ```
2858
- $ <%=cmd%> config id smtp_google update --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
3470
+ <%=cmd%> config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
2859
3471
  ```
2860
3472
 
2861
- Set this configation as global default, for instance:
3473
+ Set this configuration as global default, for instance:
2862
3474
 
2863
3475
  ```
2864
- $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
2865
- $ <%=cmd%> config id default set config cli_default
3476
+ <%=cmd%> config preset set cli_default smtp @val:@preset:smtp_google
3477
+ <%=cmd%> config preset set default config cli_default
2866
3478
  ```
2867
3479
 
3480
+ ## Email templates
3481
+
3482
+ Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
3483
+
3484
+ The template is the full SMTP message, including headers.
3485
+
3486
+ The following variables are defined by default:
3487
+
3488
+ * from_name
3489
+ * from_email
3490
+ * to
3491
+
3492
+ Other variables are defined depending on context.
3493
+
2868
3494
  ## Test
2869
3495
 
2870
3496
  Check settings with `smtp_settings` command. Send test email with `email_test`.
2871
3497
 
2872
3498
  ```
2873
- $ <%=cmd%> config --smtp=@preset:smtp_google smtp
2874
- $ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
3499
+ <%=cmd%> config --smtp=@preset:smtp_google smtp
3500
+ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3501
+ ```
3502
+
3503
+ ## Notifications for transfer status
3504
+
3505
+ An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
3506
+
3507
+ To activate, use option `notif_to`.
3508
+
3509
+ A default e-mail template is used, but it can be overridden with option `notif_template`.
3510
+
3511
+ The environment provided contains the following additional variables:
3512
+
3513
+ * subject
3514
+ * body
3515
+ * global_transfer_status
3516
+ * ts
3517
+
3518
+ Example of template:
3519
+
3520
+ ```
3521
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
3522
+ To: <<%='<'%>%=to%>>
3523
+ Subject: <%='<'%>%=subject%>
3524
+
3525
+ Transfer is: <%='<'%>%=global_transfer_status%>
2875
3526
  ```
2876
3527
 
2877
3528
  # Tool: `asession`
@@ -2879,7 +3530,7 @@ $ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
2879
3530
  This gem comes with a second executable tool providing a simplified standardized interface
2880
3531
  to start a FASP session: `asession`.
2881
3532
 
2882
- It aims at simplifying the startup of a FASP session from a programmatic stand point as formating a [_transfer-spec_](#transferspec) is:
3533
+ It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
2883
3534
 
2884
3535
  * common to Aspera Node API (HTTP POST /ops/transfer)
2885
3536
  * common to Aspera Connect API (browser javascript startTransfer)
@@ -2891,7 +3542,7 @@ This makes it easy to integrate with any language provided that one can spawn a
2891
3542
 
2892
3543
  The tool expect one single argument: a [_transfer-spec_](#transferspec).
2893
3544
 
2894
- If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formated [_transfer-spec_](#transferspec) on stdin.
3545
+ If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
2895
3546
 
2896
3547
  Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
2897
3548
 
@@ -2902,7 +3553,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
2902
3553
  * `EX_loglevel` to change log level of the tool
2903
3554
  * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
2904
3555
 
2905
- Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the "local" transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3556
+ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
2906
3557
 
2907
3558
  ## Comparison of interfaces
2908
3559
 
@@ -2929,7 +3580,7 @@ echo "${MY_TSPEC}"|asession
2929
3580
  This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
2930
3581
 
2931
3582
  ```
2932
- $ asession
3583
+ asession
2933
3584
  {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
2934
3585
  {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
2935
3586
  {"type":"DONE"}
@@ -2944,8 +3595,8 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
2944
3595
  ## Help
2945
3596
 
2946
3597
  ```
2947
- $ asession -h
2948
- <%= File.read(ENV["INCL_ASESSION"]) %>
3598
+ asession -h
3599
+ <%=File.read(ENV["INCL_ASESSION"])%>
2949
3600
  ```
2950
3601
 
2951
3602
  # Hot folder
@@ -2982,8 +3633,8 @@ Interesting ascp features are found in its arguments: (see ascp manual):
2982
3633
  Note that:
2983
3634
 
2984
3635
  * <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
2985
- * not all native ascp arguments are available as standard transfer_spec parameters
2986
- * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
3636
+ * most, but not all native ascp arguments are available as standard transfer_spec parameters
3637
+ * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
2987
3638
 
2988
3639
  ### server side and configuration
2989
3640
 
@@ -2998,18 +3649,18 @@ Once <%=tool%> parameters are defined, run the command using the OS native sched
2998
3649
  ## Example
2999
3650
 
3000
3651
  ```
3001
- $ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
3652
+ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
3002
3653
 
3003
3654
  ```
3004
3655
 
3005
- The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3656
+ The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3006
3657
 
3007
- # Aspera Health check and Nagios
3658
+ # Health check and Nagios
3008
3659
 
3009
- Each plugin provide a `health` command that will check the health status of the application. Example:
3660
+ Most plugin provide a `health` command that will check the health status of the application. Example:
3010
3661
 
3011
3662
  ```
3012
- $ <%=cmd%> console health
3663
+ <%=cmd%> console health
3013
3664
  +--------+-------------+------------+
3014
3665
  | status | component | message |
3015
3666
  +--------+-------------+------------+
@@ -3026,13 +3677,13 @@ Typically, the health check uses the REST API of the application with the follow
3026
3677
  <%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
3027
3678
 
3028
3679
  ```
3029
- $ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3680
+ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3030
3681
  OK - [transfer:ok]
3031
- $ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3682
+ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3032
3683
  OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
3033
3684
  ```
3034
3685
 
3035
- # Module: `Aspera`
3686
+ # Ruby Module: `Aspera`
3036
3687
 
3037
3688
  Main components:
3038
3689
 
@@ -3043,8 +3694,8 @@ Main components:
3043
3694
  A working example can be found in the gem, example:
3044
3695
 
3045
3696
  ```
3046
- $ <%=cmd%> config gem_path
3047
- $ cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
3697
+ <%=cmd%> config gem_path
3698
+ cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
3048
3699
  ```
3049
3700
 
3050
3701
  This sample code shows some example of use of the API as well as
@@ -3079,106 +3730,143 @@ So, it evolved into <%=tool%>:
3079
3730
 
3080
3731
  # Changes (Release notes)
3081
3732
 
3082
- * <%= gemspec.version.to_s %>
3083
-
3084
- * new: command `faspex package recv` supports link of type: `faspe:`
3085
- * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3733
+ * <%=gemspec.version.to_s%>
3734
+
3735
+ * new: support transfer agent: [Transfer SDK](#agt_trsdk)
3736
+ * new: support [http socket options](#http_options)
3737
+ * new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
3738
+ * new: `config vault` supports encrypted passwords, also macos keychain
3739
+ * new: `config preset` command for consistency with id
3740
+ * new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
3741
+ * change: when using wss, use [ruby's CA certs](#certificates)
3742
+ * change: unexpected parameter makes exit code not zero
3743
+ * change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
3744
+ * change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
3745
+ * change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
3746
+ * fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
3747
+
3748
+ * 4.4.0
3749
+
3750
+ * new: `aoc packages list` add possibility to add filter with option `query`
3751
+ * new: `aoc admin res xxx list` now get all items by default #50
3752
+ * new: `preset` option can specify name or hash value
3753
+ * new: `node` plugin accepts bearer token and access key as credential
3754
+ * new: `node` option `token_type` allows using basic token in addition to aspera type.
3755
+ * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
3756
+ * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
3757
+
3758
+ * 4.3.0
3759
+
3760
+ * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
3761
+ * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3762
+
3763
+ * 4.2.2
3764
+
3765
+ * new: `faspex package list` retrieves the whole list, not just first page
3766
+ * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3767
+ * new: the error "Remote host is not who we expected" displays a special remediation message
3768
+ * new: `conf ascp spec` displays supported transfer spec
3769
+ * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3770
+ * fix: space character in `faspe:` url are precent encoded if needed
3771
+ * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3772
+ * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3773
+ * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3774
+
3775
+ * 4.2.1
3776
+
3777
+ * new: command `faspex package recv` supports link of type: `faspe:`
3778
+ * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3086
3779
 
3087
3780
  * 4.2.0
3088
3781
 
3089
- * new: command `aoc remind` to receive organization membership by email
3090
- * new: in `preview` option `value` to filter out on file name
3091
- * new: `initdemo` to initialize for demo server
3092
- * new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3093
- * fix: on Windows `conf ascp use` expects ascp.exe
3094
- * fix: (break) multi_session_threshold is Integer, not String
3095
- * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3096
- * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3097
- * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3098
- * change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
3782
+ * new: command `aoc remind` to receive organization membership by email
3783
+ * new: in `preview` option `value` to filter out on file name
3784
+ * new: `initdemo` to initialize for demo server
3785
+ * new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3786
+ * fix: on Windows `conf ascp use` expects ascp.exe
3787
+ * fix: (break) multi_session_threshold is Integer, not String
3788
+ * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3789
+ * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3790
+ * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3791
+ * change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
3099
3792
 
3100
3793
  * 4.1.0
3101
3794
 
3102
- * fix: remove keys from transfer spec and command line when not needed
3103
- * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3104
- * new: update documentation with regard to offline and docker installation
3105
- * new: renamed command `nagios_check` to `health`
3106
- * new: agent `http_gw` now supports upload
3107
- * new: added option `sdk_url` to install SDK from local file for offline install
3108
- * new: check new gem version periodically
3109
- * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3110
- * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3111
- * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3112
- * new: AoC a password can be provided for a public link
3113
- * new: `conf doc` take an optional parameter to go to a section
3114
- * new: initial support for Faspex 5 Beta 1
3795
+ * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3796
+ * new: update documentation with regard to offline and docker installation
3797
+ * new: renamed command `nagios_check` to `health`
3798
+ * new: agent `http_gw` now supports upload
3799
+ * new: added option `sdk_url` to install SDK from local file for offline install
3800
+ * new: check new gem version periodically
3801
+ * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3802
+ * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3803
+ * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3804
+ * new: AoC a password can be provided for a public link
3805
+ * new: `conf doc` take an optional parameter to go to a section
3806
+ * new: initial support for Faspex 5 Beta 1
3115
3807
 
3116
3808
  * 4.0.0
3117
3809
 
3118
- * now available as open source at [<%= gemspec.homepage %>](<%= gemspec.homepage %>) with general cleanup
3119
- * changed default tool name from `mlia` to `ascli`
3120
- * changed `aspera` command to `aoc`
3121
- * changed gem name from `asperalm` to `aspera-cli`
3122
- * changed module name from `Asperalm` to `Aspera`
3123
- * removed command `folder` in `preview`, merged to `scan`
3124
- * persistency files go to sub folder instead of main folder
3125
- * added possibility to install SDK: `config ascp install`
3810
+ * now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
3811
+ * changed default tool name from `mlia` to `ascli`
3812
+ * changed `aspera` command to `aoc`
3813
+ * changed gem name from `asperalm` to `aspera-cli`
3814
+ * changed module name from `Asperalm` to `Aspera`
3815
+ * removed command `folder` in `preview`, merged to `scan`
3816
+ * persistency files go to sub folder instead of main folder
3817
+ * added possibility to install SDK: `config ascp install`
3126
3818
 
3127
3819
  * 0.11.8
3128
3820
 
3129
- * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
3821
+ * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
3130
3822
 
3131
3823
  * 0.11.7
3132
3824
 
3133
- * rework on rest call error handling
3134
- * use option `display` with value `data` to remove out of extraneous information
3135
- * fixed option `lock_port` not working
3136
- * generate special icon if preview failed
3137
- * possibility to choose transfer progress bar type with option `progress`
3138
- * AoC package creation now output package id
3825
+ * rework on rest call error handling
3826
+ * use option `display` with value `data` to remove out of extraneous information
3827
+ * fixed option `lock_port` not working
3828
+ * generate special icon if preview failed
3829
+ * possibility to choose transfer progress bar type with option `progress`
3830
+ * AoC package creation now output package id
3139
3831
 
3140
3832
  * 0.11.6
3141
3833
 
3142
- * orchestrator : added more choice in auth type
3143
- * preview: cleanup in generator (removed and renamed parameters)
3144
- * preview: better documentation
3145
- * preview: animated thumbnails for video (option: `video_png_conv=animated`)
3146
- * preview: new event trigger: `trevents` (`events` seems broken)
3147
- * preview: unique tmp folder to avoid clash of multiple instances
3148
- * repo: added template for secrets used for testing
3834
+ * orchestrator : added more choice in auth type
3835
+ * preview: cleanup in generator (removed and renamed parameters)
3836
+ * preview: better documentation
3837
+ * preview: animated thumbnails for video (option: `video_png_conv=animated`)
3838
+ * preview: new event trigger: `trevents` (`events` seems broken)
3839
+ * preview: unique tmp folder to avoid clash of multiple instances
3840
+ * repo: added template for secrets used for testing
3149
3841
 
3150
3842
  * 0.11.5
3151
3843
 
3152
- * added option `default_ports` for AoC (see manual)
3153
- * allow bulk delete in `aspera files` with option `bulk=yes`
3154
- * fix getting connect versions
3155
- * added section for Aix
3156
- * support all ciphers for `local`ascp (incl. gcm, etc..)
3157
- * added transfer spec param `apply_local_docroot` for `local`
3844
+ * added option `default_ports` for AoC (see manual)
3845
+ * allow bulk delete in `aspera files` with option `bulk=yes`
3846
+ * fix getting connect versions
3847
+ * added section for Aix
3848
+ * support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
3849
+ * added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
3158
3850
 
3159
3851
  * 0.11.4
3160
3852
 
3161
- * possibility to give shared inbox name when sending a package (else use id and type)
3853
+ * possibility to give shared inbox name when sending a package (else use id and type)
3162
3854
 
3163
3855
  * 0.11.3
3164
3856
 
3165
- * minor fixes on multi-session: avoid exception on progress bar
3857
+ * minor fixes on multi-session: avoid exception on progress bar
3166
3858
 
3167
3859
  * 0.11.2
3168
3860
 
3169
- * fixes on multi-session: progress bat and transfer spec param for "direct"
3861
+ * fixes on multi-session: progress bat and transfer spec param for "direct"
3170
3862
 
3171
3863
  * 0.11.1
3172
3864
 
3173
- * enhanced short_link creation commands (see examples)
3865
+ * enhanced short_link creation commands (see examples)
3174
3866
 
3175
3867
  * 0.11
3176
3868
 
3177
- * add option to provide file list directly to ascp like this (only for direct transfer agent):
3178
-
3179
- ```
3180
- ... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
3181
- ```
3869
+ * add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
3182
3870
 
3183
3871
  * 0.10.18
3184
3872
 
@@ -3187,7 +3875,7 @@ So, it evolved into <%=tool%>:
3187
3875
  * 0.10.17
3188
3876
 
3189
3877
  * fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
3190
- * new modifier: `@list:<saparator>val1<separator>...`
3878
+ * new modifier: `@list:<separator>val1<separator>...`
3191
3879
 
3192
3880
  * 0.10.16
3193
3881
 
@@ -3209,7 +3897,7 @@ So, it evolved into <%=tool%>:
3209
3897
  * 0.10.12
3210
3898
 
3211
3899
  * added support for AoC node registration keys
3212
- * replaced option : `local_resume` with `transfer_info` for agent `direct`
3900
+ * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
3213
3901
  * Transfer agent is no more a Singleton instance, but only one is used in CLI
3214
3902
  * `@incps` : new extended value modifier
3215
3903
  * ATS: no more provides access keys secrets: now user must provide it
@@ -3241,7 +3929,7 @@ So, it evolved into <%=tool%>:
3241
3929
 
3242
3930
  * 0.10.6
3243
3931
 
3244
- * FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
3932
+ * FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
3245
3933
  * gem version requirements made more open
3246
3934
 
3247
3935
  * 0.10.5
@@ -3259,9 +3947,9 @@ So, it evolved into <%=tool%>:
3259
3947
 
3260
3948
  * 0.10.2
3261
3949
 
3262
- * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
3263
- * added doc for "cargo" like actions
3264
- * added doc for multi-session
3950
+ * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
3951
+ * added doc for "cargo" like actions
3952
+ * added doc for multi-session
3265
3953
 
3266
3954
  * 0.10.1
3267
3955
 
@@ -3289,22 +3977,22 @@ So, it evolved into <%=tool%>:
3289
3977
 
3290
3978
  * 0.9.33
3291
3979
 
3292
- * new command to display basic token of node
3293
- * new command to display bearer token of node in AoC
3294
- * the --fields= option, support +_fieldname_ to add a field to default fields
3295
- * many small changes
3980
+ * new command to display basic token of node
3981
+ * new command to display bearer token of node in AoC
3982
+ * the --fields= option, support +_fieldname_ to add a field to default fields
3983
+ * many small changes
3296
3984
 
3297
3985
  * 0.9.32
3298
3986
 
3299
- * all Faspex public links are now supported
3300
- * removed faspex operation recv_publink
3301
- * replaced with option `link` (consistent with AoC)
3987
+ * all Faspex public links are now supported
3988
+ * removed faspex operation recv_publink
3989
+ * replaced with option `link` (consistent with AoC)
3302
3990
 
3303
3991
  * 0.9.31
3304
3992
 
3305
- * added more support for public link: receive and send package, to user or dropbox and files view.
3306
- * delete expired file lists
3307
- * changed text table gem from text-table to terminal-table because it supports multiline values
3993
+ * added more support for public link: receive and send package, to user or dropbox and files view.
3994
+ * delete expired file lists
3995
+ * changed text table gem from text-table to terminal-table because it supports multiline values
3308
3996
 
3309
3997
  * 0.9.27
3310
3998
 
@@ -3324,149 +4012,149 @@ So, it evolved into <%=tool%>:
3324
4012
 
3325
4013
  * 0.9.24
3326
4014
 
3327
- * fix bug where AoC node to node transfer did not work
3328
- * fix bug on error if ED25519 private key is defined in .ssh
4015
+ * fix bug where AoC node to node transfer did not work
4016
+ * fix bug on error if ED25519 private key is defined in .ssh
3329
4017
 
3330
4018
  * 0.9.23
3331
4019
 
3332
- * defined REST error handlers, more error conditions detected
3333
- * commands to select specific ascp location
4020
+ * defined REST error handlers, more error conditions detected
4021
+ * commands to select specific ascp location
3334
4022
 
3335
4023
  * 0.9.21
3336
4024
 
3337
- * supports simplified wizard using global client
3338
- * only ascp binary is required, other SDK (keys) files are now generated
4025
+ * supports simplified wizard using global client
4026
+ * only ascp binary is required, other SDK (keys) files are now generated
3339
4027
 
3340
4028
  * 0.9.20
3341
4029
 
3342
- * improved wizard (prepare for AoC global client id)
3343
- * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
3344
- * removed outdated pictures from this doc
4030
+ * improved wizard (prepare for AoC global client id)
4031
+ * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
4032
+ * removed outdated pictures from this doc
3345
4033
 
3346
4034
  * 0.9.19
3347
4035
 
3348
- * added command aspera bearer --scope=xx
4036
+ * added command aspera bearer --scope=xx
3349
4037
 
3350
4038
  * 0.9.18
3351
4039
 
3352
- * enhanced aspera admin events to support query
4040
+ * enhanced aspera admin events to support query
3353
4041
 
3354
4042
  * 0.9.16
3355
4043
 
3356
- * AoC transfers are now reported in activity app
3357
- * new interface for Rest class authentication (keep backward compatibility)
4044
+ * AoC transfers are now reported in activity app
4045
+ * new interface for Rest class authentication (keep backward compatibility)
3358
4046
 
3359
4047
  * 0.9.15
3360
4048
 
3361
- * new feature: "find" command in aspera files
3362
- * sample code for transfer API
4049
+ * new feature: "find" command in aspera files
4050
+ * sample code for transfer API
3363
4051
 
3364
4052
  * 0.9.12
3365
4053
 
3366
- * add nagios commands
3367
- * support of ATS for IBM Cloud, removed old version based on aspera id
4054
+ * add nagios commands
4055
+ * support of ATS for IBM Cloud, removed old version based on aspera id
3368
4056
 
3369
4057
  * 0.9.11
3370
4058
 
3371
- * Breaking change: @stdin is now @stdin:
3372
- * support of ATS for IBM Cloud, removed old version based on aspera id
4059
+ * Breaking change: @stdin is now @stdin:
4060
+ * support of ATS for IBM Cloud, removed old version based on aspera id
3373
4061
 
3374
4062
 
3375
4063
  * 0.9.10
3376
4064
 
3377
- * Breaking change: parameter transfer-node becomes more generic: transfer-info
3378
- * Display SaaS storage usage with command: aspera admin res node --id=nn info
3379
- * cleaner way of specifying source file list for transfers
3380
- * Breaking change: replaced download_mode option with http_download action
4065
+ * Breaking change: parameter transfer-node becomes more generic: transfer-info
4066
+ * Display SaaS storage usage with command: aspera admin res node --id=nn info
4067
+ * cleaner way of specifying source file list for transfers
4068
+ * Breaking change: replaced download_mode option with http_download action
3381
4069
 
3382
4070
  * 0.9.9
3383
4071
 
3384
- * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
3385
- * Now supports "cargo" for Aspera on Cloud (automatic package download)
4072
+ * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
4073
+ * Now supports "cargo" for Aspera on Cloud (automatic package download)
3386
4074
 
3387
4075
  * 0.9.8
3388
4076
 
3389
- * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
3390
- * AoC: share to share transfer with command "transfer"
4077
+ * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
4078
+ * AoC: share to share transfer with command "transfer"
3391
4079
 
3392
4080
  * 0.9.7
3393
4081
 
3394
- * homogeneous [_transfer-spec_](#transferspec) for node and local
3395
- * preview persistency goes to unique file by default
3396
- * catch mxf extension in preview as video
3397
- * Faspex: possibility to download all paclages by specifying id=ALL
3398
- * Faspex: to come: cargo-like function to download only new packages with id=NEW
4082
+ * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
4083
+ * preview persistency goes to unique file by default
4084
+ * catch mxf extension in preview as video
4085
+ * Faspex: possibility to download all packages by specifying id=ALL
4086
+ * Faspex: to come: cargo-like function to download only new packages with id=NEW
3399
4087
 
3400
4088
  * 0.9.6
3401
4089
 
3402
- * Breaking change: `@param:`is now `@preset:` and is generic
3403
- * AoC: added command to display current workspace information
4090
+ * Breaking change: `@param:`is now `@preset:` and is generic
4091
+ * AoC: added command to display current workspace information
3404
4092
 
3405
4093
  * 0.9.5
3406
4094
 
3407
- * new parameter: new_user_option used to choose between public_link and invite of external users.
3408
- * fixed bug in wizard, and wizard uses now product detection
4095
+ * new parameter: new_user_option used to choose between public_link and invite of external users.
4096
+ * fixed bug in wizard, and wizard uses now product detection
3409
4097
 
3410
4098
  * 0.9.4
3411
4099
 
3412
- * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
3413
- * AoC Package send supports external users
3414
- * new command to export AoC config to Aspera CLI config
4100
+ * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
4101
+ * AoC Package send supports external users
4102
+ * new command to export AoC config to Aspera CLI config
3415
4103
 
3416
4104
  * 0.9.3
3417
4105
 
3418
- * REST error message show host and code
3419
- * option for quiet display
3420
- * modified transfer interface and allow token re-generation on error
3421
- * async add admin command
3422
- * async add db parameters
3423
- * Breaking change: new option "sources" to specify files to transfer
4106
+ * REST error message show host and code
4107
+ * option for quiet display
4108
+ * modified transfer interface and allow token re-generation on error
4109
+ * async add admin command
4110
+ * async add db parameters
4111
+ * Breaking change: new option "sources" to specify files to transfer
3424
4112
 
3425
4113
  * 0.9.2
3426
4114
 
3427
- * Breaking change: changed AoC package creation to match API, see AoC section
4115
+ * Breaking change: changed AoC package creation to match API, see AoC section
3428
4116
 
3429
4117
  * 0.9.1
3430
4118
 
3431
- * Breaking change: changed faspex package creation to match API, see Faspex section
4119
+ * Breaking change: changed faspex package creation to match API, see Faspex section
3432
4120
 
3433
4121
  * 0.9
3434
4122
 
3435
- * Renamed the CLI from aslmcli to <%=tool%>
3436
- * Automatic rename and conversion of former config folder from aslmcli to <%=tool%>
4123
+ * Renamed the CLI from aslmcli to <%=tool%>
4124
+ * Automatic rename and conversion of former config folder from aslmcli to <%=tool%>
3437
4125
 
3438
4126
  * 0.7.6
3439
4127
 
3440
- * add "sync" plugin
4128
+ * add "sync" plugin
3441
4129
 
3442
4130
  * 0.7
3443
4131
 
3444
- * Breaking change: AoC package recv take option if for package instead of argument.
3445
- * Breaking change: Rest class and Oauth class changed init parameters
3446
- * AoC: receive package from public link
3447
- * select by col value on output
3448
- * added rename (AoC, node)
4132
+ * Breaking change: AoC package recv take option if for package instead of argument.
4133
+ * Breaking change: Rest class and Oauth class changed init parameters
4134
+ * AoC: receive package from public link
4135
+ * select by col value on output
4136
+ * added rename (AoC, node)
3449
4137
 
3450
4138
  * 0.6.19
3451
4139
 
3452
4140
  Breaking change:
3453
4141
 
3454
- * ats server list provisioned &rarr; ats cluster list
3455
- * ats server list clouds &rarr; ats cluster clouds
3456
- * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
3457
- * ats server id xxx &rarr; ats cluster show --id=xxx
3458
- * ats subscriptions &rarr; ats credential subscriptions
3459
- * ats api_key repository list &rarr; ats credential cache list
3460
- * ats api_key list &rarr; ats credential list
3461
- * ats access_key id xxx &rarr; ats access_key --id=xxx
4142
+ * ats server list provisioned &rarr; ats cluster list
4143
+ * ats server list clouds &rarr; ats cluster clouds
4144
+ * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4145
+ * ats server id xxx &rarr; ats cluster show --id=xxx
4146
+ * ats subscriptions &rarr; ats credential subscriptions
4147
+ * ats api_key repository list &rarr; ats credential cache list
4148
+ * ats api_key list &rarr; ats credential list
4149
+ * ats access_key id xxx &rarr; ats access_key --id=xxx
3462
4150
 
3463
4151
  * 0.6.18
3464
4152
 
3465
- * some commands take now --id option instead of id command.
4153
+ * some commands take now --id option instead of id command.
3466
4154
 
3467
4155
  * 0.6.15
3468
4156
 
3469
- * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
4157
+ * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
3470
4158
 
3471
4159
  # BUGS, FEATURES, CONTRIBUTION
3472
4160
 
@@ -3474,16 +4162,16 @@ For issues or feature requests use the Github repository and issues.
3474
4162
 
3475
4163
  You can also contribute to this open source project.
3476
4164
 
3477
- One can also create one's own command nplugin.
4165
+ One can also [create one's own plugin](#createownplugin).
3478
4166
 
3479
4167
  ## Only one value for any option
3480
4168
 
3481
4169
  Some commands and sub commands may ask for the same option name.
3482
- Currently, since option definition is position independant (last one wins), it is not possible
4170
+ Currently, since option definition is position independent (last one wins), it is not possible
3483
4171
  to give an option to a command and the same option with different value to a sub command.
3484
4172
 
3485
- For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
3486
- As a workaround use another option, if available, to identify the entity.
4173
+ For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
4174
+ As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
3487
4175
 
3488
4176
  This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
3489
4177
 
@@ -3503,20 +4191,24 @@ You may either install the suggested Gems, or remove your ed25519 key from your
3503
4191
 
3504
4192
  ## Error "Remote host is not who we expected"
3505
4193
 
3506
- `ascp` version 4.x changed the algorithm used to check the SSH server certificate. To ignore the certificate (SSH fingerprint) add option on client side:
4194
+ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
4195
+
4196
+ Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
3507
4197
 
3508
4198
  ```
3509
4199
  --ts=@json:'{"sshfp":null}'
3510
4200
  ```
3511
4201
 
3512
- Refer to ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server): if you have access to server side, basically disable other SSH host keys than RSA.
4202
+ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
4203
+
4204
+ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
3513
4205
 
3514
- ## Miscelaneous
4206
+ ## Miscellaneous
3515
4207
 
3516
4208
  * remove rest and oauth classes and use ruby standard gems:
3517
4209
 
3518
- * oauth
3519
- * https://github.com/rest-client/rest-client
4210
+ * oauth
4211
+ * https://github.com/rest-client/rest-client
3520
4212
 
3521
4213
  * use Thor or any standard Ruby CLI manager
3522
4214