aspera-cli 4.0.0 → 4.2.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +843 -304
  3. data/bin/dascli +13 -0
  4. data/docs/Makefile +4 -4
  5. data/docs/README.erb.md +805 -172
  6. data/docs/test_env.conf +22 -3
  7. data/examples/aoc.rb +14 -3
  8. data/examples/faspex4.rb +89 -0
  9. data/lib/aspera/aoc.rb +87 -108
  10. data/lib/aspera/cli/formater.rb +2 -0
  11. data/lib/aspera/cli/main.rb +89 -49
  12. data/lib/aspera/cli/plugin.rb +9 -4
  13. data/lib/aspera/cli/plugins/alee.rb +1 -1
  14. data/lib/aspera/cli/plugins/aoc.rb +188 -173
  15. data/lib/aspera/cli/plugins/ats.rb +2 -2
  16. data/lib/aspera/cli/plugins/config.rb +218 -145
  17. data/lib/aspera/cli/plugins/console.rb +2 -2
  18. data/lib/aspera/cli/plugins/faspex.rb +114 -61
  19. data/lib/aspera/cli/plugins/faspex5.rb +85 -43
  20. data/lib/aspera/cli/plugins/node.rb +3 -3
  21. data/lib/aspera/cli/plugins/preview.rb +59 -45
  22. data/lib/aspera/cli/plugins/server.rb +23 -8
  23. data/lib/aspera/cli/transfer_agent.rb +77 -49
  24. data/lib/aspera/cli/version.rb +1 -1
  25. data/lib/aspera/command_line_builder.rb +49 -31
  26. data/lib/aspera/cos_node.rb +33 -28
  27. data/lib/aspera/environment.rb +2 -2
  28. data/lib/aspera/fasp/connect.rb +28 -21
  29. data/lib/aspera/fasp/http_gw.rb +140 -28
  30. data/lib/aspera/fasp/installation.rb +93 -46
  31. data/lib/aspera/fasp/local.rb +88 -45
  32. data/lib/aspera/fasp/manager.rb +15 -0
  33. data/lib/aspera/fasp/node.rb +4 -4
  34. data/lib/aspera/fasp/parameters.rb +59 -101
  35. data/lib/aspera/fasp/parameters.yaml +531 -0
  36. data/lib/aspera/fasp/resume_policy.rb +13 -12
  37. data/lib/aspera/fasp/uri.rb +1 -1
  38. data/lib/aspera/log.rb +1 -1
  39. data/lib/aspera/node.rb +61 -1
  40. data/lib/aspera/oauth.rb +49 -46
  41. data/lib/aspera/persistency_folder.rb +9 -4
  42. data/lib/aspera/preview/file_types.rb +53 -21
  43. data/lib/aspera/preview/generator.rb +3 -3
  44. data/lib/aspera/rest.rb +29 -18
  45. data/lib/aspera/secrets.rb +20 -0
  46. data/lib/aspera/sync.rb +40 -35
  47. data/lib/aspera/temp_file_manager.rb +19 -0
  48. data/lib/aspera/web_auth.rb +105 -0
  49. metadata +54 -20
  50. data/docs/transfer_spec.html +0 -99
data/docs/README.erb.md CHANGED
@@ -1,7 +1,7 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
2
  <%
3
3
  # check that required env vars exist, and files
4
- %w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_TRSPEC}.each do |e|
4
+ %w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_DIR_GEM}.each do |e|
5
5
  raise "missing env var #{e}" unless ENV.has_key?(e)
6
6
  raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
7
7
  end
@@ -14,28 +14,102 @@ prsts='['+opprst+'s](#lprt)'
14
14
  prstt=opprst.capitalize # in title
15
15
  gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
16
16
  geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
17
+ $LOAD_PATH.unshift(ENV["INCL_DIR_GEM"])
18
+ require 'aspera/fasp/parameters'
19
+ def spec_table
20
+ r='<table><tr><th>Field</th><th>Type</th>'
21
+ Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
22
+ r << '<th>'<<c.to_s.upcase<<'</th>'
23
+ end
24
+ r << '<th>Description</th></tr>'
25
+ Aspera::Fasp::Parameters.man_table.each do |p|
26
+ p[:description] << (p[:description].empty? ? '' : "\n") << "(" << p[:cli] << ")" unless p[:cli].to_s.empty?
27
+ p.delete(:cli)
28
+ p.keys.each{|c|p[c]='&nbsp;' if p[c].to_s.empty?}
29
+ p[:description].gsub!("\n",'<br/>')
30
+ p[:type].gsub!(',','<br/>')
31
+ r << '<tr><td>'<<p[:name]<<'</td><td>'<<p[:type]<<'</td>'
32
+ Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
33
+ r << '<td>'<<p[c]<<'</td>'
34
+ end
35
+ r << '<td>'<<p[:description]<<'</td></tr>'
36
+ end
37
+ r << '</table>'
38
+ return r
39
+ end
17
40
  -%>
18
- # <%=tool%> : a Command Line for IBM Aspera products
41
+ # <%=tool%> : Command Line Interface for IBM Aspera products
19
42
 
20
43
  Version : <%= gemspec.version.to_s %>
21
44
 
22
45
  _Laurent/2016-<%=Time.new.year%>_
23
46
 
24
- This gem provides a command line interface to Aspera Applications.
47
+ This gem provides <%=tool%>: a command line interface to Aspera Applications.
48
+
49
+ <%=tool%> is a also great tool to learn Aspera APIs.
50
+
51
+ Ruby Gem: [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubygems_uri'] %>)
52
+
53
+ Ruby Doc: [<%= gemspec.metadata['documentation_uri'] %>](<%= gemspec.metadata['documentation_uri'] %>)
54
+
55
+ Ruby version must be >= <%= gemspec.required_ruby_version %>
56
+
57
+ # <a name="when_to_use"></a>When to use and when not to use
58
+
59
+ <%=tool%> is designed to be used as a command line tool to:
60
+
61
+ * execute commands on Aspera products
62
+ * transfer to/from Aspera products
63
+
64
+ So it is designed for:
65
+
66
+ * Interactive operations on a text terminal (typically, VT100 compatible)
67
+ * Batch operations in (shell) scripts (e.g. cron job)
25
68
 
26
- Location (once released):
27
- [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubygems_uri'] %>)
69
+ <%=tool%> can be seen as a command line tool integrating:
28
70
 
29
- Disclaimers:
71
+ * a configuration file (config.yaml) and advanced command line options
72
+ * cURL (for REST calls)
73
+ * Aspera transfer (ascp)
30
74
 
31
- * This has not yet been officially released so things may change
75
+ One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea.
76
+ For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
32
77
 
33
- <%=tool%> is also a great tool to learn Aspera APIs.
78
+ * Product APIs (REST) : e.g. AoC, Faspex, node
79
+ * Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
80
+
81
+ Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
82
+
83
+ For scripting and ad'hoc command line operations, <%=tool%> is perfect.
84
+
85
+ # Notations
34
86
 
35
87
  In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
88
+ Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
36
89
 
37
90
  Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
38
91
 
92
+ # <a name="parsing"></a>Shell and Command line parsing
93
+
94
+ <%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
95
+
96
+ On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
97
+
98
+ On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
99
+
100
+ * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
101
+ * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
102
+
103
+ In case of doubt of argument values after parsing test like this:
104
+
105
+ ```
106
+ $ <%=cmd%> conf echo "Hello World" arg2 3
107
+ "Hello World"
108
+ ERROR: Argument: unprocessed values: ["arg2", "3"]
109
+ ```
110
+
111
+ `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
112
+
39
113
  # Quick Start
40
114
 
41
115
  This section guides you from installation, first use and advanced use.
@@ -55,9 +129,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
55
129
 
56
130
  If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
57
131
 
58
- If you want to test with Aspera demo transfer server, a default configuration is created on first use:
132
+ To test with Aspera demo transfer server, setup the environment and then test:
59
133
 
60
134
  ```
135
+ $ <%=cmd%> config initdemo
61
136
  $ <%=cmd%> server browse /
62
137
  :............:...........:......:........:...........................:.......................:
63
138
  : zmode : zuid : zgid : size : mtime : name :
@@ -77,7 +152,7 @@ If you want to use <%=tool%> with another server, and in order to make further c
77
152
  * download a file
78
153
 
79
154
  ```
80
- $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
155
+ $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
81
156
  updated: myserver
82
157
  $ <%=cmd%> config id default set server myserver
83
158
  updated: default&rarr;server to myserver
@@ -111,77 +186,159 @@ Then, follow the section relative to the product you want to interact with ( Asp
111
186
 
112
187
  # <a name="installation"></a>Installation
113
188
 
114
- In order to use the tool or the gem, it is necessary to install those components:
189
+ It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
115
190
 
116
- * [Ruby](#ruby)
191
+ The direct installation is recommended and consists in installing:
192
+
193
+ * [Ruby](#ruby) version >= <%= gemspec.required_ruby_version %>
117
194
  * [<%= gemspec.name %>](#the_gem)
118
- * [FASP](#fasp_prot)
195
+ * [Aspera SDK (ascp)](#fasp_prot)
196
+
197
+ The following sections provide information on the various installation methods.
119
198
 
120
- The following sections provide information on the installation.
199
+ An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
200
+
201
+ ## Docker container
202
+
203
+ Use this method only if you know what you do, else use the standard recommended method as described here above.
204
+
205
+ This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
206
+
207
+ Ensure that you have Docker installed.
208
+
209
+ ```
210
+ $ docker --version
211
+ ```
212
+
213
+ Download the wrapping script:
214
+
215
+ ```
216
+ $ curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
217
+ $ chmod a+x <%=cmd%>
218
+ ```
219
+
220
+ Install the container image:
221
+
222
+ ```
223
+ $ ./<%=cmd%> install
224
+ ```
225
+
226
+ Start using it !
227
+
228
+ Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
229
+
230
+ The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
231
+
232
+ To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
233
+ To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
121
234
 
122
235
  ## <a name="ruby"></a>Ruby
123
236
 
124
- A ruby interpreter is required to run the tool or to use the gem and tool.
237
+ Use this method to install on the native host.
125
238
 
126
- Ruby minimum version: <%= gemspec.required_ruby_version %>
239
+ A ruby interpreter is required to run the tool or to use the gem and tool.
127
240
 
128
- Ruby 3 is also supported.
241
+ Ruby minimum version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
129
242
 
130
- Any type of Ruby installation can be used (installer, rpm, rvm, ...).
243
+ *Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
131
244
 
132
245
  Refer to the following sections for a proposed method for specific operating systems.
133
246
 
134
- ### macOS
247
+ The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
248
+ If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
249
+ If you have a simpler better way to install Ruby version >= <%= gemspec.required_ruby_version %> : use it !
250
+
251
+ ### Generic: RVM: single user installation (not root)
252
+
253
+ Use this method which provides more flexibility.
254
+
255
+ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
135
256
 
136
- MacOS 10.13+ (High Sierra) comes with a recent Ruby, so you can use it directly, you will need to install <%= gemspec.name %> using `sudo` :
257
+ Install the 2 keys
137
258
 
138
259
  ```
139
- $ sudo gem install <%= gemspec.name %><%=geminstadd%>
260
+ $ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
140
261
  ```
141
262
 
142
- Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
263
+ Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
143
264
 
144
265
  ```
145
- $ brew install ruby
266
+ $ \curl -sSL https://get.rvm.io | bash -s stable
267
+ ```
268
+
269
+ If you keep the same terminal (ont needed if re-login):
270
+
271
+ ```
272
+ $ source ~/.rvm/scripts/rvm
273
+ ```
274
+
275
+ It is advised to get one of the pre-compiled ruby version, you can list with:
276
+
277
+ ```
278
+ $ rvm list --remote
146
279
  ```
147
280
 
148
- It is also possib le to use `rvm`
281
+ Install the chosen pre-compiled Ruby version:
149
282
 
150
- ### Windows
283
+ ```
284
+ $ rvm install 2.7.2 --binary
285
+ ```
151
286
 
152
- Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/).
287
+ Ruby is now installed for the user, go on to Gem installation.
153
288
 
154
- Go to "Downloads".
289
+ ### Generic: RVM: global installation (as root)
155
290
 
156
- Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
291
+ Follow the same method as single user install, but execute as "root".
157
292
 
158
- At the end of the installer uncheck the box to skip the installation of "MSys2".
293
+ As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
294
+ One can install in another location with :
159
295
 
160
- ### Linux
296
+ ```
297
+ # curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
298
+ ```
161
299
 
162
- Install Latest Ruby 2 using "rvm" [https://rvm.io/](https://rvm.io/) .
163
- It installs by default in /usr/local/rvm , but you can install in another location:
300
+ As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
301
+ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok`.
302
+ To activate ruby (and <%=cmd%>) later, source it:
164
303
 
165
304
  ```
166
- curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
305
+ # source /etc/profile.d/rvm.sh.ok
306
+ # rvm version
167
307
  ```
168
308
 
169
- Once installed, you can install latest ruby:
309
+ ### Windows: Installer
310
+
311
+ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/) :
312
+
313
+ * Go to "Downloads".
314
+ * Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
315
+ * At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.
316
+
317
+ ### macOS: pre-installed or `brew`
318
+
319
+ MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%= gemspec.name %> using `sudo` :
170
320
 
171
321
  ```
172
- # rvm install ruby
322
+ $ sudo gem install <%= gemspec.name %><%=geminstadd%>
173
323
  ```
174
324
 
175
- If you dont want all users to have ruby by default,
176
- rename the file: `/etc/profile.d/rvm.sh` with another extension, and source it to get rvm.
325
+ Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
177
326
 
178
- Alternatively, only if you know what you do, on RPM based systems (CentOs, Redhat), install the ruby provided by yum which may be 2.0.
327
+ ```
328
+ $ brew install ruby
329
+ ```
330
+
331
+ ### Linux: package
332
+
333
+ If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
334
+
335
+ Example:
179
336
 
180
337
  ```
181
338
  # yum install -y ruby rubygems ruby-json
182
339
  ```
183
340
 
184
- One can cleanup your whole yum-installed ruby environment like this to uninstall:
341
+ One can cleanup the whole yum-installed ruby environment like this to uninstall:
185
342
 
186
343
  ```
187
344
  gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
@@ -206,6 +363,50 @@ For instance to build from source, and install in `/opt/ruby` :
206
363
  # make install
207
364
  ```
208
365
 
366
+ ### <a name="offline_install"></a>Installation without internet access
367
+
368
+ Note that currently no pre-packaged version exist yet.
369
+ A method to build one provided here:
370
+
371
+ On a server with the same OS version and with internet access follow the "Generic single user installation" method.
372
+
373
+ Then create an archive:
374
+
375
+ ```
376
+ $ cd
377
+ $ tar zcvf rvm-<%=cmd%>.tgz .rvm
378
+ ```
379
+
380
+ Get the Aspera SDK. Execute:
381
+
382
+ ```
383
+ $ <%=cmd%> conf --show-config|grep sdk_url
384
+ ```
385
+
386
+ Then download the SDK archive from that URL.
387
+
388
+ Another method for the SDK is to install the SDK (`<%=cmd%> conf ascp install`) on the first system, and archive `$HOME/.aspera`.
389
+
390
+ Transfer those 2 archives to the target system without internet access.
391
+
392
+ On the target system:
393
+
394
+ * Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
395
+ * in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
396
+
397
+ ```
398
+ source path_to_rvm_root/scripts/rvm
399
+ rvm use 2.7.2
400
+ ```
401
+
402
+ For the SDK, either install from archive:
403
+
404
+ ```
405
+ $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
406
+ ```
407
+
408
+ or restore the `$HOME/.aspera` folder for the user.
409
+
209
410
  ## <a name="the_gem"></a>`<%= gemspec.name %>` gem
210
411
 
211
412
  Once you have Ruby and rights to install gems: Install the gem and its dependencies:
@@ -220,21 +421,40 @@ To upgrade to the latest version:
220
421
  # gem update <%= gemspec.name %>
221
422
  ```
222
423
 
424
+ <%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
425
+
426
+ To check manually:
427
+
428
+ ```
429
+ # <%=cmd%> conf check_update
430
+ ```
431
+
432
+
433
+
223
434
  ## <a name="fasp_prot"></a>FASP Protocol
224
435
 
225
- Most file transfers will be done using the FASP protocol. Only two additional files are required to perform
226
- an Aspera Transfer:
436
+ Most file transfers will be done using the FASP protocol, using `ascp`.
437
+ Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
227
438
 
228
439
  * ascp
229
440
  * aspera-license (in same folder, or ../etc)
230
441
 
231
- This can be installed directly with
442
+ This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
232
443
 
233
444
  ```
234
445
  $ <%=cmd%> conf ascp install
235
446
  ```
236
447
 
237
- Those can be found in one of IBM Aspera transfer server or client with its license file (some are free):
448
+ If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
449
+
450
+ ```
451
+ $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
452
+ $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
453
+ ```
454
+
455
+ The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
456
+
457
+ If the embedded method is not used, the following packages are also suitable:
238
458
 
239
459
  * IBM Aspera Connect Client (Free)
240
460
  * IBM Aspera Desktop Client (Free)
@@ -243,7 +463,7 @@ Those can be found in one of IBM Aspera transfer server or client with its licen
243
463
  * IBM Aspera High Speed Transfer EndPoint (Licensed)
244
464
 
245
465
  For instance, Aspera Connect Client can be installed
246
- by visiting the page: [http://downloads.asperasoft.com/connect2/](http://downloads.asperasoft.com/connect2/).
466
+ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
247
467
 
248
468
  <%=tool%> will detect most of Aspera transfer products in standard locations and use the first one found.
249
469
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
@@ -251,6 +471,33 @@ Refer to section [FASP](#client) for details on how to select a client or set pa
251
471
  Several methods are provided on how to start a transfer. Use of a local client is one of them, but
252
472
  other methods are available. Refer to section: [Transfer Agents](#agents)
253
473
 
474
+ ## <a name="offline_install"></a>Offline Installation (without internet)
475
+
476
+ The procedure consists in:
477
+
478
+ * Follow the non-root installation procedure with RVM, including gem
479
+ * archive (zip, tar) the main RVM folder (includes <%=cmd%>):
480
+
481
+ ```
482
+ $ cd ~
483
+ $ tar zcvf rvm_<%=cmd%>.tgz .rvm
484
+ ```
485
+
486
+ * retrieve the SDK:
487
+
488
+ ```
489
+ $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
490
+ ```
491
+
492
+ * on the system without internet access:
493
+
494
+ ```
495
+ $ cd ~
496
+ $ tar zxvf rvm_<%=cmd%>.tgz
497
+ $ source ~/.rvm/scripts/rvm
498
+ $ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
499
+ ```
500
+
254
501
  # <a name="cli"></a>Command Line Interface: <%=tool%>
255
502
 
256
503
  The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
@@ -431,6 +678,7 @@ By default, a table output will display one line per entry, and columns for each
431
678
  * a,b,c : the list of attributes specified by the comma separated list
432
679
  * Array extended value: for instance, @json:'["a","b","c"]' same as above
433
680
  * +a,b,c : add selected properties to the default selection.
681
+ * -a,b,c : remove selected properties from the default selection.
434
682
 
435
683
  ## <a name="extended"></a>Extended Value Syntax
436
684
 
@@ -444,25 +692,25 @@ The extended value syntax is:
444
692
 
445
693
  The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
446
694
 
447
- The following "readers" are supported:
695
+ The following "readers" are supported (returns value in []):
448
696
 
449
- * @val:VALUE , prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
450
- * @file:PATH , read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
451
- * @path:PATH , performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
452
- * @env:ENVVAR , read from a named env var, e.g.--password=@env:MYPASSVAR
453
- * @stdin: , read from stdin (no value on right)
454
- * @preset:NAME , get whole <%=opprst%> value by name
697
+ * @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
698
+ * @file:PATH : [String] read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
699
+ * @path:PATH : [String] performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
700
+ * @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
701
+ * @stdin: : [String] read from stdin (no value on right)
702
+ * @preset:NAME : [Hash] get whole <%=opprst%> value by name
455
703
 
456
704
  In addition it is possible to decode a value, using one or multiple decoders :
457
705
 
458
- * @base64: decode a base64 encoded string
459
- * @json: decode JSON values (convenient to provide complex structures)
460
- * @zlib: uncompress data
461
- * @ruby: execute ruby code
462
- * @csvt: decode a titled CSV value
463
- * @lines: split a string in multiple lines and return an array
464
- * @list: split a string in multiple items taking first character as separator and return an array
465
- * @incps: include values of presets specified by key include_presets in hash
706
+ * @base64: [String] decode a base64 encoded string
707
+ * @json: [any] decode JSON values (convenient to provide complex structures)
708
+ * @zlib: [String] uncompress data
709
+ * @ruby: [any] execute ruby code
710
+ * @csvt: [Array] decode a titled CSV value
711
+ * @lines: [Array] split a string in multiple lines and return an array
712
+ * @list: [Array] split a string in multiple items taking first character as separator and return an array
713
+ * @incps: [Hash] include values of presets specified by key `incps` in input hash
466
714
 
467
715
  To display the result of an extended value, use the `config echo` command.
468
716
 
@@ -494,7 +742,7 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
494
742
  :......:.....................:
495
743
  ```
496
744
 
497
- Example: create a hash and include values from preset named "config" of config file
745
+ Example: create a hash and include values from preset named "config" of config file in this hash
498
746
 
499
747
  ```
500
748
  $ <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
@@ -533,9 +781,9 @@ It can be overriden using the envinonment variable `<%=evp%>HOME`.
533
781
  Example (Windows):
534
782
 
535
783
  ```
536
- $ set <%=evp%>HOME=C:\Users\Kenji\.aspera\ascli
784
+ $ set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
537
785
  $ <%=cmd%> config folder
538
- C:\Users\Kenji\.aspera\ascli
786
+ C:\Users\Kenji\.aspera\<%=cmd%>
539
787
  ```
540
788
 
541
789
  ## <a name="configfile"></a>Configuration file
@@ -570,7 +818,7 @@ $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
570
818
  The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
571
819
 
572
820
  ```
573
- $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera --ts=@json:'{"precalculate_job_size":true}'
821
+ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
574
822
  ```
575
823
 
576
824
  * This creates a <%=prst%> `demo_server` with all provided options.
@@ -578,13 +826,13 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
578
826
  The command `set` allows setting individual options in a <%=prst%>.
579
827
 
580
828
  ```
581
- $ <%=cmd%> config id demo_server set password demoaspera
829
+ $ <%=cmd%> config id demo_server set password _demo_pass_
582
830
  ```
583
831
 
584
832
  The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
585
833
 
586
834
  ```
587
- $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"demoaspera","ts":{"precalculate_job_size":true}}'
835
+ $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
588
836
  ```
589
837
 
590
838
  A good practice is to not manually edit the configuration file and use modification commands instead.
@@ -621,11 +869,19 @@ Note that special plugin name: `config` can be associated with a preset that is
621
869
  Operations on this preset are done using regular `config` operations:
622
870
 
623
871
  ```
624
- $ <%=cmd%> config id default set _plugin_name_ _defauklt_preset_for_plugin_
872
+ $ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
625
873
  $ <%=cmd%> config id default get _plugin_name_
626
- "_defauklt_preset_for_plugin_"
874
+ "_default_preset_for_plugin_"
627
875
  ```
628
876
 
877
+ ### <a name="lprtdef"></a>Special Plugin: config
878
+
879
+ Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
880
+
881
+ When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
882
+
883
+ If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
884
+
629
885
  ### Format of file
630
886
 
631
887
  The configuration file is a hash in a YAML file. Example:
@@ -641,7 +897,7 @@ cli_default:
641
897
  demo_server:
642
898
  url: ssh://demo.asperasoft.com:33001
643
899
  username: asperaweb
644
- password: demoaspera
900
+ password: _demo_pass_
645
901
  ```
646
902
 
647
903
  We can see here:
@@ -710,7 +966,6 @@ A <%=prst%> value can be removed with `unset`:
710
966
  $ <%=cmd%> config id cli_default unset interactive
711
967
  ```
712
968
 
713
-
714
969
  ### Examples
715
970
 
716
971
  For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
@@ -830,21 +1085,42 @@ The `config` plugin also allows specification for the use of a local FASP client
830
1085
 
831
1086
  ```
832
1087
  $ <%=cmd%> config ascp show
833
- /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
1088
+ /Users/laurent/.aspera/ascli/sdk/ascp
1089
+ $ <%=cmd%> config ascp info
1090
+ +--------------------+-----------------------------------------------------------+
1091
+ | key | value |
1092
+ +--------------------+-----------------------------------------------------------+
1093
+ | ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
1094
+ ...
834
1095
  ```
835
1096
 
836
1097
  ### Selection of local `ascp`
837
1098
 
1099
+ By default, <%=tool%> uses any found local product with ascp, including SDK.
1100
+
838
1101
  To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
839
1102
 
840
- To permanently use another ascp:
1103
+ For a permanent change, the command `config ascp use` sets the same parameter for the global default.
1104
+
1105
+ Using a POSIX shell:
841
1106
 
842
1107
  ```
843
1108
  $ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
844
- saved to default global preset /Users/laurent/Applications/Aspera CLI/bin/ascp
1109
+ ascp version: 4.0.0.182279
1110
+ Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
1111
+ Saved to default global preset global_common_defaults
1112
+ ```
1113
+
1114
+ Windows:
1115
+
1116
+ ```
1117
+ $ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1118
+ ascp version: 4.0.0.182279
1119
+ Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1120
+ Saved to default global preset global_common_defaults
845
1121
  ```
846
1122
 
847
- This sets up a global default.
1123
+ If the path has spaces, read section: [Shell and Command line parsing](#parsing).
848
1124
 
849
1125
  ### List locally installed Aspera Transfer products
850
1126
 
@@ -931,7 +1207,7 @@ will effectively push files to the related server from the agent node.
931
1207
 
932
1208
  ### <a name="direct"></a>Direct (local ascp using FASPManager API)
933
1209
 
934
- By default the CLI will use a local FASP protocol, equivalent to specifying `--transfer=direct`.
1210
+ By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
935
1211
  <%=tool%> will detect locally installed Aspera products.
936
1212
  Refer to section [FASP](#client).
937
1213
 
@@ -941,17 +1217,33 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
941
1217
  * `EX_http_proxy_url` (proxy for legacy http fallback)
942
1218
  * `EX_ascp_args`
943
1219
 
944
- The `transfer-info` optionally provides the following auto resume parameters:
1220
+ The `transfer-info` accepts the following optional parameters:
945
1221
 
946
1222
  <table>
947
- <tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
948
- <tr><td>iter_max</td>. <td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
949
- <tr><td>sleep_initial</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
950
- <tr><td>sleep_factor</td> <td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
951
- <tr><td>sleep_max</td>. <td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
952
- <tr><td>wss</td> <td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1223
+ <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1224
+ <tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
1225
+ <tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
1226
+ <tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1227
+ <tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
953
1228
  </table>
954
1229
 
1230
+ Resume parameters:
1231
+
1232
+ <table>
1233
+ <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1234
+ <tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
1235
+ <tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
1236
+ <tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
1237
+ <tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
1238
+ </table>
1239
+
1240
+ Examples:
1241
+
1242
+ ```
1243
+ $ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1244
+ $ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5}'
1245
+ ```
1246
+
955
1247
  ### IBM Aspera Connect Client GUI
956
1248
 
957
1249
  By specifying option: `--transfer=connect`, <%=tool%> will start transfers
@@ -983,9 +1275,11 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
983
1275
  Example:
984
1276
 
985
1277
  ```
986
- $ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://eudemo.asperademo.com:9443/aspera/http-gwy/v1"}'
1278
+ $ <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
987
1279
  ```
988
1280
 
1281
+ Note that the gateway only supports transfers authorized with a token.
1282
+
989
1283
  ## <a name="transferspec"></a>Transfer Specification
990
1284
 
991
1285
  Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
@@ -999,7 +1293,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
999
1293
 
1000
1294
  <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
1001
1295
 
1002
- If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
1296
+ If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1003
1297
 
1004
1298
  It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1005
1299
 
@@ -1012,11 +1306,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
1012
1306
 
1013
1307
  ## <a name="transferparams"></a>Transfer Parameters
1014
1308
 
1015
- All standard _transfer-spec_ parameters can be overloaded. To display parameters,
1016
- run in debug mode (--log-level=debug). [_transfer-spec_](#transferspec) can
1017
- also be saved/overridden in the config file.
1309
+ All standard _transfer-spec_ parameters can be speficied.
1310
+ [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1311
+
1312
+ References:
1313
+
1314
+ * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1315
+ * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1316
+
1317
+ Parameters can be displayed with commands:
1018
1318
 
1019
- <%= File.read(ENV['INCL_TRSPEC']).gsub(/.*<body>(.*)<\/body>.*/m,'\1') %>
1319
+ ```
1320
+ $ <%=cmd%> config ascp spec
1321
+ $ <%=cmd%> config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
1322
+ ```
1323
+
1324
+ Columns:
1325
+
1326
+ * D=Direct (local `ascp` execution)
1327
+ * N=Node API
1328
+ * C=Connect Client
1329
+ * arg=`ascp` argument or environment variable
1330
+
1331
+ Fields with EX_ prefix are extensions to transfer agent `direct`. (only in <%=tool%>).
1332
+
1333
+ <%= spec_table %>
1020
1334
 
1021
1335
  ### Destination folder for transfers
1022
1336
 
@@ -1032,8 +1346,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
1032
1346
 
1033
1347
  ### List of files for transfers
1034
1348
 
1035
- When uploading, downloading or sending files, the user must specify
1036
- the list of files to transfer. Most of the time, the list of files to transfer will be simply specified on the command line:
1349
+ When uploading, downloading or sending files, the user must specify the list of files to transfer.
1350
+ Most of the time, the list of files to transfer will be simply specified on the command line:
1037
1351
 
1038
1352
  ```
1039
1353
  $ <%=cmd%> server upload ~/mysample.file secondfile
@@ -1047,7 +1361,8 @@ $ <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1047
1361
 
1048
1362
  More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1049
1363
 
1050
- Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is: all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
1364
+ Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
1365
+ all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
1051
1366
 
1052
1367
  For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. Accepted values are:
1053
1368
 
@@ -1097,7 +1412,7 @@ Source files are located on "Aspera on cloud", when :
1097
1412
 
1098
1413
  ### <a name="multisession"></a>Support of multi-session
1099
1414
 
1100
- Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
1415
+ Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1101
1416
 
1102
1417
  * when agent=node :
1103
1418
 
@@ -1118,6 +1433,7 @@ shall be preferred.
1118
1433
 
1119
1434
  Multi-session spawn is done by <%=tool%>.
1120
1435
 
1436
+ When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
1121
1437
 
1122
1438
  ### Examples
1123
1439
 
@@ -1290,7 +1606,7 @@ updated: my_aoc_org
1290
1606
  Define this <%=prst%> as default configuration for the `aspera` plugin:
1291
1607
 
1292
1608
  ```
1293
- $ <%=cmd%> config id default set aspera my_aoc_org
1609
+ $ <%=cmd%> config id default set aoc my_aoc_org
1294
1610
  ```
1295
1611
 
1296
1612
  Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
@@ -1393,7 +1709,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
1393
1709
 
1394
1710
  * change auth method to JWT
1395
1711
  * provide location of private key
1396
- * provide username to login as (OAuthg "subject")
1712
+ * provide username to login as (OAuth "subject")
1397
1713
 
1398
1714
  Execute:
1399
1715
 
@@ -1664,7 +1980,7 @@ Then, create two shared folders located in two regions, in your files home, in a
1664
1980
  Then, transfer between those:
1665
1981
 
1666
1982
  ```
1667
- $ <%=cmd%> -Paoc_show aspera files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
1983
+ $ <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
1668
1984
  ```
1669
1985
 
1670
1986
  * create registration key to register a node
@@ -1744,10 +2060,10 @@ $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
1744
2060
  * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
1745
2061
  * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
1746
2062
 
1747
- Typically, one would regularly execute this command on a regular basis, using the method oif your choice:
2063
+ Typically, one would regularly execute this command on a regular basis, using the method of your choice:
1748
2064
 
1749
- * Windows scheduler
1750
- * cron
2065
+ * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2066
+ * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
1751
2067
  * etc...
1752
2068
 
1753
2069
  ## Download Files
@@ -1804,19 +2120,40 @@ The activity app can be queried with:
1804
2120
  $ <%=cmd%> aoc admin analytics transfers
1805
2121
  ```
1806
2122
 
1807
- It can also support filters and send notification email with a template:
2123
+ It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2124
+
2125
+ `mytemplate.erb`:
2126
+
2127
+ ```
2128
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2129
+ To: <<%='<'%>%=ev['user_email']%>>
2130
+ Subject: <%='<'%>%=ev['files_completed']%> files received
2131
+
2132
+ Dear <%='<'%>%=ev[:user_email.to_s]%>,
2133
+ We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2134
+ <%='<'%>%=ev['content']%>
1808
2135
 
2136
+ Thank you.
1809
2137
  ```
1810
- $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=123455 \
2138
+ The environment provided contains the following additional variable:
2139
+
2140
+ * ev : all details on the transfer event
2141
+
2142
+ Example:
2143
+
2144
+ ```
2145
+ $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
1811
2146
  --query=@json:'{"status":"completed","direction":"receive"}' \
1812
- --notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
2147
+ --notif-to=active --notif-template=@file:mytemplate.erb
1813
2148
  ```
1814
2149
 
2150
+ Options:
2151
+
1815
2152
  * `once_only` keep track of last date it was called, so next call will get only new events
1816
2153
  * `query` filter (on API call)
1817
2154
  * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
1818
2155
 
1819
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
2156
+ Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
1820
2157
 
1821
2158
  ## Using specific transfer ports
1822
2159
 
@@ -1967,7 +2304,7 @@ This can also be set as default using a preset
1967
2304
  One can test the "server" application using the well known demo server:
1968
2305
 
1969
2306
  ```
1970
- $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
2307
+ $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
1971
2308
  $ <%=cmd%> config id default set server aspera_demo_server
1972
2309
  $ <%=cmd%> server browse /aspera-test-dir-large
1973
2310
  $ <%=cmd%> server download /aspera-test-dir-large/200MB
@@ -2063,27 +2400,130 @@ to download files.
2063
2400
  $ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2064
2401
  ```
2065
2402
 
2066
- # Plugin: IBM Aspera Faspex
2403
+ # Plugin: IBM Aspera Faspex5
2404
+
2405
+ 3 authentication methods are supported:
2406
+
2407
+ * jwt
2408
+ * web
2409
+ * boot
2410
+
2411
+ For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
2412
+
2413
+ For web method, create an API client in Faspex, and use: --auth=web
2414
+
2415
+ For boot method: (will be removed in future)
2416
+
2417
+ * open a browser
2418
+ * start developer mode
2419
+ * login to faspex 5
2420
+ * find the first API call with `Authorization` token, and copy it (kind of base64 long string)
2421
+
2422
+ Use it as password and use `--auth=boot`.
2423
+
2424
+ ```
2425
+ $ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2426
+ ```
2427
+
2428
+ Ready to use Faspex5 with CLI.
2429
+
2430
+ # Plugin: IBM Aspera Faspex (4.x)
2431
+
2432
+ Notes:
2433
+
2434
+ * The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2435
+ * For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
2436
+
2437
+ ## Listing Packages
2438
+
2439
+ Command: `faspex package list`
2440
+
2441
+ ### Option `box`
2442
+
2443
+ By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
2444
+
2445
+ ### Option `recipient`
2446
+
2447
+ A user can receive a package because the recipient is:
2448
+
2449
+ * the user himself (default)
2450
+ * the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
2451
+
2452
+ ### Option `query`
2067
2453
 
2068
- Note that the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2454
+ As inboxes may be large, it is possible to use the following query parameters:
2455
+
2456
+ * `count` : (native) number items in one API call (default=0, equivalent to 10)
2457
+ * `page` : (native) id of page in call (default=0)
2458
+ * `startIndex` : (native) index of item to start, default=0, oldest index=0
2459
+ * `max` : maximum number of items
2460
+ * `pmax` : maximum number of pages
2461
+
2462
+ (SQL query is `LIMIT <startIndex>, <count>`)
2463
+
2464
+ The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
2465
+
2466
+ If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
2467
+
2468
+ ### Example
2469
+
2470
+ ```
2471
+ $ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2472
+ ```
2473
+
2474
+ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
2475
+
2476
+ ## Receiving a Package
2477
+
2478
+ The command is `package recv`, possible methods are:
2479
+
2480
+ * provide a package id with option `id`
2481
+ * provide a public link with option `link`
2482
+ * provide a `faspe:` URI with option `link`
2483
+
2484
+ ```
2485
+ $ <%=cmd%> faspex package recv --id=12345
2486
+ $ <%=cmd%> faspex package recv --link=faspe://...
2487
+ ```
2488
+
2489
+ If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
2490
+
2491
+ ```
2492
+ $ <%=cmd%> faspex package list --recipient='*thedropboxname'
2493
+ ```
2494
+
2495
+ if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
2069
2496
 
2070
2497
  ## Sending a Package
2071
2498
 
2072
- Provide delivery info in JSON, example:
2499
+ The command is `faspex package send`. Package information (title, note, metadata, options) is provided in option `delivery_info`. (Refer to Faspex API).
2500
+
2501
+ Example:
2073
2502
 
2074
2503
  ```
2075
- --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}'
2504
+ $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2076
2505
  ```
2077
2506
 
2078
- a note can be added: `"note":"Please ..."`
2507
+ If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
2508
+
2509
+ Additional optional parameters in `delivery_info`:
2510
+
2511
+ * Package Note: : `"note":"note this and that"`
2512
+ * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2079
2513
 
2080
- metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2514
+ ## Email notification on transfer
2081
2515
 
2516
+ Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
2517
+
2518
+ Example:
2519
+
2520
+ ```
2521
+ $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
2522
+ ```
2082
2523
 
2083
- Note for full details, refer to:
2084
- [Reference on Developer Site](https://developer.asperasoft.com/web/faspex/sending)
2524
+ In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
2085
2525
 
2086
- ## operation on dropboxes
2526
+ ## Operation on dropboxes
2087
2527
 
2088
2528
  Example:
2089
2529
 
@@ -2144,6 +2584,8 @@ $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p
2144
2584
  # Plugin: IBM Cloud Object Storage
2145
2585
 
2146
2586
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2587
+ It uses the same transfer service as Aspera on Cloud.
2588
+ see [https://status.aspera.io](https://status.aspera.io)
2147
2589
 
2148
2590
  Required options are either:
2149
2591
 
@@ -2200,7 +2642,14 @@ Endpoints for regions can be found by querying the `endpoints` URL.
2200
2642
  For convenience, let us create a default configuration, for example:
2201
2643
 
2202
2644
  ```
2203
- $ <%=cmd%> conf id mycos update --service-credentials=@val:@json:@file:$HOME/service_creds.json --region=us-south --bucket=laurent
2645
+ $ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
2646
+ $ <%=cmd%> conf id default set cos mycos
2647
+ ```
2648
+
2649
+ or using direct parameters:
2650
+
2651
+ ```
2652
+ $ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2204
2653
  $ <%=cmd%> conf id default set cos mycos
2205
2654
  ```
2206
2655
 
@@ -2218,7 +2667,7 @@ to start from ma configuration file, using <%=tool%> standard options.
2218
2667
 
2219
2668
  # Plugin: Preview
2220
2669
 
2221
- The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on an Aspera HSTS for use primarily in the Aspera on Cloud application.
2670
+ The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
2222
2671
  This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
2223
2672
  Several parameters can be used to tune several aspects:
2224
2673
 
@@ -2266,6 +2715,7 @@ The tool requires the following external tools available in the `PATH`:
2266
2715
  * OptiPNG : `optipng`
2267
2716
  * FFmpeg : `ffmpeg` `ffprobe`
2268
2717
  * Libreoffice : `libreoffice`
2718
+ * ruby gem `mimemagic`
2269
2719
 
2270
2720
  Here shown on Redhat/CentOS.
2271
2721
 
@@ -2277,6 +2727,26 @@ To check if all tools are found properly, execute:
2277
2727
  $ <%=cmd%> preview check
2278
2728
  ```
2279
2729
 
2730
+ ### mimemagic
2731
+
2732
+ To benefit from extra mime type detection install gem mimemagic:
2733
+
2734
+ ```
2735
+ # gem install mimemagic
2736
+ ```
2737
+
2738
+ or to install an earlier version if any problem:
2739
+
2740
+ ```
2741
+ # gem install mimemagic -v '~> 0.3.0'
2742
+ ```
2743
+
2744
+ To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
2745
+
2746
+ If not used, Mime type used for conversion is the one provided by the node API.
2747
+
2748
+ If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
2749
+
2280
2750
  ### Image: Imagemagick and optipng
2281
2751
 
2282
2752
  ```
@@ -2301,90 +2771,111 @@ The generation of preview in based on the use of `unoconv` and `libreoffice`
2301
2771
  # dnf install unoconv
2302
2772
  ```
2303
2773
 
2774
+ * Amazon Linux
2775
+
2776
+ ```
2777
+ # amazon-linux-extras enable libreoffice
2778
+ # yum clean metadata
2779
+ # yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
2780
+ # wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
2781
+ # mv unoconv /usr/bin
2782
+ # chmod a+x /usr/bin/unoconv
2783
+ ```
2304
2784
 
2305
2785
  ## Configuration
2306
2786
 
2307
- Like any <%=tool%> commands, parameters can be passed on command line or using a configuration <%=prst%>. Note that if you use the <%=tool%> run as `xfer` user, like here, the configuration file must be created as the same user. Example using a <%=prst%> named `my_preset_name` (choose any name relevant to you, e.g. the AoC node name, and replace in the following lines):
2787
+ The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)
2788
+
2789
+ Like any <%=tool%> commands, parameters can be passed on command line or using a configuration <%=prst%>. The configuration file must be created with the same user used to run so that it is properly used on runtime.
2790
+
2791
+ Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
2308
2792
 
2309
2793
  ```
2310
2794
  # su -s /bin/bash - xfer
2311
- $ <%=cmd%> config id my_preset_name update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
2312
- $ <%=cmd%> config id default set preview my_preset_name
2795
+ $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
2796
+ $ <%=cmd%> config id default set preview previewconf
2313
2797
  ```
2314
2798
 
2315
- Here we assume that Office file generation is disabled, else remove the option. For the `lock_port` option refer to a previous section in thsi manual.
2799
+ Here we assume that Office file generation is disabled, else remove this option.
2800
+ `lock_port` prevents concurrent execution of generation when using a scheduler.
2316
2801
 
2317
- Once can check if the access key is well configured using:
2802
+ One can check if the access key is well configured using:
2318
2803
 
2319
2804
  ```
2320
- $ <%=cmd%> -Pmy_preset_name node browse /
2805
+ $ <%=cmd%> -Ppreviewconf node browse /
2321
2806
  ```
2322
2807
 
2323
2808
  This shall list the contents of the storage root of the access key.
2324
2809
 
2325
2810
  ## Execution
2326
2811
 
2327
- The tool intentionally supports only a "one shot" mode in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
2328
- It needs to be run regularly to create or update preview files. For that use your best
2812
+ The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
2813
+ It needs to be run on a regular basis to create or update preview files. For that use your best
2329
2814
  reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.
2330
2815
 
2331
- Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consiustent have generate the appropriate access rights, the generation process
2332
- should be run as user `xfer`.
2816
+ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
2333
2817
 
2334
2818
  Lets do a one shot test, using the configuration previously created:
2335
2819
 
2336
2820
  ```
2337
2821
  # su -s /bin/bash - xfer
2338
- $ <%=cmd%> preview scan --overwrite=always
2822
+ xfer$ <%=cmd%> preview scan --overwrite=always
2339
2823
  ```
2340
2824
 
2341
2825
  When the preview generator is first executed it will create a file: `.aspera_access_key`
2342
- which contains the access key used.
2826
+ in the previews folder which contains the access key used.
2343
2827
  On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
2344
2828
 
2345
2829
  ## Configuration for Execution in scheduler
2346
2830
 
2347
- Here is an example of configuration for use with cron on Linux. Adapt the scripts to your own preference.
2831
+ Here is an example of configuration for use with cron on Linux.
2832
+ Adapt the scripts to your own preference.
2348
2833
 
2349
2834
  We assume here that a configuration preset was created as shown previously.
2350
2835
 
2351
- Here the cronjob is created for `root`, and changes the user to `xfer`, also overriding the shell which should be `aspshell`. (adapt the command below, as it would override existing crontab). It is also up to you to use directly the `xfer` user's crontab. This is an example only.
2836
+ Lets first setup a script that will be used in the sceduler and sets up the environment.
2352
2837
 
2838
+ Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
2839
+
2840
+ ```
2841
+ #!/bin/bash
2842
+ # set a timeout protection, just in case
2843
+ case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
2844
+ . /etc/profile.d/rvm.sh
2845
+ rvm use 2.6 --quiet
2846
+ exec timeout ${tmout} <%=cmd%> "${@}"
2353
2847
  ```
2354
- # crontab<<EOF
2355
- 2-59 * * * * su -s /bin/bash - xfer -c 'nice +10 timeout 10m <%=cmd%> preview event --log-level=info --logger=syslog --iteration-file=/tmp/preview_restart.txt'
2356
- 0 * * * * su -s /bin/bash - xfer -c 'nice +10 timeout 30m <%=cmd%> preview scan --log-level=info --logger=syslog'
2848
+
2849
+ Here the cronjob is created for user `xfer`.
2850
+
2851
+ ```
2852
+ xfer$ crontab<<EOF
2853
+ 0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
2854
+ 2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
2357
2855
  EOF
2358
2856
  ```
2359
2857
 
2360
- Nopte that the options here may be located in the config preset, but it was left on the command line to keep stdout for command line execution of preview.
2858
+ Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
2361
2859
 
2362
2860
  ## Candidate detection for creation or update (or deletion)
2363
2861
 
2364
- The tool will find candidates for preview generation using three commands:
2862
+ The tool generates preview files using those commands:
2365
2863
 
2366
2864
  * `trevents` : only recently uploaded files will be tested (transfer events)
2367
2865
  * `events` : only recently uploaded files will be tested (file events: not working)
2368
- * `scan` : deeply scan all files under the access key&apos;s "storage root"
2369
- * `folder` : same as `scan`, but only on the specified folder&apos;s "file identifier"
2370
- * `file` : for an individual file generation
2371
-
2372
- Note that for the `event`, the option `iteration_file` should be specified so that
2373
- successive calls only process new events. This file will hold an identifier
2374
- telling from where to get new events.
2375
-
2376
- It is also possible to test a local file, using the `test` command.
2866
+ * `scan` : recursively scan all files under the access key&apos;s "storage root"
2867
+ * `test` : test using a local file
2377
2868
 
2378
2869
  Once candidate are selected, once candidates are selected,
2379
2870
  a preview is always generated if it does not exist already,
2380
2871
  else if a preview already exist, it will be generated
2381
- using one of three overwrite method:
2872
+ using one of three values for the `overwrite` option:
2382
2873
 
2383
2874
  * `always` : preview is always generated, even if it already exists and is newer than original
2384
2875
  * `never` : preview is generated only if it does not exist already
2385
2876
  * `mtime` : preview is generated only if the original file is newer than the existing
2386
2877
 
2387
- Deletion of preview for deleted source files: not implemented yet.
2878
+ Deletion of preview for deleted source files: not implemented yet (TODO).
2388
2879
 
2389
2880
  If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
2390
2881
 
@@ -2394,6 +2885,14 @@ $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
2394
2885
 
2395
2886
  The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
2396
2887
 
2888
+ When scanning the option `value` has the same behaviour as for the `node find` command.
2889
+
2890
+ For instance to filter out files beginning with `._` do:
2891
+
2892
+ ```
2893
+ ... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
2894
+ ```
2895
+
2397
2896
  ## Preview File types
2398
2897
 
2399
2898
  Two types of preview can be generated:
@@ -2442,14 +2941,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
2442
2941
  The `smtp` option is a hash table (extended value) with the following fields:
2443
2942
  <table>
2444
2943
  <tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
2445
- <tr><td>server</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
2446
- <tr><td>tls</td><td>true</td><td>false</td><td>use of TLS</td></tr>
2447
- <tr><td>port</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
2448
- <tr><td>domain</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
2449
- <tr><td>username</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
2450
- <tr><td>password</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
2451
- <tr><td>from\_email</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
2452
- <tr><td>from\_name</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
2944
+ <tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
2945
+ <tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
2946
+ <tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
2947
+ <tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
2948
+ <tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
2949
+ <tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
2950
+ <tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
2951
+ <tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
2453
2952
  </table>
2454
2953
 
2455
2954
  ## Example of configuration:
@@ -2479,13 +2978,52 @@ $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
2479
2978
  $ <%=cmd%> config id default set config cli_default
2480
2979
  ```
2481
2980
 
2981
+ ## Email templates
2982
+
2983
+ Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
2984
+
2985
+ The template is the full SMTP message, including headers.
2986
+
2987
+ The following variables are defined by default:
2988
+
2989
+ * from_name
2990
+ * from_email
2991
+ * to
2992
+
2993
+ Other variables are defined depending on context.
2994
+
2482
2995
  ## Test
2483
2996
 
2484
2997
  Check settings with `smtp_settings` command. Send test email with `email_test`.
2485
2998
 
2486
2999
  ```
2487
3000
  $ <%=cmd%> config --smtp=@preset:smtp_google smtp
2488
- $ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
3001
+ $ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3002
+ ```
3003
+
3004
+ ## Notifications for transfer status
3005
+
3006
+ An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
3007
+
3008
+ To activate, use option `notif_to`.
3009
+
3010
+ A default e-mail template is used, but it can be overriden with option `notif_template`.
3011
+
3012
+ The environment provided contains the following additional variables:
3013
+
3014
+ * subject
3015
+ * body
3016
+ * global_transfer_status
3017
+ * ts
3018
+
3019
+ Example of template:
3020
+
3021
+ ```
3022
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
3023
+ To: <<%='<'%>%=to%>>
3024
+ Subject: <%='<'%>%=subject%>
3025
+
3026
+ Transfer is: <%='<'%>%=global_transfer_status%>
2489
3027
  ```
2490
3028
 
2491
3029
  # Tool: `asession`
@@ -2531,7 +3069,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
2531
3069
  ## Simple session
2532
3070
 
2533
3071
  ```
2534
- MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
3072
+ MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
2535
3073
 
2536
3074
  echo "${MY_TSPEC}"|asession
2537
3075
  ```
@@ -2544,7 +3082,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
2544
3082
 
2545
3083
  ```
2546
3084
  $ asession
2547
- {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"demoaspera","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
3085
+ {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
2548
3086
  {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
2549
3087
  {"type":"DONE"}
2550
3088
  ```
@@ -2596,7 +3134,7 @@ Interesting ascp features are found in its arguments: (see ascp manual):
2596
3134
  Note that:
2597
3135
 
2598
3136
  * <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
2599
- * not all native ascp arguments are available as standard transfer_spec parameters
3137
+ * most, but not all native ascp arguments are available as standard transfer_spec parameters
2600
3138
  * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
2601
3139
 
2602
3140
  ### server side and configuration
@@ -2618,6 +3156,34 @@ $ <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=1
2618
3156
 
2619
3157
  The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
2620
3158
 
3159
+ # Aspera Health check and Nagios
3160
+
3161
+ Each plugin provide a `health` command that will check the health status of the application. Example:
3162
+
3163
+ ```
3164
+ $ <%=cmd%> console health
3165
+ +--------+-------------+------------+
3166
+ | status | component | message |
3167
+ +--------+-------------+------------+
3168
+ | ok | console api | accessible |
3169
+ +--------+-------------+------------+
3170
+ ```
3171
+
3172
+ Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
3173
+
3174
+ * issuing a transfer to the server
3175
+ * checking web app status with `asctl all:status`
3176
+ * checking daemons process status
3177
+
3178
+ <%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
3179
+
3180
+ ```
3181
+ $ <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3182
+ OK - [transfer:ok]
3183
+ $ <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3184
+ OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
3185
+ ```
3186
+
2621
3187
  # Module: `Aspera`
2622
3188
 
2623
3189
  Main components:
@@ -2637,6 +3203,16 @@ This sample code shows some example of use of the API as well as
2637
3203
  REST API.
2638
3204
  Note: although nice, it's probably a good idea to use RestClient for REST.
2639
3205
 
3206
+ Example of use of the API of Aspera on Cloud:
3207
+
3208
+ ```
3209
+ require 'aspera/aoc'
3210
+
3211
+ aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/<%=cmd%>/aspera_on_cloud_key')),username: 'laurent.martin.aspera@fr.ibm.com',subpath: 'api/v1')
3212
+
3213
+ aoc.read('self')
3214
+ ```
3215
+
2640
3216
  # History
2641
3217
 
2642
3218
  When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
@@ -2653,11 +3229,55 @@ So, it evolved into <%=tool%>:
2653
3229
  * supports transfers with multiple [Transfer Agents](#agents), that&apos;s why transfer parameters moved from ascp command line to [_transfer-spec_](#transferspec) (more reliable , more standard)
2654
3230
  * `ruby` is consistent with other Aspera products
2655
3231
 
2656
-
2657
-
2658
- # Release Notes
2659
-
2660
- * 4.0.0.pre2
3232
+ # Changes (Release notes)
3233
+
3234
+ * <%= gemspec.version.to_s %>
3235
+
3236
+ * new: `faspex package list` retrieves the whole list, not just first page
3237
+ * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3238
+ * new: the error "Remote host is not who we expected" displays a special remediation message
3239
+ * new: `conf ascp spec` displays supported transfer spec
3240
+ * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3241
+ * fix: space character in `faspe:` url are precent encoded if needed
3242
+ * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3243
+ * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3244
+ * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3245
+
3246
+ * 4.2.1
3247
+
3248
+ * new: command `faspex package recv` supports link of type: `faspe:`
3249
+ * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3250
+
3251
+ * 4.2.0
3252
+
3253
+ * new: command `aoc remind` to receive organization membership by email
3254
+ * new: in `preview` option `value` to filter out on file name
3255
+ * new: `initdemo` to initialize for demo server
3256
+ * new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3257
+ * fix: on Windows `conf ascp use` expects ascp.exe
3258
+ * fix: (break) multi_session_threshold is Integer, not String
3259
+ * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3260
+ * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3261
+ * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3262
+ * change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
3263
+
3264
+ * 4.1.0
3265
+
3266
+ * fix: remove keys from transfer spec and command line when not needed
3267
+ * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3268
+ * new: update documentation with regard to offline and docker installation
3269
+ * new: renamed command `nagios_check` to `health`
3270
+ * new: agent `http_gw` now supports upload
3271
+ * new: added option `sdk_url` to install SDK from local file for offline install
3272
+ * new: check new gem version periodically
3273
+ * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3274
+ * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3275
+ * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3276
+ * new: AoC a password can be provided for a public link
3277
+ * new: `conf doc` take an optional parameter to go to a section
3278
+ * new: initial support for Faspex 5 Beta 1
3279
+
3280
+ * 4.0.0
2661
3281
 
2662
3282
  * now available as open source at [<%= gemspec.homepage %>](<%= gemspec.homepage %>) with general cleanup
2663
3283
  * changed default tool name from `mlia` to `ascli`
@@ -2785,7 +3405,7 @@ So, it evolved into <%=tool%>:
2785
3405
 
2786
3406
  * 0.10.6
2787
3407
 
2788
- * FaspManager: transfer spec `authentication` no more needed for local tranfer to use aspera public keys. public keys will be used if there is a token and no key or password is provided.
3408
+ * FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
2789
3409
  * gem version requirements made more open
2790
3410
 
2791
3411
  * 0.10.5
@@ -3012,11 +3632,15 @@ Breaking change:
3012
3632
 
3013
3633
  * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
3014
3634
 
3015
- # BUGS
3635
+ # BUGS, FEATURES, CONTRIBUTION
3016
3636
 
3017
- * This is best effort code without official support, dont expect full capabilities. This code is not supported by IBM/Aspera. You can contact the author for bugs or features.
3637
+ For issues or feature requests use the Github repository and issues.
3018
3638
 
3019
- ## only one value for any option
3639
+ You can also contribute to this open source project.
3640
+
3641
+ One can also create one's own command nplugin.
3642
+
3643
+ ## Only one value for any option
3020
3644
 
3021
3645
  Some commands and sub commands may ask for the same option name.
3022
3646
  Currently, since option definition is position independant (last one wins), it is not possible
@@ -3030,7 +3654,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
3030
3654
 
3031
3655
  ## ED255519 key not supported
3032
3656
 
3033
- ED255519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
3657
+ ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
3658
+
3034
3659
  Without this deactivation, if such key was present the following error was generated:
3035
3660
 
3036
3661
  ```
@@ -3040,7 +3665,21 @@ OpenSSH keys only supported if ED25519 is available
3040
3665
  Which meant that you do not have ruby support for ED25519 SSH keys.
3041
3666
  You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
3042
3667
 
3043
- # TODO
3668
+ ## Error "Remote host is not who we expected"
3669
+
3670
+ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
3671
+
3672
+ Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
3673
+
3674
+ ```
3675
+ --ts=@json:'{"sshfp":null}'
3676
+ ```
3677
+
3678
+ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
3679
+
3680
+ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
3681
+
3682
+ ## Miscelaneous
3044
3683
 
3045
3684
  * remove rest and oauth classes and use ruby standard gems:
3046
3685
 
@@ -3056,9 +3695,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
3056
3695
  * Going through proxy: use env var http_proxy and https_proxy, no_proxy
3057
3696
 
3058
3697
  * easier use with https://github.com/pmq20/ruby-packer
3059
-
3060
- # Contribution
3061
-
3062
- Send comments !
3063
-
3064
- Create your own plugin !