aspera-cli 4.1.0 → 4.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +455 -229
  3. data/docs/Makefile +4 -4
  4. data/docs/README.erb.md +457 -126
  5. data/docs/test_env.conf +19 -2
  6. data/examples/aoc.rb +14 -3
  7. data/examples/faspex4.rb +89 -0
  8. data/lib/aspera/aoc.rb +38 -40
  9. data/lib/aspera/cli/main.rb +65 -33
  10. data/lib/aspera/cli/plugins/aoc.rb +54 -65
  11. data/lib/aspera/cli/plugins/ats.rb +2 -2
  12. data/lib/aspera/cli/plugins/config.rb +158 -137
  13. data/lib/aspera/cli/plugins/faspex.rb +111 -64
  14. data/lib/aspera/cli/plugins/faspex5.rb +35 -48
  15. data/lib/aspera/cli/plugins/node.rb +3 -2
  16. data/lib/aspera/cli/plugins/preview.rb +88 -55
  17. data/lib/aspera/cli/transfer_agent.rb +98 -62
  18. data/lib/aspera/cli/version.rb +1 -1
  19. data/lib/aspera/command_line_builder.rb +48 -31
  20. data/lib/aspera/cos_node.rb +34 -28
  21. data/lib/aspera/environment.rb +2 -2
  22. data/lib/aspera/fasp/aoc.rb +1 -1
  23. data/lib/aspera/fasp/installation.rb +68 -45
  24. data/lib/aspera/fasp/local.rb +89 -45
  25. data/lib/aspera/fasp/manager.rb +3 -0
  26. data/lib/aspera/fasp/node.rb +23 -1
  27. data/lib/aspera/fasp/parameters.rb +57 -86
  28. data/lib/aspera/fasp/parameters.yaml +531 -0
  29. data/lib/aspera/fasp/resume_policy.rb +13 -12
  30. data/lib/aspera/fasp/uri.rb +1 -1
  31. data/lib/aspera/id_generator.rb +22 -0
  32. data/lib/aspera/node.rb +14 -3
  33. data/lib/aspera/oauth.rb +135 -129
  34. data/lib/aspera/persistency_action_once.rb +11 -7
  35. data/lib/aspera/persistency_folder.rb +6 -26
  36. data/lib/aspera/rest.rb +3 -12
  37. data/lib/aspera/secrets.rb +20 -0
  38. data/lib/aspera/sync.rb +40 -35
  39. data/lib/aspera/timer_limiter.rb +22 -0
  40. data/lib/aspera/web_auth.rb +105 -0
  41. metadata +22 -3
  42. data/docs/transfer_spec.html +0 -99
data/docs/README.erb.md CHANGED
@@ -1,21 +1,44 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
- <%
3
- # check that required env vars exist, and files
4
- %w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_TRSPEC}.each do |e|
2
+ <% #```ruby
3
+ # check that required env vars exist, and files
4
+ %w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_DIR_GEM}.each do |e|
5
5
  raise "missing env var #{e}" unless ENV.has_key?(e)
6
6
  raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
7
7
  end
8
- cmd=ENV["EXENAME"] # just command name
8
+ cmd=ENV['EXENAME'] # just command name
9
9
  tool='`'+cmd+'`' # used in text with formatting of command
10
10
  evp=cmd.upcase+'_' # prefix for env vars
11
11
  opprst='option preset' # just the name for "option preset"
12
- prst='['+opprst+'](#lprt)'
13
- prsts='['+opprst+'s](#lprt)'
12
+ prst='['+opprst+'](#lprt)' # name with link
13
+ prsts='['+opprst+'s](#lprt)' # name with link (plural)
14
14
  prstt=opprst.capitalize # in title
15
15
  gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
16
16
  geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
17
+ $LOAD_PATH.unshift(ENV["INCL_DIR_GEM"])
18
+ require 'aspera/fasp/parameters'
19
+ def spec_table
20
+ r='<table><tr><th>Field</th><th>Type</th>'
21
+ Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
22
+ r << '<th>'<<c.to_s.upcase<<'</th>'
23
+ end
24
+ r << '<th>Description</th></tr>'
25
+ Aspera::Fasp::Parameters.man_table.each do |p|
26
+ p[:description] << (p[:description].empty? ? '' : "\n") << "(" << p[:cli] << ")" unless p[:cli].to_s.empty?
27
+ p.delete(:cli)
28
+ p.keys.each{|c|p[c]='&nbsp;' if p[c].to_s.empty?}
29
+ p[:description].gsub!("\n",'<br/>')
30
+ p[:type].gsub!(',','<br/>')
31
+ r << '<tr><td>'<<p[:name]<<'</td><td>'<<p[:type]<<'</td>'
32
+ Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
33
+ r << '<td>'<<p[c]<<'</td>'
34
+ end
35
+ r << '<td>'<<p[:description]<<'</td></tr>'
36
+ end
37
+ r << '</table>'
38
+ return r
39
+ end #```
17
40
  -%>
18
- # <%=tool%> : Command Line Interface for IBM Aspera products
41
+ <font size="+12"><center><%=tool%> : Command Line Interface for IBM Aspera products</center></font>
19
42
 
20
43
  Version : <%= gemspec.version.to_s %>
21
44
 
@@ -29,7 +52,7 @@ Ruby Gem: [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubyge
29
52
 
30
53
  Ruby Doc: [<%= gemspec.metadata['documentation_uri'] %>](<%= gemspec.metadata['documentation_uri'] %>)
31
54
 
32
- Ruby version must be >= <%= gemspec.required_ruby_version %>
55
+ Required Ruby version: <%= gemspec.required_ruby_version %>
33
56
 
34
57
  # <a name="when_to_use"></a>When to use and when not to use
35
58
 
@@ -45,7 +68,8 @@ So it is designed for:
45
68
 
46
69
  <%=tool%> can be seen as a command line tool integrating:
47
70
 
48
- * a configuration file (config.yaml) and advanced command line options
71
+ * a configuration file (config.yaml)
72
+ * advanced command line options
49
73
  * cURL (for REST calls)
50
74
  * Aspera transfer (ascp)
51
75
 
@@ -106,9 +130,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
106
130
 
107
131
  If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
108
132
 
109
- If you want to test with Aspera demo transfer server, a default configuration is created on first use:
133
+ To test with Aspera demo transfer server, setup the environment and then test:
110
134
 
111
135
  ```
136
+ $ <%=cmd%> config initdemo
112
137
  $ <%=cmd%> server browse /
113
138
  :............:...........:......:........:...........................:.......................:
114
139
  : zmode : zuid : zgid : size : mtime : name :
@@ -128,7 +153,7 @@ If you want to use <%=tool%> with another server, and in order to make further c
128
153
  * download a file
129
154
 
130
155
  ```
131
- $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
156
+ $ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
132
157
  updated: myserver
133
158
  $ <%=cmd%> config id default set server myserver
134
159
  updated: default&rarr;server to myserver
@@ -162,18 +187,22 @@ Then, follow the section relative to the product you want to interact with ( Asp
162
187
 
163
188
  # <a name="installation"></a>Installation
164
189
 
165
- It is possible to install *either* as a docker container or step by step on the native host:
190
+ It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
166
191
 
167
- * [Ruby](#ruby) version >= <%= gemspec.required_ruby_version %>
192
+ The direct installation is recommended and consists in installing:
193
+
194
+ * [Ruby](#ruby) version <%= gemspec.required_ruby_version %>
168
195
  * [<%= gemspec.name %>](#the_gem)
169
196
  * [Aspera SDK (ascp)](#fasp_prot)
170
197
 
171
- The following sections provide information on the installation.
198
+ The following sections provide information on the various installation methods.
172
199
 
173
200
  An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
174
201
 
175
202
  ## Docker container
176
203
 
204
+ Use this method only if you know what you do, else use the standard recommended method as described here above.
205
+
177
206
  This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
178
207
 
179
208
  Ensure that you have Docker installed.
@@ -197,11 +226,12 @@ $ ./<%=cmd%> install
197
226
 
198
227
  Start using it !
199
228
 
200
- Note that the tool is run in the container.
229
+ Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
201
230
 
202
- The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>`.
231
+ The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
203
232
 
204
233
  To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
234
+ To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
205
235
 
206
236
  ## <a name="ruby"></a>Ruby
207
237
 
@@ -209,15 +239,15 @@ Use this method to install on the native host.
209
239
 
210
240
  A ruby interpreter is required to run the tool or to use the gem and tool.
211
241
 
212
- Ruby minimum version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
242
+ Required Ruby version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
213
243
 
214
- *Any type of Ruby installation can be used* : rpm, yum, dnf, rvm, brew, windows installer, ... .
244
+ *Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
215
245
 
216
246
  Refer to the following sections for a proposed method for specific operating systems.
217
247
 
218
248
  The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
219
249
  If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
220
- If you have a simpler better way to install Ruby version >= <%= gemspec.required_ruby_version %> : use it !
250
+ If you have a simpler better way to install Ruby version <%= gemspec.required_ruby_version %> : use it !
221
251
 
222
252
  ### Generic: RVM: single user installation (not root)
223
253
 
@@ -789,7 +819,7 @@ $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
789
819
  The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
790
820
 
791
821
  ```
792
- $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera --ts=@json:'{"precalculate_job_size":true}'
822
+ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
793
823
  ```
794
824
 
795
825
  * This creates a <%=prst%> `demo_server` with all provided options.
@@ -797,13 +827,13 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
797
827
  The command `set` allows setting individual options in a <%=prst%>.
798
828
 
799
829
  ```
800
- $ <%=cmd%> config id demo_server set password demoaspera
830
+ $ <%=cmd%> config id demo_server set password _demo_pass_
801
831
  ```
802
832
 
803
833
  The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
804
834
 
805
835
  ```
806
- $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"demoaspera","ts":{"precalculate_job_size":true}}'
836
+ $ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
807
837
  ```
808
838
 
809
839
  A good practice is to not manually edit the configuration file and use modification commands instead.
@@ -840,11 +870,19 @@ Note that special plugin name: `config` can be associated with a preset that is
840
870
  Operations on this preset are done using regular `config` operations:
841
871
 
842
872
  ```
843
- $ <%=cmd%> config id default set _plugin_name_ _defauklt_preset_for_plugin_
873
+ $ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
844
874
  $ <%=cmd%> config id default get _plugin_name_
845
- "_defauklt_preset_for_plugin_"
875
+ "_default_preset_for_plugin_"
846
876
  ```
847
877
 
878
+ ### <a name="lprtdef"></a>Special Plugin: config
879
+
880
+ Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
881
+
882
+ When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
883
+
884
+ If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
885
+
848
886
  ### Format of file
849
887
 
850
888
  The configuration file is a hash in a YAML file. Example:
@@ -860,7 +898,7 @@ cli_default:
860
898
  demo_server:
861
899
  url: ssh://demo.asperasoft.com:33001
862
900
  username: asperaweb
863
- password: demoaspera
901
+ password: _demo_pass_
864
902
  ```
865
903
 
866
904
  We can see here:
@@ -929,7 +967,6 @@ A <%=prst%> value can be removed with `unset`:
929
967
  $ <%=cmd%> config id cli_default unset interactive
930
968
  ```
931
969
 
932
-
933
970
  ### Examples
934
971
 
935
972
  For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
@@ -1049,21 +1086,42 @@ The `config` plugin also allows specification for the use of a local FASP client
1049
1086
 
1050
1087
  ```
1051
1088
  $ <%=cmd%> config ascp show
1052
- /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
1089
+ /Users/laurent/.aspera/ascli/sdk/ascp
1090
+ $ <%=cmd%> config ascp info
1091
+ +--------------------+-----------------------------------------------------------+
1092
+ | key | value |
1093
+ +--------------------+-----------------------------------------------------------+
1094
+ | ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
1095
+ ...
1053
1096
  ```
1054
1097
 
1055
1098
  ### Selection of local `ascp`
1056
1099
 
1100
+ By default, <%=tool%> uses any found local product with ascp, including SDK.
1101
+
1057
1102
  To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
1058
1103
 
1059
- To permanently use another ascp:
1104
+ For a permanent change, the command `config ascp use` sets the same parameter for the global default.
1105
+
1106
+ Using a POSIX shell:
1060
1107
 
1061
1108
  ```
1062
1109
  $ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
1063
- saved to default global preset /Users/laurent/Applications/Aspera CLI/bin/ascp
1110
+ ascp version: 4.0.0.182279
1111
+ Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
1112
+ Saved to default global preset global_common_defaults
1064
1113
  ```
1065
1114
 
1066
- This sets up a global default.
1115
+ Windows:
1116
+
1117
+ ```
1118
+ $ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1119
+ ascp version: 4.0.0.182279
1120
+ Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1121
+ Saved to default global preset global_common_defaults
1122
+ ```
1123
+
1124
+ If the path has spaces, read section: [Shell and Command line parsing](#parsing).
1067
1125
 
1068
1126
  ### List locally installed Aspera Transfer products
1069
1127
 
@@ -1150,7 +1208,7 @@ will effectively push files to the related server from the agent node.
1150
1208
 
1151
1209
  ### <a name="direct"></a>Direct (local ascp using FASPManager API)
1152
1210
 
1153
- By default the CLI will use a local FASP protocol, equivalent to specifying `--transfer=direct`.
1211
+ By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
1154
1212
  <%=tool%> will detect locally installed Aspera products.
1155
1213
  Refer to section [FASP](#client).
1156
1214
 
@@ -1160,17 +1218,41 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
1160
1218
  * `EX_http_proxy_url` (proxy for legacy http fallback)
1161
1219
  * `EX_ascp_args`
1162
1220
 
1163
- The `transfer-info` optionally provides the following auto resume parameters:
1221
+ The `transfer-info` accepts the following optional parameters:
1164
1222
 
1165
1223
  <table>
1166
- <tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1167
- <tr><td>iter_max</td>. <td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
1168
- <tr><td>sleep_initial</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
1169
- <tr><td>sleep_factor</td> <td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
1170
- <tr><td>sleep_max</td>. <td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
1171
- <tr><td>wss</td> <td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1224
+ <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1225
+ <tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
1226
+ <tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
1227
+ <tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
1228
+ <tr><td>multi_incr_udp</td><td>Bool</td><td>true</td><td>Multi Session</td><td>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)</td></tr>
1229
+ <tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
1172
1230
  </table>
1173
1231
 
1232
+ Resume parameters:
1233
+
1234
+ In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1235
+ Sleep between iteration is:
1236
+
1237
+ ```
1238
+ max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
1239
+ ```
1240
+
1241
+ <table>
1242
+ <tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
1243
+ <tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
1244
+ <tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
1245
+ <tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
1246
+ <tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
1247
+ </table>
1248
+
1249
+ Examples:
1250
+
1251
+ ```
1252
+ $ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1253
+ $ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1254
+ ```
1255
+
1174
1256
  ### IBM Aspera Connect Client GUI
1175
1257
 
1176
1258
  By specifying option: `--transfer=connect`, <%=tool%> will start transfers
@@ -1220,7 +1302,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
1220
1302
 
1221
1303
  <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
1222
1304
 
1223
- If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
1305
+ If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
1224
1306
 
1225
1307
  It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
1226
1308
 
@@ -1233,11 +1315,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
1233
1315
 
1234
1316
  ## <a name="transferparams"></a>Transfer Parameters
1235
1317
 
1236
- All standard _transfer-spec_ parameters can be overloaded. To display parameters,
1237
- run in debug mode (--log-level=debug). [_transfer-spec_](#transferspec) can
1238
- also be saved/overridden in the config file.
1318
+ All standard _transfer-spec_ parameters can be speficied.
1319
+ [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1320
+
1321
+ References:
1322
+
1323
+ * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1324
+ * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1325
+
1326
+ Parameters can be displayed with commands:
1239
1327
 
1240
- <%= File.read(ENV['INCL_TRSPEC']).gsub(/.*<body>(.*)<\/body>.*/m,'\1') %>
1328
+ ```
1329
+ $ <%=cmd%> config ascp spec
1330
+ $ <%=cmd%> config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
1331
+ ```
1332
+
1333
+ Columns:
1334
+
1335
+ * D=Direct (local `ascp` execution)
1336
+ * N=Node API
1337
+ * C=Connect Client
1338
+ * arg=`ascp` argument or environment variable
1339
+
1340
+ Fields with EX_ prefix are extensions to transfer agent `direct`. (only in <%=tool%>).
1341
+
1342
+ <%= spec_table %>
1241
1343
 
1242
1344
  ### Destination folder for transfers
1243
1345
 
@@ -1319,7 +1421,7 @@ Source files are located on "Aspera on cloud", when :
1319
1421
 
1320
1422
  ### <a name="multisession"></a>Support of multi-session
1321
1423
 
1322
- Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
1424
+ Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1323
1425
 
1324
1426
  * when agent=node :
1325
1427
 
@@ -1340,6 +1442,7 @@ shall be preferred.
1340
1442
 
1341
1443
  Multi-session spawn is done by <%=tool%>.
1342
1444
 
1445
+ When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
1343
1446
 
1344
1447
  ### Examples
1345
1448
 
@@ -1449,7 +1552,7 @@ For this, specify the option: `--use-generic-client=no`.
1449
1552
 
1450
1553
  This will guide you through the steps to create.
1451
1554
 
1452
- ## <a name="aocwizard"></a>Configuration: using manual setup
1555
+ ## <a name="aocmanual"></a>Configuration: using manual setup
1453
1556
 
1454
1557
  If you used the wizard (recommended): skip this section.
1455
1558
 
@@ -1615,7 +1718,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
1615
1718
 
1616
1719
  * change auth method to JWT
1617
1720
  * provide location of private key
1618
- * provide username to login as (OAuthg "subject")
1721
+ * provide username to login as (OAuth "subject")
1619
1722
 
1620
1723
  Execute:
1621
1724
 
@@ -1713,10 +1816,10 @@ $ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id
1713
1816
  : 98398 : dummyuser1@example.com :
1714
1817
  : 98399 : dummyuser2@example.com :
1715
1818
  :.......:........................:
1716
- $ thelist=$(echo $(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email --field=id --format=csv)|tr ' ' ,)
1819
+ $ thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
1717
1820
  $ echo $thelist
1718
- 98398,98399
1719
- $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
1821
+ ["113501","354061"]
1822
+ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
1720
1823
  :.......:.........:
1721
1824
  : id : status :
1722
1825
  :.......:.........:
@@ -1725,6 +1828,14 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
1725
1828
  :.......:.........:
1726
1829
  ```
1727
1830
 
1831
+ * <a name="deactuser"></a>Find deactivated users since more than 2 years
1832
+
1833
+ ```
1834
+ ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
1835
+ ```
1836
+
1837
+ To delete them use the same method as before
1838
+
1728
1839
  * Display current user's workspaces
1729
1840
 
1730
1841
  ```
@@ -1966,10 +2077,10 @@ $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
1966
2077
  * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
1967
2078
  * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
1968
2079
 
1969
- Typically, one would regularly execute this command on a regular basis, using the method oif your choice:
2080
+ Typically, one would regularly execute this command on a regular basis, using the method of your choice:
1970
2081
 
1971
- * Windows scheduler
1972
- * cron
2082
+ * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2083
+ * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
1973
2084
  * etc...
1974
2085
 
1975
2086
  ## Download Files
@@ -2026,19 +2137,40 @@ The activity app can be queried with:
2026
2137
  $ <%=cmd%> aoc admin analytics transfers
2027
2138
  ```
2028
2139
 
2029
- It can also support filters and send notification email with a template:
2140
+ It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2141
+
2142
+ `mytemplate.erb`:
2143
+
2144
+ ```
2145
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2146
+ To: <<%='<'%>%=ev['user_email']%>>
2147
+ Subject: <%='<'%>%=ev['files_completed']%> files received
2148
+
2149
+ Dear <%='<'%>%=ev[:user_email.to_s]%>,
2150
+ We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2151
+ <%='<'%>%=ev['content']%>
2152
+
2153
+ Thank you.
2154
+ ```
2155
+ The environment provided contains the following additional variable:
2156
+
2157
+ * ev : all details on the transfer event
2158
+
2159
+ Example:
2030
2160
 
2031
2161
  ```
2032
- $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=123455 \
2162
+ $ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2033
2163
  --query=@json:'{"status":"completed","direction":"receive"}' \
2034
- --notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
2164
+ --notif-to=active --notif-template=@file:mytemplate.erb
2035
2165
  ```
2036
2166
 
2167
+ Options:
2168
+
2037
2169
  * `once_only` keep track of last date it was called, so next call will get only new events
2038
2170
  * `query` filter (on API call)
2039
2171
  * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2040
2172
 
2041
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
2173
+ Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
2042
2174
 
2043
2175
  ## Using specific transfer ports
2044
2176
 
@@ -2099,7 +2231,37 @@ $ <%=cmd%> ats api_key create
2099
2231
  +--------+----------------------------------------------+
2100
2232
  $ <%=cmd%> config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2101
2233
  ```
2234
+ ## Cross Organization transfers
2235
+
2236
+ It is possible to transfer files directly between organizations without having to first download locally and then upload...
2237
+
2238
+ Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
2239
+
2240
+ Procedure to send a file from org1 to org2:
2241
+
2242
+ * Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2243
+ * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2244
+ * Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
2245
+ * Check that access works and locate the destination folder `mydestfolder`
2246
+ * execute the following:
2247
+
2248
+ ```
2249
+ $ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2250
+ ```
2251
+
2252
+ Explanation:
2253
+
2254
+ * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2255
+ * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2256
+ * `--format=json` format the output in JSON (instead of default text table)
2257
+ * `--display=data` display only the result, and remove other information, such as workspace name
2258
+ * `|` the standard output of the first command is fed into the second one
2259
+ * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2260
+ * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2261
+ * `--transfer=node` use transfer agent type `node` instead of default `direct`
2262
+ * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2102
2263
 
2264
+ Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
2103
2265
  ## Examples
2104
2266
 
2105
2267
  Example: create access key on softlayer:
@@ -2189,7 +2351,7 @@ This can also be set as default using a preset
2189
2351
  One can test the "server" application using the well known demo server:
2190
2352
 
2191
2353
  ```
2192
- $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
2354
+ $ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
2193
2355
  $ <%=cmd%> config id default set server aspera_demo_server
2194
2356
  $ <%=cmd%> server browse /aspera-test-dir-large
2195
2357
  $ <%=cmd%> server download /aspera-test-dir-large/200MB
@@ -2285,16 +2447,19 @@ to download files.
2285
2447
  $ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
2286
2448
  ```
2287
2449
 
2288
- # Plugin: IBM Aspera Faspex
2450
+ # Plugin: IBM Aspera Faspex5
2289
2451
 
2290
- Notes:
2452
+ 3 authentication methods are supported:
2453
+
2454
+ * jwt
2455
+ * web
2456
+ * boot
2291
2457
 
2292
- * the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2293
- * for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
2458
+ For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
2294
2459
 
2295
- ## Faspex 5 Beta1
2460
+ For web method, create an API client in Faspex, and use: --auth=web
2296
2461
 
2297
- As the web UI does not yet allow adding API client yet, the way to use CLI is:
2462
+ For boot method: (will be removed in future)
2298
2463
 
2299
2464
  * open a browser
2300
2465
  * start developer mode
@@ -2307,30 +2472,74 @@ Use it as password and use `--auth=boot`.
2307
2472
  $ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2308
2473
  ```
2309
2474
 
2310
- An JWT client can then be created with a private key:
2475
+ Ready to use Faspex5 with CLI.
2476
+
2477
+ # Plugin: IBM Aspera Faspex (4.x)
2478
+
2479
+ Notes:
2480
+
2481
+ * The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2482
+ * For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
2483
+
2484
+ ## Listing Packages
2485
+
2486
+ Command: `faspex package list`
2487
+
2488
+ ### Option `box`
2489
+
2490
+ By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
2491
+
2492
+ ### Option `recipient`
2493
+
2494
+ A user can receive a package because the recipient is:
2495
+
2496
+ * the user himself (default)
2497
+ * the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
2498
+
2499
+ ### Option `query`
2500
+
2501
+ As inboxes may be large, it is possible to use the following query parameters:
2502
+
2503
+ * `count` : (native) number items in one API call (default=0, equivalent to 10)
2504
+ * `page` : (native) id of page in call (default=0)
2505
+ * `startIndex` : (native) index of item to start, default=0, oldest index=0
2506
+ * `max` : maximum number of items
2507
+ * `pmax` : maximum number of pages
2508
+
2509
+ (SQL query is `LIMIT <startIndex>, <count>`)
2510
+
2511
+ The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
2512
+
2513
+ If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
2514
+
2515
+ ### Example
2311
2516
 
2312
2517
  ```
2313
- $ jsonk=$(openssl rsa -in ~/.aspera/ascli/aspera_on_cloud_key -pubout 2> /dev/null | sed -e :a -e N -e '$!ba' -e 's/\n/\\n/g')
2314
- $ <%=cmd%> faspex5 -Pf5boot auth_client create --value=@json:'{"name":"hello","client_type":"public","redirect_uris":["https://localhost:12345"],"allow_jwt_grant":true,"public_key":"'$jsonk'"}'
2518
+ $ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
2315
2519
  ```
2316
2520
 
2317
- or deleted by name:
2521
+ List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
2522
+
2523
+ ## Receiving a Package
2524
+
2525
+ The command is `package recv`, possible methods are:
2526
+
2527
+ * provide a package id with option `id`
2528
+ * provide a public link with option `link`
2529
+ * provide a `faspe:` URI with option `link`
2318
2530
 
2319
2531
  ```
2320
- $ id=$(<%=cmd%> faspex5 auth_client list --select=@json:'{"name":"hello"}' --fields=client_id --format=csv)
2321
- $ <%=cmd%> faspex5 auth_client delete --id=$id
2532
+ $ <%=cmd%> faspex package recv --id=12345
2533
+ $ <%=cmd%> faspex package recv --link=faspe://...
2322
2534
  ```
2323
2535
 
2324
- Once the API client is created with a client_id and secret (result of create command), create a configuration:
2536
+ If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
2325
2537
 
2326
2538
  ```
2327
- $ <%=cmd%> conf id f5 update --url=https://localhost/aspera/faspex --auth=jwt --client-id=abcd --client-secret=def --username=pierre@example.com --private-key=@val:@file:~/.aspera/ascli/aspera_on_cloud_key
2328
- $ <%=cmd%> conf id default set faspex5 f5
2329
- ```
2330
-
2331
- Ready to use Faspex5 with CLI.
2539
+ $ <%=cmd%> faspex package list --recipient='*thedropboxname'
2540
+ ```
2332
2541
 
2333
- Once the graphical registration form exist, ther bootstrap method can be removed.
2542
+ if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
2334
2543
 
2335
2544
  ## Sending a Package
2336
2545
 
@@ -2349,7 +2558,19 @@ Additional optional parameters in `delivery_info`:
2349
2558
  * Package Note: : `"note":"note this and that"`
2350
2559
  * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2351
2560
 
2352
- ## operation on dropboxes
2561
+ ## Email notification on transfer
2562
+
2563
+ Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
2564
+
2565
+ Example:
2566
+
2567
+ ```
2568
+ $ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
2569
+ ```
2570
+
2571
+ In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
2572
+
2573
+ ## Operation on dropboxes
2353
2574
 
2354
2575
  Example:
2355
2576
 
@@ -2359,7 +2580,7 @@ $ <%=cmd%> faspex v4 dropbox list
2359
2580
  $ <%=cmd%> faspex v4 dropbox delete --id=36
2360
2581
  ```
2361
2582
 
2362
- ## remote sources
2583
+ ## Remote sources
2363
2584
 
2364
2585
  Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
2365
2586
  the node API is used, for this it is required to add a section ":storage" that links
@@ -2410,29 +2631,38 @@ $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p
2410
2631
  # Plugin: IBM Cloud Object Storage
2411
2632
 
2412
2633
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2413
- It uses the same transfer service as Aspera on Cloud.
2414
- see [https://status.aspera.io](https://status.aspera.io)
2634
+ It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
2635
+ Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
2415
2636
 
2416
- Required options are either:
2637
+ There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the forst method. If you dont have credentials but have access to the IBM Cloud console, then use the second method.
2638
+
2639
+ ## Using endpoint, apikey and Ressource Instance ID (CRN)
2640
+
2641
+ If you have those parameters already, then following options shall be provided:
2417
2642
 
2418
2643
  * `bucket` bucket name
2419
2644
  * `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
2420
2645
  * `apikey` API Key
2421
2646
  * `crn` resource instance id
2422
2647
 
2423
- or:
2648
+ For example, let us create a default configuration:
2424
2649
 
2425
- * `bucket` bucket name
2426
- * `region` bucket region, e.g. eu-de
2427
- * `service_credentials` see below
2650
+ ```
2651
+ $ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2652
+ $ <%=cmd%> conf id default set cos mycos
2653
+ ```
2654
+
2655
+ Then, jump to the transfer example.
2428
2656
 
2429
- Service credentials are directly created using the IBM cloud web ui. Navigate to:
2657
+ ## Using service credential file
2658
+
2659
+ If you are the COS administrator and dont have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
2430
2660
 
2431
2661
  Navigation Menu &rarr; Resource List &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
2432
2662
 
2433
2663
  Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
2434
2664
 
2435
- or using the CLI:
2665
+ or using the IBM Cloud CLI:
2436
2666
 
2437
2667
  ```
2438
2668
  $ ibmcloud resource service-keys
@@ -2463,37 +2693,42 @@ The field `resource_instance_id` is for option `crn`
2463
2693
 
2464
2694
  The field `apikey` is for option `apikey`
2465
2695
 
2466
- Endpoints for regions can be found by querying the `endpoints` URL.
2696
+ (If needed: endpoints for regions can be found by querying the `endpoints` URL.)
2697
+
2698
+ The required options for this method are:
2699
+
2700
+ * `bucket` bucket name
2701
+ * `region` bucket region, e.g. eu-de
2702
+ * `service_credentials` see below
2467
2703
 
2468
- For convenience, let us create a default configuration, for example:
2704
+ For example, let us create a default configuration:
2469
2705
 
2470
2706
  ```
2471
2707
  $ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
2472
2708
  $ <%=cmd%> conf id default set cos mycos
2473
2709
  ```
2474
2710
 
2475
- or using direct parameters:
2711
+ ## Operations, transfers
2476
2712
 
2477
- ```
2478
- $ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2479
- $ <%=cmd%> conf id default set cos mycos
2480
- ```
2713
+ Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
2481
2714
 
2482
- Now, Ready to do operations, a subset of "node" plugin operations are supported, basically node API:
2715
+ A subset of `node` plugin operations are supported, basically node API:
2483
2716
 
2484
2717
  ```
2485
- $ <%=cmd%> cos node browse /
2486
- $ <%=cmd%> cos node upload myfile.txt
2718
+ $ <%=cmd%> cos node info
2719
+ $ <%=cmd%> cos node upload 'faux:///sample1G?1g'
2487
2720
  ```
2488
2721
 
2722
+ Note: we generate a dummy file `sample1G` if size 2GB using the `faux` PVCL (man ascp), but you can of course send a real file by specifying a real file instead.
2723
+
2489
2724
  # Plugin: IBM Aspera Sync
2490
2725
 
2491
- A basic plugin to start an "async" using <%=tool%>. The main advantage is the possibility
2492
- to start from ma configuration file, using <%=tool%> standard options.
2726
+ A basic plugin to start an "async" using <%=tool%>.
2727
+ The main advantage is the possibility to start from ma configuration file, using <%=tool%> standard options.
2493
2728
 
2494
2729
  # Plugin: Preview
2495
2730
 
2496
- The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on an Aspera HSTS for use primarily in the Aspera on Cloud application.
2731
+ The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
2497
2732
  This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
2498
2733
  Several parameters can be used to tune several aspects:
2499
2734
 
@@ -2622,9 +2857,10 @@ $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=
2622
2857
  $ <%=cmd%> config id default set preview previewconf
2623
2858
  ```
2624
2859
 
2625
- Here we assume that Office file generation is disabled, else remove the option. `lock_port` prevents concurrent execution of generation when using a scheduler.
2860
+ Here we assume that Office file generation is disabled, else remove this option.
2861
+ `lock_port` prevents concurrent execution of generation when using a scheduler.
2626
2862
 
2627
- Once can check if the access key is well configured using:
2863
+ One can check if the access key is well configured using:
2628
2864
 
2629
2865
  ```
2630
2866
  $ <%=cmd%> -Ppreviewconf node browse /
@@ -2710,6 +2946,14 @@ $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
2710
2946
 
2711
2947
  The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
2712
2948
 
2949
+ When scanning the option `value` has the same behaviour as for the `node find` command.
2950
+
2951
+ For instance to filter out files beginning with `._` do:
2952
+
2953
+ ```
2954
+ ... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
2955
+ ```
2956
+
2713
2957
  ## Preview File types
2714
2958
 
2715
2959
  Two types of preview can be generated:
@@ -2758,14 +3002,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
2758
3002
  The `smtp` option is a hash table (extended value) with the following fields:
2759
3003
  <table>
2760
3004
  <tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
2761
- <tr><td>server</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
2762
- <tr><td>tls</td><td>true</td><td>false</td><td>use of TLS</td></tr>
2763
- <tr><td>port</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
2764
- <tr><td>domain</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
2765
- <tr><td>username</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
2766
- <tr><td>password</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
2767
- <tr><td>from\_email</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
2768
- <tr><td>from\_name</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
3005
+ <tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
3006
+ <tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
3007
+ <tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
3008
+ <tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
3009
+ <tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
3010
+ <tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
3011
+ <tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
3012
+ <tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
2769
3013
  </table>
2770
3014
 
2771
3015
  ## Example of configuration:
@@ -2795,13 +3039,52 @@ $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
2795
3039
  $ <%=cmd%> config id default set config cli_default
2796
3040
  ```
2797
3041
 
3042
+ ## Email templates
3043
+
3044
+ Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
3045
+
3046
+ The template is the full SMTP message, including headers.
3047
+
3048
+ The following variables are defined by default:
3049
+
3050
+ * from_name
3051
+ * from_email
3052
+ * to
3053
+
3054
+ Other variables are defined depending on context.
3055
+
2798
3056
  ## Test
2799
3057
 
2800
3058
  Check settings with `smtp_settings` command. Send test email with `email_test`.
2801
3059
 
2802
3060
  ```
2803
3061
  $ <%=cmd%> config --smtp=@preset:smtp_google smtp
2804
- $ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
3062
+ $ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3063
+ ```
3064
+
3065
+ ## Notifications for transfer status
3066
+
3067
+ An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
3068
+
3069
+ To activate, use option `notif_to`.
3070
+
3071
+ A default e-mail template is used, but it can be overriden with option `notif_template`.
3072
+
3073
+ The environment provided contains the following additional variables:
3074
+
3075
+ * subject
3076
+ * body
3077
+ * global_transfer_status
3078
+ * ts
3079
+
3080
+ Example of template:
3081
+
3082
+ ```
3083
+ From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
3084
+ To: <<%='<'%>%=to%>>
3085
+ Subject: <%='<'%>%=subject%>
3086
+
3087
+ Transfer is: <%='<'%>%=global_transfer_status%>
2805
3088
  ```
2806
3089
 
2807
3090
  # Tool: `asession`
@@ -2847,7 +3130,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
2847
3130
  ## Simple session
2848
3131
 
2849
3132
  ```
2850
- MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
3133
+ MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
2851
3134
 
2852
3135
  echo "${MY_TSPEC}"|asession
2853
3136
  ```
@@ -2860,7 +3143,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
2860
3143
 
2861
3144
  ```
2862
3145
  $ asession
2863
- {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"demoaspera","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
3146
+ {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
2864
3147
  {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
2865
3148
  {"type":"DONE"}
2866
3149
  ```
@@ -2912,7 +3195,7 @@ Interesting ascp features are found in its arguments: (see ascp manual):
2912
3195
  Note that:
2913
3196
 
2914
3197
  * <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
2915
- * not all native ascp arguments are available as standard transfer_spec parameters
3198
+ * most, but not all native ascp arguments are available as standard transfer_spec parameters
2916
3199
  * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
2917
3200
 
2918
3201
  ### server side and configuration
@@ -3009,7 +3292,42 @@ So, it evolved into <%=tool%>:
3009
3292
 
3010
3293
  # Changes (Release notes)
3011
3294
 
3012
- * 4.x
3295
+ * <%= gemspec.version.to_s %>
3296
+
3297
+ * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on `direct` transfer agent.
3298
+ * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3299
+
3300
+ * 4.2.2
3301
+
3302
+ * new: `faspex package list` retrieves the whole list, not just first page
3303
+ * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3304
+ * new: the error "Remote host is not who we expected" displays a special remediation message
3305
+ * new: `conf ascp spec` displays supported transfer spec
3306
+ * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3307
+ * fix: space character in `faspe:` url are precent encoded if needed
3308
+ * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3309
+ * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3310
+ * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3311
+
3312
+ * 4.2.1
3313
+
3314
+ * new: command `faspex package recv` supports link of type: `faspe:`
3315
+ * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3316
+
3317
+ * 4.2.0
3318
+
3319
+ * new: command `aoc remind` to receive organization membership by email
3320
+ * new: in `preview` option `value` to filter out on file name
3321
+ * new: `initdemo` to initialize for demo server
3322
+ * new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3323
+ * fix: on Windows `conf ascp use` expects ascp.exe
3324
+ * fix: (break) multi_session_threshold is Integer, not String
3325
+ * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3326
+ * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3327
+ * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3328
+ * change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
3329
+
3330
+ * 4.1.0
3013
3331
 
3014
3332
  * fix: remove keys from transfer spec and command line when not needed
3015
3333
  * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
@@ -3380,11 +3698,15 @@ Breaking change:
3380
3698
 
3381
3699
  * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
3382
3700
 
3383
- # BUGS
3701
+ # BUGS, FEATURES, CONTRIBUTION
3702
+
3703
+ For issues or feature requests use the Github repository and issues.
3704
+
3705
+ You can also contribute to this open source project.
3384
3706
 
3385
- * This is best effort code without official support, dont expect full capabilities. This code is not supported by IBM/Aspera. You can contact the author for bugs or features.
3707
+ One can also create one's own command nplugin.
3386
3708
 
3387
- ## only one value for any option
3709
+ ## Only one value for any option
3388
3710
 
3389
3711
  Some commands and sub commands may ask for the same option name.
3390
3712
  Currently, since option definition is position independant (last one wins), it is not possible
@@ -3398,7 +3720,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
3398
3720
 
3399
3721
  ## ED255519 key not supported
3400
3722
 
3401
- ED255519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
3723
+ ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
3724
+
3402
3725
  Without this deactivation, if such key was present the following error was generated:
3403
3726
 
3404
3727
  ```
@@ -3408,7 +3731,21 @@ OpenSSH keys only supported if ED25519 is available
3408
3731
  Which meant that you do not have ruby support for ED25519 SSH keys.
3409
3732
  You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
3410
3733
 
3411
- # TODO
3734
+ ## Error "Remote host is not who we expected"
3735
+
3736
+ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
3737
+
3738
+ Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
3739
+
3740
+ ```
3741
+ --ts=@json:'{"sshfp":null}'
3742
+ ```
3743
+
3744
+ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
3745
+
3746
+ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
3747
+
3748
+ ## Miscelaneous
3412
3749
 
3413
3750
  * remove rest and oauth classes and use ruby standard gems:
3414
3751
 
@@ -3424,9 +3761,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
3424
3761
  * Going through proxy: use env var http_proxy and https_proxy, no_proxy
3425
3762
 
3426
3763
  * easier use with https://github.com/pmq20/ruby-packer
3427
-
3428
- # Contribution
3429
-
3430
- Send comments !
3431
-
3432
- Create your own plugin !