aspera-cli 4.1.0 → 4.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +455 -229
- data/docs/Makefile +4 -4
- data/docs/README.erb.md +457 -126
- data/docs/test_env.conf +19 -2
- data/examples/aoc.rb +14 -3
- data/examples/faspex4.rb +89 -0
- data/lib/aspera/aoc.rb +38 -40
- data/lib/aspera/cli/main.rb +65 -33
- data/lib/aspera/cli/plugins/aoc.rb +54 -65
- data/lib/aspera/cli/plugins/ats.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +158 -137
- data/lib/aspera/cli/plugins/faspex.rb +111 -64
- data/lib/aspera/cli/plugins/faspex5.rb +35 -48
- data/lib/aspera/cli/plugins/node.rb +3 -2
- data/lib/aspera/cli/plugins/preview.rb +88 -55
- data/lib/aspera/cli/transfer_agent.rb +98 -62
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +48 -31
- data/lib/aspera/cos_node.rb +34 -28
- data/lib/aspera/environment.rb +2 -2
- data/lib/aspera/fasp/aoc.rb +1 -1
- data/lib/aspera/fasp/installation.rb +68 -45
- data/lib/aspera/fasp/local.rb +89 -45
- data/lib/aspera/fasp/manager.rb +3 -0
- data/lib/aspera/fasp/node.rb +23 -1
- data/lib/aspera/fasp/parameters.rb +57 -86
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/resume_policy.rb +13 -12
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/id_generator.rb +22 -0
- data/lib/aspera/node.rb +14 -3
- data/lib/aspera/oauth.rb +135 -129
- data/lib/aspera/persistency_action_once.rb +11 -7
- data/lib/aspera/persistency_folder.rb +6 -26
- data/lib/aspera/rest.rb +3 -12
- data/lib/aspera/secrets.rb +20 -0
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/timer_limiter.rb +22 -0
- data/lib/aspera/web_auth.rb +105 -0
- metadata +22 -3
- data/docs/transfer_spec.html +0 -99
data/docs/README.erb.md
CHANGED
|
@@ -1,21 +1,44 @@
|
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
|
2
|
-
<%
|
|
3
|
-
|
|
4
|
-
%w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION
|
|
2
|
+
<% #```ruby
|
|
3
|
+
# check that required env vars exist, and files
|
|
4
|
+
%w{EXENAME GEMSPEC INCL_USAGE INCL_COMMANDS INCL_ASESSION INCL_DIR_GEM}.each do |e|
|
|
5
5
|
raise "missing env var #{e}" unless ENV.has_key?(e)
|
|
6
6
|
raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
|
|
7
7
|
end
|
|
8
|
-
cmd=ENV[
|
|
8
|
+
cmd=ENV['EXENAME'] # just command name
|
|
9
9
|
tool='`'+cmd+'`' # used in text with formatting of command
|
|
10
10
|
evp=cmd.upcase+'_' # prefix for env vars
|
|
11
11
|
opprst='option preset' # just the name for "option preset"
|
|
12
|
-
prst='['+opprst+'](#lprt)'
|
|
13
|
-
prsts='['+opprst+'s](#lprt)'
|
|
12
|
+
prst='['+opprst+'](#lprt)' # name with link
|
|
13
|
+
prsts='['+opprst+'s](#lprt)' # name with link (plural)
|
|
14
14
|
prstt=opprst.capitalize # in title
|
|
15
15
|
gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
|
|
16
16
|
geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
|
|
17
|
+
$LOAD_PATH.unshift(ENV["INCL_DIR_GEM"])
|
|
18
|
+
require 'aspera/fasp/parameters'
|
|
19
|
+
def spec_table
|
|
20
|
+
r='<table><tr><th>Field</th><th>Type</th>'
|
|
21
|
+
Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
|
|
22
|
+
r << '<th>'<<c.to_s.upcase<<'</th>'
|
|
23
|
+
end
|
|
24
|
+
r << '<th>Description</th></tr>'
|
|
25
|
+
Aspera::Fasp::Parameters.man_table.each do |p|
|
|
26
|
+
p[:description] << (p[:description].empty? ? '' : "\n") << "(" << p[:cli] << ")" unless p[:cli].to_s.empty?
|
|
27
|
+
p.delete(:cli)
|
|
28
|
+
p.keys.each{|c|p[c]=' ' if p[c].to_s.empty?}
|
|
29
|
+
p[:description].gsub!("\n",'<br/>')
|
|
30
|
+
p[:type].gsub!(',','<br/>')
|
|
31
|
+
r << '<tr><td>'<<p[:name]<<'</td><td>'<<p[:type]<<'</td>'
|
|
32
|
+
Aspera::Fasp::Parameters::SUPPORTED_AGENTS_SHORT.each do |c|
|
|
33
|
+
r << '<td>'<<p[c]<<'</td>'
|
|
34
|
+
end
|
|
35
|
+
r << '<td>'<<p[:description]<<'</td></tr>'
|
|
36
|
+
end
|
|
37
|
+
r << '</table>'
|
|
38
|
+
return r
|
|
39
|
+
end #```
|
|
17
40
|
-%>
|
|
18
|
-
|
|
41
|
+
<font size="+12"><center><%=tool%> : Command Line Interface for IBM Aspera products</center></font>
|
|
19
42
|
|
|
20
43
|
Version : <%= gemspec.version.to_s %>
|
|
21
44
|
|
|
@@ -29,7 +52,7 @@ Ruby Gem: [<%= gemspec.metadata['rubygems_uri'] %>](<%= gemspec.metadata['rubyge
|
|
|
29
52
|
|
|
30
53
|
Ruby Doc: [<%= gemspec.metadata['documentation_uri'] %>](<%= gemspec.metadata['documentation_uri'] %>)
|
|
31
54
|
|
|
32
|
-
Ruby version
|
|
55
|
+
Required Ruby version: <%= gemspec.required_ruby_version %>
|
|
33
56
|
|
|
34
57
|
# <a name="when_to_use"></a>When to use and when not to use
|
|
35
58
|
|
|
@@ -45,7 +68,8 @@ So it is designed for:
|
|
|
45
68
|
|
|
46
69
|
<%=tool%> can be seen as a command line tool integrating:
|
|
47
70
|
|
|
48
|
-
* a configuration file (config.yaml)
|
|
71
|
+
* a configuration file (config.yaml)
|
|
72
|
+
* advanced command line options
|
|
49
73
|
* cURL (for REST calls)
|
|
50
74
|
* Aspera transfer (ascp)
|
|
51
75
|
|
|
@@ -106,9 +130,10 @@ Once installation is completed, you can proceed to the first use with a demo ser
|
|
|
106
130
|
|
|
107
131
|
If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
108
132
|
|
|
109
|
-
|
|
133
|
+
To test with Aspera demo transfer server, setup the environment and then test:
|
|
110
134
|
|
|
111
135
|
```
|
|
136
|
+
$ <%=cmd%> config initdemo
|
|
112
137
|
$ <%=cmd%> server browse /
|
|
113
138
|
:............:...........:......:........:...........................:.......................:
|
|
114
139
|
: zmode : zuid : zgid : size : mtime : name :
|
|
@@ -128,7 +153,7 @@ If you want to use <%=tool%> with another server, and in order to make further c
|
|
|
128
153
|
* download a file
|
|
129
154
|
|
|
130
155
|
```
|
|
131
|
-
$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
|
156
|
+
$ <%=cmd%> config id myserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
|
132
157
|
updated: myserver
|
|
133
158
|
$ <%=cmd%> config id default set server myserver
|
|
134
159
|
updated: default→server to myserver
|
|
@@ -162,18 +187,22 @@ Then, follow the section relative to the product you want to interact with ( Asp
|
|
|
162
187
|
|
|
163
188
|
# <a name="installation"></a>Installation
|
|
164
189
|
|
|
165
|
-
It is possible to install *either*
|
|
190
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, Macos) or as a docker container.
|
|
166
191
|
|
|
167
|
-
|
|
192
|
+
The direct installation is recommended and consists in installing:
|
|
193
|
+
|
|
194
|
+
* [Ruby](#ruby) version <%= gemspec.required_ruby_version %>
|
|
168
195
|
* [<%= gemspec.name %>](#the_gem)
|
|
169
196
|
* [Aspera SDK (ascp)](#fasp_prot)
|
|
170
197
|
|
|
171
|
-
The following sections provide information on the installation.
|
|
198
|
+
The following sections provide information on the various installation methods.
|
|
172
199
|
|
|
173
200
|
An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
|
174
201
|
|
|
175
202
|
## Docker container
|
|
176
203
|
|
|
204
|
+
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
|
205
|
+
|
|
177
206
|
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
|
178
207
|
|
|
179
208
|
Ensure that you have Docker installed.
|
|
@@ -197,11 +226,12 @@ $ ./<%=cmd%> install
|
|
|
197
226
|
|
|
198
227
|
Start using it !
|
|
199
228
|
|
|
200
|
-
Note that the tool is run in the container.
|
|
229
|
+
Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
|
|
201
230
|
|
|
202
|
-
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd
|
|
231
|
+
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
|
|
203
232
|
|
|
204
233
|
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
|
234
|
+
To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
|
|
205
235
|
|
|
206
236
|
## <a name="ruby"></a>Ruby
|
|
207
237
|
|
|
@@ -209,15 +239,15 @@ Use this method to install on the native host.
|
|
|
209
239
|
|
|
210
240
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
|
211
241
|
|
|
212
|
-
Ruby
|
|
242
|
+
Required Ruby version: <%= gemspec.required_ruby_version %>. Ruby version 3 is also supported.
|
|
213
243
|
|
|
214
|
-
*
|
|
244
|
+
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
|
215
245
|
|
|
216
246
|
Refer to the following sections for a proposed method for specific operating systems.
|
|
217
247
|
|
|
218
248
|
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
|
|
219
249
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
|
220
|
-
If you have a simpler better way to install Ruby version
|
|
250
|
+
If you have a simpler better way to install Ruby version <%= gemspec.required_ruby_version %> : use it !
|
|
221
251
|
|
|
222
252
|
### Generic: RVM: single user installation (not root)
|
|
223
253
|
|
|
@@ -789,7 +819,7 @@ $ <%=cmd%> config id <<%=opprst%>> set|delete|show|initialize|update
|
|
|
789
819
|
The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
|
|
790
820
|
|
|
791
821
|
```
|
|
792
|
-
$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
|
822
|
+
$ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
|
793
823
|
```
|
|
794
824
|
|
|
795
825
|
* This creates a <%=prst%> `demo_server` with all provided options.
|
|
@@ -797,13 +827,13 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
|
|
|
797
827
|
The command `set` allows setting individual options in a <%=prst%>.
|
|
798
828
|
|
|
799
829
|
```
|
|
800
|
-
$ <%=cmd%> config id demo_server set password
|
|
830
|
+
$ <%=cmd%> config id demo_server set password _demo_pass_
|
|
801
831
|
```
|
|
802
832
|
|
|
803
833
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
|
804
834
|
|
|
805
835
|
```
|
|
806
|
-
$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"
|
|
836
|
+
$ <%=cmd%> config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
|
807
837
|
```
|
|
808
838
|
|
|
809
839
|
A good practice is to not manually edit the configuration file and use modification commands instead.
|
|
@@ -840,11 +870,19 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
|
840
870
|
Operations on this preset are done using regular `config` operations:
|
|
841
871
|
|
|
842
872
|
```
|
|
843
|
-
$ <%=cmd%> config id default set _plugin_name_
|
|
873
|
+
$ <%=cmd%> config id default set _plugin_name_ _default_preset_for_plugin_
|
|
844
874
|
$ <%=cmd%> config id default get _plugin_name_
|
|
845
|
-
"
|
|
875
|
+
"_default_preset_for_plugin_"
|
|
846
876
|
```
|
|
847
877
|
|
|
878
|
+
### <a name="lprtdef"></a>Special Plugin: config
|
|
879
|
+
|
|
880
|
+
Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
|
|
881
|
+
|
|
882
|
+
When <%=tool%> starts, it lookjs for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
|
883
|
+
|
|
884
|
+
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
|
885
|
+
|
|
848
886
|
### Format of file
|
|
849
887
|
|
|
850
888
|
The configuration file is a hash in a YAML file. Example:
|
|
@@ -860,7 +898,7 @@ cli_default:
|
|
|
860
898
|
demo_server:
|
|
861
899
|
url: ssh://demo.asperasoft.com:33001
|
|
862
900
|
username: asperaweb
|
|
863
|
-
password:
|
|
901
|
+
password: _demo_pass_
|
|
864
902
|
```
|
|
865
903
|
|
|
866
904
|
We can see here:
|
|
@@ -929,7 +967,6 @@ A <%=prst%> value can be removed with `unset`:
|
|
|
929
967
|
$ <%=cmd%> config id cli_default unset interactive
|
|
930
968
|
```
|
|
931
969
|
|
|
932
|
-
|
|
933
970
|
### Examples
|
|
934
971
|
|
|
935
972
|
For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
|
|
@@ -1049,21 +1086,42 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
|
1049
1086
|
|
|
1050
1087
|
```
|
|
1051
1088
|
$ <%=cmd%> config ascp show
|
|
1052
|
-
/Users/laurent/
|
|
1089
|
+
/Users/laurent/.aspera/ascli/sdk/ascp
|
|
1090
|
+
$ <%=cmd%> config ascp info
|
|
1091
|
+
+--------------------+-----------------------------------------------------------+
|
|
1092
|
+
| key | value |
|
|
1093
|
+
+--------------------+-----------------------------------------------------------+
|
|
1094
|
+
| ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
|
|
1095
|
+
...
|
|
1053
1096
|
```
|
|
1054
1097
|
|
|
1055
1098
|
### Selection of local `ascp`
|
|
1056
1099
|
|
|
1100
|
+
By default, <%=tool%> uses any found local product with ascp, including SDK.
|
|
1101
|
+
|
|
1057
1102
|
To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
|
|
1058
1103
|
|
|
1059
|
-
|
|
1104
|
+
For a permanent change, the command `config ascp use` sets the same parameter for the global default.
|
|
1105
|
+
|
|
1106
|
+
Using a POSIX shell:
|
|
1060
1107
|
|
|
1061
1108
|
```
|
|
1062
1109
|
$ <%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
|
1063
|
-
|
|
1110
|
+
ascp version: 4.0.0.182279
|
|
1111
|
+
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
|
1112
|
+
Saved to default global preset global_common_defaults
|
|
1064
1113
|
```
|
|
1065
1114
|
|
|
1066
|
-
|
|
1115
|
+
Windows:
|
|
1116
|
+
|
|
1117
|
+
```
|
|
1118
|
+
$ <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
|
1119
|
+
ascp version: 4.0.0.182279
|
|
1120
|
+
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
|
1121
|
+
Saved to default global preset global_common_defaults
|
|
1122
|
+
```
|
|
1123
|
+
|
|
1124
|
+
If the path has spaces, read section: [Shell and Command line parsing](#parsing).
|
|
1067
1125
|
|
|
1068
1126
|
### List locally installed Aspera Transfer products
|
|
1069
1127
|
|
|
@@ -1150,7 +1208,7 @@ will effectively push files to the related server from the agent node.
|
|
|
1150
1208
|
|
|
1151
1209
|
### <a name="direct"></a>Direct (local ascp using FASPManager API)
|
|
1152
1210
|
|
|
1153
|
-
By default
|
|
1211
|
+
By default <%=tool%> uses a local ascp, equivalent to specifying `--transfer=direct`.
|
|
1154
1212
|
<%=tool%> will detect locally installed Aspera products.
|
|
1155
1213
|
Refer to section [FASP](#client).
|
|
1156
1214
|
|
|
@@ -1160,17 +1218,41 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
|
|
|
1160
1218
|
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
|
1161
1219
|
* `EX_ascp_args`
|
|
1162
1220
|
|
|
1163
|
-
The `transfer-info`
|
|
1221
|
+
The `transfer-info` accepts the following optional parameters:
|
|
1164
1222
|
|
|
1165
1223
|
<table>
|
|
1166
|
-
<tr><th>Name</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
|
1167
|
-
<tr><td>
|
|
1168
|
-
<tr><td>
|
|
1169
|
-
<tr><td>
|
|
1170
|
-
<tr><td>
|
|
1171
|
-
<tr><td>
|
|
1224
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
|
1225
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>3</td><td>Multi session</td><td>Verification time that ascp is running</td></tr>
|
|
1226
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>2</td><td>Multi session</td><td>Delay between startup of sessions</td></tr>
|
|
1227
|
+
<tr><td>wss</td><td>Bool</td><td>false</td><td>Web Socket Session</td><td>Enable use of web socket session in case it is available</td></tr>
|
|
1228
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>true</td><td>Multi Session</td><td>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)</td></tr>
|
|
1229
|
+
<tr><td>resume</td><td>Hash</td><td>nil</td><td>Resumer parameters</td><td>See below</td></tr>
|
|
1172
1230
|
</table>
|
|
1173
1231
|
|
|
1232
|
+
Resume parameters:
|
|
1233
|
+
|
|
1234
|
+
In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
|
1235
|
+
Sleep between iteration is:
|
|
1236
|
+
|
|
1237
|
+
```
|
|
1238
|
+
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
|
1239
|
+
```
|
|
1240
|
+
|
|
1241
|
+
<table>
|
|
1242
|
+
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
|
1243
|
+
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
|
1244
|
+
<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
|
1245
|
+
<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
|
|
1246
|
+
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
|
1247
|
+
</table>
|
|
1248
|
+
|
|
1249
|
+
Examples:
|
|
1250
|
+
|
|
1251
|
+
```
|
|
1252
|
+
$ <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
|
1253
|
+
$ <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
|
1254
|
+
```
|
|
1255
|
+
|
|
1174
1256
|
### IBM Aspera Connect Client GUI
|
|
1175
1257
|
|
|
1176
1258
|
By specifying option: `--transfer=connect`, <%=tool%> will start transfers
|
|
@@ -1220,7 +1302,7 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
|
1220
1302
|
|
|
1221
1303
|
<%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
|
1222
1304
|
|
|
1223
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters.
|
|
1305
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cummulative.
|
|
1224
1306
|
|
|
1225
1307
|
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is espacially useful for ascp command line parameters not supported yet in the transfer spec.
|
|
1226
1308
|
|
|
@@ -1233,11 +1315,31 @@ A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the co
|
|
|
1233
1315
|
|
|
1234
1316
|
## <a name="transferparams"></a>Transfer Parameters
|
|
1235
1317
|
|
|
1236
|
-
All standard _transfer-spec_ parameters can be
|
|
1237
|
-
|
|
1238
|
-
|
|
1318
|
+
All standard _transfer-spec_ parameters can be speficied.
|
|
1319
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
|
1320
|
+
|
|
1321
|
+
References:
|
|
1322
|
+
|
|
1323
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
|
1324
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
|
1325
|
+
|
|
1326
|
+
Parameters can be displayed with commands:
|
|
1239
1327
|
|
|
1240
|
-
|
|
1328
|
+
```
|
|
1329
|
+
$ <%=cmd%> config ascp spec
|
|
1330
|
+
$ <%=cmd%> config ascp spec --select=@json:'{"f":"Y"}' --fields=-f,n,c
|
|
1331
|
+
```
|
|
1332
|
+
|
|
1333
|
+
Columns:
|
|
1334
|
+
|
|
1335
|
+
* D=Direct (local `ascp` execution)
|
|
1336
|
+
* N=Node API
|
|
1337
|
+
* C=Connect Client
|
|
1338
|
+
* arg=`ascp` argument or environment variable
|
|
1339
|
+
|
|
1340
|
+
Fields with EX_ prefix are extensions to transfer agent `direct`. (only in <%=tool%>).
|
|
1341
|
+
|
|
1342
|
+
<%= spec_table %>
|
|
1241
1343
|
|
|
1242
1344
|
### Destination folder for transfers
|
|
1243
1345
|
|
|
@@ -1319,7 +1421,7 @@ Source files are located on "Aspera on cloud", when :
|
|
|
1319
1421
|
|
|
1320
1422
|
### <a name="multisession"></a>Support of multi-session
|
|
1321
1423
|
|
|
1322
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
|
1424
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
|
1323
1425
|
|
|
1324
1426
|
* when agent=node :
|
|
1325
1427
|
|
|
@@ -1340,6 +1442,7 @@ shall be preferred.
|
|
|
1340
1442
|
|
|
1341
1443
|
Multi-session spawn is done by <%=tool%>.
|
|
1342
1444
|
|
|
1445
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
|
1343
1446
|
|
|
1344
1447
|
### Examples
|
|
1345
1448
|
|
|
@@ -1449,7 +1552,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
|
1449
1552
|
|
|
1450
1553
|
This will guide you through the steps to create.
|
|
1451
1554
|
|
|
1452
|
-
## <a name="
|
|
1555
|
+
## <a name="aocmanual"></a>Configuration: using manual setup
|
|
1453
1556
|
|
|
1454
1557
|
If you used the wizard (recommended): skip this section.
|
|
1455
1558
|
|
|
@@ -1615,7 +1718,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
|
|
|
1615
1718
|
|
|
1616
1719
|
* change auth method to JWT
|
|
1617
1720
|
* provide location of private key
|
|
1618
|
-
* provide username to login as (
|
|
1721
|
+
* provide username to login as (OAuth "subject")
|
|
1619
1722
|
|
|
1620
1723
|
Execute:
|
|
1621
1724
|
|
|
@@ -1713,10 +1816,10 @@ $ <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id
|
|
|
1713
1816
|
: 98398 : dummyuser1@example.com :
|
|
1714
1817
|
: 98399 : dummyuser2@example.com :
|
|
1715
1818
|
:.......:........................:
|
|
1716
|
-
$ thelist=$(
|
|
1819
|
+
$ thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
|
1717
1820
|
$ echo $thelist
|
|
1718
|
-
|
|
1719
|
-
$ <%=cmd%> aoc admin res user --bulk=yes --id=@json:
|
|
1821
|
+
["113501","354061"]
|
|
1822
|
+
$ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
1720
1823
|
:.......:.........:
|
|
1721
1824
|
: id : status :
|
|
1722
1825
|
:.......:.........:
|
|
@@ -1725,6 +1828,14 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
|
|
|
1725
1828
|
:.......:.........:
|
|
1726
1829
|
```
|
|
1727
1830
|
|
|
1831
|
+
* <a name="deactuser"></a>Find deactivated users since more than 2 years
|
|
1832
|
+
|
|
1833
|
+
```
|
|
1834
|
+
ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
|
1835
|
+
```
|
|
1836
|
+
|
|
1837
|
+
To delete them use the same method as before
|
|
1838
|
+
|
|
1728
1839
|
* Display current user's workspaces
|
|
1729
1840
|
|
|
1730
1841
|
```
|
|
@@ -1966,10 +2077,10 @@ $ <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
|
1966
2077
|
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder.
|
|
1967
2078
|
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid collisions
|
|
1968
2079
|
|
|
1969
|
-
Typically, one would regularly execute this command on a regular basis, using the method
|
|
2080
|
+
Typically, one would regularly execute this command on a regular basis, using the method of your choice:
|
|
1970
2081
|
|
|
1971
|
-
* Windows scheduler
|
|
1972
|
-
* cron
|
|
2082
|
+
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
|
2083
|
+
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
|
1973
2084
|
* etc...
|
|
1974
2085
|
|
|
1975
2086
|
## Download Files
|
|
@@ -2026,19 +2137,40 @@ The activity app can be queried with:
|
|
|
2026
2137
|
$ <%=cmd%> aoc admin analytics transfers
|
|
2027
2138
|
```
|
|
2028
2139
|
|
|
2029
|
-
It can also support filters and send notification
|
|
2140
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
|
2141
|
+
|
|
2142
|
+
`mytemplate.erb`:
|
|
2143
|
+
|
|
2144
|
+
```
|
|
2145
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
|
2146
|
+
To: <<%='<'%>%=ev['user_email']%>>
|
|
2147
|
+
Subject: <%='<'%>%=ev['files_completed']%> files received
|
|
2148
|
+
|
|
2149
|
+
Dear <%='<'%>%=ev[:user_email.to_s]%>,
|
|
2150
|
+
We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
|
|
2151
|
+
<%='<'%>%=ev['content']%>
|
|
2152
|
+
|
|
2153
|
+
Thank you.
|
|
2154
|
+
```
|
|
2155
|
+
The environment provided contains the following additional variable:
|
|
2156
|
+
|
|
2157
|
+
* ev : all details on the transfer event
|
|
2158
|
+
|
|
2159
|
+
Example:
|
|
2030
2160
|
|
|
2031
2161
|
```
|
|
2032
|
-
$ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=
|
|
2162
|
+
$ <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
|
2033
2163
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
|
2034
|
-
--
|
|
2164
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
|
2035
2165
|
```
|
|
2036
2166
|
|
|
2167
|
+
Options:
|
|
2168
|
+
|
|
2037
2169
|
* `once_only` keep track of last date it was called, so next call will get only new events
|
|
2038
2170
|
* `query` filter (on API call)
|
|
2039
2171
|
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
|
2040
2172
|
|
|
2041
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months.
|
|
2173
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
|
2042
2174
|
|
|
2043
2175
|
## Using specific transfer ports
|
|
2044
2176
|
|
|
@@ -2099,7 +2231,37 @@ $ <%=cmd%> ats api_key create
|
|
|
2099
2231
|
+--------+----------------------------------------------+
|
|
2100
2232
|
$ <%=cmd%> config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
|
2101
2233
|
```
|
|
2234
|
+
## Cross Organization transfers
|
|
2235
|
+
|
|
2236
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
|
2237
|
+
|
|
2238
|
+
Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
|
|
2239
|
+
|
|
2240
|
+
Procedure to send a file from org1 to org2:
|
|
2241
|
+
|
|
2242
|
+
* Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
|
2243
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
|
2244
|
+
* Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
|
|
2245
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
|
2246
|
+
* execute the following:
|
|
2247
|
+
|
|
2248
|
+
```
|
|
2249
|
+
$ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
|
2250
|
+
```
|
|
2251
|
+
|
|
2252
|
+
Explanation:
|
|
2253
|
+
|
|
2254
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
|
2255
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
|
2256
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
|
2257
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
|
2258
|
+
* `|` the standard output of the first command is fed into the second one
|
|
2259
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
|
2260
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
|
2261
|
+
* `--transfer=node` use transfer agent type `node` instead of default `direct`
|
|
2262
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
|
2102
2263
|
|
|
2264
|
+
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
|
2103
2265
|
## Examples
|
|
2104
2266
|
|
|
2105
2267
|
Example: create access key on softlayer:
|
|
@@ -2189,7 +2351,7 @@ This can also be set as default using a preset
|
|
|
2189
2351
|
One can test the "server" application using the well known demo server:
|
|
2190
2352
|
|
|
2191
2353
|
```
|
|
2192
|
-
$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
|
2354
|
+
$ <%=cmd%> config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
|
2193
2355
|
$ <%=cmd%> config id default set server aspera_demo_server
|
|
2194
2356
|
$ <%=cmd%> server browse /aspera-test-dir-large
|
|
2195
2357
|
$ <%=cmd%> server download /aspera-test-dir-large/200MB
|
|
@@ -2285,16 +2447,19 @@ to download files.
|
|
|
2285
2447
|
$ <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
|
2286
2448
|
```
|
|
2287
2449
|
|
|
2288
|
-
# Plugin: IBM Aspera
|
|
2450
|
+
# Plugin: IBM Aspera Faspex5
|
|
2289
2451
|
|
|
2290
|
-
|
|
2452
|
+
3 authentication methods are supported:
|
|
2453
|
+
|
|
2454
|
+
* jwt
|
|
2455
|
+
* web
|
|
2456
|
+
* boot
|
|
2291
2457
|
|
|
2292
|
-
|
|
2293
|
-
* for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
|
|
2458
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
|
2294
2459
|
|
|
2295
|
-
|
|
2460
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
|
2296
2461
|
|
|
2297
|
-
|
|
2462
|
+
For boot method: (will be removed in future)
|
|
2298
2463
|
|
|
2299
2464
|
* open a browser
|
|
2300
2465
|
* start developer mode
|
|
@@ -2307,30 +2472,74 @@ Use it as password and use `--auth=boot`.
|
|
|
2307
2472
|
$ <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
|
2308
2473
|
```
|
|
2309
2474
|
|
|
2310
|
-
|
|
2475
|
+
Ready to use Faspex5 with CLI.
|
|
2476
|
+
|
|
2477
|
+
# Plugin: IBM Aspera Faspex (4.x)
|
|
2478
|
+
|
|
2479
|
+
Notes:
|
|
2480
|
+
|
|
2481
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
|
2482
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
|
2483
|
+
|
|
2484
|
+
## Listing Packages
|
|
2485
|
+
|
|
2486
|
+
Command: `faspex package list`
|
|
2487
|
+
|
|
2488
|
+
### Option `box`
|
|
2489
|
+
|
|
2490
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
|
2491
|
+
|
|
2492
|
+
### Option `recipient`
|
|
2493
|
+
|
|
2494
|
+
A user can receive a package because the recipient is:
|
|
2495
|
+
|
|
2496
|
+
* the user himself (default)
|
|
2497
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
|
2498
|
+
|
|
2499
|
+
### Option `query`
|
|
2500
|
+
|
|
2501
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
|
2502
|
+
|
|
2503
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
|
2504
|
+
* `page` : (native) id of page in call (default=0)
|
|
2505
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
|
2506
|
+
* `max` : maximum number of items
|
|
2507
|
+
* `pmax` : maximum number of pages
|
|
2508
|
+
|
|
2509
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
|
2510
|
+
|
|
2511
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
|
2512
|
+
|
|
2513
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
|
2514
|
+
|
|
2515
|
+
### Example
|
|
2311
2516
|
|
|
2312
2517
|
```
|
|
2313
|
-
$
|
|
2314
|
-
$ <%=cmd%> faspex5 -Pf5boot auth_client create --value=@json:'{"name":"hello","client_type":"public","redirect_uris":["https://localhost:12345"],"allow_jwt_grant":true,"public_key":"'$jsonk'"}'
|
|
2518
|
+
$ <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
|
2315
2519
|
```
|
|
2316
2520
|
|
|
2317
|
-
|
|
2521
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
|
2522
|
+
|
|
2523
|
+
## Receiving a Package
|
|
2524
|
+
|
|
2525
|
+
The command is `package recv`, possible methods are:
|
|
2526
|
+
|
|
2527
|
+
* provide a package id with option `id`
|
|
2528
|
+
* provide a public link with option `link`
|
|
2529
|
+
* provide a `faspe:` URI with option `link`
|
|
2318
2530
|
|
|
2319
2531
|
```
|
|
2320
|
-
$
|
|
2321
|
-
$ <%=cmd%>
|
|
2532
|
+
$ <%=cmd%> faspex package recv --id=12345
|
|
2533
|
+
$ <%=cmd%> faspex package recv --link=faspe://...
|
|
2322
2534
|
```
|
|
2323
2535
|
|
|
2324
|
-
|
|
2536
|
+
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
|
2325
2537
|
|
|
2326
2538
|
```
|
|
2327
|
-
$ <%=cmd%>
|
|
2328
|
-
|
|
2329
|
-
```
|
|
2330
|
-
|
|
2331
|
-
Ready to use Faspex5 with CLI.
|
|
2539
|
+
$ <%=cmd%> faspex package list --recipient='*thedropboxname'
|
|
2540
|
+
```
|
|
2332
2541
|
|
|
2333
|
-
|
|
2542
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
|
2334
2543
|
|
|
2335
2544
|
## Sending a Package
|
|
2336
2545
|
|
|
@@ -2349,7 +2558,19 @@ Additional optional parameters in `delivery_info`:
|
|
|
2349
2558
|
* Package Note: : `"note":"note this and that"`
|
|
2350
2559
|
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
|
2351
2560
|
|
|
2352
|
-
##
|
|
2561
|
+
## Email notification on transfer
|
|
2562
|
+
|
|
2563
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
|
2564
|
+
|
|
2565
|
+
Example:
|
|
2566
|
+
|
|
2567
|
+
```
|
|
2568
|
+
$ <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
|
2569
|
+
```
|
|
2570
|
+
|
|
2571
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
|
2572
|
+
|
|
2573
|
+
## Operation on dropboxes
|
|
2353
2574
|
|
|
2354
2575
|
Example:
|
|
2355
2576
|
|
|
@@ -2359,7 +2580,7 @@ $ <%=cmd%> faspex v4 dropbox list
|
|
|
2359
2580
|
$ <%=cmd%> faspex v4 dropbox delete --id=36
|
|
2360
2581
|
```
|
|
2361
2582
|
|
|
2362
|
-
##
|
|
2583
|
+
## Remote sources
|
|
2363
2584
|
|
|
2364
2585
|
Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
|
|
2365
2586
|
the node API is used, for this it is required to add a section ":storage" that links
|
|
@@ -2410,29 +2631,38 @@ $ for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p
|
|
|
2410
2631
|
# Plugin: IBM Cloud Object Storage
|
|
2411
2632
|
|
|
2412
2633
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
|
2413
|
-
It uses the same transfer service as Aspera on Cloud.
|
|
2414
|
-
|
|
2634
|
+
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
|
2635
|
+
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
|
2415
2636
|
|
|
2416
|
-
|
|
2637
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the forst method. If you dont have credentials but have access to the IBM Cloud console, then use the second method.
|
|
2638
|
+
|
|
2639
|
+
## Using endpoint, apikey and Ressource Instance ID (CRN)
|
|
2640
|
+
|
|
2641
|
+
If you have those parameters already, then following options shall be provided:
|
|
2417
2642
|
|
|
2418
2643
|
* `bucket` bucket name
|
|
2419
2644
|
* `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
|
|
2420
2645
|
* `apikey` API Key
|
|
2421
2646
|
* `crn` resource instance id
|
|
2422
2647
|
|
|
2423
|
-
|
|
2648
|
+
For example, let us create a default configuration:
|
|
2424
2649
|
|
|
2425
|
-
|
|
2426
|
-
|
|
2427
|
-
|
|
2650
|
+
```
|
|
2651
|
+
$ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
|
2652
|
+
$ <%=cmd%> conf id default set cos mycos
|
|
2653
|
+
```
|
|
2654
|
+
|
|
2655
|
+
Then, jump to the transfer example.
|
|
2428
2656
|
|
|
2429
|
-
|
|
2657
|
+
## Using service credential file
|
|
2658
|
+
|
|
2659
|
+
If you are the COS administrator and dont have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
|
2430
2660
|
|
|
2431
2661
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
|
2432
2662
|
|
|
2433
2663
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
2434
2664
|
|
|
2435
|
-
or using the CLI:
|
|
2665
|
+
or using the IBM Cloud CLI:
|
|
2436
2666
|
|
|
2437
2667
|
```
|
|
2438
2668
|
$ ibmcloud resource service-keys
|
|
@@ -2463,37 +2693,42 @@ The field `resource_instance_id` is for option `crn`
|
|
|
2463
2693
|
|
|
2464
2694
|
The field `apikey` is for option `apikey`
|
|
2465
2695
|
|
|
2466
|
-
|
|
2696
|
+
(If needed: endpoints for regions can be found by querying the `endpoints` URL.)
|
|
2697
|
+
|
|
2698
|
+
The required options for this method are:
|
|
2699
|
+
|
|
2700
|
+
* `bucket` bucket name
|
|
2701
|
+
* `region` bucket region, e.g. eu-de
|
|
2702
|
+
* `service_credentials` see below
|
|
2467
2703
|
|
|
2468
|
-
For
|
|
2704
|
+
For example, let us create a default configuration:
|
|
2469
2705
|
|
|
2470
2706
|
```
|
|
2471
2707
|
$ <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
|
2472
2708
|
$ <%=cmd%> conf id default set cos mycos
|
|
2473
2709
|
```
|
|
2474
2710
|
|
|
2475
|
-
|
|
2711
|
+
## Operations, transfers
|
|
2476
2712
|
|
|
2477
|
-
|
|
2478
|
-
$ <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
|
2479
|
-
$ <%=cmd%> conf id default set cos mycos
|
|
2480
|
-
```
|
|
2713
|
+
Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
|
|
2481
2714
|
|
|
2482
|
-
|
|
2715
|
+
A subset of `node` plugin operations are supported, basically node API:
|
|
2483
2716
|
|
|
2484
2717
|
```
|
|
2485
|
-
$ <%=cmd%> cos node
|
|
2486
|
-
$ <%=cmd%> cos node upload
|
|
2718
|
+
$ <%=cmd%> cos node info
|
|
2719
|
+
$ <%=cmd%> cos node upload 'faux:///sample1G?1g'
|
|
2487
2720
|
```
|
|
2488
2721
|
|
|
2722
|
+
Note: we generate a dummy file `sample1G` if size 2GB using the `faux` PVCL (man ascp), but you can of course send a real file by specifying a real file instead.
|
|
2723
|
+
|
|
2489
2724
|
# Plugin: IBM Aspera Sync
|
|
2490
2725
|
|
|
2491
|
-
A basic plugin to start an "async" using <%=tool%>.
|
|
2492
|
-
to start from ma configuration file, using <%=tool%> standard options.
|
|
2726
|
+
A basic plugin to start an "async" using <%=tool%>.
|
|
2727
|
+
The main advantage is the possibility to start from ma configuration file, using <%=tool%> standard options.
|
|
2493
2728
|
|
|
2494
2729
|
# Plugin: Preview
|
|
2495
2730
|
|
|
2496
|
-
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on
|
|
2731
|
+
The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
|
|
2497
2732
|
This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
|
|
2498
2733
|
Several parameters can be used to tune several aspects:
|
|
2499
2734
|
|
|
@@ -2622,9 +2857,10 @@ $ <%=cmd%> config id previewconf update --url=https://localhost:9092 --username=
|
|
|
2622
2857
|
$ <%=cmd%> config id default set preview previewconf
|
|
2623
2858
|
```
|
|
2624
2859
|
|
|
2625
|
-
Here we assume that Office file generation is disabled, else remove
|
|
2860
|
+
Here we assume that Office file generation is disabled, else remove this option.
|
|
2861
|
+
`lock_port` prevents concurrent execution of generation when using a scheduler.
|
|
2626
2862
|
|
|
2627
|
-
|
|
2863
|
+
One can check if the access key is well configured using:
|
|
2628
2864
|
|
|
2629
2865
|
```
|
|
2630
2866
|
$ <%=cmd%> -Ppreviewconf node browse /
|
|
@@ -2710,6 +2946,14 @@ $ <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
|
|
|
2710
2946
|
|
|
2711
2947
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
|
2712
2948
|
|
|
2949
|
+
When scanning the option `value` has the same behaviour as for the `node find` command.
|
|
2950
|
+
|
|
2951
|
+
For instance to filter out files beginning with `._` do:
|
|
2952
|
+
|
|
2953
|
+
```
|
|
2954
|
+
... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
|
|
2955
|
+
```
|
|
2956
|
+
|
|
2713
2957
|
## Preview File types
|
|
2714
2958
|
|
|
2715
2959
|
Two types of preview can be generated:
|
|
@@ -2758,14 +3002,14 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
|
2758
3002
|
The `smtp` option is a hash table (extended value) with the following fields:
|
|
2759
3003
|
<table>
|
|
2760
3004
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
|
2761
|
-
<tr><td
|
|
2762
|
-
<tr><td
|
|
2763
|
-
<tr><td
|
|
2764
|
-
<tr><td
|
|
2765
|
-
<tr><td
|
|
2766
|
-
<tr><td
|
|
2767
|
-
<tr><td
|
|
2768
|
-
<tr><td
|
|
3005
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
|
3006
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
|
3007
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
|
3008
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
|
3009
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
|
3010
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
|
3011
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
|
3012
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
|
2769
3013
|
</table>
|
|
2770
3014
|
|
|
2771
3015
|
## Example of configuration:
|
|
@@ -2795,13 +3039,52 @@ $ <%=cmd%> config id cli_default set smtp @val:@preset:smtp_google
|
|
|
2795
3039
|
$ <%=cmd%> config id default set config cli_default
|
|
2796
3040
|
```
|
|
2797
3041
|
|
|
3042
|
+
## Email templates
|
|
3043
|
+
|
|
3044
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
|
3045
|
+
|
|
3046
|
+
The template is the full SMTP message, including headers.
|
|
3047
|
+
|
|
3048
|
+
The following variables are defined by default:
|
|
3049
|
+
|
|
3050
|
+
* from_name
|
|
3051
|
+
* from_email
|
|
3052
|
+
* to
|
|
3053
|
+
|
|
3054
|
+
Other variables are defined depending on context.
|
|
3055
|
+
|
|
2798
3056
|
## Test
|
|
2799
3057
|
|
|
2800
3058
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
|
2801
3059
|
|
|
2802
3060
|
```
|
|
2803
3061
|
$ <%=cmd%> config --smtp=@preset:smtp_google smtp
|
|
2804
|
-
$ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
|
|
3062
|
+
$ <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
|
3063
|
+
```
|
|
3064
|
+
|
|
3065
|
+
## Notifications for transfer status
|
|
3066
|
+
|
|
3067
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
|
3068
|
+
|
|
3069
|
+
To activate, use option `notif_to`.
|
|
3070
|
+
|
|
3071
|
+
A default e-mail template is used, but it can be overriden with option `notif_template`.
|
|
3072
|
+
|
|
3073
|
+
The environment provided contains the following additional variables:
|
|
3074
|
+
|
|
3075
|
+
* subject
|
|
3076
|
+
* body
|
|
3077
|
+
* global_transfer_status
|
|
3078
|
+
* ts
|
|
3079
|
+
|
|
3080
|
+
Example of template:
|
|
3081
|
+
|
|
3082
|
+
```
|
|
3083
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
|
3084
|
+
To: <<%='<'%>%=to%>>
|
|
3085
|
+
Subject: <%='<'%>%=subject%>
|
|
3086
|
+
|
|
3087
|
+
Transfer is: <%='<'%>%=global_transfer_status%>
|
|
2805
3088
|
```
|
|
2806
3089
|
|
|
2807
3090
|
# Tool: `asession`
|
|
@@ -2847,7 +3130,7 @@ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters ar
|
|
|
2847
3130
|
## Simple session
|
|
2848
3131
|
|
|
2849
3132
|
```
|
|
2850
|
-
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"
|
|
3133
|
+
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
|
|
2851
3134
|
|
|
2852
3135
|
echo "${MY_TSPEC}"|asession
|
|
2853
3136
|
```
|
|
@@ -2860,7 +3143,7 @@ This is particularly useful for a persistent session ( with the [_transfer-spec_
|
|
|
2860
3143
|
|
|
2861
3144
|
```
|
|
2862
3145
|
$ asession
|
|
2863
|
-
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"
|
|
3146
|
+
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
|
2864
3147
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
|
2865
3148
|
{"type":"DONE"}
|
|
2866
3149
|
```
|
|
@@ -2912,7 +3195,7 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
|
2912
3195
|
Note that:
|
|
2913
3196
|
|
|
2914
3197
|
* <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
|
2915
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
|
3198
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
|
2916
3199
|
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
|
|
2917
3200
|
|
|
2918
3201
|
### server side and configuration
|
|
@@ -3009,7 +3292,42 @@ So, it evolved into <%=tool%>:
|
|
|
3009
3292
|
|
|
3010
3293
|
# Changes (Release notes)
|
|
3011
3294
|
|
|
3012
|
-
*
|
|
3295
|
+
* <%= gemspec.version.to_s %>
|
|
3296
|
+
|
|
3297
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on `direct` transfer agent.
|
|
3298
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
|
3299
|
+
|
|
3300
|
+
* 4.2.2
|
|
3301
|
+
|
|
3302
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
|
3303
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
|
3304
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
|
3305
|
+
* new: `conf ascp spec` displays supported transfer spec
|
|
3306
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
|
3307
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
|
3308
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
|
3309
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
|
|
3310
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
|
3311
|
+
|
|
3312
|
+
* 4.2.1
|
|
3313
|
+
|
|
3314
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
|
3315
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
|
3316
|
+
|
|
3317
|
+
* 4.2.0
|
|
3318
|
+
|
|
3319
|
+
* new: command `aoc remind` to receive organization membership by email
|
|
3320
|
+
* new: in `preview` option `value` to filter out on file name
|
|
3321
|
+
* new: `initdemo` to initialize for demo server
|
|
3322
|
+
* new: `direct` transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
|
3323
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
|
3324
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
|
3325
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
|
3326
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
|
3327
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
|
3328
|
+
* change: (break) parameters for resume in `transfer-info` for `direct` are now in sub-key `"resume"`
|
|
3329
|
+
|
|
3330
|
+
* 4.1.0
|
|
3013
3331
|
|
|
3014
3332
|
* fix: remove keys from transfer spec and command line when not needed
|
|
3015
3333
|
* fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
|
@@ -3380,11 +3698,15 @@ Breaking change:
|
|
|
3380
3698
|
|
|
3381
3699
|
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
|
|
3382
3700
|
|
|
3383
|
-
# BUGS
|
|
3701
|
+
# BUGS, FEATURES, CONTRIBUTION
|
|
3702
|
+
|
|
3703
|
+
For issues or feature requests use the Github repository and issues.
|
|
3704
|
+
|
|
3705
|
+
You can also contribute to this open source project.
|
|
3384
3706
|
|
|
3385
|
-
|
|
3707
|
+
One can also create one's own command nplugin.
|
|
3386
3708
|
|
|
3387
|
-
##
|
|
3709
|
+
## Only one value for any option
|
|
3388
3710
|
|
|
3389
3711
|
Some commands and sub commands may ask for the same option name.
|
|
3390
3712
|
Currently, since option definition is position independant (last one wins), it is not possible
|
|
@@ -3398,7 +3720,8 @@ This happens typically for the `node` sub command, e.g. identify the node by nam
|
|
|
3398
3720
|
|
|
3399
3721
|
## ED255519 key not supported
|
|
3400
3722
|
|
|
3401
|
-
|
|
3723
|
+
ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
|
|
3724
|
+
|
|
3402
3725
|
Without this deactivation, if such key was present the following error was generated:
|
|
3403
3726
|
|
|
3404
3727
|
```
|
|
@@ -3408,7 +3731,21 @@ OpenSSH keys only supported if ED25519 is available
|
|
|
3408
3731
|
Which meant that you do not have ruby support for ED25519 SSH keys.
|
|
3409
3732
|
You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
|
|
3410
3733
|
|
|
3411
|
-
|
|
3734
|
+
## Error "Remote host is not who we expected"
|
|
3735
|
+
|
|
3736
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
|
3737
|
+
|
|
3738
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
|
3739
|
+
|
|
3740
|
+
```
|
|
3741
|
+
--ts=@json:'{"sshfp":null}'
|
|
3742
|
+
```
|
|
3743
|
+
|
|
3744
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
|
3745
|
+
|
|
3746
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
|
3747
|
+
|
|
3748
|
+
## Miscelaneous
|
|
3412
3749
|
|
|
3413
3750
|
* remove rest and oauth classes and use ruby standard gems:
|
|
3414
3751
|
|
|
@@ -3424,9 +3761,3 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
|
3424
3761
|
* Going through proxy: use env var http_proxy and https_proxy, no_proxy
|
|
3425
3762
|
|
|
3426
3763
|
* easier use with https://github.com/pmq20/ruby-packer
|
|
3427
|
-
|
|
3428
|
-
# Contribution
|
|
3429
|
-
|
|
3430
|
-
Send comments !
|
|
3431
|
-
|
|
3432
|
-
Create your own plugin !
|