aspera-cli 4.2.1 → 4.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +1580 -946
- data/bin/ascli +1 -1
- data/bin/asession +3 -5
- data/docs/Makefile +8 -11
- data/docs/README.erb.md +1521 -829
- data/docs/doc_tools.rb +58 -0
- data/docs/test_env.conf +3 -1
- data/examples/faspex4.rb +28 -19
- data/examples/transfer.rb +2 -2
- data/lib/aspera/aoc.rb +157 -134
- data/lib/aspera/cli/listener/progress_multi.rb +5 -5
- data/lib/aspera/cli/main.rb +106 -48
- data/lib/aspera/cli/manager.rb +19 -20
- data/lib/aspera/cli/plugin.rb +22 -7
- data/lib/aspera/cli/plugins/aoc.rb +260 -208
- data/lib/aspera/cli/plugins/ats.rb +11 -10
- data/lib/aspera/cli/plugins/bss.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +360 -189
- data/lib/aspera/cli/plugins/faspex.rb +119 -56
- data/lib/aspera/cli/plugins/faspex5.rb +32 -17
- data/lib/aspera/cli/plugins/node.rb +72 -31
- data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
- data/lib/aspera/cli/plugins/preview.rb +94 -68
- data/lib/aspera/cli/plugins/server.rb +16 -5
- data/lib/aspera/cli/plugins/shares.rb +17 -0
- data/lib/aspera/cli/transfer_agent.rb +64 -82
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +48 -31
- data/lib/aspera/cos_node.rb +4 -3
- data/lib/aspera/environment.rb +4 -4
- data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
- data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
- data/lib/aspera/fasp/{local.rb → agent_direct.rb} +42 -38
- data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +50 -29
- data/lib/aspera/fasp/{node.rb → agent_node.rb} +43 -4
- data/lib/aspera/fasp/agent_trsdk.rb +106 -0
- data/lib/aspera/fasp/default.rb +17 -0
- data/lib/aspera/fasp/installation.rb +64 -48
- data/lib/aspera/fasp/parameters.rb +78 -91
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/faspex_gw.rb +12 -11
- data/lib/aspera/id_generator.rb +22 -0
- data/lib/aspera/keychain/encrypted_hash.rb +120 -0
- data/lib/aspera/keychain/macos_security.rb +94 -0
- data/lib/aspera/log.rb +45 -32
- data/lib/aspera/node.rb +9 -4
- data/lib/aspera/oauth.rb +116 -100
- data/lib/aspera/persistency_action_once.rb +11 -7
- data/lib/aspera/persistency_folder.rb +6 -26
- data/lib/aspera/rest.rb +66 -50
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/timer_limiter.rb +22 -0
- metadata +86 -29
- data/docs/transfer_spec.html +0 -99
- data/lib/aspera/api_detector.rb +0 -60
- data/lib/aspera/fasp/aoc.rb +0 -24
- data/lib/aspera/secrets.rb +0 -20
data/docs/README.erb.md
CHANGED
@@ -1,37 +1,24 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
|
-
<%
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
raise "missing file #{ENV[e]}" unless File.exist?(ENV[e]) or !e.start_with?('INCL_') #_
|
7
|
-
end
|
8
|
-
cmd=ENV["EXENAME"] # just command name
|
9
|
-
tool='`'+cmd+'`' # used in text with formatting of command
|
10
|
-
evp=cmd.upcase+'_' # prefix for env vars
|
11
|
-
opprst='option preset' # just the name for "option preset"
|
12
|
-
prst='['+opprst+'](#lprt)'
|
13
|
-
prsts='['+opprst+'s](#lprt)'
|
14
|
-
prstt=opprst.capitalize # in title
|
15
|
-
gemspec=Gem::Specification::load(ENV["GEMSPEC"]) or raise "error loading #{ENV["GEMSPEC"]}"
|
16
|
-
geminstadd=gemspec.version.to_s.match(/\.[^0-9]/)?' --pre':''
|
17
|
-
-%>
|
18
|
-
# <%=tool%> : Command Line Interface for IBM Aspera products
|
19
|
-
|
20
|
-
Version : <%= gemspec.version.to_s %>
|
2
|
+
<%load File.join(File.dirname(__FILE__),'doc_tools.rb')-%>
|
3
|
+
<font size="+12"><center><%=tool%> : Command Line Interface for IBM Aspera products</center></font>
|
4
|
+
|
5
|
+
Version : <%=gemspec.version.to_s%>
|
21
6
|
|
22
7
|
_Laurent/2016-<%=Time.new.year%>_
|
23
8
|
|
24
|
-
This gem provides <%=tool
|
9
|
+
This gem provides the <%=tool%> Command Line Interface to IBM Aspera software.
|
25
10
|
|
26
11
|
<%=tool%> is a also great tool to learn Aspera APIs.
|
27
12
|
|
28
|
-
Ruby Gem: [<%=
|
13
|
+
Ruby Gem: [<%=gemspec.metadata['rubygems_uri']%>](<%=gemspec.metadata['rubygems_uri']%>)
|
14
|
+
|
15
|
+
Ruby Doc: [<%=gemspec.metadata['documentation_uri']%>](<%=gemspec.metadata['documentation_uri']%>)
|
29
16
|
|
30
|
-
Ruby
|
17
|
+
Required Ruby version: <%=gemspec.required_ruby_version%>
|
31
18
|
|
32
|
-
|
19
|
+
[Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
33
20
|
|
34
|
-
# <a
|
21
|
+
# <a id="when_to_use"></a>When to use and when not to use
|
35
22
|
|
36
23
|
<%=tool%> is designed to be used as a command line tool to:
|
37
24
|
|
@@ -45,7 +32,8 @@ So it is designed for:
|
|
45
32
|
|
46
33
|
<%=tool%> can be seen as a command line tool integrating:
|
47
34
|
|
48
|
-
* a configuration file (config.yaml)
|
35
|
+
* a configuration file (config.yaml)
|
36
|
+
* advanced command line options
|
49
37
|
* cURL (for REST calls)
|
50
38
|
* Aspera transfer (ascp)
|
51
39
|
|
@@ -53,26 +41,23 @@ One might be tempted to use it as an integration element, e.g. by building a com
|
|
53
41
|
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
54
42
|
|
55
43
|
* Product APIs (REST) : e.g. AoC, Faspex, node
|
56
|
-
* Transfer SDK : with gRPC interface and
|
44
|
+
* Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
57
45
|
|
58
46
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
59
47
|
|
60
48
|
For scripting and ad'hoc command line operations, <%=tool%> is perfect.
|
61
49
|
|
62
|
-
# Notations
|
50
|
+
# <a id="parsing"></a>Notations, Shell and Command line parsing
|
63
51
|
|
64
|
-
In examples, command line operations
|
65
|
-
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
52
|
+
In examples, command line operations are shown using a shell such: `bash` or `zsh`.
|
66
53
|
|
67
54
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
68
55
|
|
69
|
-
# <a name="parsing"></a>Shell and Command line parsing
|
70
|
-
|
71
56
|
<%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
|
72
57
|
|
73
58
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
74
59
|
|
75
|
-
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
60
|
+
On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
76
61
|
|
77
62
|
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
78
63
|
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
@@ -80,12 +65,15 @@ On Windows, `cmd` is typically used. Windows process creation does not receive t
|
|
80
65
|
In case of doubt of argument values after parsing test like this:
|
81
66
|
|
82
67
|
```
|
83
|
-
|
68
|
+
<%=cmd%> conf echo "Hello World" arg2 3
|
69
|
+
```
|
70
|
+
|
71
|
+
```
|
84
72
|
"Hello World"
|
85
73
|
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
86
74
|
```
|
87
75
|
|
88
|
-
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (
|
76
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (<%=tool%>), next command line arguments are shown in the error message.
|
89
77
|
|
90
78
|
# Quick Start
|
91
79
|
|
@@ -96,8 +84,11 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
|
|
96
84
|
Once the gem is installed, <%=tool%> shall be accessible:
|
97
85
|
|
98
86
|
```
|
99
|
-
|
100
|
-
|
87
|
+
<%=cmd%> --version
|
88
|
+
```
|
89
|
+
|
90
|
+
```
|
91
|
+
<%=gemspec.version.to_s%>
|
101
92
|
```
|
102
93
|
|
103
94
|
## First use
|
@@ -109,8 +100,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
109
100
|
To test with Aspera demo transfer server, setup the environment and then test:
|
110
101
|
|
111
102
|
```
|
112
|
-
|
113
|
-
|
103
|
+
<%=cmd%> config initdemo
|
104
|
+
```
|
105
|
+
|
106
|
+
```
|
107
|
+
<%=cmd%> server browse /
|
108
|
+
```
|
109
|
+
|
110
|
+
```
|
114
111
|
:............:...........:......:........:...........................:.......................:
|
115
112
|
: zmode : zuid : zgid : size : mtime : name :
|
116
113
|
:............:...........:......:........:...........................:.......................:
|
@@ -124,16 +121,31 @@ $ <%=cmd%> server browse /
|
|
124
121
|
If you want to use <%=tool%> with another server, and in order to make further calls more convenient, it is advised to define a <%=prst%> for the server's authentication options. The following example will:
|
125
122
|
|
126
123
|
* create a <%=prst%>
|
127
|
-
* define it as default for
|
124
|
+
* define it as default for `server` plugin
|
128
125
|
* list files in a folder
|
129
126
|
* download a file
|
130
127
|
|
131
128
|
```
|
132
|
-
|
129
|
+
<%=cmd%> config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
130
|
+
```
|
131
|
+
|
132
|
+
```
|
133
133
|
updated: myserver
|
134
|
-
|
134
|
+
```
|
135
|
+
|
136
|
+
```
|
137
|
+
<%=cmd%> config preset set default server myserver
|
138
|
+
```
|
139
|
+
|
140
|
+
```
|
135
141
|
updated: default→server to myserver
|
136
|
-
|
142
|
+
```
|
143
|
+
|
144
|
+
```
|
145
|
+
<%=cmd%> server browse /aspera-test-dir-large
|
146
|
+
```
|
147
|
+
|
148
|
+
```
|
137
149
|
:............:...........:......:..............:...........................:............................:
|
138
150
|
: zmode : zuid : zgid : size : mtime : name :
|
139
151
|
:............:...........:......:..............:...........................:............................:
|
@@ -150,7 +162,13 @@ $ <%=cmd%> server browse /aspera-test-dir-large
|
|
150
162
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
151
163
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
152
164
|
:............:...........:......:..............:...........................:............................:
|
153
|
-
|
165
|
+
```
|
166
|
+
|
167
|
+
```
|
168
|
+
<%=cmd%> server download /aspera-test-dir-large/200MB
|
169
|
+
```
|
170
|
+
|
171
|
+
```
|
154
172
|
Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
|
155
173
|
complete
|
156
174
|
```
|
@@ -161,43 +179,48 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
|
|
161
179
|
|
162
180
|
Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
163
181
|
|
164
|
-
# <a
|
182
|
+
# <a id="installation"></a>Installation
|
165
183
|
|
166
|
-
It is possible to install *either* directly on the host operating system (Linux, Windows,
|
184
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
|
167
185
|
|
168
186
|
The direct installation is recommended and consists in installing:
|
169
187
|
|
170
|
-
* [Ruby](#ruby) version
|
171
|
-
* [<%=
|
188
|
+
* [Ruby](#ruby) version <%=gemspec.required_ruby_version%>
|
189
|
+
* [<%=gemspec.name%>](#the_gem)
|
172
190
|
* [Aspera SDK (ascp)](#fasp_prot)
|
173
191
|
|
174
192
|
The following sections provide information on the various installation methods.
|
175
193
|
|
176
|
-
An internet connection is required for the installation. If you
|
194
|
+
An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
177
195
|
|
178
196
|
## Docker container
|
179
197
|
|
180
198
|
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
181
199
|
|
182
|
-
This method installs a docker image that contains: Ruby,
|
200
|
+
This method installs a docker image that contains: Ruby, <%=tool%> and the FASP sdk.
|
201
|
+
|
202
|
+
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
183
203
|
|
184
204
|
Ensure that you have Docker installed.
|
185
205
|
|
186
206
|
```
|
187
|
-
|
207
|
+
docker --version
|
188
208
|
```
|
189
209
|
|
190
210
|
Download the wrapping script:
|
191
211
|
|
192
212
|
```
|
193
|
-
|
194
|
-
|
213
|
+
curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
214
|
+
```
|
215
|
+
|
216
|
+
```
|
217
|
+
chmod a+x <%=cmd%>
|
195
218
|
```
|
196
219
|
|
197
220
|
Install the container image:
|
198
221
|
|
199
222
|
```
|
200
|
-
|
223
|
+
./<%=cmd%> install
|
201
224
|
```
|
202
225
|
|
203
226
|
Start using it !
|
@@ -207,23 +230,23 @@ Note that the tool is run in the container, so transfers are also executed in th
|
|
207
230
|
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
|
208
231
|
|
209
232
|
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
210
|
-
To add local storage as a volume edit the script:
|
233
|
+
To add local storage as a volume edit the script: <%=tool%> and add a `--volume` stanza.
|
211
234
|
|
212
|
-
## <a
|
235
|
+
## <a id="ruby"></a>Ruby
|
213
236
|
|
214
237
|
Use this method to install on the native host.
|
215
238
|
|
216
239
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
217
240
|
|
218
|
-
Ruby
|
241
|
+
Required Ruby version: <%=gemspec.required_ruby_version%>. Ruby version 3 is also supported.
|
219
242
|
|
220
243
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
221
244
|
|
222
245
|
Refer to the following sections for a proposed method for specific operating systems.
|
223
246
|
|
224
|
-
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux,
|
247
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
225
248
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
226
|
-
If you have a simpler better way to install Ruby version
|
249
|
+
If you have a simpler better way to install Ruby version <%=gemspec.required_ruby_version%> : use it !
|
227
250
|
|
228
251
|
### Generic: RVM: single user installation (not root)
|
229
252
|
|
@@ -234,31 +257,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
|
234
257
|
Install the 2 keys
|
235
258
|
|
236
259
|
```
|
237
|
-
|
260
|
+
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
238
261
|
```
|
239
262
|
|
240
263
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
241
264
|
|
242
265
|
```
|
243
|
-
|
266
|
+
\curl -sSL https://get.rvm.io | bash -s stable
|
244
267
|
```
|
245
268
|
|
246
|
-
If you keep the same terminal (
|
269
|
+
If you keep the same terminal (not needed if re-login):
|
247
270
|
|
248
271
|
```
|
249
|
-
|
272
|
+
source ~/.rvm/scripts/rvm
|
250
273
|
```
|
251
274
|
|
252
|
-
It is advised to get one of the pre-compiled ruby version, you can list with:
|
275
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
253
276
|
|
254
277
|
```
|
255
|
-
|
278
|
+
rvm list --remote
|
256
279
|
```
|
257
280
|
|
258
281
|
Install the chosen pre-compiled Ruby version:
|
259
282
|
|
260
283
|
```
|
261
|
-
|
284
|
+
rvm install 2.7.2 --binary
|
262
285
|
```
|
263
286
|
|
264
287
|
Ruby is now installed for the user, go on to Gem installation.
|
@@ -271,7 +294,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
|
|
271
294
|
One can install in another location with :
|
272
295
|
|
273
296
|
```
|
274
|
-
|
297
|
+
curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
275
298
|
```
|
276
299
|
|
277
300
|
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
@@ -279,8 +302,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
|
|
279
302
|
To activate ruby (and <%=cmd%>) later, source it:
|
280
303
|
|
281
304
|
```
|
282
|
-
|
283
|
-
|
305
|
+
source /etc/profile.d/rvm.sh.ok
|
306
|
+
```
|
307
|
+
|
308
|
+
```
|
309
|
+
rvm version
|
284
310
|
```
|
285
311
|
|
286
312
|
### Windows: Installer
|
@@ -293,16 +319,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
|
|
293
319
|
|
294
320
|
### macOS: pre-installed or `brew`
|
295
321
|
|
296
|
-
|
322
|
+
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
|
297
323
|
|
298
324
|
```
|
299
|
-
|
325
|
+
sudo gem install <%=gemspec.name%><%=geminstadd%>
|
300
326
|
```
|
301
327
|
|
302
328
|
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
303
329
|
|
304
330
|
```
|
305
|
-
|
331
|
+
brew install ruby
|
306
332
|
```
|
307
333
|
|
308
334
|
### Linux: package
|
@@ -312,90 +338,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
|
|
312
338
|
Example:
|
313
339
|
|
314
340
|
```
|
315
|
-
|
341
|
+
yum install -y ruby rubygems ruby-json
|
316
342
|
```
|
317
343
|
|
318
344
|
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
319
345
|
|
320
346
|
```
|
321
347
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
348
|
+
```
|
349
|
+
|
350
|
+
```
|
322
351
|
yum remove -y ruby ruby-libs
|
323
352
|
```
|
324
353
|
|
325
354
|
### Other Unixes: Aix, etc...
|
326
355
|
|
327
|
-
If your
|
356
|
+
If your Unix does not provide a pre-built ruby, you can get it using one of those
|
328
357
|
[methods](https://www.ruby-lang.org/en/documentation/installation/)
|
329
358
|
|
330
359
|
For instance to build from source, and install in `/opt/ruby` :
|
331
360
|
|
332
361
|
```
|
333
|
-
|
334
|
-
# gzip -d ruby-2.7.2.tar.gz
|
335
|
-
# tar xvf ruby-2.7.2.tar
|
336
|
-
# cd ruby-2.7.2
|
337
|
-
# ./configure --prefix=/opt/ruby
|
338
|
-
# make ruby.imp
|
339
|
-
# make
|
340
|
-
# make install
|
341
|
-
```
|
342
|
-
|
343
|
-
### <a name="offline_install"></a>Installation without internet access
|
344
|
-
|
345
|
-
Note that currently no pre-packaged version exist yet.
|
346
|
-
A method to build one provided here:
|
347
|
-
|
348
|
-
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
349
|
-
|
350
|
-
Then create an archive:
|
351
|
-
|
352
|
-
```
|
353
|
-
$ cd
|
354
|
-
$ tar zcvf rvm-<%=cmd%>.tgz .rvm
|
355
|
-
```
|
356
|
-
|
357
|
-
Get the Aspera SDK. Execute:
|
358
|
-
|
359
|
-
```
|
360
|
-
$ <%=cmd%> conf --show-config|grep sdk_url
|
361
|
-
```
|
362
|
+
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
|
362
363
|
|
363
|
-
|
364
|
+
gzip -d ruby-2.7.2.tar.gz
|
364
365
|
|
365
|
-
|
366
|
+
tar xvf ruby-2.7.2.tar
|
366
367
|
|
367
|
-
|
368
|
+
cd ruby-2.7.2
|
368
369
|
|
369
|
-
|
370
|
+
./configure --prefix=/opt/ruby
|
370
371
|
|
371
|
-
|
372
|
-
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
372
|
+
make ruby.imp
|
373
373
|
|
374
|
-
|
375
|
-
source path_to_rvm_root/scripts/rvm
|
376
|
-
rvm use 2.7.2
|
377
|
-
```
|
374
|
+
make
|
378
375
|
|
379
|
-
|
380
|
-
|
381
|
-
```
|
382
|
-
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
376
|
+
make install
|
383
377
|
```
|
384
378
|
|
385
|
-
|
386
|
-
|
387
|
-
## <a name="the_gem"></a>`<%= gemspec.name %>` gem
|
379
|
+
## <a id="the_gem"></a>`<%=gemspec.name%>` gem
|
388
380
|
|
389
381
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
390
382
|
|
391
383
|
```
|
392
|
-
|
384
|
+
gem install <%=gemspec.name%><%=geminstadd%>
|
393
385
|
```
|
394
386
|
|
395
387
|
To upgrade to the latest version:
|
396
388
|
|
397
389
|
```
|
398
|
-
|
390
|
+
gem update <%=gemspec.name%>
|
399
391
|
```
|
400
392
|
|
401
393
|
<%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
@@ -403,12 +395,10 @@ To upgrade to the latest version:
|
|
403
395
|
To check manually:
|
404
396
|
|
405
397
|
```
|
406
|
-
|
398
|
+
<%=cmd%> conf check_update
|
407
399
|
```
|
408
400
|
|
409
|
-
|
410
|
-
|
411
|
-
## <a name="fasp_prot"></a>FASP Protocol
|
401
|
+
## <a id="fasp_prot"></a>FASP Protocol
|
412
402
|
|
413
403
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
414
404
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
@@ -416,17 +406,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
|
|
416
406
|
* ascp
|
417
407
|
* aspera-license (in same folder, or ../etc)
|
418
408
|
|
419
|
-
This can be installed either be installing an Aspera transfer
|
409
|
+
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
420
410
|
|
421
411
|
```
|
422
|
-
|
412
|
+
<%=cmd%> conf ascp install
|
423
413
|
```
|
424
414
|
|
425
|
-
If a local SDK installation is
|
415
|
+
If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
|
426
416
|
|
427
417
|
```
|
428
|
-
|
429
|
-
|
418
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
419
|
+
```
|
420
|
+
|
421
|
+
```
|
422
|
+
<%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
430
423
|
```
|
431
424
|
|
432
425
|
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
@@ -445,39 +438,58 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
445
438
|
<%=tool%> will detect most of Aspera transfer products in standard locations and use the first one found.
|
446
439
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
447
440
|
|
448
|
-
Several methods are provided
|
449
|
-
other methods are available. Refer to section: [Transfer Agents](#agents)
|
441
|
+
Several methods are provided to start a transfer.
|
442
|
+
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
443
|
+
|
444
|
+
## <a id="offline_install"></a>Installation in air gapped environment
|
450
445
|
|
451
|
-
|
446
|
+
Note that currently no pre-packaged version exist yet.
|
447
|
+
A method to build one is provided here:
|
452
448
|
|
453
|
-
The procedure
|
449
|
+
The procedure:
|
454
450
|
|
455
451
|
* Follow the non-root installation procedure with RVM, including gem
|
456
|
-
*
|
452
|
+
* Archive (zip, tar) the main RVM folder (includes <%=cmd%>):
|
457
453
|
|
458
454
|
```
|
459
|
-
$
|
460
|
-
$ tar zcvf rvm_<%=cmd%>.tgz .rvm
|
455
|
+
cd $HOME && tar zcvf rvm-<%=cmd%>.tgz .rvm
|
461
456
|
```
|
462
457
|
|
463
|
-
*
|
458
|
+
* Get the Aspera SDK.
|
464
459
|
|
465
460
|
```
|
466
|
-
|
461
|
+
<%=cmd%> conf --show-config --fields=sdk_url
|
467
462
|
```
|
468
463
|
|
469
|
-
*
|
464
|
+
* Download the SDK archive from that URL.
|
470
465
|
|
471
466
|
```
|
472
|
-
|
473
|
-
$ tar zxvf rvm_<%=cmd%>.tgz
|
474
|
-
$ source ~/.rvm/scripts/rvm
|
475
|
-
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
467
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
476
468
|
```
|
477
469
|
|
478
|
-
|
470
|
+
* Transfer those 2 files to the target system
|
479
471
|
|
480
|
-
|
472
|
+
* On target system
|
473
|
+
|
474
|
+
```
|
475
|
+
cd $HOME
|
476
|
+
|
477
|
+
tar zxvf rvm-<%=cmd%>.tgz
|
478
|
+
|
479
|
+
source ~/.rvm/scripts/rvm
|
480
|
+
|
481
|
+
<%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
482
|
+
```
|
483
|
+
|
484
|
+
* Add those lines to shell init (`.profile`)
|
485
|
+
|
486
|
+
```
|
487
|
+
source ~/.rvm/scripts/rvm
|
488
|
+
```
|
489
|
+
|
490
|
+
# <a id="cli"></a>Command Line Interface: <%=tool%>
|
491
|
+
|
492
|
+
The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
481
493
|
|
482
494
|
* IBM Aspera High Speed Transfer Server (FASP and Node)
|
483
495
|
* IBM Aspera on Cloud (including ATS)
|
@@ -492,7 +504,7 @@ The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which inte
|
|
492
504
|
* Supports most Aspera server products (on-premise and SaaS)
|
493
505
|
* Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
494
506
|
* Supports Commands, Option values and Parameters shortcuts
|
495
|
-
* FASP [Transfer Agents](#agents) can be:
|
507
|
+
* FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
496
508
|
* Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
|
497
509
|
* Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
498
510
|
* Supports FaspStream creation (using Node API)
|
@@ -504,7 +516,7 @@ The `<%= gemspec.name %>` Gem provides a command line interface (CLI) which inte
|
|
504
516
|
Basic usage is displayed by executing:
|
505
517
|
|
506
518
|
```
|
507
|
-
|
519
|
+
<%=cmd%> -h
|
508
520
|
```
|
509
521
|
|
510
522
|
Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
|
@@ -518,7 +530,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
|
|
518
530
|
There are two types of arguments: Commands and Options. Example :
|
519
531
|
|
520
532
|
```
|
521
|
-
|
533
|
+
<%=cmd%> command --option-name=VAL1 VAL2
|
522
534
|
```
|
523
535
|
|
524
536
|
* executes _command_: `command`
|
@@ -546,13 +558,16 @@ Exceptions:
|
|
546
558
|
* the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
547
559
|
|
548
560
|
```
|
549
|
-
|
561
|
+
<%=cmd%> config echo -- --sample
|
562
|
+
```
|
563
|
+
|
564
|
+
```
|
550
565
|
"--sample"
|
551
566
|
```
|
552
567
|
|
553
568
|
Note that `--sample` is taken as an argument, and not option.
|
554
569
|
|
555
|
-
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on
|
570
|
+
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
556
571
|
|
557
572
|
The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
|
558
573
|
|
@@ -573,7 +588,7 @@ Command line arguments that are not options are either commands or arguments. If
|
|
573
588
|
|
574
589
|
Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
|
575
590
|
|
576
|
-
The
|
591
|
+
The behavior can be controlled with:
|
577
592
|
|
578
593
|
* --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
579
594
|
* yes : missing mandatory parameters/options are asked to the user
|
@@ -590,8 +605,8 @@ The information displayed depends on the action.
|
|
590
605
|
|
591
606
|
Depending on action, the output will contain:
|
592
607
|
|
593
|
-
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is
|
594
|
-
* `object_list` : displayed as a 2 dimensional table: one line per item, one
|
608
|
+
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
609
|
+
* `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
595
610
|
* `value_list` : a table with one column.
|
596
611
|
* `empty` : nothing
|
597
612
|
* `status` : a message
|
@@ -616,12 +631,15 @@ The style of output can be set using the `format` parameter, supporting:
|
|
616
631
|
* `yaml` : YAML
|
617
632
|
* `csv` : Comma Separated Values
|
618
633
|
|
619
|
-
###
|
634
|
+
### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
620
635
|
|
621
636
|
Table output can be filtered using the `select` parameter. Example:
|
622
637
|
|
623
638
|
```
|
624
|
-
|
639
|
+
<%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
640
|
+
```
|
641
|
+
|
642
|
+
```
|
625
643
|
:...............................:..................................:...........:
|
626
644
|
: name : email : ats_admin :
|
627
645
|
:...............................:..................................:...........:
|
@@ -634,7 +652,7 @@ Note that `select` filters selected elements from the result of API calls, while
|
|
634
652
|
|
635
653
|
### Verbosity of output
|
636
654
|
|
637
|
-
|
655
|
+
Output messages are categorized in 3 types:
|
638
656
|
|
639
657
|
* `info` output contain additional information, such as number of elements in a table
|
640
658
|
* `data` output contain the actual output of the command (object, or list of objects)
|
@@ -648,7 +666,7 @@ The option `display` controls the level of output:
|
|
648
666
|
|
649
667
|
### Selection of output object properties
|
650
668
|
|
651
|
-
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific
|
669
|
+
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
652
670
|
|
653
671
|
* DEF : default display of columns (that's the default, when not set)
|
654
672
|
* ALL : all columns available
|
@@ -657,7 +675,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
657
675
|
* +a,b,c : add selected properties to the default selection.
|
658
676
|
* -a,b,c : remove selected properties from the default selection.
|
659
677
|
|
660
|
-
## <a
|
678
|
+
## <a id="extended"></a>Extended Value Syntax
|
661
679
|
|
662
680
|
Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
|
663
681
|
|
@@ -694,23 +712,32 @@ To display the result of an extended value, use the `config echo` command.
|
|
694
712
|
Example: read the content of the specified file, then, base64 decode, then unzip:
|
695
713
|
|
696
714
|
```
|
697
|
-
|
715
|
+
<%=cmd%> config echo @zlib:@base64:@file:myfile.dat
|
698
716
|
```
|
699
717
|
|
700
718
|
Example: create a value as a hash, with one key and the value is read from a file:
|
701
719
|
|
702
720
|
```
|
703
|
-
|
721
|
+
<%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
|
704
722
|
```
|
705
723
|
|
706
724
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
707
725
|
|
708
726
|
```
|
709
|
-
|
727
|
+
cat test.csv
|
728
|
+
```
|
729
|
+
|
730
|
+
```
|
710
731
|
name,email
|
711
732
|
lolo,laurent@example.com
|
712
733
|
toto,titi@tutu.tata
|
713
|
-
|
734
|
+
```
|
735
|
+
|
736
|
+
```
|
737
|
+
<%=cmd%> config echo @csvt:@file:test.csv
|
738
|
+
```
|
739
|
+
|
740
|
+
```
|
714
741
|
:......:.....................:
|
715
742
|
: name : email :
|
716
743
|
:......:.....................:
|
@@ -722,13 +749,16 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
|
|
722
749
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
723
750
|
|
724
751
|
```
|
725
|
-
|
752
|
+
<%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
753
|
+
```
|
754
|
+
|
755
|
+
```
|
726
756
|
{"version"=>"0.9", "hello"=>true}
|
727
757
|
```
|
728
758
|
|
729
759
|
Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
|
730
760
|
|
731
|
-
## <a
|
761
|
+
## <a id="native"></a>Structured Value
|
732
762
|
|
733
763
|
Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
|
734
764
|
|
@@ -739,7 +769,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
|
|
739
769
|
|
740
770
|
It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
|
741
771
|
|
742
|
-
## <a
|
772
|
+
## <a id="conffolder"></a>Configuration and Persistency Folder
|
743
773
|
|
744
774
|
<%=tool%> configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/<%=cmd%>`.
|
745
775
|
|
@@ -749,53 +779,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
|
|
749
779
|
The main folder can be displayed using :
|
750
780
|
|
751
781
|
```
|
752
|
-
|
782
|
+
<%=cmd%> config folder
|
783
|
+
```
|
784
|
+
|
785
|
+
```
|
753
786
|
/Users/kenji/.aspera/<%=cmd%>
|
754
787
|
```
|
755
788
|
|
756
|
-
It can be
|
789
|
+
It can be overridden using the environment variable `<%=evp%>HOME`.
|
757
790
|
|
758
791
|
Example (Windows):
|
759
792
|
|
760
793
|
```
|
761
|
-
|
762
|
-
|
794
|
+
set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
|
795
|
+
<%=cmd%> config folder
|
763
796
|
C:\Users\Kenji\.aspera\<%=cmd%>
|
764
797
|
```
|
765
798
|
|
766
|
-
## <a
|
799
|
+
## <a id="configfile"></a>Configuration file
|
767
800
|
|
768
801
|
On the first execution of <%=tool%>, an empty configuration file is created in the configuration folder.
|
769
802
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
770
803
|
|
771
|
-
Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it
|
772
|
-
using the `config` command.
|
804
|
+
Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it using the `config` command.
|
773
805
|
|
774
|
-
All options for <%=tool%>
|
806
|
+
All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuration file.
|
775
807
|
|
776
|
-
A configuration file provides a way to define default values, especially
|
777
|
-
for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
808
|
+
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
778
809
|
|
779
|
-
The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml`
|
780
|
-
(this can be overriden with option `--config-file=path` or equivalent env var).
|
810
|
+
The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
|
781
811
|
|
782
|
-
|
783
|
-
called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
812
|
+
The configuration file is simply a catalog of pre-defined lists of options, called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
784
813
|
|
785
|
-
### <a
|
814
|
+
### <a id="lprt"></a><%=prstt%>
|
786
815
|
|
787
816
|
A <%=prst%> is simply a collection of parameters and their associated values in a named section in the configuration file.
|
788
817
|
|
789
818
|
A named <%=prst%> can be modified directly using <%=tool%>, which will update the configuration file :
|
790
819
|
|
791
820
|
```
|
792
|
-
|
821
|
+
<%=cmd%> config preset set|delete|show|initialize|update <<%=opprst%>>
|
793
822
|
```
|
794
823
|
|
795
824
|
The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
|
796
825
|
|
797
826
|
```
|
798
|
-
|
827
|
+
<%=cmd%> config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
799
828
|
```
|
800
829
|
|
801
830
|
* This creates a <%=prst%> `demo_server` with all provided options.
|
@@ -803,39 +832,48 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
|
|
803
832
|
The command `set` allows setting individual options in a <%=prst%>.
|
804
833
|
|
805
834
|
```
|
806
|
-
|
835
|
+
<%=cmd%> config preset set demo_server password _demo_pass_
|
807
836
|
```
|
808
837
|
|
809
838
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
810
839
|
|
811
840
|
```
|
812
|
-
|
841
|
+
<%=cmd%> config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
813
842
|
```
|
814
843
|
|
815
|
-
A
|
816
|
-
If necessary, the configuration file can be edited (or simply consulted) with:
|
844
|
+
A full terminal based overview of the configuration can be displayed using:
|
817
845
|
|
818
846
|
```
|
819
|
-
|
847
|
+
<%=cmd%> config preset over
|
820
848
|
```
|
821
849
|
|
822
|
-
A
|
850
|
+
A list of <%=prst%> can be displayed using:
|
823
851
|
|
824
852
|
```
|
825
|
-
|
853
|
+
<%=cmd%> config preset list
|
826
854
|
```
|
827
855
|
|
828
|
-
A
|
856
|
+
A good practice is to not manually edit the configuration file and use modification commands instead.
|
857
|
+
If necessary, the configuration file can opened in a text editor with:
|
829
858
|
|
830
859
|
```
|
831
|
-
|
860
|
+
<%=cmd%> config open
|
832
861
|
```
|
833
862
|
|
834
|
-
|
863
|
+
Older format for commands are still supported:
|
864
|
+
|
865
|
+
```
|
866
|
+
<%=cmd%> config id <name> set|delete|show|initialize|update
|
867
|
+
<%=cmd%> config over
|
868
|
+
<%=cmd%> config list
|
869
|
+
```
|
870
|
+
|
871
|
+
|
872
|
+
### <a id="lprtconf"></a>Special <%=prstt%>: config
|
835
873
|
|
836
874
|
This preset name is reserved and contains a single key: `version`. This is the version of <%=tool%> which created the file.
|
837
875
|
|
838
|
-
### <a
|
876
|
+
### <a id="lprtdef"></a>Special <%=prstt%>: default
|
839
877
|
|
840
878
|
This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
|
841
879
|
|
@@ -846,16 +884,16 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
846
884
|
Operations on this preset are done using regular `config` operations:
|
847
885
|
|
848
886
|
```
|
849
|
-
|
850
|
-
|
887
|
+
<%=cmd%> config preset set default _plugin_name_ _default_preset_for_plugin_
|
888
|
+
<%=cmd%> config preset get default _plugin_name_
|
851
889
|
"_default_preset_for_plugin_"
|
852
890
|
```
|
853
891
|
|
854
|
-
### <a
|
892
|
+
### <a id="lplugconf"></a>Special Plugin: config
|
855
893
|
|
856
894
|
Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
|
857
895
|
|
858
|
-
When <%=tool%> starts, it
|
896
|
+
When <%=tool%> starts, it looks for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
859
897
|
|
860
898
|
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
861
899
|
|
@@ -880,7 +918,7 @@ demo_server:
|
|
880
918
|
We can see here:
|
881
919
|
|
882
920
|
* The configuration was created with CLI version 0.3.7
|
883
|
-
* the default <%=prst%> to load for plugin
|
921
|
+
* the default <%=prst%> to load for `server` plugin is : `demo_server`
|
884
922
|
* the <%=prst%> `demo_server` defines some parameters: the URL and credentials
|
885
923
|
* the default <%=prst%> to load in any case is : `cli_default`
|
886
924
|
|
@@ -894,10 +932,10 @@ The user may create as many <%=prsts%> as needed. For instance, a particular <%=
|
|
894
932
|
|
895
933
|
Values in the configuration also follow the [Extended Value Syntax](#extended).
|
896
934
|
|
897
|
-
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config
|
935
|
+
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
|
898
936
|
|
899
937
|
```
|
900
|
-
|
938
|
+
<%=cmd%> config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
|
901
939
|
```
|
902
940
|
|
903
941
|
This creates the <%=prst%>:
|
@@ -911,36 +949,51 @@ my_aoc_org:
|
|
911
949
|
|
912
950
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
913
951
|
|
952
|
+
### Options evaluation order
|
953
|
+
|
954
|
+
Some options are global, some options are available only for some plugins. (the plugin is the first level command).
|
955
|
+
|
914
956
|
Options are loaded using this algorithm:
|
915
957
|
|
916
|
-
*
|
917
|
-
|
918
|
-
|
919
|
-
*
|
920
|
-
*
|
958
|
+
* If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
959
|
+
* else it looks for the name of the plugin as key in section `default`, the value is the name of the default <%=prst%> for it, and loads it.
|
960
|
+
* If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the <%=prst%> specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
961
|
+
* Environment variables are evaluated
|
962
|
+
* Command line options are evaluated
|
921
963
|
|
922
964
|
Parameters are evaluated in the order of command line.
|
923
965
|
|
924
|
-
To avoid loading the default <%=prst%> for a plugin,
|
966
|
+
To avoid loading the default <%=prst%> for a plugin, use: `-N`
|
925
967
|
|
926
968
|
On command line, words in parameter names are separated by a dash, in configuration file, separator
|
927
969
|
is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
|
928
970
|
|
929
|
-
|
930
|
-
convert olver versions, remove the leading ":" in front of keys.
|
971
|
+
The main plugin name is `config`, so it is possible to define a default <%=prst%> for the main plugin with:
|
931
972
|
|
932
|
-
|
933
|
-
|
973
|
+
```
|
974
|
+
<%=cmd%> config preset set cli_default interactive no
|
975
|
+
```
|
934
976
|
|
935
977
|
```
|
936
|
-
|
937
|
-
$ <%=cmd%> config id default set config cli_default
|
978
|
+
<%=cmd%> config preset set default config cli_default
|
938
979
|
```
|
939
980
|
|
940
981
|
A <%=prst%> value can be removed with `unset`:
|
941
982
|
|
942
983
|
```
|
943
|
-
|
984
|
+
<%=cmd%> config preset unset cli_default interactive
|
985
|
+
```
|
986
|
+
|
987
|
+
Example: Define options using command line:
|
988
|
+
|
989
|
+
```
|
990
|
+
<%=cmd%> -N --url=x --password=y --username=y node --show-config
|
991
|
+
```
|
992
|
+
|
993
|
+
Example: Define options using a hash:
|
994
|
+
|
995
|
+
```
|
996
|
+
<%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
944
997
|
```
|
945
998
|
|
946
999
|
### Examples
|
@@ -950,44 +1003,120 @@ only username/password and url are required (either on command line, or from con
|
|
950
1003
|
Those can usually be provided on the command line:
|
951
1004
|
|
952
1005
|
```
|
953
|
-
|
1006
|
+
<%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
|
954
1007
|
```
|
955
1008
|
|
956
1009
|
This can also be provisioned in a config file:
|
957
1010
|
|
1011
|
+
* Build <%=prst%>
|
1012
|
+
|
958
1013
|
```
|
959
|
-
|
960
|
-
|
961
|
-
|
962
|
-
4$ <%=cmd%> config id default set shares shares06
|
963
|
-
5$ <%=cmd%> config overview
|
964
|
-
6$ <%=cmd%> shares repo browse /
|
1014
|
+
<%=cmd%> config preset set shares06 url https://10.25.0.6
|
1015
|
+
<%=cmd%> config preset set shares06 username john
|
1016
|
+
<%=cmd%> config preset set shares06 password 4sp3ra
|
965
1017
|
```
|
966
1018
|
|
967
|
-
The three first commands build a <%=prst%>.
|
968
1019
|
Note that this can also be done with one single command:
|
969
1020
|
|
970
1021
|
```
|
971
|
-
|
1022
|
+
<%=cmd%> config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
|
1023
|
+
```
|
1024
|
+
|
1025
|
+
or
|
1026
|
+
|
1027
|
+
```
|
1028
|
+
<%=cmd%> config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
|
1029
|
+
```
|
1030
|
+
|
1031
|
+
* Define this <%=prst%> as the default <%=prst%> for the specified plugin (`shares`)
|
1032
|
+
|
1033
|
+
```
|
1034
|
+
<%=cmd%> config preset set default shares shares06
|
1035
|
+
```
|
1036
|
+
|
1037
|
+
* Display the content of configuration file in table format
|
1038
|
+
|
1039
|
+
```
|
1040
|
+
<%=cmd%> config overview
|
1041
|
+
```
|
1042
|
+
|
1043
|
+
* Execute a command on the shares application using default parameters
|
1044
|
+
|
1045
|
+
```
|
1046
|
+
<%=cmd%> shares repo browse /
|
1047
|
+
```
|
1048
|
+
|
1049
|
+
## <a id="vault"></a>Secret Vault
|
1050
|
+
|
1051
|
+
When a secret or password is needed, it is possible to store in the secret vault.
|
1052
|
+
|
1053
|
+
By default the vault is defined using option `secrets`.
|
1054
|
+
|
1055
|
+
### Using system keychain
|
1056
|
+
|
1057
|
+
Only on macOS.
|
1058
|
+
|
1059
|
+
It is possible to store secrets in macOS keychain (only read supported currently).
|
1060
|
+
|
1061
|
+
Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
|
1062
|
+
|
1063
|
+
### Modern config file format: encrypted in config file
|
1064
|
+
|
1065
|
+
It is possible to store and use secrets encrypted.
|
1066
|
+
For this use the `config vault` command.
|
1067
|
+
|
1068
|
+
The vault can be initialized with `config vault init`
|
1069
|
+
|
1070
|
+
Then secrets can be manipulated using commands:
|
1071
|
+
|
1072
|
+
* `set`
|
1073
|
+
* `get`
|
1074
|
+
* `list`
|
1075
|
+
* `delete`
|
1076
|
+
|
1077
|
+
Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
|
1078
|
+
|
1079
|
+
### Legacy config file format
|
1080
|
+
|
1081
|
+
The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
|
1082
|
+
|
1083
|
+
For example, choose a repository name, for example `my_secrets`, and populate it like this:
|
1084
|
+
|
1085
|
+
```
|
1086
|
+
<%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
|
1087
|
+
|
1088
|
+
<%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
|
1089
|
+
|
1090
|
+
<%=cmd%> conf id default get config
|
1091
|
+
|
1092
|
+
cli_default
|
972
1093
|
```
|
973
1094
|
|
974
|
-
|
975
|
-
|
976
|
-
|
977
|
-
|
1095
|
+
Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
|
1096
|
+
So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
|
1097
|
+
|
1098
|
+
```
|
1099
|
+
<%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
|
1100
|
+
```
|
978
1101
|
|
979
|
-
|
1102
|
+
A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
|
980
1103
|
|
1104
|
+
To test if a secret can be found use:
|
1105
|
+
|
1106
|
+
```
|
1107
|
+
<%=cmd%> conf vault get --username=access_key1
|
1108
|
+
```
|
981
1109
|
|
982
1110
|
## Plugins
|
983
1111
|
|
984
1112
|
The CLI tool uses a plugin mechanism. The first level command (just after <%=tool%> on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
|
985
1113
|
For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
|
986
1114
|
|
987
|
-
### Create your own plugin
|
1115
|
+
### <a id="createownplugin"></a>Create your own plugin
|
1116
|
+
|
988
1117
|
```
|
989
|
-
|
990
|
-
|
1118
|
+
mkdir -p ~/.aspera/<%=cmd%>/plugins
|
1119
|
+
cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
|
991
1120
|
require 'aspera/cli/plugin'
|
992
1121
|
module Aspera
|
993
1122
|
module Cli
|
@@ -1002,39 +1131,78 @@ end # Aspera
|
|
1002
1131
|
EOF
|
1003
1132
|
```
|
1004
1133
|
|
1005
|
-
|
1134
|
+
### <a id="plugins"></a>Plugins: Application URL and Authentication
|
1135
|
+
|
1136
|
+
<%=tool%> comes with several Aspera application plugins.
|
1137
|
+
|
1138
|
+
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
1006
1139
|
|
1007
|
-
|
1140
|
+
Those are using options:
|
1008
1141
|
|
1009
|
-
|
1142
|
+
* url
|
1143
|
+
* username
|
1144
|
+
* password
|
1145
|
+
|
1146
|
+
Those can be provided using command line, parameter set, env var, see section above.
|
1147
|
+
|
1148
|
+
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
1149
|
+
|
1150
|
+
## Logging, Debugging
|
1151
|
+
|
1152
|
+
The gem is equipped with traces. By default logging level is `warn`.
|
1153
|
+
To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `<%=evp%>LOG_LEVEL`, or parameter in con file).
|
1154
|
+
|
1155
|
+
It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
|
1156
|
+
|
1157
|
+
By default passwords and secrets are removed from logs.
|
1158
|
+
Use option `log_passwords` to change this behaviour.
|
1010
1159
|
|
1011
1160
|
## Learning Aspera Product APIs (REST)
|
1012
1161
|
|
1013
1162
|
This CLI uses REST APIs.
|
1014
|
-
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
|
1015
|
-
exact content or HTTP requests and responses.
|
1163
|
+
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
|
1016
1164
|
|
1017
1165
|
In order to get traces of execution, use argument : `--log-level=debug`
|
1018
1166
|
|
1019
|
-
## <a
|
1167
|
+
## <a id="http_options"></a>HTTP socket parameters
|
1168
|
+
|
1169
|
+
If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
|
1170
|
+
|
1171
|
+
Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
|
1172
|
+
|
1173
|
+
* `read_timeout` 60 sec
|
1174
|
+
* `write_timeout` 60 sec
|
1175
|
+
|
1176
|
+
Default values are the ones of Ruby.
|
1177
|
+
|
1178
|
+
Example:
|
1179
|
+
|
1180
|
+
```
|
1181
|
+
<%=cmd%> aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
1182
|
+
```
|
1183
|
+
|
1184
|
+
## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
|
1020
1185
|
|
1021
1186
|
Some actions may require the use of a graphical tool:
|
1022
1187
|
|
1023
1188
|
* a browser for Aspera on Cloud authentication (web auth method)
|
1024
1189
|
* a text editor for configuration file edition
|
1025
1190
|
|
1026
|
-
By default the CLI will assume that a graphical environment is available on windows,
|
1027
|
-
and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1191
|
+
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1028
1192
|
It is also possible to force the graphical mode with option --ui :
|
1029
1193
|
|
1030
|
-
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
|
1031
|
-
a text
|
1032
|
-
* `--ui=text` forces a text environment, the URL or file path to open is displayed on
|
1033
|
-
terminal.
|
1194
|
+
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
1195
|
+
* `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
1034
1196
|
|
1035
1197
|
## HTTP proxy for REST
|
1036
1198
|
|
1037
|
-
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are
|
1199
|
+
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
|
1200
|
+
|
1201
|
+
## <a id="certificates"></a>SSL CA certificate bundle
|
1202
|
+
|
1203
|
+
<%=tool%> uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
|
1204
|
+
|
1205
|
+
`ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). <%=tool%> overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
|
1038
1206
|
|
1039
1207
|
## Proxy auto config
|
1040
1208
|
|
@@ -1043,13 +1211,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
|
|
1043
1211
|
The PAC file can be tested with command: `config proxy_check` , example:
|
1044
1212
|
|
1045
1213
|
```
|
1046
|
-
|
1214
|
+
<%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
|
1047
1215
|
PROXY proxy.example.com:8080
|
1048
1216
|
```
|
1049
1217
|
|
1050
1218
|
This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
|
1051
1219
|
|
1052
|
-
## <a
|
1220
|
+
## <a id="client"></a>FASP configuration
|
1053
1221
|
|
1054
1222
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
1055
1223
|
|
@@ -1061,9 +1229,9 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
1061
1229
|
### Show path of currently used `ascp`
|
1062
1230
|
|
1063
1231
|
```
|
1064
|
-
|
1232
|
+
<%=cmd%> config ascp show
|
1065
1233
|
/Users/laurent/.aspera/ascli/sdk/ascp
|
1066
|
-
|
1234
|
+
<%=cmd%> config ascp info
|
1067
1235
|
+--------------------+-----------------------------------------------------------+
|
1068
1236
|
| key | value |
|
1069
1237
|
+--------------------+-----------------------------------------------------------+
|
@@ -1071,7 +1239,7 @@ $ <%=cmd%> config ascp info
|
|
1071
1239
|
...
|
1072
1240
|
```
|
1073
1241
|
|
1074
|
-
### Selection of
|
1242
|
+
### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
1075
1243
|
|
1076
1244
|
By default, <%=tool%> uses any found local product with ascp, including SDK.
|
1077
1245
|
|
@@ -1082,7 +1250,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
|
|
1082
1250
|
Using a POSIX shell:
|
1083
1251
|
|
1084
1252
|
```
|
1085
|
-
|
1253
|
+
<%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
1086
1254
|
ascp version: 4.0.0.182279
|
1087
1255
|
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
1088
1256
|
Saved to default global preset global_common_defaults
|
@@ -1091,7 +1259,7 @@ Saved to default global preset global_common_defaults
|
|
1091
1259
|
Windows:
|
1092
1260
|
|
1093
1261
|
```
|
1094
|
-
|
1262
|
+
<%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1095
1263
|
ascp version: 4.0.0.182279
|
1096
1264
|
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1097
1265
|
Saved to default global preset global_common_defaults
|
@@ -1104,7 +1272,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
|
|
1104
1272
|
Locally installed Aspera products can be listed with:
|
1105
1273
|
|
1106
1274
|
```
|
1107
|
-
|
1275
|
+
<%=cmd%> config ascp products list
|
1108
1276
|
:.........................................:................................................:
|
1109
1277
|
: name : app_root :
|
1110
1278
|
:.........................................:................................................:
|
@@ -1115,7 +1283,7 @@ $ <%=cmd%> config ascp products list
|
|
1115
1283
|
:.........................................:................................................:
|
1116
1284
|
```
|
1117
1285
|
|
1118
|
-
### Selection of local client
|
1286
|
+
### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
1119
1287
|
|
1120
1288
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1121
1289
|
|
@@ -1124,14 +1292,14 @@ Using the option use_product finds the ascp binary of the selected product.
|
|
1124
1292
|
To permanently use the ascp of a product:
|
1125
1293
|
|
1126
1294
|
```
|
1127
|
-
|
1295
|
+
<%=cmd%> config ascp products use 'Aspera Connect'
|
1128
1296
|
saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
|
1129
1297
|
```
|
1130
1298
|
|
1131
1299
|
### Installation of Connect Client on command line
|
1132
1300
|
|
1133
1301
|
```
|
1134
|
-
|
1302
|
+
<%=cmd%> config ascp connect list
|
1135
1303
|
:...............................................:......................................:..............:
|
1136
1304
|
: id : title : version :
|
1137
1305
|
:...............................................:......................................:..............:
|
@@ -1144,7 +1312,7 @@ $ <%=cmd%> config ascp connect list
|
|
1144
1312
|
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
|
1145
1313
|
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
|
1146
1314
|
:...............................................:......................................:..............:
|
1147
|
-
|
1315
|
+
<%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
1148
1316
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1149
1317
|
: title : type : href : hreflang : rel :
|
1150
1318
|
:.............................................:..........................:.......................................................................:..........:...............:
|
@@ -1157,11 +1325,11 @@ $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
|
1157
1325
|
: Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
|
1158
1326
|
: Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
|
1159
1327
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1160
|
-
|
1328
|
+
<%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
|
1161
1329
|
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
|
1162
1330
|
```
|
1163
1331
|
|
1164
|
-
## <a
|
1332
|
+
## <a id="agents"></a>Transfer Agents
|
1165
1333
|
|
1166
1334
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
1167
1335
|
|
@@ -1170,94 +1338,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
1170
1338
|
|
1171
1339
|
There are currently 3 agents:
|
1172
1340
|
|
1173
|
-
* `direct` : a local execution of `ascp`
|
1174
|
-
* `connect` : use of a local Connect Client
|
1175
|
-
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
1176
|
-
* `httpgw` : use of an Aspera HTTP Gateway
|
1341
|
+
* [`direct`](#agt_direct) : a local execution of `ascp`
|
1342
|
+
* [`connect`](#agt_connect) : use of a local Connect Client
|
1343
|
+
* [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
|
1344
|
+
* [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1345
|
+
* [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1177
1346
|
|
1178
1347
|
Note that all transfer operation are seen from the point of view of the agent.
|
1179
1348
|
For instance, a node agent making an "upload", or "package send" operation,
|
1180
1349
|
will effectively push files to the related server from the agent node.
|
1181
1350
|
|
1182
|
-
<%=tool%>
|
1351
|
+
<%=tool%> standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1183
1352
|
|
1184
1353
|
|
1185
|
-
### <a
|
1354
|
+
### <a id="agt_direct"></a>Direct
|
1186
1355
|
|
1187
|
-
|
1188
|
-
|
1356
|
+
The `direct` agent directly executes a local ascp.
|
1357
|
+
This is the default for <%=tool%>.
|
1358
|
+
This is equivalent to specifying `--transfer=direct`.
|
1359
|
+
<%=tool%> will detect locally installed Aspera products, including SDK.
|
1189
1360
|
Refer to section [FASP](#client).
|
1190
1361
|
|
1191
|
-
|
1192
|
-
|
1193
|
-
* `EX_fasp_proxy_url`
|
1194
|
-
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1195
|
-
* `EX_ascp_args`
|
1196
|
-
|
1197
|
-
The `transfer-info` accepts the following optional parameters:
|
1362
|
+
The `transfer-info` accepts the following optional parameters to control multi-session, WSS
|
1198
1363
|
|
1199
1364
|
<table>
|
1200
|
-
<tr><th>Name</th><th>Type</th><th>
|
1201
|
-
<tr><td>
|
1202
|
-
<tr><td>
|
1203
|
-
<tr><td>
|
1204
|
-
<tr><td>
|
1365
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1366
|
+
<tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
|
1367
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
|
1368
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
|
1369
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
|
1370
|
+
<tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
|
1371
|
+
<tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
|
1372
|
+
<tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
|
1373
|
+
<tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
|
1374
|
+
<tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
|
1205
1375
|
</table>
|
1206
1376
|
|
1207
|
-
Resume
|
1377
|
+
Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
1378
|
+
Sleep between iterations is:
|
1208
1379
|
|
1209
|
-
|
1210
|
-
|
1211
|
-
|
1212
|
-
|
1213
|
-
|
1214
|
-
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
1215
|
-
</table>
|
1380
|
+
```
|
1381
|
+
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
1382
|
+
```
|
1383
|
+
|
1384
|
+
Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
|
1216
1385
|
|
1217
1386
|
Examples:
|
1218
1387
|
|
1219
1388
|
```
|
1220
|
-
|
1221
|
-
|
1389
|
+
<%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
1390
|
+
<%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
1222
1391
|
```
|
1223
1392
|
|
1224
|
-
|
1393
|
+
To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
|
1394
|
+
|
1395
|
+
* `EX_fasp_proxy_url`
|
1396
|
+
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1397
|
+
* `EX_ascp_args`
|
1398
|
+
|
1399
|
+
### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
1225
1400
|
|
1226
|
-
By specifying option: `--transfer=connect`, <%=tool%> will start transfers
|
1227
|
-
using the locally installed Aspera Connect Client.
|
1401
|
+
By specifying option: `--transfer=connect`, <%=tool%> will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
|
1228
1402
|
|
1229
|
-
### Aspera Node API : Node to node transfers
|
1403
|
+
### <a id="agt_node"></a>Aspera Node API : Node to node transfers
|
1230
1404
|
|
1231
1405
|
By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
|
1232
1406
|
Transfer Server using the Node API, either on a local or remote node.
|
1407
|
+
Parameters provided in option `transfer_info` are:
|
1233
1408
|
|
1234
|
-
|
1235
|
-
|
1236
|
-
|
1237
|
-
|
1238
|
-
|
1409
|
+
<table>
|
1410
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1411
|
+
<tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
|
1412
|
+
<tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
|
1413
|
+
<tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
|
1414
|
+
<tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
|
1415
|
+
</table>
|
1239
1416
|
|
1240
|
-
|
1241
|
-
`--transfer-info=@preset:<psetname>` or specified using the
|
1417
|
+
Like any other option, `transfer_info` can get its value from a pre-configured <%=prst%> :
|
1418
|
+
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
1242
1419
|
`--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
|
1243
1420
|
|
1244
|
-
|
1421
|
+
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
1245
1422
|
|
1246
|
-
|
1423
|
+
If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
|
1247
1424
|
|
1248
|
-
### <a
|
1425
|
+
### <a id="agt_httpgw"></a>HTTP Gateway
|
1249
1426
|
|
1250
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
1427
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
|
1251
1428
|
|
1252
1429
|
Example:
|
1253
1430
|
|
1254
1431
|
```
|
1255
|
-
|
1432
|
+
<%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1256
1433
|
```
|
1257
1434
|
|
1258
1435
|
Note that the gateway only supports transfers authorized with a token.
|
1259
1436
|
|
1260
|
-
|
1437
|
+
### <a id="agt_trsdk"></a>Transfer SDK
|
1438
|
+
|
1439
|
+
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
1440
|
+
|
1441
|
+
By default it will listen on local port `55002` on `127.0.0.1`.
|
1442
|
+
|
1443
|
+
## <a id="transferspec"></a>Transfer Specification
|
1261
1444
|
|
1262
1445
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
1263
1446
|
is described in a _transfer-spec_ (Transfer Specification), such as:
|
@@ -1270,9 +1453,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
1270
1453
|
|
1271
1454
|
<%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1272
1455
|
|
1273
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are
|
1456
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
|
1274
1457
|
|
1275
|
-
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is
|
1458
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
1276
1459
|
|
1277
1460
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
1278
1461
|
|
@@ -1281,13 +1464,35 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
1281
1464
|
|
1282
1465
|
A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
1283
1466
|
|
1284
|
-
## <a
|
1467
|
+
## <a id="transferparams"></a>Transfer Parameters
|
1285
1468
|
|
1286
|
-
All standard _transfer-spec_ parameters can be
|
1287
|
-
|
1288
|
-
also be saved/overridden in the config file.
|
1469
|
+
All standard _transfer-spec_ parameters can be specified.
|
1470
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
1289
1471
|
|
1290
|
-
|
1472
|
+
References:
|
1473
|
+
|
1474
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
1475
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
1476
|
+
* [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
1477
|
+
|
1478
|
+
Parameters can be displayed with commands:
|
1479
|
+
|
1480
|
+
```
|
1481
|
+
<%=cmd%> config ascp spec
|
1482
|
+
<%=cmd%> config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
1483
|
+
```
|
1484
|
+
|
1485
|
+
Columns:
|
1486
|
+
|
1487
|
+
* D=Direct (local `ascp` execution)
|
1488
|
+
* N=Node API
|
1489
|
+
* C=Connect Client
|
1490
|
+
|
1491
|
+
`ascp` argument or environment variable is provided in description.
|
1492
|
+
|
1493
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in <%=tool%>).
|
1494
|
+
|
1495
|
+
<%=spec_table%>
|
1291
1496
|
|
1292
1497
|
### Destination folder for transfers
|
1293
1498
|
|
@@ -1303,20 +1508,20 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1303
1508
|
|
1304
1509
|
### List of files for transfers
|
1305
1510
|
|
1306
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
1307
|
-
|
1511
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
1512
|
+
So, by default, the list of files to transfer will be simply specified on the command line:
|
1308
1513
|
|
1309
1514
|
```
|
1310
|
-
|
1515
|
+
<%=cmd%> server upload ~/mysample.file secondfile
|
1311
1516
|
```
|
1312
1517
|
|
1313
1518
|
This is equivalent to:
|
1314
1519
|
|
1315
1520
|
```
|
1316
|
-
|
1521
|
+
<%=cmd%> server upload --sources=@args ~/mysample.file secondfile
|
1317
1522
|
```
|
1318
1523
|
|
1319
|
-
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1524
|
+
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
1320
1525
|
|
1321
1526
|
Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
|
1322
1527
|
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
@@ -1339,13 +1544,21 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
1339
1544
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
1340
1545
|
```
|
1341
1546
|
|
1342
|
-
|
1547
|
+
providing a file list directly to ascp:
|
1548
|
+
|
1549
|
+
```
|
1550
|
+
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
|
1551
|
+
```
|
1552
|
+
|
1553
|
+
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
1343
1554
|
|
1344
1555
|
```
|
1345
1556
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
1346
1557
|
```
|
1347
1558
|
|
1348
|
-
|
1559
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
1560
|
+
|
1561
|
+
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
1349
1562
|
|
1350
1563
|
* `list` : (default) the path of destination is the same as source
|
1351
1564
|
* `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
|
@@ -1353,9 +1566,11 @@ In case the file list is provided on the command line (i.e. using `--sources=@ar
|
|
1353
1566
|
Example:
|
1354
1567
|
|
1355
1568
|
```
|
1356
|
-
|
1569
|
+
<%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
1357
1570
|
```
|
1358
1571
|
|
1572
|
+
Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
|
1573
|
+
|
1359
1574
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
1360
1575
|
|
1361
1576
|
* All files must be in the same source folder.
|
@@ -1367,9 +1582,9 @@ Source files are located on "Aspera on cloud", when :
|
|
1367
1582
|
* the server is Aspera on Cloud, and making a download / recv
|
1368
1583
|
* the agent is Aspera on Cloud, and making an upload / send
|
1369
1584
|
|
1370
|
-
### <a
|
1585
|
+
### <a id="multisession"></a>Support of multi-session
|
1371
1586
|
|
1372
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
1587
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
1373
1588
|
|
1374
1589
|
* when agent=node :
|
1375
1590
|
|
@@ -1390,6 +1605,7 @@ shall be preferred.
|
|
1390
1605
|
|
1391
1606
|
Multi-session spawn is done by <%=tool%>.
|
1392
1607
|
|
1608
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
1393
1609
|
|
1394
1610
|
### Examples
|
1395
1611
|
|
@@ -1411,71 +1627,198 @@ Multi-session spawn is done by <%=tool%>.
|
|
1411
1627
|
--ts=@json:'{"http_fallback":"force"}'
|
1412
1628
|
```
|
1413
1629
|
|
1414
|
-
* Activate progress when not activated by default on server
|
1630
|
+
* Activate progress when not activated by default on server
|
1631
|
+
|
1632
|
+
```
|
1633
|
+
--ts=@json:'{"precalculate_job_size":true}'
|
1634
|
+
```
|
1635
|
+
|
1636
|
+
## <a id="scheduling"></a>Lock for exclusive execution
|
1637
|
+
|
1638
|
+
In some conditions, it may be desirable to ensure that <%=tool%> is not executed several times in parallel.
|
1639
|
+
|
1640
|
+
For instance when <%=tool%> is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
1641
|
+
|
1642
|
+
* Executing instances may pile-up and kill the system
|
1643
|
+
* The same file may be transferred by multiple instances at the same time.
|
1644
|
+
* `preview` may generate the same files in multiple instances.
|
1645
|
+
|
1646
|
+
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
1647
|
+
|
1648
|
+
* The Windows scheduler does this by default
|
1649
|
+
* Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
1650
|
+
|
1651
|
+
```
|
1652
|
+
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
1653
|
+
```
|
1654
|
+
|
1655
|
+
<%=tool%> natively supports a locking mechanism with option `lock_port`.
|
1656
|
+
(Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
|
1657
|
+
|
1658
|
+
Example:
|
1659
|
+
|
1660
|
+
Run this same command in two separate terminals within less than 30 seconds:
|
1661
|
+
|
1662
|
+
```
|
1663
|
+
<%=cmd%> config echo @ruby:'sleep(30)' --lock-port=12345
|
1664
|
+
```
|
1665
|
+
|
1666
|
+
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
1667
|
+
|
1668
|
+
```
|
1669
|
+
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
1670
|
+
```
|
1671
|
+
|
1672
|
+
## "Provençale"
|
1673
|
+
|
1674
|
+
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
1675
|
+
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
1676
|
+
|
1677
|
+
The list of supported *PVCL* adapters can be retrieved with command:
|
1678
|
+
|
1679
|
+
```
|
1680
|
+
<%=cmd%> conf ascp info
|
1681
|
+
+--------------------+-----------------------------------------------------------+
|
1682
|
+
| key | value |
|
1683
|
+
+--------------------+-----------------------------------------------------------+
|
1684
|
+
-----8<-----snip-----8<-----
|
1685
|
+
| product_name | IBM Aspera SDK |
|
1686
|
+
| product_version | 4.0.1.182389 |
|
1687
|
+
| process | pvcl |
|
1688
|
+
| shares | pvcl |
|
1689
|
+
| noded | pvcl |
|
1690
|
+
| faux | pvcl |
|
1691
|
+
| file | pvcl |
|
1692
|
+
| stdio | pvcl |
|
1693
|
+
| stdio-tar | pvcl |
|
1694
|
+
+--------------------+-----------------------------------------------------------+
|
1695
|
+
```
|
1696
|
+
|
1697
|
+
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
1698
|
+
|
1699
|
+
Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
|
1700
|
+
|
1701
|
+
The simplified format is:
|
1702
|
+
|
1703
|
+
```
|
1704
|
+
<adapter>:///<sub file path>?<arg1>=<val1>&...
|
1705
|
+
```
|
1706
|
+
|
1707
|
+
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
1708
|
+
|
1709
|
+
## <a id="faux_testing"></a>`faux:` for testing
|
1710
|
+
|
1711
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>.
|
1712
|
+
|
1713
|
+
This adapter can be used to simulate a file or a directory.
|
1714
|
+
|
1715
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
|
1716
|
+
|
1717
|
+
```
|
1718
|
+
faux:///filename?filesize
|
1719
|
+
```
|
1720
|
+
|
1721
|
+
where:
|
1722
|
+
|
1723
|
+
* `filename` is the name that will be assigned to the file on the destination
|
1724
|
+
* `filesize` is the number of bytes that will be sent (in decimal).
|
1725
|
+
|
1726
|
+
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
1727
|
+
|
1728
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
1729
|
+
|
1730
|
+
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
1731
|
+
|
1732
|
+
```
|
1733
|
+
faux:///dirname?<arg1>=<val1>&...
|
1734
|
+
```
|
1735
|
+
|
1736
|
+
where:
|
1737
|
+
|
1738
|
+
* `dirname` is the folder name and can contain `/` to specify a subfolder.
|
1739
|
+
* supported arguments are:
|
1740
|
+
|
1741
|
+
<table>
|
1742
|
+
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
1743
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
|
1744
|
+
<tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
|
1745
|
+
<tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
|
1746
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
|
1747
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
|
1748
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
|
1749
|
+
</table>
|
1750
|
+
|
1751
|
+
The sequence parameter is applied as follows:
|
1415
1752
|
|
1416
|
-
|
1417
|
-
--ts=@json:'{"precalculate_job_size":true}'
|
1418
|
-
```
|
1753
|
+
* If `seq` is `random` then each file size is:
|
1419
1754
|
|
1755
|
+
* size +/- (inc * rand())
|
1756
|
+
* Where rand is a random number between 0 and 1
|
1757
|
+
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
1758
|
+
* Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
1420
1759
|
|
1760
|
+
* If `seq` is `sequential` then each file size is:
|
1421
1761
|
|
1422
|
-
|
1762
|
+
* `size + ((fileindex - 1) * inc)`
|
1763
|
+
* Where first file is index 1
|
1764
|
+
* So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
1765
|
+
* As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
1423
1766
|
|
1424
|
-
|
1767
|
+
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
1425
1768
|
|
1426
|
-
|
1769
|
+
To discard data at the destination, the destination argument is set to `faux://` .
|
1427
1770
|
|
1428
|
-
|
1771
|
+
Examples:
|
1429
1772
|
|
1430
|
-
|
1773
|
+
* Upload 20 gibibytes of random data to file myfile to directory /Upload
|
1431
1774
|
|
1432
|
-
|
1775
|
+
```
|
1776
|
+
<%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
|
1777
|
+
```
|
1433
1778
|
|
1434
|
-
|
1779
|
+
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
1435
1780
|
|
1436
1781
|
```
|
1437
|
-
<%=
|
1438
|
-
...and more
|
1782
|
+
<%=cmd%> server upload /tmp/sample --to-folder=faux://
|
1439
1783
|
```
|
1440
1784
|
|
1441
|
-
|
1785
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
1442
1786
|
|
1443
1787
|
```
|
1444
|
-
|
1445
|
-
<%= File.read(ENV["INCL_USAGE"]) %>
|
1446
|
-
|
1788
|
+
<%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
1447
1789
|
```
|
1448
1790
|
|
1449
|
-
|
1450
|
-
|
1451
|
-
# <a name="plugins"></a>Plugins: Application URL and Authentication
|
1791
|
+
## <a id="commands"></a>Sample Commands
|
1452
1792
|
|
1453
|
-
|
1793
|
+
A non complete list of commands used in unit tests:
|
1454
1794
|
|
1455
|
-
|
1795
|
+
```
|
1796
|
+
<%=File.read(ENV["INCL_COMMANDS"])%>
|
1797
|
+
...and more
|
1798
|
+
```
|
1456
1799
|
|
1457
|
-
|
1800
|
+
## <a id="usage"></a>Usage
|
1458
1801
|
|
1459
|
-
|
1460
|
-
|
1461
|
-
|
1802
|
+
```
|
1803
|
+
<%=cmd%> -h
|
1804
|
+
<%=File.read(ENV["INCL_USAGE"])%>
|
1462
1805
|
|
1463
|
-
|
1806
|
+
```
|
1464
1807
|
|
1465
|
-
|
1808
|
+
Note that actions and parameter values can be written in short form.
|
1466
1809
|
|
1467
|
-
# <a
|
1810
|
+
# <a id="aoc"></a>Plugin: Aspera on Cloud
|
1468
1811
|
|
1469
1812
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
1470
1813
|
|
1471
1814
|
It is recommended to use the wizard to set it up, but manual configuration is also possible.
|
1472
1815
|
|
1473
|
-
## <a
|
1816
|
+
## <a id="aocwizard"></a>Configuration: using Wizard
|
1474
1817
|
|
1475
1818
|
<%=tool%> provides a configuration wizard. Here is a sample invocation :
|
1476
1819
|
|
1477
1820
|
```
|
1478
|
-
|
1821
|
+
<%=cmd%> config wizard
|
1479
1822
|
option: url> https://myorg.ibmaspera.com
|
1480
1823
|
Detected: Aspera on Cloud
|
1481
1824
|
Preparing preset: aoc_myorg
|
@@ -1491,7 +1834,7 @@ Setting config preset as default for aspera
|
|
1491
1834
|
saving config file
|
1492
1835
|
Done.
|
1493
1836
|
You can test with:
|
1494
|
-
|
1837
|
+
<%=cmd%> aoc user info show
|
1495
1838
|
```
|
1496
1839
|
|
1497
1840
|
Optionally, it is possible to create a new organization-specific "integration".
|
@@ -1499,7 +1842,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
1499
1842
|
|
1500
1843
|
This will guide you through the steps to create.
|
1501
1844
|
|
1502
|
-
## <a
|
1845
|
+
## <a id="aocmanual"></a>Configuration: using manual setup
|
1503
1846
|
|
1504
1847
|
If you used the wizard (recommended): skip this section.
|
1505
1848
|
|
@@ -1511,7 +1854,7 @@ Several types of OAuth authentication are supported:
|
|
1511
1854
|
* Web based authentication : authentication is made by user using a browser
|
1512
1855
|
* URL Token : external users authentication with url tokens (public links)
|
1513
1856
|
|
1514
|
-
The authentication method is
|
1857
|
+
The authentication method is controlled by option `auth`.
|
1515
1858
|
|
1516
1859
|
For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [<%=prst%> for Aspera on Cloud](#aocpreset).
|
1517
1860
|
|
@@ -1519,7 +1862,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
|
|
1519
1862
|
|
1520
1863
|
In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.<%=tool%> saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
1521
1864
|
|
1522
|
-
### <a
|
1865
|
+
### <a id="clientreg"></a>Optional: API Client Registration
|
1523
1866
|
|
1524
1867
|
If you use the built-in client_id and client_secret, skip this and do not set them in next section.
|
1525
1868
|
|
@@ -1543,31 +1886,31 @@ Note: for web based authentication, <%=tool%> listens on a local port (e.g. spec
|
|
1543
1886
|
|
1544
1887
|
Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
|
1545
1888
|
|
1546
|
-
### <a
|
1889
|
+
### <a id="aocpreset"></a><%=prst%> for Aspera on Cloud
|
1547
1890
|
|
1548
1891
|
If you did not use the wizard, you can also manually create a <%=prst%> for <%=tool%> in its configuration file.
|
1549
1892
|
|
1550
1893
|
Lets create an <%=prst%> called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
1551
1894
|
|
1552
1895
|
```
|
1553
|
-
|
1896
|
+
<%=cmd%> config preset ask my_aoc_org url client_id client_secret
|
1554
1897
|
option: url> https://myorg.ibmaspera.com/
|
1555
1898
|
option: client_id> BJLPObQiFw
|
1556
1899
|
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
|
1557
1900
|
updated: my_aoc_org
|
1558
1901
|
```
|
1559
1902
|
|
1560
|
-
(This can also be done in one line using the command `config
|
1903
|
+
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
1561
1904
|
|
1562
1905
|
Define this <%=prst%> as default configuration for the `aspera` plugin:
|
1563
1906
|
|
1564
1907
|
```
|
1565
|
-
|
1908
|
+
<%=cmd%> config preset set default aoc my_aoc_org
|
1566
1909
|
```
|
1567
1910
|
|
1568
1911
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
1569
1912
|
|
1570
|
-
### <a
|
1913
|
+
### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
|
1571
1914
|
|
1572
1915
|
For a Browser-less, Private Key-based authentication, use the following steps.
|
1573
1916
|
|
@@ -1582,13 +1925,13 @@ This can be done using any of the following method:
|
|
1582
1925
|
* using the CLI:
|
1583
1926
|
|
1584
1927
|
```
|
1585
|
-
|
1928
|
+
<%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
|
1586
1929
|
```
|
1587
1930
|
|
1588
1931
|
* `ssh-keygen`:
|
1589
1932
|
|
1590
1933
|
```
|
1591
|
-
|
1934
|
+
ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
|
1592
1935
|
```
|
1593
1936
|
|
1594
1937
|
* `openssl`
|
@@ -1596,11 +1939,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
|
|
1596
1939
|
(on some openssl implementation (mac) there is option: -nodes (no DES))
|
1597
1940
|
|
1598
1941
|
```
|
1599
|
-
|
1600
|
-
|
1601
|
-
|
1602
|
-
|
1603
|
-
|
1942
|
+
APIKEY=~/.aspera/<%=cmd%>/aocapikey
|
1943
|
+
openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
|
1944
|
+
openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
|
1945
|
+
openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
|
1946
|
+
rm -f ${APIKEY}.protected
|
1604
1947
|
```
|
1605
1948
|
|
1606
1949
|
#### API Client JWT activation
|
@@ -1619,13 +1962,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
1619
1962
|
* Using command line
|
1620
1963
|
|
1621
1964
|
```
|
1622
|
-
|
1965
|
+
<%=cmd%> aoc admin res client list
|
1623
1966
|
:............:.........:
|
1624
1967
|
: id : name :
|
1625
1968
|
:............:.........:
|
1626
1969
|
: BJLPObQiFw : <%=cmd%> :
|
1627
1970
|
:............:.........:
|
1628
|
-
|
1971
|
+
<%=cmd%> aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
1629
1972
|
modified
|
1630
1973
|
```
|
1631
1974
|
|
@@ -1646,14 +1989,14 @@ open the previously generated public key located here: `$HOME/.aspera/<%=cmd%>/a
|
|
1646
1989
|
* Using command line
|
1647
1990
|
|
1648
1991
|
```
|
1649
|
-
|
1992
|
+
<%=cmd%> aoc admin res user list
|
1650
1993
|
:........:................:
|
1651
1994
|
: id : name :
|
1652
1995
|
:........:................:
|
1653
1996
|
: 109952 : Tech Support :
|
1654
1997
|
: 109951 : LAURENT MARTIN :
|
1655
1998
|
:........:................:
|
1656
|
-
|
1999
|
+
<%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
|
1657
2000
|
modified
|
1658
2001
|
```
|
1659
2002
|
|
@@ -1661,16 +2004,16 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
1661
2004
|
|
1662
2005
|
### <%=prst%> modification for JWT
|
1663
2006
|
|
1664
|
-
To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the
|
2007
|
+
To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the following:
|
1665
2008
|
|
1666
2009
|
* change auth method to JWT
|
1667
2010
|
* provide location of private key
|
1668
|
-
* provide username to login as (
|
2011
|
+
* provide username to login as (OAuth "subject")
|
1669
2012
|
|
1670
2013
|
Execute:
|
1671
2014
|
|
1672
2015
|
```
|
1673
|
-
|
2016
|
+
<%=cmd%> config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
|
1674
2017
|
```
|
1675
2018
|
|
1676
2019
|
Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
|
@@ -1678,73 +2021,157 @@ Note: the private key argument represents the actual PEM string. In order to rea
|
|
1678
2021
|
After this last step, commands do not require web login anymore.
|
1679
2022
|
|
1680
2023
|
|
1681
|
-
### <a
|
2024
|
+
### <a id="aocfirst"></a>First Use
|
1682
2025
|
|
1683
2026
|
Once client has been registered and <%=prst%> created: <%=tool%> can be used:
|
1684
2027
|
|
1685
2028
|
```
|
1686
|
-
|
2029
|
+
<%=cmd%> aoc files br /
|
1687
2030
|
Current Workspace: Default Workspace (default)
|
1688
2031
|
empty
|
1689
2032
|
```
|
1690
2033
|
|
1691
2034
|
|
1692
|
-
|
2035
|
+
## Administration
|
1693
2036
|
|
1694
2037
|
The `admin` command allows several administrative tasks (and require admin privilege).
|
1695
2038
|
|
1696
2039
|
It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
|
1697
2040
|
|
1698
|
-
Bulk
|
2041
|
+
### Bulk creation and deletion of resource
|
2042
|
+
|
2043
|
+
Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
|
2044
|
+
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2045
|
+
|
2046
|
+
### Listing resources
|
2047
|
+
|
2048
|
+
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
2049
|
+
|
2050
|
+
The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
|
2051
|
+
|
2052
|
+
The following parameters are supported:
|
1699
2053
|
|
1700
|
-
|
2054
|
+
* `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
2055
|
+
* `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
2056
|
+
* `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
2057
|
+
* `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
2058
|
+
* `page` : native api parameter, in general do not use (added by
|
2059
|
+
* `per_page` : native api parameter, number of items par api call, in general do not use
|
2060
|
+
* Other specific parameters depending on resource type.
|
2061
|
+
|
2062
|
+
Both `max` and `pmax` are processed internally in <%=tool%>, not included in actual API call and limit the number of successive pages requested to API. <%=tool%> will return all values using paging if not provided.
|
2063
|
+
|
2064
|
+
Other parameters are directly sent as parameters to the GET request on API.
|
2065
|
+
|
2066
|
+
`page` and `per_page` are normally added by <%=tool%> to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
2067
|
+
|
2068
|
+
`q` and `sort` are available on most resource types.
|
2069
|
+
|
2070
|
+
Other parameters depend on the type of entity (refer to AoC API).
|
2071
|
+
|
2072
|
+
Examples:
|
2073
|
+
|
2074
|
+
* List users with `laurent` in name:
|
2075
|
+
|
2076
|
+
```
|
2077
|
+
<%=cmd%> aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2078
|
+
```
|
2079
|
+
|
2080
|
+
* List users who logged-in before a date:
|
2081
|
+
|
2082
|
+
```
|
2083
|
+
<%=cmd%> aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2084
|
+
```
|
2085
|
+
|
2086
|
+
* List external users and sort in reverse alphabetical order using name:
|
1701
2087
|
|
1702
2088
|
```
|
1703
|
-
--query=@json:'{"
|
2089
|
+
<%=cmd%> aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
1704
2090
|
```
|
1705
2091
|
|
1706
|
-
|
2092
|
+
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
2093
|
+
|
2094
|
+
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
2095
|
+
|
2096
|
+
### <a id="res_select"></a>Selecting a resource
|
2097
|
+
|
2098
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
|
2099
|
+
|
2100
|
+
To execute an action on a specific resource, select it using one of those methods:
|
2101
|
+
|
2102
|
+
* *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
|
2103
|
+
* give name on command line *after the action*: `aoc admin res node show name abc`
|
2104
|
+
* provide option `id` : `aoc admin res node show --id=123`
|
2105
|
+
* provide option `name` : `aoc admin res node show --name=abc`
|
2106
|
+
|
2107
|
+
### Access Key secrets
|
2108
|
+
|
2109
|
+
In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
|
2110
|
+
It is usually provided using the `secret` option.
|
2111
|
+
For example in a command like:
|
2112
|
+
|
1707
2113
|
```
|
1708
|
-
--
|
1709
|
-
--query=@json:'{"q":"laurent"}'
|
2114
|
+
<%=cmd%> aoc admin res node --id=123 --secret="secret1" v3 info
|
1710
2115
|
```
|
1711
2116
|
|
1712
|
-
|
2117
|
+
It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
|
1713
2118
|
|
1714
|
-
|
2119
|
+
### Activity
|
1715
2120
|
|
1716
|
-
|
1717
|
-
secret is required, it is usually provided using the `secret` option. For example in a command like:
|
2121
|
+
The activity app can be queried with:
|
1718
2122
|
|
1719
2123
|
```
|
1720
|
-
|
2124
|
+
<%=cmd%> aoc admin analytics transfers
|
1721
2125
|
```
|
1722
2126
|
|
1723
|
-
It
|
2127
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
1724
2128
|
|
1725
|
-
|
2129
|
+
`mytemplate.erb`:
|
1726
2130
|
|
1727
2131
|
```
|
1728
|
-
|
1729
|
-
|
1730
|
-
|
1731
|
-
|
2132
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
2133
|
+
To: <<%='<'%>%=ev['user_email']%>>
|
2134
|
+
Subject: <%='<'%>%=ev['files_completed']%> files received
|
2135
|
+
|
2136
|
+
Dear <%='<'%>%=ev[:user_email.to_s]%>,
|
2137
|
+
We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
|
2138
|
+
<%='<'%>%=ev['content']%>
|
2139
|
+
|
2140
|
+
Thank you.
|
1732
2141
|
```
|
2142
|
+
The environment provided contains the following additional variable:
|
2143
|
+
|
2144
|
+
* ev : all details on the transfer event
|
1733
2145
|
|
1734
|
-
|
2146
|
+
Example:
|
1735
2147
|
|
1736
2148
|
```
|
1737
|
-
|
2149
|
+
<%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
2150
|
+
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2151
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
1738
2152
|
```
|
1739
2153
|
|
1740
|
-
|
2154
|
+
Options:
|
2155
|
+
|
2156
|
+
* `once_only` keep track of last date it was called, so next call will get only new events
|
2157
|
+
* `query` filter (on API call)
|
2158
|
+
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
2159
|
+
|
2160
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
1741
2161
|
|
1742
|
-
|
2162
|
+
### Transfer: Using specific transfer ports
|
1743
2163
|
|
1744
|
-
|
2164
|
+
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
2165
|
+
The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
2166
|
+
|
2167
|
+
### Using ATS
|
2168
|
+
|
2169
|
+
Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
|
2170
|
+
|
2171
|
+
### Example: Bulk creation of users
|
1745
2172
|
|
1746
2173
|
```
|
1747
|
-
|
2174
|
+
<%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
1748
2175
|
:.......:.........:
|
1749
2176
|
: id : status :
|
1750
2177
|
:.......:.........:
|
@@ -1753,20 +2180,20 @@ $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exa
|
|
1753
2180
|
:.......:.........:
|
1754
2181
|
```
|
1755
2182
|
|
1756
|
-
|
2183
|
+
### Example: Find with filter and delete
|
1757
2184
|
|
1758
2185
|
```
|
1759
|
-
|
2186
|
+
<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
1760
2187
|
:.......:........................:
|
1761
2188
|
: id : email :
|
1762
2189
|
:.......:........................:
|
1763
2190
|
: 98398 : dummyuser1@example.com :
|
1764
2191
|
: 98399 : dummyuser2@example.com :
|
1765
2192
|
:.......:........................:
|
1766
|
-
|
1767
|
-
|
1768
|
-
|
1769
|
-
|
2193
|
+
thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
2194
|
+
echo $thelist
|
2195
|
+
["113501","354061"]
|
2196
|
+
<%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
1770
2197
|
:.......:.........:
|
1771
2198
|
: id : status :
|
1772
2199
|
:.......:.........:
|
@@ -1775,10 +2202,18 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:[$thelist] delete
|
|
1775
2202
|
:.......:.........:
|
1776
2203
|
```
|
1777
2204
|
|
1778
|
-
|
2205
|
+
### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
2206
|
+
|
2207
|
+
```
|
2208
|
+
<%=cmd%> aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
2209
|
+
```
|
2210
|
+
|
2211
|
+
To delete them use the same method as before
|
2212
|
+
|
2213
|
+
### Example: Display current user's workspaces
|
1779
2214
|
|
1780
2215
|
```
|
1781
|
-
|
2216
|
+
<%=cmd%> aoc user workspaces
|
1782
2217
|
:......:............................:
|
1783
2218
|
: id : name :
|
1784
2219
|
:......:............................:
|
@@ -1788,42 +2223,45 @@ $ <%=cmd%> aoc user workspaces
|
|
1788
2223
|
:......:............................:
|
1789
2224
|
```
|
1790
2225
|
|
1791
|
-
|
2226
|
+
### Example: Create a sub access key in a "node"
|
1792
2227
|
|
1793
2228
|
Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
1794
2229
|
|
1795
2230
|
```
|
1796
|
-
|
2231
|
+
<%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
|
1797
2232
|
```
|
1798
2233
|
|
1799
|
-
|
2234
|
+
### Example: Display transfer events (ops/transfer)
|
1800
2235
|
|
1801
2236
|
```
|
1802
|
-
|
2237
|
+
<%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
1803
2238
|
```
|
1804
2239
|
|
1805
|
-
|
1806
|
-
|
1807
|
-
|
1808
|
-
|
2240
|
+
Examples of query (TODO: cleanup):
|
2241
|
+
|
2242
|
+
```
|
2243
|
+
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
2244
|
+
|
2245
|
+
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
2246
|
+
|
1809
2247
|
# filter= 'id', 'short_summary', or 'summary'
|
1810
2248
|
# count=nnn
|
1811
2249
|
# tag=x.y.z%3Dvalue
|
1812
2250
|
# iteration_token=nnn
|
1813
2251
|
# after_time=2016-05-01T23:53:09Z
|
1814
2252
|
# active_only=true|false
|
2253
|
+
```
|
1815
2254
|
|
1816
|
-
|
1817
|
-
* Display node events (events)
|
2255
|
+
### Example: Display node events (events)
|
1818
2256
|
|
1819
2257
|
```
|
1820
|
-
|
2258
|
+
<%=cmd%> aoc admin res node --secret=_secret_ v3 events
|
1821
2259
|
```
|
1822
2260
|
|
1823
|
-
|
2261
|
+
### Example: Display members of a workspace
|
1824
2262
|
|
1825
2263
|
```
|
1826
|
-
|
2264
|
+
<%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
1827
2265
|
:.............:.........:..................................:
|
1828
2266
|
: member_type : manager : member.email :
|
1829
2267
|
:.............:.........:..................................:
|
@@ -1842,29 +2280,29 @@ other query parameters:
|
|
1842
2280
|
{"workspace_membership_through":true,"include_indirect":true}
|
1843
2281
|
```
|
1844
2282
|
|
1845
|
-
|
2283
|
+
### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
|
1846
2284
|
|
1847
|
-
a-
|
2285
|
+
a- Get id of first workspace
|
1848
2286
|
|
1849
2287
|
```
|
1850
2288
|
WS1='First Workspace'
|
1851
2289
|
WS1ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
1852
2290
|
```
|
1853
2291
|
|
1854
|
-
b-
|
2292
|
+
b- Get id of second workspace
|
1855
2293
|
|
1856
2294
|
```
|
1857
2295
|
WS2='Second Workspace'
|
1858
2296
|
WS2ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
|
1859
2297
|
```
|
1860
2298
|
|
1861
|
-
c-
|
2299
|
+
c- Extract membership information
|
1862
2300
|
|
1863
2301
|
```
|
1864
|
-
|
2302
|
+
<%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
1865
2303
|
```
|
1866
2304
|
|
1867
|
-
d-
|
2305
|
+
d- Convert to creation data for second workspace:
|
1868
2306
|
|
1869
2307
|
```
|
1870
2308
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
@@ -1876,16 +2314,16 @@ or, using jq:
|
|
1876
2314
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
1877
2315
|
```
|
1878
2316
|
|
1879
|
-
e-
|
2317
|
+
e- Add members to second workspace
|
1880
2318
|
|
1881
2319
|
```
|
1882
|
-
|
2320
|
+
<%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
1883
2321
|
```
|
1884
2322
|
|
1885
|
-
|
2323
|
+
### Example: Get users who did not log since a date
|
1886
2324
|
|
1887
2325
|
```
|
1888
|
-
|
2326
|
+
<%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
1889
2327
|
:...............................:
|
1890
2328
|
: email :
|
1891
2329
|
:...............................:
|
@@ -1894,20 +2332,20 @@ $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"per_page":100
|
|
1894
2332
|
:...............................:
|
1895
2333
|
```
|
1896
2334
|
|
1897
|
-
|
2335
|
+
### Example: List "Limited" users
|
1898
2336
|
|
1899
2337
|
```
|
1900
|
-
|
2338
|
+
<%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
1901
2339
|
```
|
1902
2340
|
|
1903
|
-
|
2341
|
+
### Example: Perform a multi Gbps transfer between two remote shared folders
|
1904
2342
|
|
1905
|
-
In this example, a user has access to a workspace where two shared folders are located on
|
2343
|
+
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
1906
2344
|
|
1907
2345
|
First, setup the environment (skip if already done)
|
1908
2346
|
|
1909
2347
|
```
|
1910
|
-
|
2348
|
+
<%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
|
1911
2349
|
Detected: Aspera on Cloud
|
1912
2350
|
Preparing preset: aoc_sedemo
|
1913
2351
|
Using existing key:
|
@@ -1926,7 +2364,7 @@ Setting config preset as default for aspera
|
|
1926
2364
|
saving config file
|
1927
2365
|
Done.
|
1928
2366
|
You can test with:
|
1929
|
-
|
2367
|
+
<%=cmd%> aoc user info show
|
1930
2368
|
```
|
1931
2369
|
|
1932
2370
|
This creates the option preset "aoc_<org name>" to allow seamless command line access and sets it as default for aspera on cloud.
|
@@ -1936,19 +2374,20 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
1936
2374
|
Then, transfer between those:
|
1937
2375
|
|
1938
2376
|
```
|
1939
|
-
|
2377
|
+
<%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
1940
2378
|
```
|
1941
2379
|
|
1942
|
-
|
2380
|
+
### Example: create registration key to register a node
|
2381
|
+
|
1943
2382
|
```
|
1944
|
-
|
2383
|
+
<%=cmd%> aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
1945
2384
|
jfqslfdjlfdjfhdjklqfhdkl
|
1946
2385
|
```
|
1947
2386
|
|
1948
|
-
|
2387
|
+
### Example: delete all registration keys
|
1949
2388
|
|
1950
2389
|
```
|
1951
|
-
|
2390
|
+
<%=cmd%> aoc admin res client list --fields=id --format=csv|<%=cmd%> aoc admin res client delete --bulk=yes --id=@lines:@stdin:
|
1952
2391
|
+-----+---------+
|
1953
2392
|
| id | status |
|
1954
2393
|
+-----+---------+
|
@@ -1959,143 +2398,215 @@ $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc
|
|
1959
2398
|
+-----+---------+
|
1960
2399
|
```
|
1961
2400
|
|
1962
|
-
|
2401
|
+
### Example: Create a node
|
1963
2402
|
|
1964
|
-
|
2403
|
+
AoC nodes as actually composed with two related entities:
|
2404
|
+
|
2405
|
+
* An access key created on the Transfer Server (HSTS/ATS)
|
2406
|
+
* a `node` resource in the AoC application.
|
2407
|
+
|
2408
|
+
The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
|
2409
|
+
Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
|
2410
|
+
|
2411
|
+
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
2412
|
+
|
2413
|
+
* create the access key on ATS
|
1965
2414
|
|
1966
2415
|
```
|
1967
|
-
|
2416
|
+
<%=cmd%> aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
1968
2417
|
```
|
1969
2418
|
|
1970
|
-
|
2419
|
+
Take a note of the randomly generated `id` and `secret`.
|
2420
|
+
|
2421
|
+
* Retrieve the ATS node address
|
1971
2422
|
|
1972
2423
|
```
|
1973
|
-
|
2424
|
+
<%=cmd%> aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
|
1974
2425
|
```
|
1975
2426
|
|
1976
|
-
*
|
2427
|
+
* Create the node entity
|
2428
|
+
|
2429
|
+
```
|
2430
|
+
<%=cmd%> aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
2431
|
+
```
|
2432
|
+
|
2433
|
+
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
2434
|
+
|
2435
|
+
### Example: List packages in a given shared inbox
|
2436
|
+
|
2437
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
2438
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2439
|
+
|
2440
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
1977
2441
|
|
1978
2442
|
```
|
1979
|
-
|
2443
|
+
shbxid=$(<%=cmd%> aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
2444
|
+
|
2445
|
+
<%=cmd%> aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
1980
2446
|
```
|
1981
2447
|
|
1982
|
-
##
|
2448
|
+
## Packages
|
2449
|
+
|
2450
|
+
The webmail-like application.
|
2451
|
+
|
2452
|
+
### Send a Package
|
1983
2453
|
|
1984
2454
|
Send a package:
|
1985
2455
|
|
1986
2456
|
```
|
1987
|
-
|
2457
|
+
<%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
|
1988
2458
|
```
|
1989
2459
|
|
1990
2460
|
Notes:
|
1991
2461
|
|
1992
|
-
*
|
1993
|
-
*
|
1994
|
-
*
|
1995
|
-
*
|
1996
|
-
*
|
2462
|
+
* The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
2463
|
+
* List allowed shared inbox destinations with: `<%=cmd%> aoc user shared_inboxes`
|
2464
|
+
* Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
2465
|
+
* Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
2466
|
+
* or just names: `"recipients":[{"The Dest"}]` . <%=cmd%> will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
2467
|
+
* If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
2468
|
+
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
1997
2469
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
1998
2470
|
|
1999
2471
|
Examples:
|
2000
2472
|
|
2473
|
+
* Send a package with one file to two users, using their email
|
2474
|
+
|
2475
|
+
```
|
2476
|
+
<%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
2477
|
+
```
|
2478
|
+
|
2479
|
+
* Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
|
2480
|
+
|
2481
|
+
```
|
2482
|
+
<%=cmd%> aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
|
2483
|
+
```
|
2484
|
+
|
2485
|
+
* Send a package with one file to a shared inbox (by name) with metadata
|
2486
|
+
|
2001
2487
|
```
|
2002
|
-
|
2003
|
-
$ <%=cmd%> aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
|
2004
|
-
$ <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2488
|
+
<%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2005
2489
|
```
|
2006
2490
|
|
2007
|
-
|
2491
|
+
### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
2008
2492
|
|
2009
2493
|
It is possible to automatically download new packages, like using Aspera Cargo:
|
2010
2494
|
|
2011
2495
|
```
|
2012
|
-
|
2496
|
+
<%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
2013
2497
|
```
|
2014
2498
|
|
2015
2499
|
* `--id=ALL` (case sensitive) will download all packages
|
2016
|
-
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
2017
|
-
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid
|
2500
|
+
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
2501
|
+
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
2018
2502
|
|
2019
|
-
Typically, one would
|
2503
|
+
Typically, one would execute this command on a regular basis, using the method of your choice:
|
2020
2504
|
|
2021
2505
|
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
2022
2506
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2023
2507
|
* etc...
|
2024
2508
|
|
2025
|
-
##
|
2026
|
-
|
2027
|
-
Download of files is straightforward with a specific syntax for the `aspera files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2028
|
-
|
2029
|
-
* if only one source is provided, it is downloaded
|
2030
|
-
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2031
|
-
|
2032
|
-
## Find Files
|
2033
|
-
|
2034
|
-
The command `aspera files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2509
|
+
## Files
|
2035
2510
|
|
2036
|
-
|
2511
|
+
Folder sharing app.
|
2037
2512
|
|
2038
|
-
|
2039
|
-
* not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
|
2513
|
+
### Download Files
|
2040
2514
|
|
2041
|
-
|
2515
|
+
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2042
2516
|
|
2043
|
-
*
|
2517
|
+
* if only one source is provided, it is downloaded
|
2518
|
+
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2044
2519
|
|
2045
|
-
|
2520
|
+
### Shared folders
|
2046
2521
|
|
2047
|
-
*
|
2522
|
+
* list shared folders in node
|
2048
2523
|
|
2049
2524
|
```
|
2050
|
-
|
2525
|
+
<%=cmd%> aoc admin res node --id=8669 shared_folders
|
2051
2526
|
```
|
2052
2527
|
|
2053
|
-
*
|
2528
|
+
* list shared folders in workspace
|
2054
2529
|
|
2055
2530
|
```
|
2056
|
-
|
2531
|
+
<%=cmd%> aoc admin res workspace --id=10818 shared_folders
|
2057
2532
|
```
|
2058
2533
|
|
2059
|
-
*
|
2534
|
+
* list members of shared folder
|
2060
2535
|
|
2061
2536
|
```
|
2062
|
-
|
2537
|
+
<%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
|
2063
2538
|
```
|
2064
2539
|
|
2065
|
-
|
2540
|
+
### Cross Organization transfers
|
2541
|
+
|
2542
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
2543
|
+
|
2544
|
+
Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
|
2545
|
+
|
2546
|
+
Procedure to send a file from org1 to org2:
|
2547
|
+
|
2548
|
+
* Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
2549
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
2550
|
+
* Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
|
2551
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
2552
|
+
* execute the following:
|
2066
2553
|
|
2067
2554
|
```
|
2068
|
-
|
2555
|
+
<%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
2069
2556
|
```
|
2070
2557
|
|
2071
|
-
|
2558
|
+
Explanation:
|
2072
2559
|
|
2073
|
-
|
2560
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
2561
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
2562
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
2563
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
2564
|
+
* `|` the standard output of the first command is fed into the second one
|
2565
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2566
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2567
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
2568
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2569
|
+
|
2570
|
+
### Find Files
|
2571
|
+
|
2572
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2573
|
+
|
2574
|
+
The expression can be of 3 formats:
|
2575
|
+
|
2576
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
2577
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2578
|
+
|
2579
|
+
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2580
|
+
|
2581
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2582
|
+
|
2583
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
2584
|
+
|
2585
|
+
* Find files more recent than 100 days
|
2074
2586
|
|
2075
2587
|
```
|
2076
|
-
|
2588
|
+
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
2077
2589
|
```
|
2078
2590
|
|
2079
|
-
|
2591
|
+
* Find files older than 1 year on a given node and store in file list
|
2080
2592
|
|
2081
2593
|
```
|
2082
|
-
|
2083
|
-
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2084
|
-
--notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
|
2594
|
+
<%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
2085
2595
|
```
|
2086
2596
|
|
2087
|
-
*
|
2088
|
-
* `query` filter (on API call)
|
2089
|
-
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
2090
|
-
|
2091
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
|
2597
|
+
* Delete the files, one by one
|
2092
2598
|
|
2093
|
-
|
2599
|
+
```
|
2600
|
+
cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
2601
|
+
```
|
2094
2602
|
|
2095
|
-
|
2603
|
+
* Delete the files in bulk
|
2096
2604
|
|
2605
|
+
```
|
2606
|
+
cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
2607
|
+
```
|
2097
2608
|
|
2098
|
-
# Plugin: Aspera Transfer Service
|
2609
|
+
# <a id="ats"></a>Plugin: Aspera Transfer Service
|
2099
2610
|
|
2100
2611
|
ATS is usable either :
|
2101
2612
|
|
@@ -2105,10 +2616,13 @@ ATS is usable either :
|
|
2105
2616
|
|
2106
2617
|
## IBM Cloud ATS : creation of api key
|
2107
2618
|
|
2619
|
+
This section is about using ATS with an IBM cloud subscription.
|
2620
|
+
If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
|
2621
|
+
|
2108
2622
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
2109
2623
|
|
2110
2624
|
```
|
2111
|
-
|
2625
|
+
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
2112
2626
|
OK
|
2113
2627
|
API key mykeyname was created
|
2114
2628
|
|
@@ -2127,48 +2641,47 @@ References:
|
|
2127
2641
|
* [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
2128
2642
|
* [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
2129
2643
|
|
2130
|
-
|
2131
2644
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
2132
2645
|
|
2133
2646
|
```
|
2134
|
-
|
2135
|
-
|
2136
|
-
|
2647
|
+
<%=cmd%> config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
|
2648
|
+
<%=cmd%> config preset set default ats my_ibm_ats
|
2649
|
+
<%=cmd%> ats api_key instances
|
2137
2650
|
+--------------------------------------+
|
2138
2651
|
| instance |
|
2139
2652
|
+--------------------------------------+
|
2140
2653
|
| aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
|
2141
2654
|
+--------------------------------------+
|
2142
|
-
|
2143
|
-
|
2655
|
+
<%=cmd%> config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
|
2656
|
+
<%=cmd%> ats api_key create
|
2144
2657
|
+--------+----------------------------------------------+
|
2145
2658
|
| key | value |
|
2146
2659
|
+--------+----------------------------------------------+
|
2147
2660
|
| id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
|
2148
2661
|
| secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
|
2149
2662
|
+--------+----------------------------------------------+
|
2150
|
-
|
2663
|
+
<%=cmd%> config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
2151
2664
|
```
|
2152
2665
|
|
2153
2666
|
## Examples
|
2154
2667
|
|
2155
|
-
Example: create access key on softlayer:
|
2668
|
+
Example: create access key on IBM Cloud (softlayer):
|
2156
2669
|
|
2157
2670
|
```
|
2158
|
-
|
2671
|
+
<%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
2159
2672
|
```
|
2160
2673
|
|
2161
2674
|
Example: create access key on AWS:
|
2162
2675
|
|
2163
2676
|
```
|
2164
|
-
|
2677
|
+
<%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
|
2165
2678
|
|
2166
2679
|
```
|
2167
2680
|
|
2168
2681
|
Example: create access key on Azure SAS:
|
2169
2682
|
|
2170
2683
|
```
|
2171
|
-
|
2684
|
+
<%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
2172
2685
|
|
2173
2686
|
```
|
2174
2687
|
|
@@ -2177,7 +2690,7 @@ $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{
|
|
2177
2690
|
Example: create access key on Azure:
|
2178
2691
|
|
2179
2692
|
```
|
2180
|
-
|
2693
|
+
<%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
2181
2694
|
|
2182
2695
|
```
|
2183
2696
|
|
@@ -2187,85 +2700,114 @@ delete all my access keys:
|
|
2187
2700
|
for k in $(<%=cmd%> ats access_key list --field=id --format=csv);do <%=cmd%> ats access_key id $k delete;done
|
2188
2701
|
```
|
2189
2702
|
|
2703
|
+
The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
|
2704
|
+
|
2190
2705
|
# Plugin: IBM Aspera High Speed Transfer Server (transfer)
|
2191
2706
|
|
2192
|
-
This plugin
|
2707
|
+
This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
|
2708
|
+
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
2709
|
+
Modern mode is to use the node API and transfer tokens.
|
2193
2710
|
|
2194
2711
|
## Authentication
|
2195
2712
|
|
2196
2713
|
Both password and SSH keys auth are supported.
|
2197
2714
|
|
2198
|
-
|
2715
|
+
If username is not provided, the default transfer user `xfer` is used.
|
2716
|
+
|
2717
|
+
If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
|
2718
|
+
|
2719
|
+
```
|
2720
|
+
<%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
2721
|
+
```
|
2722
|
+
|
2723
|
+
Multiple SSH key paths can be provided.
|
2724
|
+
The value of the parameter `ssh_keys` can be a single value or an array.
|
2725
|
+
Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
|
2199
2726
|
|
2200
2727
|
Examples:
|
2201
2728
|
|
2202
2729
|
```
|
2203
|
-
|
2204
|
-
|
2205
|
-
|
2730
|
+
<%=cmd%> server --ssh-keys=~/.ssh/id_rsa
|
2731
|
+
<%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
|
2732
|
+
<%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
2206
2733
|
```
|
2207
2734
|
|
2208
|
-
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
2735
|
+
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
2736
|
+
By default the ssh library expect that an ssh-agent is running.
|
2209
2737
|
|
2210
|
-
|
2738
|
+
On Linux, if you get an error message such as:
|
2211
2739
|
|
2212
2740
|
```
|
2213
|
-
[Linux]
|
2214
2741
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
|
2215
2742
|
```
|
2216
2743
|
|
2217
|
-
or
|
2744
|
+
or on Windows:
|
2218
2745
|
|
2219
2746
|
```
|
2220
|
-
[Windows]
|
2221
2747
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
2222
2748
|
```
|
2223
2749
|
|
2224
|
-
This means that you
|
2750
|
+
This means that you don't have such an ssh agent running, then:
|
2225
2751
|
|
2226
2752
|
* check env var: `SSH_AGENT_SOCK`
|
2227
|
-
* check if the key is protected with a passphrase
|
2753
|
+
* check if the ssh key is protected with a passphrase
|
2228
2754
|
* [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
2229
|
-
* To
|
2755
|
+
* To disable use of `ssh-agent`, use the option `ssh_option` like this:
|
2230
2756
|
|
2231
2757
|
```
|
2232
|
-
|
2758
|
+
<%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
|
2233
2759
|
```
|
2234
2760
|
|
2235
|
-
This can also be set as default using a preset
|
2761
|
+
This can also be set as default using a global preset.
|
2236
2762
|
|
2237
2763
|
## Example
|
2238
2764
|
|
2239
|
-
One can test the
|
2765
|
+
One can test the `server` application using the well known demo server:
|
2240
2766
|
|
2241
2767
|
```
|
2242
|
-
|
2243
|
-
|
2244
|
-
|
2245
|
-
$ <%=cmd%> server download /aspera-test-dir-large/200MB
|
2768
|
+
<%=cmd%> config initdemo
|
2769
|
+
<%=cmd%> server browse /aspera-test-dir-large
|
2770
|
+
<%=cmd%> server download /aspera-test-dir-large/200MB
|
2246
2771
|
```
|
2247
2772
|
|
2248
|
-
|
2249
|
-
|
2773
|
+
`initdemo` creates a <%=prst%> `demoserver` and set it as default for plugin `server`.
|
2250
2774
|
|
2251
2775
|
# Plugin: IBM Aspera High Speed Transfer Server (node)
|
2252
2776
|
|
2253
2777
|
This plugin gives access to capabilities provided by HSTS node API.
|
2254
2778
|
|
2255
|
-
##
|
2779
|
+
## File Operations
|
2256
2780
|
|
2257
2781
|
It is possible to:
|
2258
2782
|
* browse
|
2259
2783
|
* transfer (upload / download)
|
2260
2784
|
* ...
|
2261
2785
|
|
2786
|
+
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
2787
|
+
|
2788
|
+
* `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
|
2789
|
+
* `basic` : transfer spec is created like this:
|
2790
|
+
|
2791
|
+
```
|
2792
|
+
{
|
2793
|
+
"remote_host": address of node url,
|
2794
|
+
"remote_user": "xfer",
|
2795
|
+
"ssh_port": 33001,
|
2796
|
+
"token": "Basic <base 64 encoded user/pass>",
|
2797
|
+
"direction": send/recv
|
2798
|
+
}
|
2799
|
+
```
|
2800
|
+
|
2801
|
+
* `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
2802
|
+
|
2262
2803
|
## Central
|
2263
2804
|
|
2264
|
-
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and
|
2805
|
+
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
|
2265
2806
|
|
2266
2807
|
Filtering can be applied:
|
2808
|
+
|
2267
2809
|
```
|
2268
|
-
|
2810
|
+
<%=cmd%> node central file list
|
2269
2811
|
```
|
2270
2812
|
|
2271
2813
|
by providing the `validator` option, offline transfer validation can be done.
|
@@ -2277,7 +2819,7 @@ It is possible to start a FASPStream session using the node API:
|
|
2277
2819
|
Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
|
2278
2820
|
|
2279
2821
|
```
|
2280
|
-
|
2822
|
+
<%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
|
2281
2823
|
```
|
2282
2824
|
|
2283
2825
|
## Watchfolder
|
@@ -2289,11 +2831,10 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
2289
2831
|
* Start watchd and watchfolderd services running as a system user having access to files
|
2290
2832
|
* configure a watchfolder to define automated transfers
|
2291
2833
|
|
2292
|
-
|
2293
2834
|
```
|
2294
|
-
|
2295
|
-
|
2296
|
-
|
2835
|
+
<%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
2836
|
+
<%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
2837
|
+
<%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
2297
2838
|
```
|
2298
2839
|
|
2299
2840
|
## Out of Transfer File Validation
|
@@ -2301,13 +2842,13 @@ $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watc
|
|
2301
2842
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
2302
2843
|
|
2303
2844
|
```
|
2304
|
-
|
2845
|
+
<%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
2305
2846
|
:..............:..............:............:......................................:
|
2306
2847
|
: session_uuid : file_id : status : path :
|
2307
2848
|
:..............:..............:............:......................................:
|
2308
2849
|
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
|
2309
2850
|
:..............:..............:............:......................................:
|
2310
|
-
|
2851
|
+
<%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
2311
2852
|
updated
|
2312
2853
|
```
|
2313
2854
|
|
@@ -2323,7 +2864,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
|
|
2323
2864
|
Then execute the following command:
|
2324
2865
|
|
2325
2866
|
```
|
2326
|
-
|
2867
|
+
<%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
|
2327
2868
|
```
|
2328
2869
|
|
2329
2870
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance
|
@@ -2332,18 +2873,22 @@ to download files.
|
|
2332
2873
|
## Create access key
|
2333
2874
|
|
2334
2875
|
```
|
2335
|
-
|
2876
|
+
<%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
2336
2877
|
```
|
2337
2878
|
|
2338
2879
|
# Plugin: IBM Aspera Faspex5
|
2339
2880
|
|
2340
2881
|
3 authentication methods are supported:
|
2341
2882
|
|
2342
|
-
* boot
|
2343
|
-
* web
|
2344
2883
|
* jwt
|
2884
|
+
* web
|
2885
|
+
* boot
|
2886
|
+
|
2887
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
2345
2888
|
|
2346
|
-
For
|
2889
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
2890
|
+
|
2891
|
+
For boot method: (will be removed in future)
|
2347
2892
|
|
2348
2893
|
* open a browser
|
2349
2894
|
* start developer mode
|
@@ -2353,45 +2898,77 @@ For boot method:
|
|
2353
2898
|
Use it as password and use `--auth=boot`.
|
2354
2899
|
|
2355
2900
|
```
|
2356
|
-
|
2901
|
+
<%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
2357
2902
|
```
|
2358
2903
|
|
2359
|
-
For web method, create an API client in Faspex, and use: --auth=web
|
2360
|
-
|
2361
|
-
For JWT, create an API client in Faspex with jwt supporot, and use: --auth=jwt
|
2362
|
-
as of beta£3 this does not allow regular users.
|
2363
|
-
|
2364
2904
|
Ready to use Faspex5 with CLI.
|
2365
2905
|
|
2366
|
-
Once the graphical registration form exist, ther bootstrap method can be removed.
|
2367
|
-
|
2368
2906
|
# Plugin: IBM Aspera Faspex (4.x)
|
2369
2907
|
|
2370
2908
|
Notes:
|
2371
2909
|
|
2372
|
-
*
|
2373
|
-
*
|
2910
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
2911
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
2912
|
+
|
2913
|
+
## Listing Packages
|
2914
|
+
|
2915
|
+
Command: `faspex package list`
|
2916
|
+
|
2917
|
+
### Option `box`
|
2918
|
+
|
2919
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
2920
|
+
|
2921
|
+
### Option `recipient`
|
2922
|
+
|
2923
|
+
A user can receive a package because the recipient is:
|
2924
|
+
|
2925
|
+
* the user himself (default)
|
2926
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
2927
|
+
|
2928
|
+
### Option `query`
|
2929
|
+
|
2930
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
2931
|
+
|
2932
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
2933
|
+
* `page` : (native) id of page in call (default=0)
|
2934
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
2935
|
+
* `max` : maximum number of items
|
2936
|
+
* `pmax` : maximum number of pages
|
2937
|
+
|
2938
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
2939
|
+
|
2940
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
2941
|
+
|
2942
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
2943
|
+
|
2944
|
+
### Example
|
2945
|
+
|
2946
|
+
```
|
2947
|
+
<%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
2948
|
+
```
|
2949
|
+
|
2950
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
2374
2951
|
|
2375
2952
|
## Receiving a Package
|
2376
2953
|
|
2377
|
-
The command is `package recv`, possible
|
2954
|
+
The command is `package recv`, possible methods are:
|
2378
2955
|
|
2379
2956
|
* provide a package id with option `id`
|
2380
2957
|
* provide a public link with option `link`
|
2381
2958
|
* provide a `faspe:` URI with option `link`
|
2382
2959
|
|
2383
2960
|
```
|
2384
|
-
|
2385
|
-
|
2961
|
+
<%=cmd%> faspex package recv --id=12345
|
2962
|
+
<%=cmd%> faspex package recv --link=faspe://...
|
2386
2963
|
```
|
2387
2964
|
|
2388
2965
|
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
2389
2966
|
|
2390
2967
|
```
|
2391
|
-
|
2968
|
+
<%=cmd%> faspex package list --recipient='*thedropboxname'
|
2392
2969
|
```
|
2393
2970
|
|
2394
|
-
|
2971
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
2395
2972
|
|
2396
2973
|
## Sending a Package
|
2397
2974
|
|
@@ -2400,7 +2977,7 @@ The command is `faspex package send`. Package information (title, note, metadata
|
|
2400
2977
|
Example:
|
2401
2978
|
|
2402
2979
|
```
|
2403
|
-
|
2980
|
+
<%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
2404
2981
|
```
|
2405
2982
|
|
2406
2983
|
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
@@ -2410,17 +2987,29 @@ Additional optional parameters in `delivery_info`:
|
|
2410
2987
|
* Package Note: : `"note":"note this and that"`
|
2411
2988
|
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
2412
2989
|
|
2413
|
-
##
|
2990
|
+
## Email notification on transfer
|
2991
|
+
|
2992
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
2993
|
+
|
2994
|
+
Example:
|
2995
|
+
|
2996
|
+
```
|
2997
|
+
<%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
2998
|
+
```
|
2999
|
+
|
3000
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
3001
|
+
|
3002
|
+
## Operation on dropboxes
|
2414
3003
|
|
2415
3004
|
Example:
|
2416
3005
|
|
2417
3006
|
```
|
2418
|
-
|
2419
|
-
|
2420
|
-
|
3007
|
+
<%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
3008
|
+
<%=cmd%> faspex v4 dropbox list
|
3009
|
+
<%=cmd%> faspex v4 dropbox delete --id=36
|
2421
3010
|
```
|
2422
3011
|
|
2423
|
-
##
|
3012
|
+
## Remote sources
|
2424
3013
|
|
2425
3014
|
Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
|
2426
3015
|
the node API is used, for this it is required to add a section ":storage" that links
|
@@ -2455,7 +3044,7 @@ It is possible to tell <%=tool%> to download newly received packages, much like
|
|
2455
3044
|
cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
|
2456
3045
|
|
2457
3046
|
```
|
2458
|
-
|
3047
|
+
<%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
|
2459
3048
|
```
|
2460
3049
|
|
2461
3050
|
# Plugin: IBM Aspera Shares
|
@@ -2465,42 +3054,51 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
2465
3054
|
In Shares2, users, groups listing are paged, to display sequential pages:
|
2466
3055
|
|
2467
3056
|
```
|
2468
|
-
|
3057
|
+
for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
|
2469
3058
|
```
|
2470
3059
|
|
2471
3060
|
# Plugin: IBM Cloud Object Storage
|
2472
3061
|
|
2473
3062
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
2474
|
-
It uses the same transfer service as Aspera on Cloud.
|
2475
|
-
|
3063
|
+
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
3064
|
+
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
3065
|
+
|
3066
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
2476
3067
|
|
2477
|
-
|
3068
|
+
## Using endpoint, apikey and Resource Instance ID (CRN)
|
3069
|
+
|
3070
|
+
If you have those parameters already, then following options shall be provided:
|
2478
3071
|
|
2479
3072
|
* `bucket` bucket name
|
2480
3073
|
* `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
|
2481
3074
|
* `apikey` API Key
|
2482
3075
|
* `crn` resource instance id
|
2483
3076
|
|
2484
|
-
|
3077
|
+
For example, let us create a default configuration:
|
2485
3078
|
|
2486
|
-
|
2487
|
-
|
2488
|
-
|
3079
|
+
```
|
3080
|
+
<%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
3081
|
+
<%=cmd%> conf id default set cos mycos
|
3082
|
+
```
|
2489
3083
|
|
2490
|
-
|
3084
|
+
Then, jump to the transfer example.
|
3085
|
+
|
3086
|
+
## Using service credential file
|
3087
|
+
|
3088
|
+
If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
2491
3089
|
|
2492
3090
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
2493
3091
|
|
2494
3092
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
2495
3093
|
|
2496
|
-
or using the CLI:
|
3094
|
+
or using the IBM Cloud CLI:
|
2497
3095
|
|
2498
3096
|
```
|
2499
|
-
|
2500
|
-
|
3097
|
+
ibmcloud resource service-keys
|
3098
|
+
ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
2501
3099
|
```
|
2502
3100
|
|
2503
|
-
(if you
|
3101
|
+
(if you don't have `jq` installed, extract the structure as follows)
|
2504
3102
|
|
2505
3103
|
It consists in the following structure:
|
2506
3104
|
|
@@ -2524,33 +3122,38 @@ The field `resource_instance_id` is for option `crn`
|
|
2524
3122
|
|
2525
3123
|
The field `apikey` is for option `apikey`
|
2526
3124
|
|
2527
|
-
|
3125
|
+
(If needed: endpoints for regions can be found by querying the `endpoints` URL.)
|
2528
3126
|
|
2529
|
-
|
3127
|
+
The required options for this method are:
|
2530
3128
|
|
2531
|
-
|
2532
|
-
|
2533
|
-
|
2534
|
-
```
|
3129
|
+
* `bucket` bucket name
|
3130
|
+
* `region` bucket region, e.g. eu-de
|
3131
|
+
* `service_credentials` see below
|
2535
3132
|
|
2536
|
-
|
3133
|
+
For example, let us create a default configuration:
|
2537
3134
|
|
2538
3135
|
```
|
2539
|
-
|
2540
|
-
|
3136
|
+
<%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
3137
|
+
<%=cmd%> conf id default set cos mycos
|
2541
3138
|
```
|
2542
3139
|
|
2543
|
-
|
3140
|
+
## Operations, transfers
|
3141
|
+
|
3142
|
+
Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
|
3143
|
+
|
3144
|
+
A subset of `node` plugin operations are supported, basically node API:
|
2544
3145
|
|
2545
3146
|
```
|
2546
|
-
|
2547
|
-
|
3147
|
+
<%=cmd%> cos node info
|
3148
|
+
<%=cmd%> cos node upload 'faux:///sample1G?1g'
|
2548
3149
|
```
|
2549
3150
|
|
3151
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
3152
|
+
|
2550
3153
|
# Plugin: IBM Aspera Sync
|
2551
3154
|
|
2552
|
-
A basic plugin to start an "async" using <%=tool%>.
|
2553
|
-
to start from ma configuration file, using <%=tool%> standard options.
|
3155
|
+
A basic plugin to start an "async" using <%=tool%>.
|
3156
|
+
The main advantage is the possibility to start from ma configuration file, using <%=tool%> standard options.
|
2554
3157
|
|
2555
3158
|
# Plugin: Preview
|
2556
3159
|
|
@@ -2571,8 +3174,10 @@ Specify the previews folder as shown in:
|
|
2571
3174
|
By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
|
2572
3175
|
|
2573
3176
|
```
|
2574
|
-
|
2575
|
-
|
3177
|
+
PATH=/opt/aspera/bin:$PATH
|
3178
|
+
|
3179
|
+
asconfigurator -x "server;preview_dir,previews"
|
3180
|
+
asnodeadmin --reload
|
2576
3181
|
```
|
2577
3182
|
|
2578
3183
|
Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
|
@@ -2585,16 +3190,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
|
|
2585
3190
|
To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
|
2586
3191
|
|
2587
3192
|
```
|
2588
|
-
|
3193
|
+
asuserdata -a | grep max_request_file_create_size_kb
|
3194
|
+
|
2589
3195
|
max_request_file_create_size_kb: "1024"
|
2590
|
-
|
3196
|
+
|
3197
|
+
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
2591
3198
|
```
|
2592
3199
|
|
2593
3200
|
If you use a value different than 16777216, then specify it using option `max_size`.
|
2594
3201
|
|
2595
3202
|
Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
|
2596
3203
|
|
2597
|
-
## <a
|
3204
|
+
## <a id="prev_ext"></a>External tools: Linux
|
2598
3205
|
|
2599
3206
|
The tool requires the following external tools available in the `PATH`:
|
2600
3207
|
|
@@ -2611,7 +3218,7 @@ Other OSes should work as well, but are note tested.
|
|
2611
3218
|
To check if all tools are found properly, execute:
|
2612
3219
|
|
2613
3220
|
```
|
2614
|
-
|
3221
|
+
<%=cmd%> preview check
|
2615
3222
|
```
|
2616
3223
|
|
2617
3224
|
### mimemagic
|
@@ -2619,22 +3226,22 @@ $ <%=cmd%> preview check
|
|
2619
3226
|
To benefit from extra mime type detection install gem mimemagic:
|
2620
3227
|
|
2621
3228
|
```
|
2622
|
-
|
3229
|
+
gem install mimemagic
|
2623
3230
|
```
|
2624
3231
|
|
2625
3232
|
or to install an earlier version if any problem:
|
2626
3233
|
|
2627
3234
|
```
|
2628
|
-
|
3235
|
+
gem install mimemagic -v '~> 0.3.0'
|
2629
3236
|
```
|
2630
3237
|
|
2631
3238
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
2632
3239
|
|
2633
3240
|
If not used, Mime type used for conversion is the one provided by the node API.
|
2634
3241
|
|
2635
|
-
If used, it the `preview` command will first
|
3242
|
+
If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
|
2636
3243
|
|
2637
|
-
### Image:
|
3244
|
+
### Image: ImageMagick and optipng
|
2638
3245
|
|
2639
3246
|
```
|
2640
3247
|
yum install -y ImageMagick optipng
|
@@ -2642,31 +3249,33 @@ yum install -y ImageMagick optipng
|
|
2642
3249
|
|
2643
3250
|
### Video: FFmpeg
|
2644
3251
|
|
3252
|
+
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
3253
|
+
|
2645
3254
|
```
|
2646
|
-
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt &&
|
3255
|
+
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
2647
3256
|
```
|
2648
3257
|
|
2649
3258
|
### Office: Unoconv and Libreoffice
|
2650
3259
|
|
2651
|
-
If you
|
3260
|
+
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
2652
3261
|
|
2653
3262
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
2654
3263
|
|
2655
|
-
*
|
3264
|
+
* CentOS 8
|
2656
3265
|
|
2657
3266
|
```
|
2658
|
-
|
3267
|
+
dnf install unoconv
|
2659
3268
|
```
|
2660
3269
|
|
2661
3270
|
* Amazon Linux
|
2662
3271
|
|
2663
3272
|
```
|
2664
|
-
|
2665
|
-
|
2666
|
-
|
2667
|
-
|
2668
|
-
|
2669
|
-
|
3273
|
+
amazon-linux-extras enable libreoffice
|
3274
|
+
yum clean metadata
|
3275
|
+
yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
3276
|
+
wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
3277
|
+
mv unoconv /usr/bin
|
3278
|
+
chmod a+x /usr/bin/unoconv
|
2670
3279
|
```
|
2671
3280
|
|
2672
3281
|
## Configuration
|
@@ -2678,9 +3287,11 @@ Like any <%=tool%> commands, parameters can be passed on command line or using a
|
|
2678
3287
|
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
2679
3288
|
|
2680
3289
|
```
|
2681
|
-
|
2682
|
-
|
2683
|
-
|
3290
|
+
su -s /bin/bash - xfer
|
3291
|
+
|
3292
|
+
<%=cmd%> config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
3293
|
+
|
3294
|
+
<%=cmd%> config preset set default preview previewconf
|
2684
3295
|
```
|
2685
3296
|
|
2686
3297
|
Here we assume that Office file generation is disabled, else remove this option.
|
@@ -2689,7 +3300,7 @@ Here we assume that Office file generation is disabled, else remove this option.
|
|
2689
3300
|
One can check if the access key is well configured using:
|
2690
3301
|
|
2691
3302
|
```
|
2692
|
-
|
3303
|
+
<%=cmd%> -Ppreviewconf node browse /
|
2693
3304
|
```
|
2694
3305
|
|
2695
3306
|
This shall list the contents of the storage root of the access key.
|
@@ -2705,8 +3316,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
|
|
2705
3316
|
Lets do a one shot test, using the configuration previously created:
|
2706
3317
|
|
2707
3318
|
```
|
2708
|
-
|
2709
|
-
|
3319
|
+
su -s /bin/bash - xfer
|
3320
|
+
|
3321
|
+
<%=cmd%> preview scan --overwrite=always
|
2710
3322
|
```
|
2711
3323
|
|
2712
3324
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
@@ -2720,13 +3332,13 @@ Adapt the scripts to your own preference.
|
|
2720
3332
|
|
2721
3333
|
We assume here that a configuration preset was created as shown previously.
|
2722
3334
|
|
2723
|
-
Lets first setup a script that will be used in the
|
3335
|
+
Lets first setup a script that will be used in the scheduler and sets up the environment.
|
2724
3336
|
|
2725
3337
|
Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
|
2726
3338
|
|
2727
3339
|
```
|
2728
|
-
#!/bin/bash
|
2729
|
-
# set a timeout protection, just in case
|
3340
|
+
#!/bin/bash
|
3341
|
+
# set a timeout protection, just in case
|
2730
3342
|
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
2731
3343
|
. /etc/profile.d/rvm.sh
|
2732
3344
|
rvm use 2.6 --quiet
|
@@ -2736,13 +3348,13 @@ exec timeout ${tmout} <%=cmd%> "${@}"
|
|
2736
3348
|
Here the cronjob is created for user `xfer`.
|
2737
3349
|
|
2738
3350
|
```
|
2739
|
-
|
3351
|
+
crontab<<EOF
|
2740
3352
|
0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
|
2741
3353
|
2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
|
2742
3354
|
EOF
|
2743
3355
|
```
|
2744
3356
|
|
2745
|
-
Note that the
|
3357
|
+
Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
2746
3358
|
|
2747
3359
|
## Candidate detection for creation or update (or deletion)
|
2748
3360
|
|
@@ -2767,12 +3379,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
|
|
2767
3379
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
2768
3380
|
|
2769
3381
|
```
|
2770
|
-
|
3382
|
+
<%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
|
2771
3383
|
```
|
2772
3384
|
|
2773
3385
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
2774
3386
|
|
2775
|
-
When scanning the option `value` has the same
|
3387
|
+
When scanning the option `value` has the same behavior as for the `node find` command.
|
2776
3388
|
|
2777
3389
|
For instance to filter out files beginning with `._` do:
|
2778
3390
|
|
@@ -2791,7 +3403,7 @@ Use option `skip_format` to skip generation of a format.
|
|
2791
3403
|
|
2792
3404
|
## Supported input Files types
|
2793
3405
|
|
2794
|
-
The preview generator supports
|
3406
|
+
The preview generator supports rendering of those file categories:
|
2795
3407
|
|
2796
3408
|
* image
|
2797
3409
|
* pdf
|
@@ -2811,7 +3423,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
|
|
2811
3423
|
|
2812
3424
|
## Access to original files and preview creation
|
2813
3425
|
|
2814
|
-
Standard open source tools are used to create
|
3426
|
+
Standard open source tools are used to create thumbnails and video previews.
|
2815
3427
|
Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
|
2816
3428
|
The tool provides 2 ways to read and write files with the option: `file_access`
|
2817
3429
|
|
@@ -2828,50 +3440,89 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
2828
3440
|
The `smtp` option is a hash table (extended value) with the following fields:
|
2829
3441
|
<table>
|
2830
3442
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
2831
|
-
<tr><td
|
2832
|
-
<tr><td
|
2833
|
-
<tr><td
|
2834
|
-
<tr><td
|
2835
|
-
<tr><td
|
2836
|
-
<tr><td
|
2837
|
-
<tr><td
|
2838
|
-
<tr><td
|
3443
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
3444
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
3445
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
3446
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
3447
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
3448
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
3449
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
3450
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
2839
3451
|
</table>
|
2840
3452
|
|
2841
3453
|
## Example of configuration:
|
2842
3454
|
|
2843
3455
|
```
|
2844
|
-
|
2845
|
-
|
2846
|
-
|
3456
|
+
<%=cmd%> config preset set smtp_google server smtp.google.com
|
3457
|
+
<%=cmd%> config preset set smtp_google username john@gmail.com
|
3458
|
+
<%=cmd%> config preset set smtp_google password P@ssw0rd
|
2847
3459
|
```
|
2848
3460
|
|
2849
3461
|
or
|
2850
3462
|
|
2851
3463
|
```
|
2852
|
-
|
3464
|
+
<%=cmd%> config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
|
2853
3465
|
```
|
2854
3466
|
|
2855
3467
|
or
|
2856
3468
|
|
2857
3469
|
```
|
2858
|
-
|
3470
|
+
<%=cmd%> config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
|
2859
3471
|
```
|
2860
3472
|
|
2861
|
-
Set this
|
3473
|
+
Set this configuration as global default, for instance:
|
2862
3474
|
|
2863
3475
|
```
|
2864
|
-
|
2865
|
-
|
3476
|
+
<%=cmd%> config preset set cli_default smtp @val:@preset:smtp_google
|
3477
|
+
<%=cmd%> config preset set default config cli_default
|
2866
3478
|
```
|
2867
3479
|
|
3480
|
+
## Email templates
|
3481
|
+
|
3482
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
3483
|
+
|
3484
|
+
The template is the full SMTP message, including headers.
|
3485
|
+
|
3486
|
+
The following variables are defined by default:
|
3487
|
+
|
3488
|
+
* from_name
|
3489
|
+
* from_email
|
3490
|
+
* to
|
3491
|
+
|
3492
|
+
Other variables are defined depending on context.
|
3493
|
+
|
2868
3494
|
## Test
|
2869
3495
|
|
2870
3496
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
2871
3497
|
|
2872
3498
|
```
|
2873
|
-
|
2874
|
-
|
3499
|
+
<%=cmd%> config --smtp=@preset:smtp_google smtp
|
3500
|
+
<%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
3501
|
+
```
|
3502
|
+
|
3503
|
+
## Notifications for transfer status
|
3504
|
+
|
3505
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
3506
|
+
|
3507
|
+
To activate, use option `notif_to`.
|
3508
|
+
|
3509
|
+
A default e-mail template is used, but it can be overridden with option `notif_template`.
|
3510
|
+
|
3511
|
+
The environment provided contains the following additional variables:
|
3512
|
+
|
3513
|
+
* subject
|
3514
|
+
* body
|
3515
|
+
* global_transfer_status
|
3516
|
+
* ts
|
3517
|
+
|
3518
|
+
Example of template:
|
3519
|
+
|
3520
|
+
```
|
3521
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
3522
|
+
To: <<%='<'%>%=to%>>
|
3523
|
+
Subject: <%='<'%>%=subject%>
|
3524
|
+
|
3525
|
+
Transfer is: <%='<'%>%=global_transfer_status%>
|
2875
3526
|
```
|
2876
3527
|
|
2877
3528
|
# Tool: `asession`
|
@@ -2879,7 +3530,7 @@ $ <%=cmd%> config --smtp=@preset:smtp_google email sample.dest@example.com
|
|
2879
3530
|
This gem comes with a second executable tool providing a simplified standardized interface
|
2880
3531
|
to start a FASP session: `asession`.
|
2881
3532
|
|
2882
|
-
It aims at simplifying the startup of a FASP session from a programmatic stand point as
|
3533
|
+
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
|
2883
3534
|
|
2884
3535
|
* common to Aspera Node API (HTTP POST /ops/transfer)
|
2885
3536
|
* common to Aspera Connect API (browser javascript startTransfer)
|
@@ -2891,7 +3542,7 @@ This makes it easy to integrate with any language provided that one can spawn a
|
|
2891
3542
|
|
2892
3543
|
The tool expect one single argument: a [_transfer-spec_](#transferspec).
|
2893
3544
|
|
2894
|
-
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON
|
3545
|
+
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
|
2895
3546
|
|
2896
3547
|
Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
|
2897
3548
|
|
@@ -2902,7 +3553,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
2902
3553
|
* `EX_loglevel` to change log level of the tool
|
2903
3554
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
2904
3555
|
|
2905
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the
|
3556
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
2906
3557
|
|
2907
3558
|
## Comparison of interfaces
|
2908
3559
|
|
@@ -2929,7 +3580,7 @@ echo "${MY_TSPEC}"|asession
|
|
2929
3580
|
This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
|
2930
3581
|
|
2931
3582
|
```
|
2932
|
-
|
3583
|
+
asession
|
2933
3584
|
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
2934
3585
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
2935
3586
|
{"type":"DONE"}
|
@@ -2944,8 +3595,8 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
|
|
2944
3595
|
## Help
|
2945
3596
|
|
2946
3597
|
```
|
2947
|
-
|
2948
|
-
<%=
|
3598
|
+
asession -h
|
3599
|
+
<%=File.read(ENV["INCL_ASESSION"])%>
|
2949
3600
|
```
|
2950
3601
|
|
2951
3602
|
# Hot folder
|
@@ -2982,8 +3633,8 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
2982
3633
|
Note that:
|
2983
3634
|
|
2984
3635
|
* <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
2985
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
2986
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the
|
3636
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3637
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
|
2987
3638
|
|
2988
3639
|
### server side and configuration
|
2989
3640
|
|
@@ -2998,18 +3649,18 @@ Once <%=tool%> parameters are defined, run the command using the OS native sched
|
|
2998
3649
|
## Example
|
2999
3650
|
|
3000
3651
|
```
|
3001
|
-
|
3652
|
+
<%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
|
3002
3653
|
|
3003
3654
|
```
|
3004
3655
|
|
3005
|
-
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they
|
3656
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3006
3657
|
|
3007
|
-
#
|
3658
|
+
# Health check and Nagios
|
3008
3659
|
|
3009
|
-
|
3660
|
+
Most plugin provide a `health` command that will check the health status of the application. Example:
|
3010
3661
|
|
3011
3662
|
```
|
3012
|
-
|
3663
|
+
<%=cmd%> console health
|
3013
3664
|
+--------+-------------+------------+
|
3014
3665
|
| status | component | message |
|
3015
3666
|
+--------+-------------+------------+
|
@@ -3026,13 +3677,13 @@ Typically, the health check uses the REST API of the application with the follow
|
|
3026
3677
|
<%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
3027
3678
|
|
3028
3679
|
```
|
3029
|
-
|
3680
|
+
<%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
|
3030
3681
|
OK - [transfer:ok]
|
3031
|
-
|
3682
|
+
<%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
3032
3683
|
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
3033
3684
|
```
|
3034
3685
|
|
3035
|
-
# Module: `Aspera`
|
3686
|
+
# Ruby Module: `Aspera`
|
3036
3687
|
|
3037
3688
|
Main components:
|
3038
3689
|
|
@@ -3043,8 +3694,8 @@ Main components:
|
|
3043
3694
|
A working example can be found in the gem, example:
|
3044
3695
|
|
3045
3696
|
```
|
3046
|
-
|
3047
|
-
|
3697
|
+
<%=cmd%> config gem_path
|
3698
|
+
cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
|
3048
3699
|
```
|
3049
3700
|
|
3050
3701
|
This sample code shows some example of use of the API as well as
|
@@ -3079,106 +3730,143 @@ So, it evolved into <%=tool%>:
|
|
3079
3730
|
|
3080
3731
|
# Changes (Release notes)
|
3081
3732
|
|
3082
|
-
* <%=
|
3083
|
-
|
3084
|
-
|
3085
|
-
|
3733
|
+
* <%=gemspec.version.to_s%>
|
3734
|
+
|
3735
|
+
* new: support transfer agent: [Transfer SDK](#agt_trsdk)
|
3736
|
+
* new: support [http socket options](#http_options)
|
3737
|
+
* new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
|
3738
|
+
* new: `config vault` supports encrypted passwords, also macos keychain
|
3739
|
+
* new: `config preset` command for consistency with id
|
3740
|
+
* new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
|
3741
|
+
* change: when using wss, use [ruby's CA certs](#certificates)
|
3742
|
+
* change: unexpected parameter makes exit code not zero
|
3743
|
+
* change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
|
3744
|
+
* change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
|
3745
|
+
* change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
|
3746
|
+
* fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
|
3747
|
+
|
3748
|
+
* 4.4.0
|
3749
|
+
|
3750
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
3751
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
3752
|
+
* new: `preset` option can specify name or hash value
|
3753
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
3754
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
3755
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
3756
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
3757
|
+
|
3758
|
+
* 4.3.0
|
3759
|
+
|
3760
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
|
3761
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
3762
|
+
|
3763
|
+
* 4.2.2
|
3764
|
+
|
3765
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
3766
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
3767
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
3768
|
+
* new: `conf ascp spec` displays supported transfer spec
|
3769
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
3770
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
3771
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
3772
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
|
3773
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
3774
|
+
|
3775
|
+
* 4.2.1
|
3776
|
+
|
3777
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
3778
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
3086
3779
|
|
3087
3780
|
* 4.2.0
|
3088
3781
|
|
3089
|
-
|
3090
|
-
|
3091
|
-
|
3092
|
-
|
3093
|
-
|
3094
|
-
|
3095
|
-
|
3096
|
-
|
3097
|
-
|
3098
|
-
|
3782
|
+
* new: command `aoc remind` to receive organization membership by email
|
3783
|
+
* new: in `preview` option `value` to filter out on file name
|
3784
|
+
* new: `initdemo` to initialize for demo server
|
3785
|
+
* new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3786
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
3787
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
3788
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3789
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3790
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3791
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
|
3099
3792
|
|
3100
3793
|
* 4.1.0
|
3101
3794
|
|
3102
|
-
|
3103
|
-
|
3104
|
-
|
3105
|
-
|
3106
|
-
|
3107
|
-
|
3108
|
-
|
3109
|
-
|
3110
|
-
|
3111
|
-
|
3112
|
-
|
3113
|
-
|
3114
|
-
* new: initial support for Faspex 5 Beta 1
|
3795
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
3796
|
+
* new: update documentation with regard to offline and docker installation
|
3797
|
+
* new: renamed command `nagios_check` to `health`
|
3798
|
+
* new: agent `http_gw` now supports upload
|
3799
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
3800
|
+
* new: check new gem version periodically
|
3801
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
3802
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
3803
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
3804
|
+
* new: AoC a password can be provided for a public link
|
3805
|
+
* new: `conf doc` take an optional parameter to go to a section
|
3806
|
+
* new: initial support for Faspex 5 Beta 1
|
3115
3807
|
|
3116
3808
|
* 4.0.0
|
3117
3809
|
|
3118
|
-
|
3119
|
-
|
3120
|
-
|
3121
|
-
|
3122
|
-
|
3123
|
-
|
3124
|
-
|
3125
|
-
|
3810
|
+
* now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
|
3811
|
+
* changed default tool name from `mlia` to `ascli`
|
3812
|
+
* changed `aspera` command to `aoc`
|
3813
|
+
* changed gem name from `asperalm` to `aspera-cli`
|
3814
|
+
* changed module name from `Asperalm` to `Aspera`
|
3815
|
+
* removed command `folder` in `preview`, merged to `scan`
|
3816
|
+
* persistency files go to sub folder instead of main folder
|
3817
|
+
* added possibility to install SDK: `config ascp install`
|
3126
3818
|
|
3127
3819
|
* 0.11.8
|
3128
3820
|
|
3129
|
-
|
3821
|
+
* Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
|
3130
3822
|
|
3131
3823
|
* 0.11.7
|
3132
3824
|
|
3133
|
-
|
3134
|
-
|
3135
|
-
|
3136
|
-
|
3137
|
-
|
3138
|
-
|
3825
|
+
* rework on rest call error handling
|
3826
|
+
* use option `display` with value `data` to remove out of extraneous information
|
3827
|
+
* fixed option `lock_port` not working
|
3828
|
+
* generate special icon if preview failed
|
3829
|
+
* possibility to choose transfer progress bar type with option `progress`
|
3830
|
+
* AoC package creation now output package id
|
3139
3831
|
|
3140
3832
|
* 0.11.6
|
3141
3833
|
|
3142
|
-
|
3143
|
-
|
3144
|
-
|
3145
|
-
|
3146
|
-
|
3147
|
-
|
3148
|
-
|
3834
|
+
* orchestrator : added more choice in auth type
|
3835
|
+
* preview: cleanup in generator (removed and renamed parameters)
|
3836
|
+
* preview: better documentation
|
3837
|
+
* preview: animated thumbnails for video (option: `video_png_conv=animated`)
|
3838
|
+
* preview: new event trigger: `trevents` (`events` seems broken)
|
3839
|
+
* preview: unique tmp folder to avoid clash of multiple instances
|
3840
|
+
* repo: added template for secrets used for testing
|
3149
3841
|
|
3150
3842
|
* 0.11.5
|
3151
3843
|
|
3152
|
-
|
3153
|
-
|
3154
|
-
|
3155
|
-
|
3156
|
-
|
3157
|
-
|
3844
|
+
* added option `default_ports` for AoC (see manual)
|
3845
|
+
* allow bulk delete in `aspera files` with option `bulk=yes`
|
3846
|
+
* fix getting connect versions
|
3847
|
+
* added section for Aix
|
3848
|
+
* support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
|
3849
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
|
3158
3850
|
|
3159
3851
|
* 0.11.4
|
3160
3852
|
|
3161
|
-
|
3853
|
+
* possibility to give shared inbox name when sending a package (else use id and type)
|
3162
3854
|
|
3163
3855
|
* 0.11.3
|
3164
3856
|
|
3165
|
-
|
3857
|
+
* minor fixes on multi-session: avoid exception on progress bar
|
3166
3858
|
|
3167
3859
|
* 0.11.2
|
3168
3860
|
|
3169
|
-
|
3861
|
+
* fixes on multi-session: progress bat and transfer spec param for "direct"
|
3170
3862
|
|
3171
3863
|
* 0.11.1
|
3172
3864
|
|
3173
|
-
|
3865
|
+
* enhanced short_link creation commands (see examples)
|
3174
3866
|
|
3175
3867
|
* 0.11
|
3176
3868
|
|
3177
|
-
|
3178
|
-
|
3179
|
-
```
|
3180
|
-
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
|
3181
|
-
```
|
3869
|
+
* add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
|
3182
3870
|
|
3183
3871
|
* 0.10.18
|
3184
3872
|
|
@@ -3187,7 +3875,7 @@ So, it evolved into <%=tool%>:
|
|
3187
3875
|
* 0.10.17
|
3188
3876
|
|
3189
3877
|
* fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
|
3190
|
-
* new modifier: `@list:<
|
3878
|
+
* new modifier: `@list:<separator>val1<separator>...`
|
3191
3879
|
|
3192
3880
|
* 0.10.16
|
3193
3881
|
|
@@ -3209,7 +3897,7 @@ So, it evolved into <%=tool%>:
|
|
3209
3897
|
* 0.10.12
|
3210
3898
|
|
3211
3899
|
* added support for AoC node registration keys
|
3212
|
-
* replaced option : `local_resume` with `transfer_info` for agent `direct`
|
3900
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
|
3213
3901
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
3214
3902
|
* `@incps` : new extended value modifier
|
3215
3903
|
* ATS: no more provides access keys secrets: now user must provide it
|
@@ -3241,7 +3929,7 @@ So, it evolved into <%=tool%>:
|
|
3241
3929
|
|
3242
3930
|
* 0.10.6
|
3243
3931
|
|
3244
|
-
* FaspManager: transfer spec `authentication` no more needed for local
|
3932
|
+
* FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
3245
3933
|
* gem version requirements made more open
|
3246
3934
|
|
3247
3935
|
* 0.10.5
|
@@ -3259,9 +3947,9 @@ So, it evolved into <%=tool%>:
|
|
3259
3947
|
|
3260
3948
|
* 0.10.2
|
3261
3949
|
|
3262
|
-
|
3263
|
-
|
3264
|
-
|
3950
|
+
* updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
|
3951
|
+
* added doc for "cargo" like actions
|
3952
|
+
* added doc for multi-session
|
3265
3953
|
|
3266
3954
|
* 0.10.1
|
3267
3955
|
|
@@ -3289,22 +3977,22 @@ So, it evolved into <%=tool%>:
|
|
3289
3977
|
|
3290
3978
|
* 0.9.33
|
3291
3979
|
|
3292
|
-
|
3293
|
-
|
3294
|
-
|
3295
|
-
|
3980
|
+
* new command to display basic token of node
|
3981
|
+
* new command to display bearer token of node in AoC
|
3982
|
+
* the --fields= option, support +_fieldname_ to add a field to default fields
|
3983
|
+
* many small changes
|
3296
3984
|
|
3297
3985
|
* 0.9.32
|
3298
3986
|
|
3299
|
-
|
3300
|
-
|
3301
|
-
|
3987
|
+
* all Faspex public links are now supported
|
3988
|
+
* removed faspex operation recv_publink
|
3989
|
+
* replaced with option `link` (consistent with AoC)
|
3302
3990
|
|
3303
3991
|
* 0.9.31
|
3304
3992
|
|
3305
|
-
|
3306
|
-
|
3307
|
-
|
3993
|
+
* added more support for public link: receive and send package, to user or dropbox and files view.
|
3994
|
+
* delete expired file lists
|
3995
|
+
* changed text table gem from text-table to terminal-table because it supports multiline values
|
3308
3996
|
|
3309
3997
|
* 0.9.27
|
3310
3998
|
|
@@ -3324,149 +4012,149 @@ So, it evolved into <%=tool%>:
|
|
3324
4012
|
|
3325
4013
|
* 0.9.24
|
3326
4014
|
|
3327
|
-
|
3328
|
-
|
4015
|
+
* fix bug where AoC node to node transfer did not work
|
4016
|
+
* fix bug on error if ED25519 private key is defined in .ssh
|
3329
4017
|
|
3330
4018
|
* 0.9.23
|
3331
4019
|
|
3332
|
-
|
3333
|
-
|
4020
|
+
* defined REST error handlers, more error conditions detected
|
4021
|
+
* commands to select specific ascp location
|
3334
4022
|
|
3335
4023
|
* 0.9.21
|
3336
4024
|
|
3337
|
-
|
3338
|
-
|
4025
|
+
* supports simplified wizard using global client
|
4026
|
+
* only ascp binary is required, other SDK (keys) files are now generated
|
3339
4027
|
|
3340
4028
|
* 0.9.20
|
3341
4029
|
|
3342
|
-
|
3343
|
-
|
3344
|
-
|
4030
|
+
* improved wizard (prepare for AoC global client id)
|
4031
|
+
* preview generator: addedoption : --skip-format=<png,mp4>
|
4032
|
+
* removed outdated pictures from this doc
|
3345
4033
|
|
3346
4034
|
* 0.9.19
|
3347
4035
|
|
3348
|
-
|
4036
|
+
* added command aspera bearer --scope=xx
|
3349
4037
|
|
3350
4038
|
* 0.9.18
|
3351
4039
|
|
3352
|
-
|
4040
|
+
* enhanced aspera admin events to support query
|
3353
4041
|
|
3354
4042
|
* 0.9.16
|
3355
4043
|
|
3356
|
-
|
3357
|
-
|
4044
|
+
* AoC transfers are now reported in activity app
|
4045
|
+
* new interface for Rest class authentication (keep backward compatibility)
|
3358
4046
|
|
3359
4047
|
* 0.9.15
|
3360
4048
|
|
3361
|
-
|
3362
|
-
|
4049
|
+
* new feature: "find" command in aspera files
|
4050
|
+
* sample code for transfer API
|
3363
4051
|
|
3364
4052
|
* 0.9.12
|
3365
4053
|
|
3366
|
-
|
3367
|
-
|
4054
|
+
* add nagios commands
|
4055
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
3368
4056
|
|
3369
4057
|
* 0.9.11
|
3370
4058
|
|
3371
|
-
|
3372
|
-
|
4059
|
+
* Breaking change: @stdin is now @stdin:
|
4060
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
3373
4061
|
|
3374
4062
|
|
3375
4063
|
* 0.9.10
|
3376
4064
|
|
3377
|
-
|
3378
|
-
|
3379
|
-
|
3380
|
-
|
4065
|
+
* Breaking change: parameter transfer-node becomes more generic: transfer-info
|
4066
|
+
* Display SaaS storage usage with command: aspera admin res node --id=nn info
|
4067
|
+
* cleaner way of specifying source file list for transfers
|
4068
|
+
* Breaking change: replaced download_mode option with http_download action
|
3381
4069
|
|
3382
4070
|
* 0.9.9
|
3383
4071
|
|
3384
|
-
|
3385
|
-
|
4072
|
+
* Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
|
4073
|
+
* Now supports "cargo" for Aspera on Cloud (automatic package download)
|
3386
4074
|
|
3387
4075
|
* 0.9.8
|
3388
4076
|
|
3389
|
-
|
3390
|
-
|
4077
|
+
* Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
|
4078
|
+
* AoC: share to share transfer with command "transfer"
|
3391
4079
|
|
3392
4080
|
* 0.9.7
|
3393
4081
|
|
3394
|
-
|
3395
|
-
|
3396
|
-
|
3397
|
-
|
3398
|
-
|
4082
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
|
4083
|
+
* preview persistency goes to unique file by default
|
4084
|
+
* catch mxf extension in preview as video
|
4085
|
+
* Faspex: possibility to download all packages by specifying id=ALL
|
4086
|
+
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
3399
4087
|
|
3400
4088
|
* 0.9.6
|
3401
4089
|
|
3402
|
-
|
3403
|
-
|
4090
|
+
* Breaking change: `@param:`is now `@preset:` and is generic
|
4091
|
+
* AoC: added command to display current workspace information
|
3404
4092
|
|
3405
4093
|
* 0.9.5
|
3406
4094
|
|
3407
|
-
|
3408
|
-
|
4095
|
+
* new parameter: new_user_option used to choose between public_link and invite of external users.
|
4096
|
+
* fixed bug in wizard, and wizard uses now product detection
|
3409
4097
|
|
3410
4098
|
* 0.9.4
|
3411
4099
|
|
3412
|
-
|
3413
|
-
|
3414
|
-
|
4100
|
+
* Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
|
4101
|
+
* AoC Package send supports external users
|
4102
|
+
* new command to export AoC config to Aspera CLI config
|
3415
4103
|
|
3416
4104
|
* 0.9.3
|
3417
4105
|
|
3418
|
-
|
3419
|
-
|
3420
|
-
|
3421
|
-
|
3422
|
-
|
3423
|
-
|
4106
|
+
* REST error message show host and code
|
4107
|
+
* option for quiet display
|
4108
|
+
* modified transfer interface and allow token re-generation on error
|
4109
|
+
* async add admin command
|
4110
|
+
* async add db parameters
|
4111
|
+
* Breaking change: new option "sources" to specify files to transfer
|
3424
4112
|
|
3425
4113
|
* 0.9.2
|
3426
4114
|
|
3427
|
-
|
4115
|
+
* Breaking change: changed AoC package creation to match API, see AoC section
|
3428
4116
|
|
3429
4117
|
* 0.9.1
|
3430
4118
|
|
3431
|
-
|
4119
|
+
* Breaking change: changed faspex package creation to match API, see Faspex section
|
3432
4120
|
|
3433
4121
|
* 0.9
|
3434
4122
|
|
3435
|
-
|
3436
|
-
|
4123
|
+
* Renamed the CLI from aslmcli to <%=tool%>
|
4124
|
+
* Automatic rename and conversion of former config folder from aslmcli to <%=tool%>
|
3437
4125
|
|
3438
4126
|
* 0.7.6
|
3439
4127
|
|
3440
|
-
|
4128
|
+
* add "sync" plugin
|
3441
4129
|
|
3442
4130
|
* 0.7
|
3443
4131
|
|
3444
|
-
|
3445
|
-
|
3446
|
-
|
3447
|
-
|
3448
|
-
|
4132
|
+
* Breaking change: AoC package recv take option if for package instead of argument.
|
4133
|
+
* Breaking change: Rest class and Oauth class changed init parameters
|
4134
|
+
* AoC: receive package from public link
|
4135
|
+
* select by col value on output
|
4136
|
+
* added rename (AoC, node)
|
3449
4137
|
|
3450
4138
|
* 0.6.19
|
3451
4139
|
|
3452
4140
|
Breaking change:
|
3453
4141
|
|
3454
|
-
|
3455
|
-
|
3456
|
-
|
3457
|
-
|
3458
|
-
|
3459
|
-
|
3460
|
-
|
3461
|
-
|
4142
|
+
* ats server list provisioned → ats cluster list
|
4143
|
+
* ats server list clouds → ats cluster clouds
|
4144
|
+
* ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
|
4145
|
+
* ats server id xxx → ats cluster show --id=xxx
|
4146
|
+
* ats subscriptions → ats credential subscriptions
|
4147
|
+
* ats api_key repository list → ats credential cache list
|
4148
|
+
* ats api_key list → ats credential list
|
4149
|
+
* ats access_key id xxx → ats access_key --id=xxx
|
3462
4150
|
|
3463
4151
|
* 0.6.18
|
3464
4152
|
|
3465
|
-
|
4153
|
+
* some commands take now --id option instead of id command.
|
3466
4154
|
|
3467
4155
|
* 0.6.15
|
3468
4156
|
|
3469
|
-
|
4157
|
+
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
|
3470
4158
|
|
3471
4159
|
# BUGS, FEATURES, CONTRIBUTION
|
3472
4160
|
|
@@ -3474,16 +4162,16 @@ For issues or feature requests use the Github repository and issues.
|
|
3474
4162
|
|
3475
4163
|
You can also contribute to this open source project.
|
3476
4164
|
|
3477
|
-
One can also create one's own
|
4165
|
+
One can also [create one's own plugin](#createownplugin).
|
3478
4166
|
|
3479
4167
|
## Only one value for any option
|
3480
4168
|
|
3481
4169
|
Some commands and sub commands may ask for the same option name.
|
3482
|
-
Currently, since option definition is position
|
4170
|
+
Currently, since option definition is position independent (last one wins), it is not possible
|
3483
4171
|
to give an option to a command and the same option with different value to a sub command.
|
3484
4172
|
|
3485
|
-
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
|
3486
|
-
As a
|
4173
|
+
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
|
4174
|
+
As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
|
3487
4175
|
|
3488
4176
|
This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
|
3489
4177
|
|
@@ -3503,20 +4191,24 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
3503
4191
|
|
3504
4192
|
## Error "Remote host is not who we expected"
|
3505
4193
|
|
3506
|
-
`ascp`
|
4194
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
4195
|
+
|
4196
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
3507
4197
|
|
3508
4198
|
```
|
3509
4199
|
--ts=@json:'{"sshfp":null}'
|
3510
4200
|
```
|
3511
4201
|
|
3512
|
-
|
4202
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
4203
|
+
|
4204
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
3513
4205
|
|
3514
|
-
##
|
4206
|
+
## Miscellaneous
|
3515
4207
|
|
3516
4208
|
* remove rest and oauth classes and use ruby standard gems:
|
3517
4209
|
|
3518
|
-
|
3519
|
-
|
4210
|
+
* oauth
|
4211
|
+
* https://github.com/rest-client/rest-client
|
3520
4212
|
|
3521
4213
|
* use Thor or any standard Ruby CLI manager
|
3522
4214
|
|