aspera-cli 4.4.0 → 4.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +1042 -787
- data/bin/ascli +1 -1
- data/bin/asession +3 -5
- data/docs/Makefile +4 -7
- data/docs/README.erb.md +988 -740
- data/examples/faspex4.rb +4 -6
- data/examples/transfer.rb +2 -2
- data/lib/aspera/aoc.rb +139 -118
- data/lib/aspera/cli/listener/progress_multi.rb +5 -5
- data/lib/aspera/cli/main.rb +64 -34
- data/lib/aspera/cli/manager.rb +19 -20
- data/lib/aspera/cli/plugin.rb +9 -1
- data/lib/aspera/cli/plugins/aoc.rb +156 -143
- data/lib/aspera/cli/plugins/ats.rb +11 -10
- data/lib/aspera/cli/plugins/bss.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +236 -112
- data/lib/aspera/cli/plugins/faspex.rb +29 -7
- data/lib/aspera/cli/plugins/faspex5.rb +21 -8
- data/lib/aspera/cli/plugins/node.rb +21 -9
- data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
- data/lib/aspera/cli/plugins/preview.rb +2 -2
- data/lib/aspera/cli/plugins/server.rb +3 -3
- data/lib/aspera/cli/plugins/shares.rb +17 -0
- data/lib/aspera/cli/transfer_agent.rb +47 -85
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/environment.rb +4 -4
- data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
- data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
- data/lib/aspera/fasp/{local.rb → agent_direct.rb} +14 -17
- data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +4 -4
- data/lib/aspera/fasp/{node.rb → agent_node.rb} +25 -8
- data/lib/aspera/fasp/agent_trsdk.rb +106 -0
- data/lib/aspera/fasp/default.rb +17 -0
- data/lib/aspera/fasp/installation.rb +64 -48
- data/lib/aspera/fasp/parameters.rb +7 -3
- data/lib/aspera/faspex_gw.rb +6 -6
- data/lib/aspera/keychain/encrypted_hash.rb +120 -0
- data/lib/aspera/keychain/macos_security.rb +94 -0
- data/lib/aspera/log.rb +45 -32
- data/lib/aspera/node.rb +3 -6
- data/lib/aspera/rest.rb +65 -49
- metadata +68 -27
- data/lib/aspera/api_detector.rb +0 -60
- data/lib/aspera/secrets.rb +0 -20
data/docs/README.erb.md
CHANGED
@@ -16,7 +16,9 @@ Ruby Doc: [<%=gemspec.metadata['documentation_uri']%>](<%=gemspec.metadata['docu
|
|
16
16
|
|
17
17
|
Required Ruby version: <%=gemspec.required_ruby_version%>
|
18
18
|
|
19
|
-
|
19
|
+
[Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
20
|
+
|
21
|
+
# <a id="when_to_use"></a>When to use and when not to use
|
20
22
|
|
21
23
|
<%=tool%> is designed to be used as a command line tool to:
|
22
24
|
|
@@ -39,21 +41,18 @@ One might be tempted to use it as an integration element, e.g. by building a com
|
|
39
41
|
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
40
42
|
|
41
43
|
* Product APIs (REST) : e.g. AoC, Faspex, node
|
42
|
-
* Transfer SDK : with gRPC interface and
|
44
|
+
* Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
43
45
|
|
44
46
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
45
47
|
|
46
48
|
For scripting and ad'hoc command line operations, <%=tool%> is perfect.
|
47
49
|
|
48
|
-
# Notations
|
50
|
+
# <a id="parsing"></a>Notations, Shell and Command line parsing
|
49
51
|
|
50
|
-
In examples, command line operations
|
51
|
-
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
52
|
+
In examples, command line operations are shown using a shell such: `bash` or `zsh`.
|
52
53
|
|
53
54
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
54
55
|
|
55
|
-
# <a name="parsing"></a>Shell and Command line parsing
|
56
|
-
|
57
56
|
<%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
|
58
57
|
|
59
58
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
@@ -66,12 +65,15 @@ On Windows, `cmd.exe` is typically used. Windows process creation does not recei
|
|
66
65
|
In case of doubt of argument values after parsing test like this:
|
67
66
|
|
68
67
|
```
|
69
|
-
|
68
|
+
<%=cmd%> conf echo "Hello World" arg2 3
|
69
|
+
```
|
70
|
+
|
71
|
+
```
|
70
72
|
"Hello World"
|
71
73
|
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
72
74
|
```
|
73
75
|
|
74
|
-
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (
|
76
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (<%=tool%>), next command line arguments are shown in the error message.
|
75
77
|
|
76
78
|
# Quick Start
|
77
79
|
|
@@ -82,7 +84,10 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
|
|
82
84
|
Once the gem is installed, <%=tool%> shall be accessible:
|
83
85
|
|
84
86
|
```
|
85
|
-
|
87
|
+
<%=cmd%> --version
|
88
|
+
```
|
89
|
+
|
90
|
+
```
|
86
91
|
<%=gemspec.version.to_s%>
|
87
92
|
```
|
88
93
|
|
@@ -95,8 +100,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
95
100
|
To test with Aspera demo transfer server, setup the environment and then test:
|
96
101
|
|
97
102
|
```
|
98
|
-
|
99
|
-
|
103
|
+
<%=cmd%> config initdemo
|
104
|
+
```
|
105
|
+
|
106
|
+
```
|
107
|
+
<%=cmd%> server browse /
|
108
|
+
```
|
109
|
+
|
110
|
+
```
|
100
111
|
:............:...........:......:........:...........................:.......................:
|
101
112
|
: zmode : zuid : zgid : size : mtime : name :
|
102
113
|
:............:...........:......:........:...........................:.......................:
|
@@ -115,11 +126,26 @@ If you want to use <%=tool%> with another server, and in order to make further c
|
|
115
126
|
* download a file
|
116
127
|
|
117
128
|
```
|
118
|
-
|
129
|
+
<%=cmd%> config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
130
|
+
```
|
131
|
+
|
132
|
+
```
|
119
133
|
updated: myserver
|
120
|
-
|
134
|
+
```
|
135
|
+
|
136
|
+
```
|
137
|
+
<%=cmd%> config preset set default server myserver
|
138
|
+
```
|
139
|
+
|
140
|
+
```
|
121
141
|
updated: default→server to myserver
|
122
|
-
|
142
|
+
```
|
143
|
+
|
144
|
+
```
|
145
|
+
<%=cmd%> server browse /aspera-test-dir-large
|
146
|
+
```
|
147
|
+
|
148
|
+
```
|
123
149
|
:............:...........:......:..............:...........................:............................:
|
124
150
|
: zmode : zuid : zgid : size : mtime : name :
|
125
151
|
:............:...........:......:..............:...........................:............................:
|
@@ -136,7 +162,13 @@ $ <%=cmd%> server browse /aspera-test-dir-large
|
|
136
162
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
137
163
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
138
164
|
:............:...........:......:..............:...........................:............................:
|
139
|
-
|
165
|
+
```
|
166
|
+
|
167
|
+
```
|
168
|
+
<%=cmd%> server download /aspera-test-dir-large/200MB
|
169
|
+
```
|
170
|
+
|
171
|
+
```
|
140
172
|
Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
|
141
173
|
complete
|
142
174
|
```
|
@@ -147,9 +179,9 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
|
|
147
179
|
|
148
180
|
Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
149
181
|
|
150
|
-
# <a
|
182
|
+
# <a id="installation"></a>Installation
|
151
183
|
|
152
|
-
It is possible to install *either* directly on the host operating system (Linux, Windows,
|
184
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
|
153
185
|
|
154
186
|
The direct installation is recommended and consists in installing:
|
155
187
|
|
@@ -159,33 +191,36 @@ The direct installation is recommended and consists in installing:
|
|
159
191
|
|
160
192
|
The following sections provide information on the various installation methods.
|
161
193
|
|
162
|
-
An internet connection is required for the installation. If you
|
194
|
+
An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
163
195
|
|
164
196
|
## Docker container
|
165
197
|
|
166
198
|
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
167
199
|
|
168
|
-
This method installs a docker image that contains: Ruby,
|
200
|
+
This method installs a docker image that contains: Ruby, <%=tool%> and the FASP sdk.
|
169
201
|
|
170
202
|
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
171
203
|
|
172
204
|
Ensure that you have Docker installed.
|
173
205
|
|
174
206
|
```
|
175
|
-
|
207
|
+
docker --version
|
176
208
|
```
|
177
209
|
|
178
210
|
Download the wrapping script:
|
179
211
|
|
180
212
|
```
|
181
|
-
|
182
|
-
|
213
|
+
curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
214
|
+
```
|
215
|
+
|
216
|
+
```
|
217
|
+
chmod a+x <%=cmd%>
|
183
218
|
```
|
184
219
|
|
185
220
|
Install the container image:
|
186
221
|
|
187
222
|
```
|
188
|
-
|
223
|
+
./<%=cmd%> install
|
189
224
|
```
|
190
225
|
|
191
226
|
Start using it !
|
@@ -195,9 +230,9 @@ Note that the tool is run in the container, so transfers are also executed in th
|
|
195
230
|
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
|
196
231
|
|
197
232
|
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
198
|
-
To add local storage as a volume edit the script:
|
233
|
+
To add local storage as a volume edit the script: <%=tool%> and add a `--volume` stanza.
|
199
234
|
|
200
|
-
## <a
|
235
|
+
## <a id="ruby"></a>Ruby
|
201
236
|
|
202
237
|
Use this method to install on the native host.
|
203
238
|
|
@@ -209,7 +244,7 @@ Required Ruby version: <%=gemspec.required_ruby_version%>. Ruby version 3 is als
|
|
209
244
|
|
210
245
|
Refer to the following sections for a proposed method for specific operating systems.
|
211
246
|
|
212
|
-
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux,
|
247
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
213
248
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
214
249
|
If you have a simpler better way to install Ruby version <%=gemspec.required_ruby_version%> : use it !
|
215
250
|
|
@@ -222,31 +257,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
|
222
257
|
Install the 2 keys
|
223
258
|
|
224
259
|
```
|
225
|
-
|
260
|
+
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
226
261
|
```
|
227
262
|
|
228
263
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
229
264
|
|
230
265
|
```
|
231
|
-
|
266
|
+
\curl -sSL https://get.rvm.io | bash -s stable
|
232
267
|
```
|
233
268
|
|
234
|
-
If you keep the same terminal (
|
269
|
+
If you keep the same terminal (not needed if re-login):
|
235
270
|
|
236
271
|
```
|
237
|
-
|
272
|
+
source ~/.rvm/scripts/rvm
|
238
273
|
```
|
239
274
|
|
240
275
|
It is advised to get one of the pre-compiled ruby version, you can list with:
|
241
276
|
|
242
277
|
```
|
243
|
-
|
278
|
+
rvm list --remote
|
244
279
|
```
|
245
280
|
|
246
281
|
Install the chosen pre-compiled Ruby version:
|
247
282
|
|
248
283
|
```
|
249
|
-
|
284
|
+
rvm install 2.7.2 --binary
|
250
285
|
```
|
251
286
|
|
252
287
|
Ruby is now installed for the user, go on to Gem installation.
|
@@ -259,7 +294,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
|
|
259
294
|
One can install in another location with :
|
260
295
|
|
261
296
|
```
|
262
|
-
|
297
|
+
curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
263
298
|
```
|
264
299
|
|
265
300
|
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
@@ -267,8 +302,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
|
|
267
302
|
To activate ruby (and <%=cmd%>) later, source it:
|
268
303
|
|
269
304
|
```
|
270
|
-
|
271
|
-
|
305
|
+
source /etc/profile.d/rvm.sh.ok
|
306
|
+
```
|
307
|
+
|
308
|
+
```
|
309
|
+
rvm version
|
272
310
|
```
|
273
311
|
|
274
312
|
### Windows: Installer
|
@@ -281,16 +319,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
|
|
281
319
|
|
282
320
|
### macOS: pre-installed or `brew`
|
283
321
|
|
284
|
-
|
322
|
+
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
|
285
323
|
|
286
324
|
```
|
287
|
-
|
325
|
+
sudo gem install <%=gemspec.name%><%=geminstadd%>
|
288
326
|
```
|
289
327
|
|
290
328
|
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
291
329
|
|
292
330
|
```
|
293
|
-
|
331
|
+
brew install ruby
|
294
332
|
```
|
295
333
|
|
296
334
|
### Linux: package
|
@@ -300,90 +338,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
|
|
300
338
|
Example:
|
301
339
|
|
302
340
|
```
|
303
|
-
|
341
|
+
yum install -y ruby rubygems ruby-json
|
304
342
|
```
|
305
343
|
|
306
344
|
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
307
345
|
|
308
346
|
```
|
309
347
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
348
|
+
```
|
349
|
+
|
350
|
+
```
|
310
351
|
yum remove -y ruby ruby-libs
|
311
352
|
```
|
312
353
|
|
313
354
|
### Other Unixes: Aix, etc...
|
314
355
|
|
315
|
-
If your
|
356
|
+
If your Unix does not provide a pre-built ruby, you can get it using one of those
|
316
357
|
[methods](https://www.ruby-lang.org/en/documentation/installation/)
|
317
358
|
|
318
359
|
For instance to build from source, and install in `/opt/ruby` :
|
319
360
|
|
320
361
|
```
|
321
|
-
|
322
|
-
# gzip -d ruby-2.7.2.tar.gz
|
323
|
-
# tar xvf ruby-2.7.2.tar
|
324
|
-
# cd ruby-2.7.2
|
325
|
-
# ./configure --prefix=/opt/ruby
|
326
|
-
# make ruby.imp
|
327
|
-
# make
|
328
|
-
# make install
|
329
|
-
```
|
330
|
-
|
331
|
-
### <a name="offline_install"></a>Installation without internet access
|
332
|
-
|
333
|
-
Note that currently no pre-packaged version exist yet.
|
334
|
-
A method to build one provided here:
|
335
|
-
|
336
|
-
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
337
|
-
|
338
|
-
Then create an archive:
|
339
|
-
|
340
|
-
```
|
341
|
-
$ cd
|
342
|
-
$ tar zcvf rvm-<%=cmd%>.tgz .rvm
|
343
|
-
```
|
344
|
-
|
345
|
-
Get the Aspera SDK. Execute:
|
346
|
-
|
347
|
-
```
|
348
|
-
$ <%=cmd%> conf --show-config|grep sdk_url
|
349
|
-
```
|
362
|
+
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
|
350
363
|
|
351
|
-
|
364
|
+
gzip -d ruby-2.7.2.tar.gz
|
352
365
|
|
353
|
-
|
366
|
+
tar xvf ruby-2.7.2.tar
|
354
367
|
|
355
|
-
|
368
|
+
cd ruby-2.7.2
|
356
369
|
|
357
|
-
|
370
|
+
./configure --prefix=/opt/ruby
|
358
371
|
|
359
|
-
|
360
|
-
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
372
|
+
make ruby.imp
|
361
373
|
|
362
|
-
|
363
|
-
source path_to_rvm_root/scripts/rvm
|
364
|
-
rvm use 2.7.2
|
365
|
-
```
|
366
|
-
|
367
|
-
For the SDK, either install from archive:
|
374
|
+
make
|
368
375
|
|
369
|
-
|
370
|
-
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
376
|
+
make install
|
371
377
|
```
|
372
378
|
|
373
|
-
|
374
|
-
|
375
|
-
## <a name="the_gem"></a>`<%=gemspec.name%>` gem
|
379
|
+
## <a id="the_gem"></a>`<%=gemspec.name%>` gem
|
376
380
|
|
377
381
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
378
382
|
|
379
383
|
```
|
380
|
-
|
384
|
+
gem install <%=gemspec.name%><%=geminstadd%>
|
381
385
|
```
|
382
386
|
|
383
387
|
To upgrade to the latest version:
|
384
388
|
|
385
389
|
```
|
386
|
-
|
390
|
+
gem update <%=gemspec.name%>
|
387
391
|
```
|
388
392
|
|
389
393
|
<%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
@@ -391,12 +395,10 @@ To upgrade to the latest version:
|
|
391
395
|
To check manually:
|
392
396
|
|
393
397
|
```
|
394
|
-
|
398
|
+
<%=cmd%> conf check_update
|
395
399
|
```
|
396
400
|
|
397
|
-
|
398
|
-
|
399
|
-
## <a name="fasp_prot"></a>FASP Protocol
|
401
|
+
## <a id="fasp_prot"></a>FASP Protocol
|
400
402
|
|
401
403
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
402
404
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
@@ -404,17 +406,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
|
|
404
406
|
* ascp
|
405
407
|
* aspera-license (in same folder, or ../etc)
|
406
408
|
|
407
|
-
This can be installed either be installing an Aspera transfer
|
409
|
+
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
408
410
|
|
409
411
|
```
|
410
|
-
|
412
|
+
<%=cmd%> conf ascp install
|
411
413
|
```
|
412
414
|
|
413
|
-
If a local SDK installation is
|
415
|
+
If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
|
414
416
|
|
415
417
|
```
|
416
|
-
|
417
|
-
|
418
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
419
|
+
```
|
420
|
+
|
421
|
+
```
|
422
|
+
<%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
418
423
|
```
|
419
424
|
|
420
425
|
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
@@ -434,36 +439,55 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
434
439
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
435
440
|
|
436
441
|
Several methods are provided to start a transfer.
|
437
|
-
Use of a local client ([`direct`](#
|
442
|
+
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
438
443
|
|
439
|
-
## <a
|
444
|
+
## <a id="offline_install"></a>Installation in air gapped environment
|
440
445
|
|
441
|
-
|
446
|
+
Note that currently no pre-packaged version exist yet.
|
447
|
+
A method to build one is provided here:
|
448
|
+
|
449
|
+
The procedure:
|
442
450
|
|
443
451
|
* Follow the non-root installation procedure with RVM, including gem
|
444
|
-
*
|
452
|
+
* Archive (zip, tar) the main RVM folder (includes <%=cmd%>):
|
445
453
|
|
446
454
|
```
|
447
|
-
$
|
448
|
-
$ tar zcvf rvm_<%=cmd%>.tgz .rvm
|
455
|
+
cd $HOME && tar zcvf rvm-<%=cmd%>.tgz .rvm
|
449
456
|
```
|
450
457
|
|
451
|
-
*
|
458
|
+
* Get the Aspera SDK.
|
452
459
|
|
453
460
|
```
|
454
|
-
|
461
|
+
<%=cmd%> conf --show-config --fields=sdk_url
|
462
|
+
```
|
463
|
+
|
464
|
+
* Download the SDK archive from that URL.
|
465
|
+
|
466
|
+
```
|
467
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
468
|
+
```
|
469
|
+
|
470
|
+
* Transfer those 2 files to the target system
|
471
|
+
|
472
|
+
* On target system
|
473
|
+
|
474
|
+
```
|
475
|
+
cd $HOME
|
476
|
+
|
477
|
+
tar zxvf rvm-<%=cmd%>.tgz
|
478
|
+
|
479
|
+
source ~/.rvm/scripts/rvm
|
480
|
+
|
481
|
+
<%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
455
482
|
```
|
456
483
|
|
457
|
-
*
|
484
|
+
* Add those lines to shell init (`.profile`)
|
458
485
|
|
459
486
|
```
|
460
|
-
|
461
|
-
$ tar zxvf rvm_<%=cmd%>.tgz
|
462
|
-
$ source ~/.rvm/scripts/rvm
|
463
|
-
$ <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
|
487
|
+
source ~/.rvm/scripts/rvm
|
464
488
|
```
|
465
489
|
|
466
|
-
# <a
|
490
|
+
# <a id="cli"></a>Command Line Interface: <%=tool%>
|
467
491
|
|
468
492
|
The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
469
493
|
|
@@ -492,7 +516,7 @@ The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which intera
|
|
492
516
|
Basic usage is displayed by executing:
|
493
517
|
|
494
518
|
```
|
495
|
-
|
519
|
+
<%=cmd%> -h
|
496
520
|
```
|
497
521
|
|
498
522
|
Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
|
@@ -506,7 +530,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
|
|
506
530
|
There are two types of arguments: Commands and Options. Example :
|
507
531
|
|
508
532
|
```
|
509
|
-
|
533
|
+
<%=cmd%> command --option-name=VAL1 VAL2
|
510
534
|
```
|
511
535
|
|
512
536
|
* executes _command_: `command`
|
@@ -534,13 +558,16 @@ Exceptions:
|
|
534
558
|
* the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
535
559
|
|
536
560
|
```
|
537
|
-
|
561
|
+
<%=cmd%> config echo -- --sample
|
562
|
+
```
|
563
|
+
|
564
|
+
```
|
538
565
|
"--sample"
|
539
566
|
```
|
540
567
|
|
541
568
|
Note that `--sample` is taken as an argument, and not option.
|
542
569
|
|
543
|
-
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on
|
570
|
+
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
544
571
|
|
545
572
|
The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
|
546
573
|
|
@@ -561,7 +588,7 @@ Command line arguments that are not options are either commands or arguments. If
|
|
561
588
|
|
562
589
|
Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
|
563
590
|
|
564
|
-
The
|
591
|
+
The behavior can be controlled with:
|
565
592
|
|
566
593
|
* --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
567
594
|
* yes : missing mandatory parameters/options are asked to the user
|
@@ -578,8 +605,8 @@ The information displayed depends on the action.
|
|
578
605
|
|
579
606
|
Depending on action, the output will contain:
|
580
607
|
|
581
|
-
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is
|
582
|
-
* `object_list` : displayed as a 2 dimensional table: one line per item, one
|
608
|
+
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
609
|
+
* `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
583
610
|
* `value_list` : a table with one column.
|
584
611
|
* `empty` : nothing
|
585
612
|
* `status` : a message
|
@@ -604,12 +631,15 @@ The style of output can be set using the `format` parameter, supporting:
|
|
604
631
|
* `yaml` : YAML
|
605
632
|
* `csv` : Comma Separated Values
|
606
633
|
|
607
|
-
### <a
|
634
|
+
### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
608
635
|
|
609
636
|
Table output can be filtered using the `select` parameter. Example:
|
610
637
|
|
611
638
|
```
|
612
|
-
|
639
|
+
<%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
640
|
+
```
|
641
|
+
|
642
|
+
```
|
613
643
|
:...............................:..................................:...........:
|
614
644
|
: name : email : ats_admin :
|
615
645
|
:...............................:..................................:...........:
|
@@ -622,7 +652,7 @@ Note that `select` filters selected elements from the result of API calls, while
|
|
622
652
|
|
623
653
|
### Verbosity of output
|
624
654
|
|
625
|
-
|
655
|
+
Output messages are categorized in 3 types:
|
626
656
|
|
627
657
|
* `info` output contain additional information, such as number of elements in a table
|
628
658
|
* `data` output contain the actual output of the command (object, or list of objects)
|
@@ -636,7 +666,7 @@ The option `display` controls the level of output:
|
|
636
666
|
|
637
667
|
### Selection of output object properties
|
638
668
|
|
639
|
-
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific
|
669
|
+
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
640
670
|
|
641
671
|
* DEF : default display of columns (that's the default, when not set)
|
642
672
|
* ALL : all columns available
|
@@ -645,7 +675,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
645
675
|
* +a,b,c : add selected properties to the default selection.
|
646
676
|
* -a,b,c : remove selected properties from the default selection.
|
647
677
|
|
648
|
-
## <a
|
678
|
+
## <a id="extended"></a>Extended Value Syntax
|
649
679
|
|
650
680
|
Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
|
651
681
|
|
@@ -682,23 +712,32 @@ To display the result of an extended value, use the `config echo` command.
|
|
682
712
|
Example: read the content of the specified file, then, base64 decode, then unzip:
|
683
713
|
|
684
714
|
```
|
685
|
-
|
715
|
+
<%=cmd%> config echo @zlib:@base64:@file:myfile.dat
|
686
716
|
```
|
687
717
|
|
688
718
|
Example: create a value as a hash, with one key and the value is read from a file:
|
689
719
|
|
690
720
|
```
|
691
|
-
|
721
|
+
<%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
|
692
722
|
```
|
693
723
|
|
694
724
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
695
725
|
|
696
726
|
```
|
697
|
-
|
727
|
+
cat test.csv
|
728
|
+
```
|
729
|
+
|
730
|
+
```
|
698
731
|
name,email
|
699
732
|
lolo,laurent@example.com
|
700
733
|
toto,titi@tutu.tata
|
701
|
-
|
734
|
+
```
|
735
|
+
|
736
|
+
```
|
737
|
+
<%=cmd%> config echo @csvt:@file:test.csv
|
738
|
+
```
|
739
|
+
|
740
|
+
```
|
702
741
|
:......:.....................:
|
703
742
|
: name : email :
|
704
743
|
:......:.....................:
|
@@ -710,13 +749,16 @@ $ <%=cmd%> config echo @csvt:@file:test.csv
|
|
710
749
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
711
750
|
|
712
751
|
```
|
713
|
-
|
752
|
+
<%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
753
|
+
```
|
754
|
+
|
755
|
+
```
|
714
756
|
{"version"=>"0.9", "hello"=>true}
|
715
757
|
```
|
716
758
|
|
717
759
|
Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
|
718
760
|
|
719
|
-
## <a
|
761
|
+
## <a id="native"></a>Structured Value
|
720
762
|
|
721
763
|
Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
|
722
764
|
|
@@ -727,7 +769,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
|
|
727
769
|
|
728
770
|
It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
|
729
771
|
|
730
|
-
## <a
|
772
|
+
## <a id="conffolder"></a>Configuration and Persistency Folder
|
731
773
|
|
732
774
|
<%=tool%> configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/<%=cmd%>`.
|
733
775
|
|
@@ -737,49 +779,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
|
|
737
779
|
The main folder can be displayed using :
|
738
780
|
|
739
781
|
```
|
740
|
-
|
782
|
+
<%=cmd%> config folder
|
783
|
+
```
|
784
|
+
|
785
|
+
```
|
741
786
|
/Users/kenji/.aspera/<%=cmd%>
|
742
787
|
```
|
743
788
|
|
744
|
-
It can be
|
789
|
+
It can be overridden using the environment variable `<%=evp%>HOME`.
|
745
790
|
|
746
791
|
Example (Windows):
|
747
792
|
|
748
793
|
```
|
749
|
-
|
750
|
-
|
794
|
+
set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
|
795
|
+
<%=cmd%> config folder
|
751
796
|
C:\Users\Kenji\.aspera\<%=cmd%>
|
752
797
|
```
|
753
798
|
|
754
|
-
## <a
|
799
|
+
## <a id="configfile"></a>Configuration file
|
755
800
|
|
756
801
|
On the first execution of <%=tool%>, an empty configuration file is created in the configuration folder.
|
757
802
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
758
803
|
|
759
804
|
Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it using the `config` command.
|
760
805
|
|
761
|
-
All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the
|
806
|
+
All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuration file.
|
762
807
|
|
763
808
|
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
764
809
|
|
765
|
-
The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be
|
810
|
+
The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
|
766
811
|
|
767
812
|
The configuration file is simply a catalog of pre-defined lists of options, called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
768
813
|
|
769
|
-
### <a
|
814
|
+
### <a id="lprt"></a><%=prstt%>
|
770
815
|
|
771
816
|
A <%=prst%> is simply a collection of parameters and their associated values in a named section in the configuration file.
|
772
817
|
|
773
818
|
A named <%=prst%> can be modified directly using <%=tool%>, which will update the configuration file :
|
774
819
|
|
775
820
|
```
|
776
|
-
|
821
|
+
<%=cmd%> config preset set|delete|show|initialize|update <<%=opprst%>>
|
777
822
|
```
|
778
823
|
|
779
824
|
The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
|
780
825
|
|
781
826
|
```
|
782
|
-
|
827
|
+
<%=cmd%> config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
783
828
|
```
|
784
829
|
|
785
830
|
* This creates a <%=prst%> `demo_server` with all provided options.
|
@@ -787,39 +832,48 @@ $ <%=cmd%> config id demo_server update --url=ssh://demo.asperasoft.com:33001 --
|
|
787
832
|
The command `set` allows setting individual options in a <%=prst%>.
|
788
833
|
|
789
834
|
```
|
790
|
-
|
835
|
+
<%=cmd%> config preset set demo_server password _demo_pass_
|
791
836
|
```
|
792
837
|
|
793
838
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
794
839
|
|
795
840
|
```
|
796
|
-
|
841
|
+
<%=cmd%> config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
797
842
|
```
|
798
843
|
|
799
|
-
A
|
800
|
-
If necessary, the configuration file can be edited (or simply consulted) with:
|
844
|
+
A full terminal based overview of the configuration can be displayed using:
|
801
845
|
|
802
846
|
```
|
803
|
-
|
847
|
+
<%=cmd%> config preset over
|
804
848
|
```
|
805
849
|
|
806
|
-
A
|
850
|
+
A list of <%=prst%> can be displayed using:
|
807
851
|
|
808
852
|
```
|
809
|
-
|
853
|
+
<%=cmd%> config preset list
|
810
854
|
```
|
811
855
|
|
812
|
-
A
|
856
|
+
A good practice is to not manually edit the configuration file and use modification commands instead.
|
857
|
+
If necessary, the configuration file can opened in a text editor with:
|
813
858
|
|
814
859
|
```
|
815
|
-
|
860
|
+
<%=cmd%> config open
|
816
861
|
```
|
817
862
|
|
818
|
-
|
863
|
+
Older format for commands are still supported:
|
864
|
+
|
865
|
+
```
|
866
|
+
<%=cmd%> config id <name> set|delete|show|initialize|update
|
867
|
+
<%=cmd%> config over
|
868
|
+
<%=cmd%> config list
|
869
|
+
```
|
870
|
+
|
871
|
+
|
872
|
+
### <a id="lprtconf"></a>Special <%=prstt%>: config
|
819
873
|
|
820
874
|
This preset name is reserved and contains a single key: `version`. This is the version of <%=tool%> which created the file.
|
821
875
|
|
822
|
-
### <a
|
876
|
+
### <a id="lprtdef"></a>Special <%=prstt%>: default
|
823
877
|
|
824
878
|
This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
|
825
879
|
|
@@ -830,16 +884,16 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
830
884
|
Operations on this preset are done using regular `config` operations:
|
831
885
|
|
832
886
|
```
|
833
|
-
|
834
|
-
|
887
|
+
<%=cmd%> config preset set default _plugin_name_ _default_preset_for_plugin_
|
888
|
+
<%=cmd%> config preset get default _plugin_name_
|
835
889
|
"_default_preset_for_plugin_"
|
836
890
|
```
|
837
891
|
|
838
|
-
### <a
|
892
|
+
### <a id="lplugconf"></a>Special Plugin: config
|
839
893
|
|
840
894
|
Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
|
841
895
|
|
842
|
-
When <%=tool%> starts, it
|
896
|
+
When <%=tool%> starts, it looks for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
843
897
|
|
844
898
|
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
845
899
|
|
@@ -878,10 +932,10 @@ The user may create as many <%=prsts%> as needed. For instance, a particular <%=
|
|
878
932
|
|
879
933
|
Values in the configuration also follow the [Extended Value Syntax](#extended).
|
880
934
|
|
881
|
-
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config
|
935
|
+
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
|
882
936
|
|
883
937
|
```
|
884
|
-
|
938
|
+
<%=cmd%> config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
|
885
939
|
```
|
886
940
|
|
887
941
|
This creates the <%=prst%>:
|
@@ -917,26 +971,29 @@ is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration
|
|
917
971
|
The main plugin name is `config`, so it is possible to define a default <%=prst%> for the main plugin with:
|
918
972
|
|
919
973
|
```
|
920
|
-
|
921
|
-
|
974
|
+
<%=cmd%> config preset set cli_default interactive no
|
975
|
+
```
|
976
|
+
|
977
|
+
```
|
978
|
+
<%=cmd%> config preset set default config cli_default
|
922
979
|
```
|
923
980
|
|
924
981
|
A <%=prst%> value can be removed with `unset`:
|
925
982
|
|
926
983
|
```
|
927
|
-
|
984
|
+
<%=cmd%> config preset unset cli_default interactive
|
928
985
|
```
|
929
986
|
|
930
987
|
Example: Define options using command line:
|
931
988
|
|
932
989
|
```
|
933
|
-
|
990
|
+
<%=cmd%> -N --url=x --password=y --username=y node --show-config
|
934
991
|
```
|
935
992
|
|
936
993
|
Example: Define options using a hash:
|
937
994
|
|
938
995
|
```
|
939
|
-
|
996
|
+
<%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
940
997
|
```
|
941
998
|
|
942
999
|
### Examples
|
@@ -946,44 +1003,120 @@ only username/password and url are required (either on command line, or from con
|
|
946
1003
|
Those can usually be provided on the command line:
|
947
1004
|
|
948
1005
|
```
|
949
|
-
|
1006
|
+
<%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
|
950
1007
|
```
|
951
1008
|
|
952
1009
|
This can also be provisioned in a config file:
|
953
1010
|
|
1011
|
+
* Build <%=prst%>
|
1012
|
+
|
954
1013
|
```
|
955
|
-
|
956
|
-
|
957
|
-
|
958
|
-
4$ <%=cmd%> config id default set shares shares06
|
959
|
-
5$ <%=cmd%> config overview
|
960
|
-
6$ <%=cmd%> shares repo browse /
|
1014
|
+
<%=cmd%> config preset set shares06 url https://10.25.0.6
|
1015
|
+
<%=cmd%> config preset set shares06 username john
|
1016
|
+
<%=cmd%> config preset set shares06 password 4sp3ra
|
961
1017
|
```
|
962
1018
|
|
963
|
-
The three first commands build a <%=prst%>.
|
964
1019
|
Note that this can also be done with one single command:
|
965
1020
|
|
966
1021
|
```
|
967
|
-
|
1022
|
+
<%=cmd%> config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
|
1023
|
+
```
|
1024
|
+
|
1025
|
+
or
|
1026
|
+
|
1027
|
+
```
|
1028
|
+
<%=cmd%> config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
|
1029
|
+
```
|
1030
|
+
|
1031
|
+
* Define this <%=prst%> as the default <%=prst%> for the specified plugin (`shares`)
|
1032
|
+
|
1033
|
+
```
|
1034
|
+
<%=cmd%> config preset set default shares shares06
|
1035
|
+
```
|
1036
|
+
|
1037
|
+
* Display the content of configuration file in table format
|
1038
|
+
|
1039
|
+
```
|
1040
|
+
<%=cmd%> config overview
|
1041
|
+
```
|
1042
|
+
|
1043
|
+
* Execute a command on the shares application using default parameters
|
1044
|
+
|
1045
|
+
```
|
1046
|
+
<%=cmd%> shares repo browse /
|
1047
|
+
```
|
1048
|
+
|
1049
|
+
## <a id="vault"></a>Secret Vault
|
1050
|
+
|
1051
|
+
When a secret or password is needed, it is possible to store in the secret vault.
|
1052
|
+
|
1053
|
+
By default the vault is defined using option `secrets`.
|
1054
|
+
|
1055
|
+
### Using system keychain
|
1056
|
+
|
1057
|
+
Only on macOS.
|
1058
|
+
|
1059
|
+
It is possible to store secrets in macOS keychain (only read supported currently).
|
1060
|
+
|
1061
|
+
Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
|
1062
|
+
|
1063
|
+
### Modern config file format: encrypted in config file
|
1064
|
+
|
1065
|
+
It is possible to store and use secrets encrypted.
|
1066
|
+
For this use the `config vault` command.
|
1067
|
+
|
1068
|
+
The vault can be initialized with `config vault init`
|
1069
|
+
|
1070
|
+
Then secrets can be manipulated using commands:
|
1071
|
+
|
1072
|
+
* `set`
|
1073
|
+
* `get`
|
1074
|
+
* `list`
|
1075
|
+
* `delete`
|
1076
|
+
|
1077
|
+
Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
|
1078
|
+
|
1079
|
+
### Legacy config file format
|
1080
|
+
|
1081
|
+
The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
|
1082
|
+
|
1083
|
+
For example, choose a repository name, for example `my_secrets`, and populate it like this:
|
1084
|
+
|
1085
|
+
```
|
1086
|
+
<%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
|
1087
|
+
|
1088
|
+
<%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
|
1089
|
+
|
1090
|
+
<%=cmd%> conf id default get config
|
1091
|
+
|
1092
|
+
cli_default
|
968
1093
|
```
|
969
1094
|
|
970
|
-
|
971
|
-
|
972
|
-
Alternative <%=prsts%> can be used with option "-P<<%=prst%>>"
|
973
|
-
(or --preset=<<%=prst%>>)
|
1095
|
+
Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
|
1096
|
+
So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
|
974
1097
|
|
975
|
-
|
1098
|
+
```
|
1099
|
+
<%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
|
1100
|
+
```
|
1101
|
+
|
1102
|
+
A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
|
976
1103
|
|
1104
|
+
To test if a secret can be found use:
|
1105
|
+
|
1106
|
+
```
|
1107
|
+
<%=cmd%> conf vault get --username=access_key1
|
1108
|
+
```
|
977
1109
|
|
978
1110
|
## Plugins
|
979
1111
|
|
980
1112
|
The CLI tool uses a plugin mechanism. The first level command (just after <%=tool%> on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
|
981
1113
|
For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
|
982
1114
|
|
983
|
-
### Create your own plugin
|
1115
|
+
### <a id="createownplugin"></a>Create your own plugin
|
1116
|
+
|
984
1117
|
```
|
985
|
-
|
986
|
-
|
1118
|
+
mkdir -p ~/.aspera/<%=cmd%>/plugins
|
1119
|
+
cat<<EOF>~/.aspera/<%=cmd%>/plugins/test.rb
|
987
1120
|
require 'aspera/cli/plugin'
|
988
1121
|
module Aspera
|
989
1122
|
module Cli
|
@@ -998,39 +1131,78 @@ end # Aspera
|
|
998
1131
|
EOF
|
999
1132
|
```
|
1000
1133
|
|
1001
|
-
|
1134
|
+
### <a id="plugins"></a>Plugins: Application URL and Authentication
|
1135
|
+
|
1136
|
+
<%=tool%> comes with several Aspera application plugins.
|
1137
|
+
|
1138
|
+
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
1139
|
+
|
1140
|
+
Those are using options:
|
1002
1141
|
|
1003
|
-
|
1142
|
+
* url
|
1143
|
+
* username
|
1144
|
+
* password
|
1004
1145
|
|
1005
|
-
|
1146
|
+
Those can be provided using command line, parameter set, env var, see section above.
|
1147
|
+
|
1148
|
+
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
1149
|
+
|
1150
|
+
## Logging, Debugging
|
1151
|
+
|
1152
|
+
The gem is equipped with traces. By default logging level is `warn`.
|
1153
|
+
To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `<%=evp%>LOG_LEVEL`, or parameter in con file).
|
1154
|
+
|
1155
|
+
It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
|
1156
|
+
|
1157
|
+
By default passwords and secrets are removed from logs.
|
1158
|
+
Use option `log_passwords` to change this behaviour.
|
1006
1159
|
|
1007
1160
|
## Learning Aspera Product APIs (REST)
|
1008
1161
|
|
1009
1162
|
This CLI uses REST APIs.
|
1010
|
-
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
|
1011
|
-
exact content or HTTP requests and responses.
|
1163
|
+
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
|
1012
1164
|
|
1013
1165
|
In order to get traces of execution, use argument : `--log-level=debug`
|
1014
1166
|
|
1015
|
-
## <a
|
1167
|
+
## <a id="http_options"></a>HTTP socket parameters
|
1168
|
+
|
1169
|
+
If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
|
1170
|
+
|
1171
|
+
Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
|
1172
|
+
|
1173
|
+
* `read_timeout` 60 sec
|
1174
|
+
* `write_timeout` 60 sec
|
1175
|
+
|
1176
|
+
Default values are the ones of Ruby.
|
1177
|
+
|
1178
|
+
Example:
|
1179
|
+
|
1180
|
+
```
|
1181
|
+
<%=cmd%> aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
1182
|
+
```
|
1183
|
+
|
1184
|
+
## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
|
1016
1185
|
|
1017
1186
|
Some actions may require the use of a graphical tool:
|
1018
1187
|
|
1019
1188
|
* a browser for Aspera on Cloud authentication (web auth method)
|
1020
1189
|
* a text editor for configuration file edition
|
1021
1190
|
|
1022
|
-
By default the CLI will assume that a graphical environment is available on windows,
|
1023
|
-
and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1191
|
+
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1024
1192
|
It is also possible to force the graphical mode with option --ui :
|
1025
1193
|
|
1026
|
-
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
|
1027
|
-
a text
|
1028
|
-
* `--ui=text` forces a text environment, the URL or file path to open is displayed on
|
1029
|
-
terminal.
|
1194
|
+
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
1195
|
+
* `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
1030
1196
|
|
1031
1197
|
## HTTP proxy for REST
|
1032
1198
|
|
1033
|
-
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are
|
1199
|
+
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
|
1200
|
+
|
1201
|
+
## <a id="certificates"></a>SSL CA certificate bundle
|
1202
|
+
|
1203
|
+
<%=tool%> uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
|
1204
|
+
|
1205
|
+
`ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). <%=tool%> overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
|
1034
1206
|
|
1035
1207
|
## Proxy auto config
|
1036
1208
|
|
@@ -1039,13 +1211,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
|
|
1039
1211
|
The PAC file can be tested with command: `config proxy_check` , example:
|
1040
1212
|
|
1041
1213
|
```
|
1042
|
-
|
1214
|
+
<%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
|
1043
1215
|
PROXY proxy.example.com:8080
|
1044
1216
|
```
|
1045
1217
|
|
1046
1218
|
This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
|
1047
1219
|
|
1048
|
-
## <a
|
1220
|
+
## <a id="client"></a>FASP configuration
|
1049
1221
|
|
1050
1222
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
1051
1223
|
|
@@ -1057,9 +1229,9 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
1057
1229
|
### Show path of currently used `ascp`
|
1058
1230
|
|
1059
1231
|
```
|
1060
|
-
|
1232
|
+
<%=cmd%> config ascp show
|
1061
1233
|
/Users/laurent/.aspera/ascli/sdk/ascp
|
1062
|
-
|
1234
|
+
<%=cmd%> config ascp info
|
1063
1235
|
+--------------------+-----------------------------------------------------------+
|
1064
1236
|
| key | value |
|
1065
1237
|
+--------------------+-----------------------------------------------------------+
|
@@ -1067,7 +1239,7 @@ $ <%=cmd%> config ascp info
|
|
1067
1239
|
...
|
1068
1240
|
```
|
1069
1241
|
|
1070
|
-
### Selection of `ascp` location for [`direct`](#
|
1242
|
+
### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
1071
1243
|
|
1072
1244
|
By default, <%=tool%> uses any found local product with ascp, including SDK.
|
1073
1245
|
|
@@ -1078,7 +1250,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
|
|
1078
1250
|
Using a POSIX shell:
|
1079
1251
|
|
1080
1252
|
```
|
1081
|
-
|
1253
|
+
<%=cmd%> config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
1082
1254
|
ascp version: 4.0.0.182279
|
1083
1255
|
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
1084
1256
|
Saved to default global preset global_common_defaults
|
@@ -1087,7 +1259,7 @@ Saved to default global preset global_common_defaults
|
|
1087
1259
|
Windows:
|
1088
1260
|
|
1089
1261
|
```
|
1090
|
-
|
1262
|
+
<%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1091
1263
|
ascp version: 4.0.0.182279
|
1092
1264
|
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1093
1265
|
Saved to default global preset global_common_defaults
|
@@ -1100,7 +1272,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
|
|
1100
1272
|
Locally installed Aspera products can be listed with:
|
1101
1273
|
|
1102
1274
|
```
|
1103
|
-
|
1275
|
+
<%=cmd%> config ascp products list
|
1104
1276
|
:.........................................:................................................:
|
1105
1277
|
: name : app_root :
|
1106
1278
|
:.........................................:................................................:
|
@@ -1111,7 +1283,7 @@ $ <%=cmd%> config ascp products list
|
|
1111
1283
|
:.........................................:................................................:
|
1112
1284
|
```
|
1113
1285
|
|
1114
|
-
### Selection of local client for `ascp` for [`direct`](#
|
1286
|
+
### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
1115
1287
|
|
1116
1288
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1117
1289
|
|
@@ -1120,14 +1292,14 @@ Using the option use_product finds the ascp binary of the selected product.
|
|
1120
1292
|
To permanently use the ascp of a product:
|
1121
1293
|
|
1122
1294
|
```
|
1123
|
-
|
1295
|
+
<%=cmd%> config ascp products use 'Aspera Connect'
|
1124
1296
|
saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
|
1125
1297
|
```
|
1126
1298
|
|
1127
1299
|
### Installation of Connect Client on command line
|
1128
1300
|
|
1129
1301
|
```
|
1130
|
-
|
1302
|
+
<%=cmd%> config ascp connect list
|
1131
1303
|
:...............................................:......................................:..............:
|
1132
1304
|
: id : title : version :
|
1133
1305
|
:...............................................:......................................:..............:
|
@@ -1140,7 +1312,7 @@ $ <%=cmd%> config ascp connect list
|
|
1140
1312
|
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
|
1141
1313
|
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
|
1142
1314
|
:...............................................:......................................:..............:
|
1143
|
-
|
1315
|
+
<%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
1144
1316
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1145
1317
|
: title : type : href : hreflang : rel :
|
1146
1318
|
:.............................................:..........................:.......................................................................:..........:...............:
|
@@ -1153,11 +1325,11 @@ $ <%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
|
1153
1325
|
: Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
|
1154
1326
|
: Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
|
1155
1327
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1156
|
-
|
1328
|
+
<%=cmd%> config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
|
1157
1329
|
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
|
1158
1330
|
```
|
1159
1331
|
|
1160
|
-
## <a
|
1332
|
+
## <a id="agents"></a>Transfer Agents
|
1161
1333
|
|
1162
1334
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
1163
1335
|
|
@@ -1166,102 +1338,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
1166
1338
|
|
1167
1339
|
There are currently 3 agents:
|
1168
1340
|
|
1169
|
-
* [`direct`](#
|
1170
|
-
* `connect` : use of a local Connect Client
|
1171
|
-
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
1172
|
-
* `httpgw` : use of an Aspera HTTP Gateway
|
1341
|
+
* [`direct`](#agt_direct) : a local execution of `ascp`
|
1342
|
+
* [`connect`](#agt_connect) : use of a local Connect Client
|
1343
|
+
* [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
|
1344
|
+
* [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1345
|
+
* [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1173
1346
|
|
1174
1347
|
Note that all transfer operation are seen from the point of view of the agent.
|
1175
1348
|
For instance, a node agent making an "upload", or "package send" operation,
|
1176
1349
|
will effectively push files to the related server from the agent node.
|
1177
1350
|
|
1178
|
-
<%=tool%>
|
1351
|
+
<%=tool%> standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1179
1352
|
|
1180
1353
|
|
1181
|
-
### <a
|
1354
|
+
### <a id="agt_direct"></a>Direct
|
1182
1355
|
|
1183
|
-
|
1184
|
-
|
1356
|
+
The `direct` agent directly executes a local ascp.
|
1357
|
+
This is the default for <%=tool%>.
|
1358
|
+
This is equivalent to specifying `--transfer=direct`.
|
1359
|
+
<%=tool%> will detect locally installed Aspera products, including SDK.
|
1185
1360
|
Refer to section [FASP](#client).
|
1186
1361
|
|
1187
|
-
|
1188
|
-
|
1189
|
-
* `EX_fasp_proxy_url`
|
1190
|
-
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1191
|
-
* `EX_ascp_args`
|
1192
|
-
|
1193
|
-
The `transfer-info` accepts the following optional parameters:
|
1362
|
+
The `transfer-info` accepts the following optional parameters to control multi-session, WSS
|
1194
1363
|
|
1195
1364
|
<table>
|
1196
|
-
<tr><th>Name</th><th>Type</th><th>
|
1197
|
-
<tr><td>
|
1198
|
-
<tr><td>
|
1199
|
-
<tr><td>
|
1200
|
-
<tr><td>multi_incr_udp</td><td>Bool</td><td>
|
1201
|
-
<tr><td>resume</td><td>Hash</td><td>
|
1365
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1366
|
+
<tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
|
1367
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
|
1368
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
|
1369
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
|
1370
|
+
<tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
|
1371
|
+
<tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
|
1372
|
+
<tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
|
1373
|
+
<tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
|
1374
|
+
<tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
|
1202
1375
|
</table>
|
1203
1376
|
|
1204
|
-
Resume
|
1205
|
-
|
1206
|
-
In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
1207
|
-
Sleep between iteration is:
|
1377
|
+
Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
1378
|
+
Sleep between iterations is:
|
1208
1379
|
|
1209
1380
|
```
|
1210
1381
|
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
1211
1382
|
```
|
1212
1383
|
|
1213
|
-
|
1214
|
-
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1215
|
-
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
1216
|
-
<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
1217
|
-
<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
|
1218
|
-
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
1219
|
-
</table>
|
1384
|
+
Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
|
1220
1385
|
|
1221
1386
|
Examples:
|
1222
1387
|
|
1223
1388
|
```
|
1224
|
-
|
1225
|
-
|
1389
|
+
<%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
1390
|
+
<%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
1226
1391
|
```
|
1227
1392
|
|
1228
|
-
|
1393
|
+
To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
|
1229
1394
|
|
1230
|
-
|
1231
|
-
|
1395
|
+
* `EX_fasp_proxy_url`
|
1396
|
+
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1397
|
+
* `EX_ascp_args`
|
1232
1398
|
|
1233
|
-
###
|
1399
|
+
### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
1400
|
+
|
1401
|
+
By specifying option: `--transfer=connect`, <%=tool%> will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
|
1402
|
+
|
1403
|
+
### <a id="agt_node"></a>Aspera Node API : Node to node transfers
|
1234
1404
|
|
1235
1405
|
By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
|
1236
1406
|
Transfer Server using the Node API, either on a local or remote node.
|
1407
|
+
Parameters provided in option `transfer_info` are:
|
1237
1408
|
|
1238
|
-
|
1239
|
-
|
1240
|
-
|
1241
|
-
|
1242
|
-
|
1409
|
+
<table>
|
1410
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1411
|
+
<tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
|
1412
|
+
<tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
|
1413
|
+
<tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
|
1414
|
+
<tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
|
1415
|
+
</table>
|
1243
1416
|
|
1244
|
-
|
1245
|
-
`--transfer-info=@preset:<psetname>` or specified using the
|
1417
|
+
Like any other option, `transfer_info` can get its value from a pre-configured <%=prst%> :
|
1418
|
+
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
1246
1419
|
`--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
|
1247
1420
|
|
1248
|
-
|
1421
|
+
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
1249
1422
|
|
1250
|
-
|
1423
|
+
If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
|
1251
1424
|
|
1252
|
-
### <a
|
1425
|
+
### <a id="agt_httpgw"></a>HTTP Gateway
|
1253
1426
|
|
1254
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
1427
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
|
1255
1428
|
|
1256
1429
|
Example:
|
1257
1430
|
|
1258
1431
|
```
|
1259
|
-
|
1432
|
+
<%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1260
1433
|
```
|
1261
1434
|
|
1262
1435
|
Note that the gateway only supports transfers authorized with a token.
|
1263
1436
|
|
1264
|
-
|
1437
|
+
### <a id="agt_trsdk"></a>Transfer SDK
|
1438
|
+
|
1439
|
+
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
1440
|
+
|
1441
|
+
By default it will listen on local port `55002` on `127.0.0.1`.
|
1442
|
+
|
1443
|
+
## <a id="transferspec"></a>Transfer Specification
|
1265
1444
|
|
1266
1445
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
1267
1446
|
is described in a _transfer-spec_ (Transfer Specification), such as:
|
@@ -1274,9 +1453,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
1274
1453
|
|
1275
1454
|
<%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1276
1455
|
|
1277
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are
|
1456
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
|
1278
1457
|
|
1279
|
-
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#
|
1458
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
1280
1459
|
|
1281
1460
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
1282
1461
|
|
@@ -1285,21 +1464,22 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
1285
1464
|
|
1286
1465
|
A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
1287
1466
|
|
1288
|
-
## <a
|
1467
|
+
## <a id="transferparams"></a>Transfer Parameters
|
1289
1468
|
|
1290
|
-
All standard _transfer-spec_ parameters can be
|
1469
|
+
All standard _transfer-spec_ parameters can be specified.
|
1291
1470
|
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
1292
1471
|
|
1293
1472
|
References:
|
1294
1473
|
|
1295
1474
|
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
1296
1475
|
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
1476
|
+
* [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
1297
1477
|
|
1298
1478
|
Parameters can be displayed with commands:
|
1299
1479
|
|
1300
1480
|
```
|
1301
|
-
|
1302
|
-
|
1481
|
+
<%=cmd%> config ascp spec
|
1482
|
+
<%=cmd%> config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
1303
1483
|
```
|
1304
1484
|
|
1305
1485
|
Columns:
|
@@ -1307,9 +1487,10 @@ Columns:
|
|
1307
1487
|
* D=Direct (local `ascp` execution)
|
1308
1488
|
* N=Node API
|
1309
1489
|
* C=Connect Client
|
1310
|
-
* arg=`ascp` argument or environment variable
|
1311
1490
|
|
1312
|
-
|
1491
|
+
`ascp` argument or environment variable is provided in description.
|
1492
|
+
|
1493
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in <%=tool%>).
|
1313
1494
|
|
1314
1495
|
<%=spec_table%>
|
1315
1496
|
|
@@ -1327,17 +1508,17 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1327
1508
|
|
1328
1509
|
### List of files for transfers
|
1329
1510
|
|
1330
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files
|
1511
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
1331
1512
|
So, by default, the list of files to transfer will be simply specified on the command line:
|
1332
1513
|
|
1333
1514
|
```
|
1334
|
-
|
1515
|
+
<%=cmd%> server upload ~/mysample.file secondfile
|
1335
1516
|
```
|
1336
1517
|
|
1337
1518
|
This is equivalent to:
|
1338
1519
|
|
1339
1520
|
```
|
1340
|
-
|
1521
|
+
<%=cmd%> server upload --sources=@args ~/mysample.file secondfile
|
1341
1522
|
```
|
1342
1523
|
|
1343
1524
|
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
@@ -1363,13 +1544,19 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
1363
1544
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
1364
1545
|
```
|
1365
1546
|
|
1547
|
+
providing a file list directly to ascp:
|
1548
|
+
|
1549
|
+
```
|
1550
|
+
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
|
1551
|
+
```
|
1552
|
+
|
1366
1553
|
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
1367
1554
|
|
1368
1555
|
```
|
1369
1556
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
1370
1557
|
```
|
1371
1558
|
|
1372
|
-
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#
|
1559
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
1373
1560
|
|
1374
1561
|
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
1375
1562
|
|
@@ -1379,10 +1566,10 @@ In case the file list is provided on the command line i.e. using `--sources=@arg
|
|
1379
1566
|
Example:
|
1380
1567
|
|
1381
1568
|
```
|
1382
|
-
|
1569
|
+
<%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
1383
1570
|
```
|
1384
1571
|
|
1385
|
-
Internally, when transfer agent [`direct`](#
|
1572
|
+
Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
|
1386
1573
|
|
1387
1574
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
1388
1575
|
|
@@ -1395,7 +1582,7 @@ Source files are located on "Aspera on cloud", when :
|
|
1395
1582
|
* the server is Aspera on Cloud, and making a download / recv
|
1396
1583
|
* the agent is Aspera on Cloud, and making an upload / send
|
1397
1584
|
|
1398
|
-
### <a
|
1585
|
+
### <a id="multisession"></a>Support of multi-session
|
1399
1586
|
|
1400
1587
|
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
1401
1588
|
|
@@ -1446,16 +1633,14 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
1446
1633
|
--ts=@json:'{"precalculate_job_size":true}'
|
1447
1634
|
```
|
1448
1635
|
|
1449
|
-
|
1450
|
-
|
1451
|
-
## <a name="scheduling"></a>Lock for exclusive execution
|
1636
|
+
## <a id="scheduling"></a>Lock for exclusive execution
|
1452
1637
|
|
1453
1638
|
In some conditions, it may be desirable to ensure that <%=tool%> is not executed several times in parallel.
|
1454
1639
|
|
1455
1640
|
For instance when <%=tool%> is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
1456
1641
|
|
1457
1642
|
* Executing instances may pile-up and kill the system
|
1458
|
-
* The same file may be
|
1643
|
+
* The same file may be transferred by multiple instances at the same time.
|
1459
1644
|
* `preview` may generate the same files in multiple instances.
|
1460
1645
|
|
1461
1646
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
@@ -1475,7 +1660,7 @@ Example:
|
|
1475
1660
|
Run this same command in two separate terminals within less than 30 seconds:
|
1476
1661
|
|
1477
1662
|
```
|
1478
|
-
|
1663
|
+
<%=cmd%> config echo @ruby:'sleep(30)' --lock-port=12345
|
1479
1664
|
```
|
1480
1665
|
|
1481
1666
|
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
@@ -1489,14 +1674,14 @@ WARN -- : Another instance is already running (Address already in use - bind(2)
|
|
1489
1674
|
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
1490
1675
|
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
1491
1676
|
|
1492
|
-
The list of supported *PVCL* adapters can be
|
1677
|
+
The list of supported *PVCL* adapters can be retrieved with command:
|
1493
1678
|
|
1494
1679
|
```
|
1495
|
-
|
1680
|
+
<%=cmd%> conf ascp info
|
1496
1681
|
+--------------------+-----------------------------------------------------------+
|
1497
1682
|
| key | value |
|
1498
1683
|
+--------------------+-----------------------------------------------------------+
|
1499
|
-
-----8
|
1684
|
+
-----8<-----snip-----8<-----
|
1500
1685
|
| product_name | IBM Aspera SDK |
|
1501
1686
|
| product_version | 4.0.1.182389 |
|
1502
1687
|
| process | pvcl |
|
@@ -1521,20 +1706,26 @@ The simplified format is:
|
|
1521
1706
|
|
1522
1707
|
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
1523
1708
|
|
1524
|
-
## <a
|
1709
|
+
## <a id="faux_testing"></a>`faux:` for testing
|
1525
1710
|
|
1526
|
-
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool
|
1711
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>.
|
1527
1712
|
|
1528
1713
|
This adapter can be used to simulate a file or a directory.
|
1529
1714
|
|
1530
|
-
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form
|
1715
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
|
1716
|
+
|
1717
|
+
```
|
1718
|
+
faux:///filename?filesize
|
1719
|
+
```
|
1720
|
+
|
1721
|
+
where:
|
1531
1722
|
|
1532
|
-
* `
|
1533
|
-
* `
|
1723
|
+
* `filename` is the name that will be assigned to the file on the destination
|
1724
|
+
* `filesize` is the number of bytes that will be sent (in decimal).
|
1534
1725
|
|
1535
|
-
Note
|
1726
|
+
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
1536
1727
|
|
1537
|
-
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2
|
1728
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
1538
1729
|
|
1539
1730
|
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
1540
1731
|
|
@@ -1542,21 +1733,21 @@ To send uninitialized data in place of a source directory, the source argument i
|
|
1542
1733
|
faux:///dirname?<arg1>=<val1>&...
|
1543
1734
|
```
|
1544
1735
|
|
1545
|
-
|
1736
|
+
where:
|
1546
1737
|
|
1547
|
-
|
1738
|
+
* `dirname` is the folder name and can contain `/` to specify a subfolder.
|
1739
|
+
* supported arguments are:
|
1548
1740
|
|
1549
1741
|
<table>
|
1550
1742
|
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
1551
|
-
<tr><td>count</td><td>int</td><td>mandatory</td><td>
|
1552
|
-
<tr><td>file</td><td>string</td><td>file</td><td>
|
1553
|
-
<tr><td>size</td><td>int</td><td>0</td><td>
|
1554
|
-
<tr><td>inc</td><td>int</td><td>0</td><td>
|
1555
|
-
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>
|
1556
|
-
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>
|
1743
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
|
1744
|
+
<tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
|
1745
|
+
<tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
|
1746
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
|
1747
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
|
1748
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
|
1557
1749
|
</table>
|
1558
1750
|
|
1559
|
-
|
1560
1751
|
The sequence parameter is applied as follows:
|
1561
1752
|
|
1562
1753
|
* If `seq` is `random` then each file size is:
|
@@ -1564,40 +1755,40 @@ The sequence parameter is applied as follows:
|
|
1564
1755
|
* size +/- (inc * rand())
|
1565
1756
|
* Where rand is a random number between 0 and 1
|
1566
1757
|
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
1567
|
-
* Similarly, overall file size must be less than 8
|
1758
|
+
* Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
1568
1759
|
|
1569
1760
|
* If `seq` is `sequential` then each file size is:
|
1570
1761
|
|
1571
|
-
* size + ((fileindex - 1) * inc)
|
1762
|
+
* `size + ((fileindex - 1) * inc)`
|
1572
1763
|
* Where first file is index 1
|
1573
|
-
* So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
|
1574
|
-
* As with random
|
1764
|
+
* So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
1765
|
+
* As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
1575
1766
|
|
1576
|
-
Filenames generated are of the form: `<file>_<00000
|
1767
|
+
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
1577
1768
|
|
1578
1769
|
To discard data at the destination, the destination argument is set to `faux://` .
|
1579
1770
|
|
1580
1771
|
Examples:
|
1581
1772
|
|
1582
|
-
* Upload 20
|
1773
|
+
* Upload 20 gibibytes of random data to file myfile to directory /Upload
|
1583
1774
|
|
1584
1775
|
```
|
1585
|
-
|
1776
|
+
<%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
|
1586
1777
|
```
|
1587
1778
|
|
1588
1779
|
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
1589
1780
|
|
1590
1781
|
```
|
1591
|
-
|
1782
|
+
<%=cmd%> server upload /tmp/sample --to-folder=faux://
|
1592
1783
|
```
|
1593
1784
|
|
1594
|
-
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2
|
1785
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
1595
1786
|
|
1596
1787
|
```
|
1597
|
-
|
1788
|
+
<%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
1598
1789
|
```
|
1599
1790
|
|
1600
|
-
## <a
|
1791
|
+
## <a id="commands"></a>Sample Commands
|
1601
1792
|
|
1602
1793
|
A non complete list of commands used in unit tests:
|
1603
1794
|
|
@@ -1606,44 +1797,28 @@ A non complete list of commands used in unit tests:
|
|
1606
1797
|
...and more
|
1607
1798
|
```
|
1608
1799
|
|
1609
|
-
## <a
|
1800
|
+
## <a id="usage"></a>Usage
|
1610
1801
|
|
1611
1802
|
```
|
1612
|
-
|
1803
|
+
<%=cmd%> -h
|
1613
1804
|
<%=File.read(ENV["INCL_USAGE"])%>
|
1614
1805
|
|
1615
1806
|
```
|
1616
1807
|
|
1617
1808
|
Note that actions and parameter values can be written in short form.
|
1618
1809
|
|
1619
|
-
# <a
|
1620
|
-
|
1621
|
-
<%=tool%> comes with several Aspera application plugins.
|
1622
|
-
|
1623
|
-
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
1624
|
-
|
1625
|
-
Those are using options:
|
1626
|
-
|
1627
|
-
* url
|
1628
|
-
* username
|
1629
|
-
* password
|
1630
|
-
|
1631
|
-
Those can be provided using command line, parameter set, env var, see section above.
|
1632
|
-
|
1633
|
-
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
1634
|
-
|
1635
|
-
# <a name="aoc"></a>Plugin: Aspera on Cloud
|
1810
|
+
# <a id="aoc"></a>Plugin: Aspera on Cloud
|
1636
1811
|
|
1637
1812
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
1638
1813
|
|
1639
1814
|
It is recommended to use the wizard to set it up, but manual configuration is also possible.
|
1640
1815
|
|
1641
|
-
## <a
|
1816
|
+
## <a id="aocwizard"></a>Configuration: using Wizard
|
1642
1817
|
|
1643
1818
|
<%=tool%> provides a configuration wizard. Here is a sample invocation :
|
1644
1819
|
|
1645
1820
|
```
|
1646
|
-
|
1821
|
+
<%=cmd%> config wizard
|
1647
1822
|
option: url> https://myorg.ibmaspera.com
|
1648
1823
|
Detected: Aspera on Cloud
|
1649
1824
|
Preparing preset: aoc_myorg
|
@@ -1659,7 +1834,7 @@ Setting config preset as default for aspera
|
|
1659
1834
|
saving config file
|
1660
1835
|
Done.
|
1661
1836
|
You can test with:
|
1662
|
-
|
1837
|
+
<%=cmd%> aoc user info show
|
1663
1838
|
```
|
1664
1839
|
|
1665
1840
|
Optionally, it is possible to create a new organization-specific "integration".
|
@@ -1667,7 +1842,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
1667
1842
|
|
1668
1843
|
This will guide you through the steps to create.
|
1669
1844
|
|
1670
|
-
## <a
|
1845
|
+
## <a id="aocmanual"></a>Configuration: using manual setup
|
1671
1846
|
|
1672
1847
|
If you used the wizard (recommended): skip this section.
|
1673
1848
|
|
@@ -1679,7 +1854,7 @@ Several types of OAuth authentication are supported:
|
|
1679
1854
|
* Web based authentication : authentication is made by user using a browser
|
1680
1855
|
* URL Token : external users authentication with url tokens (public links)
|
1681
1856
|
|
1682
|
-
The authentication method is
|
1857
|
+
The authentication method is controlled by option `auth`.
|
1683
1858
|
|
1684
1859
|
For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [<%=prst%> for Aspera on Cloud](#aocpreset).
|
1685
1860
|
|
@@ -1687,7 +1862,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
|
|
1687
1862
|
|
1688
1863
|
In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.<%=tool%> saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
1689
1864
|
|
1690
|
-
### <a
|
1865
|
+
### <a id="clientreg"></a>Optional: API Client Registration
|
1691
1866
|
|
1692
1867
|
If you use the built-in client_id and client_secret, skip this and do not set them in next section.
|
1693
1868
|
|
@@ -1711,31 +1886,31 @@ Note: for web based authentication, <%=tool%> listens on a local port (e.g. spec
|
|
1711
1886
|
|
1712
1887
|
Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
|
1713
1888
|
|
1714
|
-
### <a
|
1889
|
+
### <a id="aocpreset"></a><%=prst%> for Aspera on Cloud
|
1715
1890
|
|
1716
1891
|
If you did not use the wizard, you can also manually create a <%=prst%> for <%=tool%> in its configuration file.
|
1717
1892
|
|
1718
1893
|
Lets create an <%=prst%> called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
1719
1894
|
|
1720
1895
|
```
|
1721
|
-
|
1896
|
+
<%=cmd%> config preset ask my_aoc_org url client_id client_secret
|
1722
1897
|
option: url> https://myorg.ibmaspera.com/
|
1723
1898
|
option: client_id> BJLPObQiFw
|
1724
1899
|
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
|
1725
1900
|
updated: my_aoc_org
|
1726
1901
|
```
|
1727
1902
|
|
1728
|
-
(This can also be done in one line using the command `config
|
1903
|
+
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
1729
1904
|
|
1730
1905
|
Define this <%=prst%> as default configuration for the `aspera` plugin:
|
1731
1906
|
|
1732
1907
|
```
|
1733
|
-
|
1908
|
+
<%=cmd%> config preset set default aoc my_aoc_org
|
1734
1909
|
```
|
1735
1910
|
|
1736
1911
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
1737
1912
|
|
1738
|
-
### <a
|
1913
|
+
### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
|
1739
1914
|
|
1740
1915
|
For a Browser-less, Private Key-based authentication, use the following steps.
|
1741
1916
|
|
@@ -1750,13 +1925,13 @@ This can be done using any of the following method:
|
|
1750
1925
|
* using the CLI:
|
1751
1926
|
|
1752
1927
|
```
|
1753
|
-
|
1928
|
+
<%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
|
1754
1929
|
```
|
1755
1930
|
|
1756
1931
|
* `ssh-keygen`:
|
1757
1932
|
|
1758
1933
|
```
|
1759
|
-
|
1934
|
+
ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
|
1760
1935
|
```
|
1761
1936
|
|
1762
1937
|
* `openssl`
|
@@ -1764,11 +1939,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
|
|
1764
1939
|
(on some openssl implementation (mac) there is option: -nodes (no DES))
|
1765
1940
|
|
1766
1941
|
```
|
1767
|
-
|
1768
|
-
|
1769
|
-
|
1770
|
-
|
1771
|
-
|
1942
|
+
APIKEY=~/.aspera/<%=cmd%>/aocapikey
|
1943
|
+
openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
|
1944
|
+
openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
|
1945
|
+
openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
|
1946
|
+
rm -f ${APIKEY}.protected
|
1772
1947
|
```
|
1773
1948
|
|
1774
1949
|
#### API Client JWT activation
|
@@ -1787,13 +1962,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
1787
1962
|
* Using command line
|
1788
1963
|
|
1789
1964
|
```
|
1790
|
-
|
1965
|
+
<%=cmd%> aoc admin res client list
|
1791
1966
|
:............:.........:
|
1792
1967
|
: id : name :
|
1793
1968
|
:............:.........:
|
1794
1969
|
: BJLPObQiFw : <%=cmd%> :
|
1795
1970
|
:............:.........:
|
1796
|
-
|
1971
|
+
<%=cmd%> aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
1797
1972
|
modified
|
1798
1973
|
```
|
1799
1974
|
|
@@ -1814,14 +1989,14 @@ open the previously generated public key located here: `$HOME/.aspera/<%=cmd%>/a
|
|
1814
1989
|
* Using command line
|
1815
1990
|
|
1816
1991
|
```
|
1817
|
-
|
1992
|
+
<%=cmd%> aoc admin res user list
|
1818
1993
|
:........:................:
|
1819
1994
|
: id : name :
|
1820
1995
|
:........:................:
|
1821
1996
|
: 109952 : Tech Support :
|
1822
1997
|
: 109951 : LAURENT MARTIN :
|
1823
1998
|
:........:................:
|
1824
|
-
|
1999
|
+
<%=cmd%> aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
|
1825
2000
|
modified
|
1826
2001
|
```
|
1827
2002
|
|
@@ -1829,7 +2004,7 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
1829
2004
|
|
1830
2005
|
### <%=prst%> modification for JWT
|
1831
2006
|
|
1832
|
-
To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the
|
2007
|
+
To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the following:
|
1833
2008
|
|
1834
2009
|
* change auth method to JWT
|
1835
2010
|
* provide location of private key
|
@@ -1838,7 +2013,7 @@ To activate default use of JWT authentication for <%=tool%> using the <%=prst%>,
|
|
1838
2013
|
Execute:
|
1839
2014
|
|
1840
2015
|
```
|
1841
|
-
|
2016
|
+
<%=cmd%> config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
|
1842
2017
|
```
|
1843
2018
|
|
1844
2019
|
Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
|
@@ -1846,26 +2021,29 @@ Note: the private key argument represents the actual PEM string. In order to rea
|
|
1846
2021
|
After this last step, commands do not require web login anymore.
|
1847
2022
|
|
1848
2023
|
|
1849
|
-
### <a
|
2024
|
+
### <a id="aocfirst"></a>First Use
|
1850
2025
|
|
1851
2026
|
Once client has been registered and <%=prst%> created: <%=tool%> can be used:
|
1852
2027
|
|
1853
2028
|
```
|
1854
|
-
|
2029
|
+
<%=cmd%> aoc files br /
|
1855
2030
|
Current Workspace: Default Workspace (default)
|
1856
2031
|
empty
|
1857
2032
|
```
|
1858
2033
|
|
1859
2034
|
|
1860
|
-
|
2035
|
+
## Administration
|
1861
2036
|
|
1862
2037
|
The `admin` command allows several administrative tasks (and require admin privilege).
|
1863
2038
|
|
1864
2039
|
It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
|
1865
2040
|
|
1866
|
-
Bulk
|
2041
|
+
### Bulk creation and deletion of resource
|
2042
|
+
|
2043
|
+
Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
|
2044
|
+
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
1867
2045
|
|
1868
|
-
|
2046
|
+
### Listing resources
|
1869
2047
|
|
1870
2048
|
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
1871
2049
|
|
@@ -1887,7 +2065,7 @@ Other parameters are directly sent as parameters to the GET request on API.
|
|
1887
2065
|
|
1888
2066
|
`page` and `per_page` are normally added by <%=tool%> to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
1889
2067
|
|
1890
|
-
`q` and `sort` are available on most
|
2068
|
+
`q` and `sort` are available on most resource types.
|
1891
2069
|
|
1892
2070
|
Other parameters depend on the type of entity (refer to AoC API).
|
1893
2071
|
|
@@ -1899,7 +2077,7 @@ Examples:
|
|
1899
2077
|
<%=cmd%> aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
1900
2078
|
```
|
1901
2079
|
|
1902
|
-
* List users who
|
2080
|
+
* List users who logged-in before a date:
|
1903
2081
|
|
1904
2082
|
```
|
1905
2083
|
<%=cmd%> aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
@@ -1915,40 +2093,85 @@ Refer to the AoC API for full list of query parameters, or use the browser in de
|
|
1915
2093
|
|
1916
2094
|
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
1917
2095
|
|
1918
|
-
|
2096
|
+
### <a id="res_select"></a>Selecting a resource
|
2097
|
+
|
2098
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
|
1919
2099
|
|
1920
|
-
|
1921
|
-
|
2100
|
+
To execute an action on a specific resource, select it using one of those methods:
|
2101
|
+
|
2102
|
+
* *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
|
2103
|
+
* give name on command line *after the action*: `aoc admin res node show name abc`
|
2104
|
+
* provide option `id` : `aoc admin res node show --id=123`
|
2105
|
+
* provide option `name` : `aoc admin res node show --name=abc`
|
2106
|
+
|
2107
|
+
### Access Key secrets
|
2108
|
+
|
2109
|
+
In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
|
2110
|
+
It is usually provided using the `secret` option.
|
2111
|
+
For example in a command like:
|
1922
2112
|
|
1923
2113
|
```
|
1924
|
-
|
2114
|
+
<%=cmd%> aoc admin res node --id=123 --secret="secret1" v3 info
|
1925
2115
|
```
|
1926
2116
|
|
1927
|
-
It is also possible to provide a set of secrets used on a regular basis
|
2117
|
+
It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
|
1928
2118
|
|
1929
|
-
|
2119
|
+
### Activity
|
2120
|
+
|
2121
|
+
The activity app can be queried with:
|
1930
2122
|
|
1931
2123
|
```
|
1932
|
-
|
1933
|
-
$ <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
|
1934
|
-
$ <%=cmd%> conf id default get config
|
1935
|
-
"cli_default"
|
2124
|
+
<%=cmd%> aoc admin analytics transfers
|
1936
2125
|
```
|
1937
2126
|
|
1938
|
-
|
2127
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
2128
|
+
|
2129
|
+
`mytemplate.erb`:
|
1939
2130
|
|
1940
2131
|
```
|
1941
|
-
|
2132
|
+
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
2133
|
+
To: <<%='<'%>%=ev['user_email']%>>
|
2134
|
+
Subject: <%='<'%>%=ev['files_completed']%> files received
|
2135
|
+
|
2136
|
+
Dear <%='<'%>%=ev[:user_email.to_s]%>,
|
2137
|
+
We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
|
2138
|
+
<%='<'%>%=ev['content']%>
|
2139
|
+
|
2140
|
+
Thank you.
|
1942
2141
|
```
|
2142
|
+
The environment provided contains the following additional variable:
|
1943
2143
|
|
1944
|
-
|
2144
|
+
* ev : all details on the transfer event
|
2145
|
+
|
2146
|
+
Example:
|
2147
|
+
|
2148
|
+
```
|
2149
|
+
<%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
2150
|
+
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2151
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
2152
|
+
```
|
1945
2153
|
|
1946
|
-
|
2154
|
+
Options:
|
2155
|
+
|
2156
|
+
* `once_only` keep track of last date it was called, so next call will get only new events
|
2157
|
+
* `query` filter (on API call)
|
2158
|
+
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
2159
|
+
|
2160
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
1947
2161
|
|
1948
|
-
|
2162
|
+
### Transfer: Using specific transfer ports
|
2163
|
+
|
2164
|
+
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
2165
|
+
The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
2166
|
+
|
2167
|
+
### Using ATS
|
2168
|
+
|
2169
|
+
Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
|
2170
|
+
|
2171
|
+
### Example: Bulk creation of users
|
1949
2172
|
|
1950
2173
|
```
|
1951
|
-
|
2174
|
+
<%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
1952
2175
|
:.......:.........:
|
1953
2176
|
: id : status :
|
1954
2177
|
:.......:.........:
|
@@ -1957,20 +2180,20 @@ $ <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exa
|
|
1957
2180
|
:.......:.........:
|
1958
2181
|
```
|
1959
2182
|
|
1960
|
-
|
2183
|
+
### Example: Find with filter and delete
|
1961
2184
|
|
1962
2185
|
```
|
1963
|
-
|
2186
|
+
<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
1964
2187
|
:.......:........................:
|
1965
2188
|
: id : email :
|
1966
2189
|
:.......:........................:
|
1967
2190
|
: 98398 : dummyuser1@example.com :
|
1968
2191
|
: 98399 : dummyuser2@example.com :
|
1969
2192
|
:.......:........................:
|
1970
|
-
|
1971
|
-
|
2193
|
+
thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
2194
|
+
echo $thelist
|
1972
2195
|
["113501","354061"]
|
1973
|
-
|
2196
|
+
<%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
1974
2197
|
:.......:.........:
|
1975
2198
|
: id : status :
|
1976
2199
|
:.......:.........:
|
@@ -1979,18 +2202,18 @@ $ <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
1979
2202
|
:.......:.........:
|
1980
2203
|
```
|
1981
2204
|
|
1982
|
-
|
2205
|
+
### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
1983
2206
|
|
1984
2207
|
```
|
1985
|
-
|
2208
|
+
<%=cmd%> aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
1986
2209
|
```
|
1987
2210
|
|
1988
2211
|
To delete them use the same method as before
|
1989
2212
|
|
1990
|
-
|
2213
|
+
### Example: Display current user's workspaces
|
1991
2214
|
|
1992
2215
|
```
|
1993
|
-
|
2216
|
+
<%=cmd%> aoc user workspaces
|
1994
2217
|
:......:............................:
|
1995
2218
|
: id : name :
|
1996
2219
|
:......:............................:
|
@@ -2000,18 +2223,18 @@ $ <%=cmd%> aoc user workspaces
|
|
2000
2223
|
:......:............................:
|
2001
2224
|
```
|
2002
2225
|
|
2003
|
-
|
2226
|
+
### Example: Create a sub access key in a "node"
|
2004
2227
|
|
2005
2228
|
Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
2006
2229
|
|
2007
2230
|
```
|
2008
|
-
|
2231
|
+
<%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
|
2009
2232
|
```
|
2010
2233
|
|
2011
|
-
|
2234
|
+
### Example: Display transfer events (ops/transfer)
|
2012
2235
|
|
2013
2236
|
```
|
2014
|
-
|
2237
|
+
<%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
2015
2238
|
```
|
2016
2239
|
|
2017
2240
|
Examples of query (TODO: cleanup):
|
@@ -2029,16 +2252,16 @@ Examples of query (TODO: cleanup):
|
|
2029
2252
|
# active_only=true|false
|
2030
2253
|
```
|
2031
2254
|
|
2032
|
-
|
2255
|
+
### Example: Display node events (events)
|
2033
2256
|
|
2034
2257
|
```
|
2035
|
-
|
2258
|
+
<%=cmd%> aoc admin res node --secret=_secret_ v3 events
|
2036
2259
|
```
|
2037
2260
|
|
2038
|
-
|
2261
|
+
### Example: Display members of a workspace
|
2039
2262
|
|
2040
2263
|
```
|
2041
|
-
|
2264
|
+
<%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
2042
2265
|
:.............:.........:..................................:
|
2043
2266
|
: member_type : manager : member.email :
|
2044
2267
|
:.............:.........:..................................:
|
@@ -2057,7 +2280,7 @@ other query parameters:
|
|
2057
2280
|
{"workspace_membership_through":true,"include_indirect":true}
|
2058
2281
|
```
|
2059
2282
|
|
2060
|
-
|
2283
|
+
### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
|
2061
2284
|
|
2062
2285
|
a- Get id of first workspace
|
2063
2286
|
|
@@ -2076,7 +2299,7 @@ WS2ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' -
|
|
2076
2299
|
c- Extract membership information
|
2077
2300
|
|
2078
2301
|
```
|
2079
|
-
|
2302
|
+
<%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
2080
2303
|
```
|
2081
2304
|
|
2082
2305
|
d- Convert to creation data for second workspace:
|
@@ -2094,13 +2317,13 @@ jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"
|
|
2094
2317
|
e- Add members to second workspace
|
2095
2318
|
|
2096
2319
|
```
|
2097
|
-
|
2320
|
+
<%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
2098
2321
|
```
|
2099
2322
|
|
2100
|
-
|
2323
|
+
### Example: Get users who did not log since a date
|
2101
2324
|
|
2102
2325
|
```
|
2103
|
-
|
2326
|
+
<%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2104
2327
|
:...............................:
|
2105
2328
|
: email :
|
2106
2329
|
:...............................:
|
@@ -2109,20 +2332,20 @@ $ <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_logi
|
|
2109
2332
|
:...............................:
|
2110
2333
|
```
|
2111
2334
|
|
2112
|
-
|
2335
|
+
### Example: List "Limited" users
|
2113
2336
|
|
2114
2337
|
```
|
2115
|
-
|
2338
|
+
<%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
2116
2339
|
```
|
2117
2340
|
|
2118
|
-
|
2341
|
+
### Example: Perform a multi Gbps transfer between two remote shared folders
|
2119
2342
|
|
2120
|
-
In this example, a user has access to a workspace where two shared folders are located on
|
2343
|
+
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
2121
2344
|
|
2122
2345
|
First, setup the environment (skip if already done)
|
2123
2346
|
|
2124
2347
|
```
|
2125
|
-
|
2348
|
+
<%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
|
2126
2349
|
Detected: Aspera on Cloud
|
2127
2350
|
Preparing preset: aoc_sedemo
|
2128
2351
|
Using existing key:
|
@@ -2141,7 +2364,7 @@ Setting config preset as default for aspera
|
|
2141
2364
|
saving config file
|
2142
2365
|
Done.
|
2143
2366
|
You can test with:
|
2144
|
-
|
2367
|
+
<%=cmd%> aoc user info show
|
2145
2368
|
```
|
2146
2369
|
|
2147
2370
|
This creates the option preset "aoc_<org name>" to allow seamless command line access and sets it as default for aspera on cloud.
|
@@ -2151,20 +2374,20 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
2151
2374
|
Then, transfer between those:
|
2152
2375
|
|
2153
2376
|
```
|
2154
|
-
|
2377
|
+
<%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
2155
2378
|
```
|
2156
2379
|
|
2157
|
-
|
2380
|
+
### Example: create registration key to register a node
|
2158
2381
|
|
2159
2382
|
```
|
2160
|
-
|
2383
|
+
<%=cmd%> aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
2161
2384
|
jfqslfdjlfdjfhdjklqfhdkl
|
2162
2385
|
```
|
2163
2386
|
|
2164
|
-
|
2387
|
+
### Example: delete all registration keys
|
2165
2388
|
|
2166
2389
|
```
|
2167
|
-
|
2390
|
+
<%=cmd%> aoc admin res client list --fields=id --format=csv|<%=cmd%> aoc admin res client delete --bulk=yes --id=@lines:@stdin:
|
2168
2391
|
+-----+---------+
|
2169
2392
|
| id | status |
|
2170
2393
|
+-----+---------+
|
@@ -2175,81 +2398,107 @@ $ <%=cmd%> aoc admin res admin/client list --fields=id --format=csv|<%=cmd%> aoc
|
|
2175
2398
|
+-----+---------+
|
2176
2399
|
```
|
2177
2400
|
|
2178
|
-
|
2401
|
+
### Example: Create a node
|
2179
2402
|
|
2180
|
-
|
2181
|
-
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2403
|
+
AoC nodes as actually composed with two related entities:
|
2182
2404
|
|
2183
|
-
|
2405
|
+
* An access key created on the Transfer Server (HSTS/ATS)
|
2406
|
+
* a `node` resource in the AoC application.
|
2184
2407
|
|
2185
|
-
|
2186
|
-
|
2408
|
+
The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
|
2409
|
+
Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
|
2410
|
+
|
2411
|
+
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
2187
2412
|
|
2188
|
-
|
2413
|
+
* create the access key on ATS
|
2414
|
+
|
2415
|
+
```
|
2416
|
+
<%=cmd%> aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
2189
2417
|
```
|
2190
2418
|
|
2191
|
-
|
2419
|
+
Take a note of the randomly generated `id` and `secret`.
|
2192
2420
|
|
2193
|
-
*
|
2421
|
+
* Retrieve the ATS node address
|
2194
2422
|
|
2195
2423
|
```
|
2196
|
-
|
2424
|
+
<%=cmd%> aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
|
2197
2425
|
```
|
2198
2426
|
|
2199
|
-
*
|
2427
|
+
* Create the node entity
|
2200
2428
|
|
2201
2429
|
```
|
2202
|
-
|
2430
|
+
<%=cmd%> aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
2203
2431
|
```
|
2204
2432
|
|
2205
|
-
|
2433
|
+
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
2434
|
+
|
2435
|
+
### Example: List packages in a given shared inbox
|
2436
|
+
|
2437
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
2438
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2439
|
+
|
2440
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
2206
2441
|
|
2207
2442
|
```
|
2208
|
-
|
2443
|
+
shbxid=$(<%=cmd%> aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
2444
|
+
|
2445
|
+
<%=cmd%> aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
2209
2446
|
```
|
2210
2447
|
|
2211
|
-
##
|
2448
|
+
## Packages
|
2449
|
+
|
2450
|
+
The webmail-like application.
|
2451
|
+
|
2452
|
+
### Send a Package
|
2212
2453
|
|
2213
2454
|
Send a package:
|
2214
2455
|
|
2215
2456
|
```
|
2216
|
-
|
2457
|
+
<%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
|
2217
2458
|
```
|
2218
2459
|
|
2219
2460
|
Notes:
|
2220
2461
|
|
2221
|
-
*
|
2222
|
-
*
|
2223
|
-
*
|
2224
|
-
*
|
2225
|
-
*
|
2462
|
+
* The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
2463
|
+
* List allowed shared inbox destinations with: `<%=cmd%> aoc user shared_inboxes`
|
2464
|
+
* Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
2465
|
+
* Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
2466
|
+
* or just names: `"recipients":[{"The Dest"}]` . <%=cmd%> will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
2467
|
+
* If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
2468
|
+
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
2226
2469
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
2227
2470
|
|
2228
2471
|
Examples:
|
2229
2472
|
|
2473
|
+
* Send a package with one file to two users, using their email
|
2474
|
+
|
2230
2475
|
```
|
2231
|
-
|
2476
|
+
<%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
2232
2477
|
```
|
2233
2478
|
|
2479
|
+
* Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
|
2480
|
+
|
2234
2481
|
```
|
2235
|
-
|
2482
|
+
<%=cmd%> aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
|
2236
2483
|
```
|
2237
2484
|
|
2485
|
+
* Send a package with one file to a shared inbox (by name) with metadata
|
2486
|
+
|
2238
2487
|
```
|
2239
|
-
|
2488
|
+
<%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2240
2489
|
```
|
2241
2490
|
|
2242
|
-
|
2491
|
+
### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
2243
2492
|
|
2244
2493
|
It is possible to automatically download new packages, like using Aspera Cargo:
|
2245
2494
|
|
2246
2495
|
```
|
2247
|
-
|
2496
|
+
<%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
2248
2497
|
```
|
2249
2498
|
|
2250
2499
|
* `--id=ALL` (case sensitive) will download all packages
|
2251
|
-
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
2252
|
-
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid
|
2500
|
+
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
2501
|
+
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
2253
2502
|
|
2254
2503
|
Typically, one would execute this command on a regular basis, using the method of your choice:
|
2255
2504
|
|
@@ -2257,101 +2506,107 @@ Typically, one would execute this command on a regular basis, using the method o
|
|
2257
2506
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2258
2507
|
* etc...
|
2259
2508
|
|
2260
|
-
##
|
2509
|
+
## Files
|
2510
|
+
|
2511
|
+
Folder sharing app.
|
2512
|
+
|
2513
|
+
### Download Files
|
2261
2514
|
|
2262
2515
|
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2263
2516
|
|
2264
2517
|
* if only one source is provided, it is downloaded
|
2265
2518
|
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2266
2519
|
|
2267
|
-
|
2268
|
-
|
2269
|
-
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2520
|
+
### Shared folders
|
2270
2521
|
|
2271
|
-
|
2272
|
-
|
2273
|
-
* empty (default) : all files, equivalent to value: `exec:true`
|
2274
|
-
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2275
|
-
|
2276
|
-
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2277
|
-
|
2278
|
-
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2279
|
-
|
2280
|
-
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
2281
|
-
|
2282
|
-
* Find files more recent than 100 days
|
2522
|
+
* list shared folders in node
|
2283
2523
|
|
2284
2524
|
```
|
2285
|
-
|
2525
|
+
<%=cmd%> aoc admin res node --id=8669 shared_folders
|
2286
2526
|
```
|
2287
2527
|
|
2288
|
-
*
|
2528
|
+
* list shared folders in workspace
|
2289
2529
|
|
2290
2530
|
```
|
2291
|
-
|
2531
|
+
<%=cmd%> aoc admin res workspace --id=10818 shared_folders
|
2292
2532
|
```
|
2293
2533
|
|
2294
|
-
*
|
2534
|
+
* list members of shared folder
|
2295
2535
|
|
2296
2536
|
```
|
2297
|
-
|
2537
|
+
<%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
|
2298
2538
|
```
|
2299
2539
|
|
2300
|
-
|
2540
|
+
### Cross Organization transfers
|
2301
2541
|
|
2302
|
-
|
2303
|
-
cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
2304
|
-
```
|
2542
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
2305
2543
|
|
2306
|
-
|
2544
|
+
Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
|
2307
2545
|
|
2308
|
-
|
2546
|
+
Procedure to send a file from org1 to org2:
|
2547
|
+
|
2548
|
+
* Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
2549
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
2550
|
+
* Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
|
2551
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
2552
|
+
* execute the following:
|
2309
2553
|
|
2310
2554
|
```
|
2311
|
-
|
2555
|
+
<%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
2312
2556
|
```
|
2313
2557
|
|
2314
|
-
|
2558
|
+
Explanation:
|
2315
2559
|
|
2316
|
-
`
|
2560
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
2561
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
2562
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
2563
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
2564
|
+
* `|` the standard output of the first command is fed into the second one
|
2565
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2566
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2567
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
2568
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2317
2569
|
|
2318
|
-
|
2319
|
-
From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
|
2320
|
-
To: <<%='<'%>%=ev['user_email']%>>
|
2321
|
-
Subject: <%='<'%>%=ev['files_completed']%> files received
|
2570
|
+
### Find Files
|
2322
2571
|
|
2323
|
-
|
2324
|
-
We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
|
2325
|
-
<%='<'%>%=ev['content']%>
|
2572
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2326
2573
|
|
2327
|
-
|
2328
|
-
```
|
2329
|
-
The environment provided contains the following additional variable:
|
2574
|
+
The expression can be of 3 formats:
|
2330
2575
|
|
2331
|
-
*
|
2576
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
2577
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2332
2578
|
|
2333
|
-
|
2579
|
+
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2580
|
+
|
2581
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2582
|
+
|
2583
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
2584
|
+
|
2585
|
+
* Find files more recent than 100 days
|
2334
2586
|
|
2335
2587
|
```
|
2336
|
-
|
2337
|
-
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2338
|
-
--notif-to=active --notif-template=@file:mytemplate.erb
|
2588
|
+
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
2339
2589
|
```
|
2340
2590
|
|
2341
|
-
|
2591
|
+
* Find files older than 1 year on a given node and store in file list
|
2342
2592
|
|
2343
|
-
|
2344
|
-
|
2345
|
-
|
2593
|
+
```
|
2594
|
+
<%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
2595
|
+
```
|
2346
2596
|
|
2347
|
-
|
2597
|
+
* Delete the files, one by one
|
2348
2598
|
|
2349
|
-
|
2599
|
+
```
|
2600
|
+
cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
2601
|
+
```
|
2350
2602
|
|
2351
|
-
|
2603
|
+
* Delete the files in bulk
|
2352
2604
|
|
2605
|
+
```
|
2606
|
+
cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
2607
|
+
```
|
2353
2608
|
|
2354
|
-
# Plugin: Aspera Transfer Service
|
2609
|
+
# <a id="ats"></a>Plugin: Aspera Transfer Service
|
2355
2610
|
|
2356
2611
|
ATS is usable either :
|
2357
2612
|
|
@@ -2361,10 +2616,13 @@ ATS is usable either :
|
|
2361
2616
|
|
2362
2617
|
## IBM Cloud ATS : creation of api key
|
2363
2618
|
|
2619
|
+
This section is about using ATS with an IBM cloud subscription.
|
2620
|
+
If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
|
2621
|
+
|
2364
2622
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
2365
2623
|
|
2366
2624
|
```
|
2367
|
-
|
2625
|
+
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
2368
2626
|
OK
|
2369
2627
|
API key mykeyname was created
|
2370
2628
|
|
@@ -2383,78 +2641,47 @@ References:
|
|
2383
2641
|
* [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
2384
2642
|
* [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
2385
2643
|
|
2386
|
-
|
2387
2644
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
2388
2645
|
|
2389
2646
|
```
|
2390
|
-
|
2391
|
-
|
2392
|
-
|
2647
|
+
<%=cmd%> config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
|
2648
|
+
<%=cmd%> config preset set default ats my_ibm_ats
|
2649
|
+
<%=cmd%> ats api_key instances
|
2393
2650
|
+--------------------------------------+
|
2394
2651
|
| instance |
|
2395
2652
|
+--------------------------------------+
|
2396
2653
|
| aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
|
2397
2654
|
+--------------------------------------+
|
2398
|
-
|
2399
|
-
|
2655
|
+
<%=cmd%> config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
|
2656
|
+
<%=cmd%> ats api_key create
|
2400
2657
|
+--------+----------------------------------------------+
|
2401
2658
|
| key | value |
|
2402
2659
|
+--------+----------------------------------------------+
|
2403
2660
|
| id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
|
2404
2661
|
| secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
|
2405
2662
|
+--------+----------------------------------------------+
|
2406
|
-
|
2663
|
+
<%=cmd%> config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
2407
2664
|
```
|
2408
|
-
## Cross Organization transfers
|
2409
|
-
|
2410
|
-
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
2411
|
-
|
2412
|
-
Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
|
2413
|
-
|
2414
|
-
Procedure to send a file from org1 to org2:
|
2415
|
-
|
2416
|
-
* Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
2417
|
-
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
2418
|
-
* Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
|
2419
|
-
* Check that access works and locate the destination folder `mydestfolder`
|
2420
|
-
* execute the following:
|
2421
2665
|
|
2422
|
-
```
|
2423
|
-
$ <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
2424
|
-
```
|
2425
|
-
|
2426
|
-
Explanation:
|
2427
|
-
|
2428
|
-
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
2429
|
-
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
2430
|
-
* `--format=json` format the output in JSON (instead of default text table)
|
2431
|
-
* `--display=data` display only the result, and remove other information, such as workspace name
|
2432
|
-
* `|` the standard output of the first command is fed into the second one
|
2433
|
-
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2434
|
-
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2435
|
-
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
|
2436
|
-
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2437
|
-
|
2438
|
-
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
2439
2666
|
## Examples
|
2440
2667
|
|
2441
|
-
Example: create access key on softlayer:
|
2668
|
+
Example: create access key on IBM Cloud (softlayer):
|
2442
2669
|
|
2443
2670
|
```
|
2444
|
-
|
2671
|
+
<%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
2445
2672
|
```
|
2446
2673
|
|
2447
2674
|
Example: create access key on AWS:
|
2448
2675
|
|
2449
2676
|
```
|
2450
|
-
|
2677
|
+
<%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
|
2451
2678
|
|
2452
2679
|
```
|
2453
2680
|
|
2454
2681
|
Example: create access key on Azure SAS:
|
2455
2682
|
|
2456
2683
|
```
|
2457
|
-
|
2684
|
+
<%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
2458
2685
|
|
2459
2686
|
```
|
2460
2687
|
|
@@ -2463,7 +2690,7 @@ $ <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{
|
|
2463
2690
|
Example: create access key on Azure:
|
2464
2691
|
|
2465
2692
|
```
|
2466
|
-
|
2693
|
+
<%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
2467
2694
|
|
2468
2695
|
```
|
2469
2696
|
|
@@ -2473,74 +2700,77 @@ delete all my access keys:
|
|
2473
2700
|
for k in $(<%=cmd%> ats access_key list --field=id --format=csv);do <%=cmd%> ats access_key id $k delete;done
|
2474
2701
|
```
|
2475
2702
|
|
2703
|
+
The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
|
2704
|
+
|
2476
2705
|
# Plugin: IBM Aspera High Speed Transfer Server (transfer)
|
2477
2706
|
|
2478
|
-
This plugin
|
2707
|
+
This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
|
2708
|
+
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
2709
|
+
Modern mode is to use the node API and transfer tokens.
|
2479
2710
|
|
2480
2711
|
## Authentication
|
2481
2712
|
|
2482
2713
|
Both password and SSH keys auth are supported.
|
2483
2714
|
|
2484
|
-
If
|
2715
|
+
If username is not provided, the default transfer user `xfer` is used.
|
2485
2716
|
|
2486
|
-
If no
|
2717
|
+
If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
|
2487
2718
|
|
2488
2719
|
```
|
2489
|
-
|
2720
|
+
<%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
2490
2721
|
```
|
2491
2722
|
|
2492
|
-
Multiple SSH key paths can be provided.
|
2723
|
+
Multiple SSH key paths can be provided.
|
2724
|
+
The value of the parameter `ssh_keys` can be a single value or an array.
|
2725
|
+
Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
|
2493
2726
|
|
2494
2727
|
Examples:
|
2495
2728
|
|
2496
2729
|
```
|
2497
|
-
|
2498
|
-
|
2499
|
-
|
2730
|
+
<%=cmd%> server --ssh-keys=~/.ssh/id_rsa
|
2731
|
+
<%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
|
2732
|
+
<%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
2500
2733
|
```
|
2501
2734
|
|
2502
|
-
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
2735
|
+
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
2736
|
+
By default the ssh library expect that an ssh-agent is running.
|
2503
2737
|
|
2504
|
-
|
2738
|
+
On Linux, if you get an error message such as:
|
2505
2739
|
|
2506
2740
|
```
|
2507
|
-
[Linux]
|
2508
2741
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
|
2509
2742
|
```
|
2510
2743
|
|
2511
|
-
or
|
2744
|
+
or on Windows:
|
2512
2745
|
|
2513
2746
|
```
|
2514
|
-
[Windows]
|
2515
2747
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
2516
2748
|
```
|
2517
2749
|
|
2518
|
-
This means that you
|
2750
|
+
This means that you don't have such an ssh agent running, then:
|
2519
2751
|
|
2520
2752
|
* check env var: `SSH_AGENT_SOCK`
|
2521
|
-
* check if the key is protected with a passphrase
|
2753
|
+
* check if the ssh key is protected with a passphrase
|
2522
2754
|
* [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
2523
|
-
* To
|
2755
|
+
* To disable use of `ssh-agent`, use the option `ssh_option` like this:
|
2524
2756
|
|
2525
2757
|
```
|
2526
|
-
|
2758
|
+
<%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
|
2527
2759
|
```
|
2528
2760
|
|
2529
|
-
This can also be set as default using a preset.
|
2761
|
+
This can also be set as default using a global preset.
|
2530
2762
|
|
2531
2763
|
## Example
|
2532
2764
|
|
2533
|
-
One can test the
|
2765
|
+
One can test the `server` application using the well known demo server:
|
2534
2766
|
|
2535
2767
|
```
|
2536
|
-
|
2537
|
-
|
2538
|
-
|
2539
|
-
$ <%=cmd%> server download /aspera-test-dir-large/200MB
|
2768
|
+
<%=cmd%> config initdemo
|
2769
|
+
<%=cmd%> server browse /aspera-test-dir-large
|
2770
|
+
<%=cmd%> server download /aspera-test-dir-large/200MB
|
2540
2771
|
```
|
2541
2772
|
|
2542
|
-
|
2543
|
-
|
2773
|
+
`initdemo` creates a <%=prst%> `demoserver` and set it as default for plugin `server`.
|
2544
2774
|
|
2545
2775
|
# Plugin: IBM Aspera High Speed Transfer Server (node)
|
2546
2776
|
|
@@ -2572,12 +2802,12 @@ For transfers, it is possible to control how transfer is authorized using option
|
|
2572
2802
|
|
2573
2803
|
## Central
|
2574
2804
|
|
2575
|
-
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and
|
2805
|
+
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
|
2576
2806
|
|
2577
2807
|
Filtering can be applied:
|
2578
2808
|
|
2579
2809
|
```
|
2580
|
-
|
2810
|
+
<%=cmd%> node central file list
|
2581
2811
|
```
|
2582
2812
|
|
2583
2813
|
by providing the `validator` option, offline transfer validation can be done.
|
@@ -2589,7 +2819,7 @@ It is possible to start a FASPStream session using the node API:
|
|
2589
2819
|
Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
|
2590
2820
|
|
2591
2821
|
```
|
2592
|
-
|
2822
|
+
<%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
|
2593
2823
|
```
|
2594
2824
|
|
2595
2825
|
## Watchfolder
|
@@ -2602,9 +2832,9 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
2602
2832
|
* configure a watchfolder to define automated transfers
|
2603
2833
|
|
2604
2834
|
```
|
2605
|
-
|
2606
|
-
|
2607
|
-
|
2835
|
+
<%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
2836
|
+
<%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
2837
|
+
<%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
2608
2838
|
```
|
2609
2839
|
|
2610
2840
|
## Out of Transfer File Validation
|
@@ -2612,13 +2842,13 @@ $ <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watc
|
|
2612
2842
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
2613
2843
|
|
2614
2844
|
```
|
2615
|
-
|
2845
|
+
<%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
2616
2846
|
:..............:..............:............:......................................:
|
2617
2847
|
: session_uuid : file_id : status : path :
|
2618
2848
|
:..............:..............:............:......................................:
|
2619
2849
|
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
|
2620
2850
|
:..............:..............:............:......................................:
|
2621
|
-
|
2851
|
+
<%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
2622
2852
|
updated
|
2623
2853
|
```
|
2624
2854
|
|
@@ -2634,7 +2864,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
|
|
2634
2864
|
Then execute the following command:
|
2635
2865
|
|
2636
2866
|
```
|
2637
|
-
|
2867
|
+
<%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
|
2638
2868
|
```
|
2639
2869
|
|
2640
2870
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance
|
@@ -2643,7 +2873,7 @@ to download files.
|
|
2643
2873
|
## Create access key
|
2644
2874
|
|
2645
2875
|
```
|
2646
|
-
|
2876
|
+
<%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
2647
2877
|
```
|
2648
2878
|
|
2649
2879
|
# Plugin: IBM Aspera Faspex5
|
@@ -2668,7 +2898,7 @@ For boot method: (will be removed in future)
|
|
2668
2898
|
Use it as password and use `--auth=boot`.
|
2669
2899
|
|
2670
2900
|
```
|
2671
|
-
|
2901
|
+
<%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
2672
2902
|
```
|
2673
2903
|
|
2674
2904
|
Ready to use Faspex5 with CLI.
|
@@ -2714,7 +2944,7 @@ If no parameter `max` or `pmax` is provided, then all packages will be listed in
|
|
2714
2944
|
### Example
|
2715
2945
|
|
2716
2946
|
```
|
2717
|
-
|
2947
|
+
<%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
2718
2948
|
```
|
2719
2949
|
|
2720
2950
|
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
@@ -2728,14 +2958,14 @@ The command is `package recv`, possible methods are:
|
|
2728
2958
|
* provide a `faspe:` URI with option `link`
|
2729
2959
|
|
2730
2960
|
```
|
2731
|
-
|
2732
|
-
|
2961
|
+
<%=cmd%> faspex package recv --id=12345
|
2962
|
+
<%=cmd%> faspex package recv --link=faspe://...
|
2733
2963
|
```
|
2734
2964
|
|
2735
2965
|
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
2736
2966
|
|
2737
2967
|
```
|
2738
|
-
|
2968
|
+
<%=cmd%> faspex package list --recipient='*thedropboxname'
|
2739
2969
|
```
|
2740
2970
|
|
2741
2971
|
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
@@ -2747,7 +2977,7 @@ The command is `faspex package send`. Package information (title, note, metadata
|
|
2747
2977
|
Example:
|
2748
2978
|
|
2749
2979
|
```
|
2750
|
-
|
2980
|
+
<%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
2751
2981
|
```
|
2752
2982
|
|
2753
2983
|
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
@@ -2764,7 +2994,7 @@ Like for any transfer, a notification can be sent by email using parameters: `no
|
|
2764
2994
|
Example:
|
2765
2995
|
|
2766
2996
|
```
|
2767
|
-
|
2997
|
+
<%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
2768
2998
|
```
|
2769
2999
|
|
2770
3000
|
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
@@ -2774,9 +3004,9 @@ In this example the notification template is directly provided on command line.
|
|
2774
3004
|
Example:
|
2775
3005
|
|
2776
3006
|
```
|
2777
|
-
|
2778
|
-
|
2779
|
-
|
3007
|
+
<%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
3008
|
+
<%=cmd%> faspex v4 dropbox list
|
3009
|
+
<%=cmd%> faspex v4 dropbox delete --id=36
|
2780
3010
|
```
|
2781
3011
|
|
2782
3012
|
## Remote sources
|
@@ -2814,7 +3044,7 @@ It is possible to tell <%=tool%> to download newly received packages, much like
|
|
2814
3044
|
cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
|
2815
3045
|
|
2816
3046
|
```
|
2817
|
-
|
3047
|
+
<%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
|
2818
3048
|
```
|
2819
3049
|
|
2820
3050
|
# Plugin: IBM Aspera Shares
|
@@ -2824,7 +3054,7 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
2824
3054
|
In Shares2, users, groups listing are paged, to display sequential pages:
|
2825
3055
|
|
2826
3056
|
```
|
2827
|
-
|
3057
|
+
for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
|
2828
3058
|
```
|
2829
3059
|
|
2830
3060
|
# Plugin: IBM Cloud Object Storage
|
@@ -2833,9 +3063,9 @@ The IBM Cloud Object Storage provides the possibility to execute transfers using
|
|
2833
3063
|
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
2834
3064
|
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
2835
3065
|
|
2836
|
-
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the
|
3066
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
2837
3067
|
|
2838
|
-
## Using endpoint, apikey and
|
3068
|
+
## Using endpoint, apikey and Resource Instance ID (CRN)
|
2839
3069
|
|
2840
3070
|
If you have those parameters already, then following options shall be provided:
|
2841
3071
|
|
@@ -2847,15 +3077,15 @@ If you have those parameters already, then following options shall be provided:
|
|
2847
3077
|
For example, let us create a default configuration:
|
2848
3078
|
|
2849
3079
|
```
|
2850
|
-
|
2851
|
-
|
3080
|
+
<%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
3081
|
+
<%=cmd%> conf id default set cos mycos
|
2852
3082
|
```
|
2853
3083
|
|
2854
3084
|
Then, jump to the transfer example.
|
2855
3085
|
|
2856
3086
|
## Using service credential file
|
2857
3087
|
|
2858
|
-
If you are the COS administrator and
|
3088
|
+
If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
2859
3089
|
|
2860
3090
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
2861
3091
|
|
@@ -2864,11 +3094,11 @@ Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
2864
3094
|
or using the IBM Cloud CLI:
|
2865
3095
|
|
2866
3096
|
```
|
2867
|
-
|
2868
|
-
|
3097
|
+
ibmcloud resource service-keys
|
3098
|
+
ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
2869
3099
|
```
|
2870
3100
|
|
2871
|
-
(if you
|
3101
|
+
(if you don't have `jq` installed, extract the structure as follows)
|
2872
3102
|
|
2873
3103
|
It consists in the following structure:
|
2874
3104
|
|
@@ -2903,8 +3133,8 @@ The required options for this method are:
|
|
2903
3133
|
For example, let us create a default configuration:
|
2904
3134
|
|
2905
3135
|
```
|
2906
|
-
|
2907
|
-
|
3136
|
+
<%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
3137
|
+
<%=cmd%> conf id default set cos mycos
|
2908
3138
|
```
|
2909
3139
|
|
2910
3140
|
## Operations, transfers
|
@@ -2914,8 +3144,8 @@ Let's assume you created a default configuration from once of the two previous s
|
|
2914
3144
|
A subset of `node` plugin operations are supported, basically node API:
|
2915
3145
|
|
2916
3146
|
```
|
2917
|
-
|
2918
|
-
|
3147
|
+
<%=cmd%> cos node info
|
3148
|
+
<%=cmd%> cos node upload 'faux:///sample1G?1g'
|
2919
3149
|
```
|
2920
3150
|
|
2921
3151
|
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
@@ -2944,8 +3174,10 @@ Specify the previews folder as shown in:
|
|
2944
3174
|
By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
|
2945
3175
|
|
2946
3176
|
```
|
2947
|
-
|
2948
|
-
|
3177
|
+
PATH=/opt/aspera/bin:$PATH
|
3178
|
+
|
3179
|
+
asconfigurator -x "server;preview_dir,previews"
|
3180
|
+
asnodeadmin --reload
|
2949
3181
|
```
|
2950
3182
|
|
2951
3183
|
Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
|
@@ -2958,16 +3190,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
|
|
2958
3190
|
To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
|
2959
3191
|
|
2960
3192
|
```
|
2961
|
-
|
3193
|
+
asuserdata -a | grep max_request_file_create_size_kb
|
3194
|
+
|
2962
3195
|
max_request_file_create_size_kb: "1024"
|
2963
|
-
|
3196
|
+
|
3197
|
+
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
2964
3198
|
```
|
2965
3199
|
|
2966
3200
|
If you use a value different than 16777216, then specify it using option `max_size`.
|
2967
3201
|
|
2968
3202
|
Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
|
2969
3203
|
|
2970
|
-
## <a
|
3204
|
+
## <a id="prev_ext"></a>External tools: Linux
|
2971
3205
|
|
2972
3206
|
The tool requires the following external tools available in the `PATH`:
|
2973
3207
|
|
@@ -2984,7 +3218,7 @@ Other OSes should work as well, but are note tested.
|
|
2984
3218
|
To check if all tools are found properly, execute:
|
2985
3219
|
|
2986
3220
|
```
|
2987
|
-
|
3221
|
+
<%=cmd%> preview check
|
2988
3222
|
```
|
2989
3223
|
|
2990
3224
|
### mimemagic
|
@@ -2992,22 +3226,22 @@ $ <%=cmd%> preview check
|
|
2992
3226
|
To benefit from extra mime type detection install gem mimemagic:
|
2993
3227
|
|
2994
3228
|
```
|
2995
|
-
|
3229
|
+
gem install mimemagic
|
2996
3230
|
```
|
2997
3231
|
|
2998
3232
|
or to install an earlier version if any problem:
|
2999
3233
|
|
3000
3234
|
```
|
3001
|
-
|
3235
|
+
gem install mimemagic -v '~> 0.3.0'
|
3002
3236
|
```
|
3003
3237
|
|
3004
3238
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
3005
3239
|
|
3006
3240
|
If not used, Mime type used for conversion is the one provided by the node API.
|
3007
3241
|
|
3008
|
-
If used, it the `preview` command will first
|
3242
|
+
If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
|
3009
3243
|
|
3010
|
-
### Image:
|
3244
|
+
### Image: ImageMagick and optipng
|
3011
3245
|
|
3012
3246
|
```
|
3013
3247
|
yum install -y ImageMagick optipng
|
@@ -3023,25 +3257,25 @@ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.ta
|
|
3023
3257
|
|
3024
3258
|
### Office: Unoconv and Libreoffice
|
3025
3259
|
|
3026
|
-
If you
|
3260
|
+
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
3027
3261
|
|
3028
3262
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
3029
3263
|
|
3030
|
-
*
|
3264
|
+
* CentOS 8
|
3031
3265
|
|
3032
3266
|
```
|
3033
|
-
|
3267
|
+
dnf install unoconv
|
3034
3268
|
```
|
3035
3269
|
|
3036
3270
|
* Amazon Linux
|
3037
3271
|
|
3038
3272
|
```
|
3039
|
-
|
3040
|
-
|
3041
|
-
|
3042
|
-
|
3043
|
-
|
3044
|
-
|
3273
|
+
amazon-linux-extras enable libreoffice
|
3274
|
+
yum clean metadata
|
3275
|
+
yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
3276
|
+
wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
3277
|
+
mv unoconv /usr/bin
|
3278
|
+
chmod a+x /usr/bin/unoconv
|
3045
3279
|
```
|
3046
3280
|
|
3047
3281
|
## Configuration
|
@@ -3053,9 +3287,11 @@ Like any <%=tool%> commands, parameters can be passed on command line or using a
|
|
3053
3287
|
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
3054
3288
|
|
3055
3289
|
```
|
3056
|
-
|
3057
|
-
|
3058
|
-
|
3290
|
+
su -s /bin/bash - xfer
|
3291
|
+
|
3292
|
+
<%=cmd%> config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
3293
|
+
|
3294
|
+
<%=cmd%> config preset set default preview previewconf
|
3059
3295
|
```
|
3060
3296
|
|
3061
3297
|
Here we assume that Office file generation is disabled, else remove this option.
|
@@ -3064,7 +3300,7 @@ Here we assume that Office file generation is disabled, else remove this option.
|
|
3064
3300
|
One can check if the access key is well configured using:
|
3065
3301
|
|
3066
3302
|
```
|
3067
|
-
|
3303
|
+
<%=cmd%> -Ppreviewconf node browse /
|
3068
3304
|
```
|
3069
3305
|
|
3070
3306
|
This shall list the contents of the storage root of the access key.
|
@@ -3080,8 +3316,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
|
|
3080
3316
|
Lets do a one shot test, using the configuration previously created:
|
3081
3317
|
|
3082
3318
|
```
|
3083
|
-
|
3084
|
-
|
3319
|
+
su -s /bin/bash - xfer
|
3320
|
+
|
3321
|
+
<%=cmd%> preview scan --overwrite=always
|
3085
3322
|
```
|
3086
3323
|
|
3087
3324
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
@@ -3095,13 +3332,13 @@ Adapt the scripts to your own preference.
|
|
3095
3332
|
|
3096
3333
|
We assume here that a configuration preset was created as shown previously.
|
3097
3334
|
|
3098
|
-
Lets first setup a script that will be used in the
|
3335
|
+
Lets first setup a script that will be used in the scheduler and sets up the environment.
|
3099
3336
|
|
3100
3337
|
Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
|
3101
3338
|
|
3102
3339
|
```
|
3103
|
-
#!/bin/bash
|
3104
|
-
# set a timeout protection, just in case
|
3340
|
+
#!/bin/bash
|
3341
|
+
# set a timeout protection, just in case
|
3105
3342
|
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
3106
3343
|
. /etc/profile.d/rvm.sh
|
3107
3344
|
rvm use 2.6 --quiet
|
@@ -3111,13 +3348,13 @@ exec timeout ${tmout} <%=cmd%> "${@}"
|
|
3111
3348
|
Here the cronjob is created for user `xfer`.
|
3112
3349
|
|
3113
3350
|
```
|
3114
|
-
|
3351
|
+
crontab<<EOF
|
3115
3352
|
0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
|
3116
3353
|
2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
|
3117
3354
|
EOF
|
3118
3355
|
```
|
3119
3356
|
|
3120
|
-
Note that the
|
3357
|
+
Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
3121
3358
|
|
3122
3359
|
## Candidate detection for creation or update (or deletion)
|
3123
3360
|
|
@@ -3142,12 +3379,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
|
|
3142
3379
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
3143
3380
|
|
3144
3381
|
```
|
3145
|
-
|
3382
|
+
<%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
|
3146
3383
|
```
|
3147
3384
|
|
3148
3385
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
3149
3386
|
|
3150
|
-
When scanning the option `value` has the same
|
3387
|
+
When scanning the option `value` has the same behavior as for the `node find` command.
|
3151
3388
|
|
3152
3389
|
For instance to filter out files beginning with `._` do:
|
3153
3390
|
|
@@ -3166,7 +3403,7 @@ Use option `skip_format` to skip generation of a format.
|
|
3166
3403
|
|
3167
3404
|
## Supported input Files types
|
3168
3405
|
|
3169
|
-
The preview generator supports
|
3406
|
+
The preview generator supports rendering of those file categories:
|
3170
3407
|
|
3171
3408
|
* image
|
3172
3409
|
* pdf
|
@@ -3186,7 +3423,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
|
|
3186
3423
|
|
3187
3424
|
## Access to original files and preview creation
|
3188
3425
|
|
3189
|
-
Standard open source tools are used to create
|
3426
|
+
Standard open source tools are used to create thumbnails and video previews.
|
3190
3427
|
Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
|
3191
3428
|
The tool provides 2 ways to read and write files with the option: `file_access`
|
3192
3429
|
|
@@ -3216,28 +3453,28 @@ The `smtp` option is a hash table (extended value) with the following fields:
|
|
3216
3453
|
## Example of configuration:
|
3217
3454
|
|
3218
3455
|
```
|
3219
|
-
|
3220
|
-
|
3221
|
-
|
3456
|
+
<%=cmd%> config preset set smtp_google server smtp.google.com
|
3457
|
+
<%=cmd%> config preset set smtp_google username john@gmail.com
|
3458
|
+
<%=cmd%> config preset set smtp_google password P@ssw0rd
|
3222
3459
|
```
|
3223
3460
|
|
3224
3461
|
or
|
3225
3462
|
|
3226
3463
|
```
|
3227
|
-
|
3464
|
+
<%=cmd%> config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
|
3228
3465
|
```
|
3229
3466
|
|
3230
3467
|
or
|
3231
3468
|
|
3232
3469
|
```
|
3233
|
-
|
3470
|
+
<%=cmd%> config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
|
3234
3471
|
```
|
3235
3472
|
|
3236
|
-
Set this
|
3473
|
+
Set this configuration as global default, for instance:
|
3237
3474
|
|
3238
3475
|
```
|
3239
|
-
|
3240
|
-
|
3476
|
+
<%=cmd%> config preset set cli_default smtp @val:@preset:smtp_google
|
3477
|
+
<%=cmd%> config preset set default config cli_default
|
3241
3478
|
```
|
3242
3479
|
|
3243
3480
|
## Email templates
|
@@ -3259,8 +3496,8 @@ Other variables are defined depending on context.
|
|
3259
3496
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
3260
3497
|
|
3261
3498
|
```
|
3262
|
-
|
3263
|
-
|
3499
|
+
<%=cmd%> config --smtp=@preset:smtp_google smtp
|
3500
|
+
<%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
3264
3501
|
```
|
3265
3502
|
|
3266
3503
|
## Notifications for transfer status
|
@@ -3269,7 +3506,7 @@ An e-mail notification can be sent upon transfer success and failure (one email
|
|
3269
3506
|
|
3270
3507
|
To activate, use option `notif_to`.
|
3271
3508
|
|
3272
|
-
A default e-mail template is used, but it can be
|
3509
|
+
A default e-mail template is used, but it can be overridden with option `notif_template`.
|
3273
3510
|
|
3274
3511
|
The environment provided contains the following additional variables:
|
3275
3512
|
|
@@ -3293,7 +3530,7 @@ Transfer is: <%='<'%>%=global_transfer_status%>
|
|
3293
3530
|
This gem comes with a second executable tool providing a simplified standardized interface
|
3294
3531
|
to start a FASP session: `asession`.
|
3295
3532
|
|
3296
|
-
It aims at simplifying the startup of a FASP session from a programmatic stand point as
|
3533
|
+
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
|
3297
3534
|
|
3298
3535
|
* common to Aspera Node API (HTTP POST /ops/transfer)
|
3299
3536
|
* common to Aspera Connect API (browser javascript startTransfer)
|
@@ -3305,7 +3542,7 @@ This makes it easy to integrate with any language provided that one can spawn a
|
|
3305
3542
|
|
3306
3543
|
The tool expect one single argument: a [_transfer-spec_](#transferspec).
|
3307
3544
|
|
3308
|
-
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON
|
3545
|
+
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
|
3309
3546
|
|
3310
3547
|
Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
|
3311
3548
|
|
@@ -3316,7 +3553,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
3316
3553
|
* `EX_loglevel` to change log level of the tool
|
3317
3554
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
3318
3555
|
|
3319
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#
|
3556
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
3320
3557
|
|
3321
3558
|
## Comparison of interfaces
|
3322
3559
|
|
@@ -3343,7 +3580,7 @@ echo "${MY_TSPEC}"|asession
|
|
3343
3580
|
This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
|
3344
3581
|
|
3345
3582
|
```
|
3346
|
-
|
3583
|
+
asession
|
3347
3584
|
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
3348
3585
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
3349
3586
|
{"type":"DONE"}
|
@@ -3358,7 +3595,7 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
|
|
3358
3595
|
## Help
|
3359
3596
|
|
3360
3597
|
```
|
3361
|
-
|
3598
|
+
asession -h
|
3362
3599
|
<%=File.read(ENV["INCL_ASESSION"])%>
|
3363
3600
|
```
|
3364
3601
|
|
@@ -3397,7 +3634,7 @@ Note that:
|
|
3397
3634
|
|
3398
3635
|
* <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
3399
3636
|
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3400
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#
|
3637
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
|
3401
3638
|
|
3402
3639
|
### server side and configuration
|
3403
3640
|
|
@@ -3412,18 +3649,18 @@ Once <%=tool%> parameters are defined, run the command using the OS native sched
|
|
3412
3649
|
## Example
|
3413
3650
|
|
3414
3651
|
```
|
3415
|
-
|
3652
|
+
<%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
|
3416
3653
|
|
3417
3654
|
```
|
3418
3655
|
|
3419
|
-
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they
|
3656
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3420
3657
|
|
3421
|
-
#
|
3658
|
+
# Health check and Nagios
|
3422
3659
|
|
3423
|
-
|
3660
|
+
Most plugin provide a `health` command that will check the health status of the application. Example:
|
3424
3661
|
|
3425
3662
|
```
|
3426
|
-
|
3663
|
+
<%=cmd%> console health
|
3427
3664
|
+--------+-------------+------------+
|
3428
3665
|
| status | component | message |
|
3429
3666
|
+--------+-------------+------------+
|
@@ -3440,13 +3677,13 @@ Typically, the health check uses the REST API of the application with the follow
|
|
3440
3677
|
<%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
3441
3678
|
|
3442
3679
|
```
|
3443
|
-
|
3680
|
+
<%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
|
3444
3681
|
OK - [transfer:ok]
|
3445
|
-
|
3682
|
+
<%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
3446
3683
|
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
3447
3684
|
```
|
3448
3685
|
|
3449
|
-
# Module: `Aspera`
|
3686
|
+
# Ruby Module: `Aspera`
|
3450
3687
|
|
3451
3688
|
Main components:
|
3452
3689
|
|
@@ -3457,8 +3694,8 @@ Main components:
|
|
3457
3694
|
A working example can be found in the gem, example:
|
3458
3695
|
|
3459
3696
|
```
|
3460
|
-
|
3461
|
-
|
3697
|
+
<%=cmd%> config gem_path
|
3698
|
+
cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
|
3462
3699
|
```
|
3463
3700
|
|
3464
3701
|
This sample code shows some example of use of the API as well as
|
@@ -3495,130 +3732,141 @@ So, it evolved into <%=tool%>:
|
|
3495
3732
|
|
3496
3733
|
* <%=gemspec.version.to_s%>
|
3497
3734
|
|
3498
|
-
|
3499
|
-
|
3500
|
-
|
3501
|
-
|
3502
|
-
|
3503
|
-
|
3504
|
-
|
3735
|
+
* new: support transfer agent: [Transfer SDK](#agt_trsdk)
|
3736
|
+
* new: support [http socket options](#http_options)
|
3737
|
+
* new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
|
3738
|
+
* new: `config vault` supports encrypted passwords, also macos keychain
|
3739
|
+
* new: `config preset` command for consistency with id
|
3740
|
+
* new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
|
3741
|
+
* change: when using wss, use [ruby's CA certs](#certificates)
|
3742
|
+
* change: unexpected parameter makes exit code not zero
|
3743
|
+
* change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
|
3744
|
+
* change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
|
3745
|
+
* change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
|
3746
|
+
* fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
|
3747
|
+
|
3748
|
+
* 4.4.0
|
3749
|
+
|
3750
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
3751
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
3752
|
+
* new: `preset` option can specify name or hash value
|
3753
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
3754
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
3755
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
3756
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
3505
3757
|
|
3506
3758
|
* 4.3.0
|
3507
3759
|
|
3508
|
-
|
3509
|
-
|
3760
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
|
3761
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
3510
3762
|
|
3511
3763
|
* 4.2.2
|
3512
3764
|
|
3513
|
-
|
3514
|
-
|
3515
|
-
|
3516
|
-
|
3517
|
-
|
3518
|
-
|
3519
|
-
|
3520
|
-
|
3521
|
-
|
3765
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
3766
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
3767
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
3768
|
+
* new: `conf ascp spec` displays supported transfer spec
|
3769
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
3770
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
3771
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
3772
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
|
3773
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
3522
3774
|
|
3523
3775
|
* 4.2.1
|
3524
3776
|
|
3525
|
-
|
3526
|
-
|
3777
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
3778
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
3527
3779
|
|
3528
3780
|
* 4.2.0
|
3529
3781
|
|
3530
|
-
|
3531
|
-
|
3532
|
-
|
3533
|
-
|
3534
|
-
|
3535
|
-
|
3536
|
-
|
3537
|
-
|
3538
|
-
|
3539
|
-
|
3782
|
+
* new: command `aoc remind` to receive organization membership by email
|
3783
|
+
* new: in `preview` option `value` to filter out on file name
|
3784
|
+
* new: `initdemo` to initialize for demo server
|
3785
|
+
* new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
3786
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
3787
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
3788
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
3789
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
3790
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
3791
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
|
3540
3792
|
|
3541
3793
|
* 4.1.0
|
3542
3794
|
|
3543
|
-
|
3544
|
-
|
3545
|
-
|
3546
|
-
|
3547
|
-
|
3548
|
-
|
3549
|
-
|
3550
|
-
|
3551
|
-
|
3552
|
-
|
3553
|
-
|
3554
|
-
|
3795
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
3796
|
+
* new: update documentation with regard to offline and docker installation
|
3797
|
+
* new: renamed command `nagios_check` to `health`
|
3798
|
+
* new: agent `http_gw` now supports upload
|
3799
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
3800
|
+
* new: check new gem version periodically
|
3801
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
3802
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
3803
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
3804
|
+
* new: AoC a password can be provided for a public link
|
3805
|
+
* new: `conf doc` take an optional parameter to go to a section
|
3806
|
+
* new: initial support for Faspex 5 Beta 1
|
3555
3807
|
|
3556
3808
|
* 4.0.0
|
3557
3809
|
|
3558
|
-
|
3559
|
-
|
3560
|
-
|
3561
|
-
|
3562
|
-
|
3563
|
-
|
3564
|
-
|
3565
|
-
|
3810
|
+
* now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
|
3811
|
+
* changed default tool name from `mlia` to `ascli`
|
3812
|
+
* changed `aspera` command to `aoc`
|
3813
|
+
* changed gem name from `asperalm` to `aspera-cli`
|
3814
|
+
* changed module name from `Asperalm` to `Aspera`
|
3815
|
+
* removed command `folder` in `preview`, merged to `scan`
|
3816
|
+
* persistency files go to sub folder instead of main folder
|
3817
|
+
* added possibility to install SDK: `config ascp install`
|
3566
3818
|
|
3567
3819
|
* 0.11.8
|
3568
3820
|
|
3569
|
-
|
3821
|
+
* Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
|
3570
3822
|
|
3571
3823
|
* 0.11.7
|
3572
3824
|
|
3573
|
-
|
3574
|
-
|
3575
|
-
|
3576
|
-
|
3577
|
-
|
3578
|
-
|
3825
|
+
* rework on rest call error handling
|
3826
|
+
* use option `display` with value `data` to remove out of extraneous information
|
3827
|
+
* fixed option `lock_port` not working
|
3828
|
+
* generate special icon if preview failed
|
3829
|
+
* possibility to choose transfer progress bar type with option `progress`
|
3830
|
+
* AoC package creation now output package id
|
3579
3831
|
|
3580
3832
|
* 0.11.6
|
3581
3833
|
|
3582
|
-
|
3583
|
-
|
3584
|
-
|
3585
|
-
|
3586
|
-
|
3587
|
-
|
3588
|
-
|
3834
|
+
* orchestrator : added more choice in auth type
|
3835
|
+
* preview: cleanup in generator (removed and renamed parameters)
|
3836
|
+
* preview: better documentation
|
3837
|
+
* preview: animated thumbnails for video (option: `video_png_conv=animated`)
|
3838
|
+
* preview: new event trigger: `trevents` (`events` seems broken)
|
3839
|
+
* preview: unique tmp folder to avoid clash of multiple instances
|
3840
|
+
* repo: added template for secrets used for testing
|
3589
3841
|
|
3590
3842
|
* 0.11.5
|
3591
3843
|
|
3592
|
-
|
3593
|
-
|
3594
|
-
|
3595
|
-
|
3596
|
-
|
3597
|
-
|
3844
|
+
* added option `default_ports` for AoC (see manual)
|
3845
|
+
* allow bulk delete in `aspera files` with option `bulk=yes`
|
3846
|
+
* fix getting connect versions
|
3847
|
+
* added section for Aix
|
3848
|
+
* support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
|
3849
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
|
3598
3850
|
|
3599
3851
|
* 0.11.4
|
3600
3852
|
|
3601
|
-
|
3853
|
+
* possibility to give shared inbox name when sending a package (else use id and type)
|
3602
3854
|
|
3603
3855
|
* 0.11.3
|
3604
3856
|
|
3605
|
-
|
3857
|
+
* minor fixes on multi-session: avoid exception on progress bar
|
3606
3858
|
|
3607
3859
|
* 0.11.2
|
3608
3860
|
|
3609
|
-
|
3861
|
+
* fixes on multi-session: progress bat and transfer spec param for "direct"
|
3610
3862
|
|
3611
3863
|
* 0.11.1
|
3612
3864
|
|
3613
|
-
|
3865
|
+
* enhanced short_link creation commands (see examples)
|
3614
3866
|
|
3615
3867
|
* 0.11
|
3616
3868
|
|
3617
|
-
|
3618
|
-
|
3619
|
-
```
|
3620
|
-
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
|
3621
|
-
```
|
3869
|
+
* add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
|
3622
3870
|
|
3623
3871
|
* 0.10.18
|
3624
3872
|
|
@@ -3627,7 +3875,7 @@ So, it evolved into <%=tool%>:
|
|
3627
3875
|
* 0.10.17
|
3628
3876
|
|
3629
3877
|
* fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
|
3630
|
-
* new modifier: `@list:<
|
3878
|
+
* new modifier: `@list:<separator>val1<separator>...`
|
3631
3879
|
|
3632
3880
|
* 0.10.16
|
3633
3881
|
|
@@ -3649,7 +3897,7 @@ So, it evolved into <%=tool%>:
|
|
3649
3897
|
* 0.10.12
|
3650
3898
|
|
3651
3899
|
* added support for AoC node registration keys
|
3652
|
-
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#
|
3900
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
|
3653
3901
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
3654
3902
|
* `@incps` : new extended value modifier
|
3655
3903
|
* ATS: no more provides access keys secrets: now user must provide it
|
@@ -3681,7 +3929,7 @@ So, it evolved into <%=tool%>:
|
|
3681
3929
|
|
3682
3930
|
* 0.10.6
|
3683
3931
|
|
3684
|
-
* FaspManager: transfer spec `authentication` no more needed for local
|
3932
|
+
* FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
3685
3933
|
* gem version requirements made more open
|
3686
3934
|
|
3687
3935
|
* 0.10.5
|
@@ -3831,10 +4079,10 @@ So, it evolved into <%=tool%>:
|
|
3831
4079
|
|
3832
4080
|
* 0.9.7
|
3833
4081
|
|
3834
|
-
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#
|
4082
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
|
3835
4083
|
* preview persistency goes to unique file by default
|
3836
4084
|
* catch mxf extension in preview as video
|
3837
|
-
* Faspex: possibility to download all
|
4085
|
+
* Faspex: possibility to download all packages by specifying id=ALL
|
3838
4086
|
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
3839
4087
|
|
3840
4088
|
* 0.9.6
|
@@ -3914,16 +4162,16 @@ For issues or feature requests use the Github repository and issues.
|
|
3914
4162
|
|
3915
4163
|
You can also contribute to this open source project.
|
3916
4164
|
|
3917
|
-
One can also create one's own
|
4165
|
+
One can also [create one's own plugin](#createownplugin).
|
3918
4166
|
|
3919
4167
|
## Only one value for any option
|
3920
4168
|
|
3921
4169
|
Some commands and sub commands may ask for the same option name.
|
3922
|
-
Currently, since option definition is position
|
4170
|
+
Currently, since option definition is position independent (last one wins), it is not possible
|
3923
4171
|
to give an option to a command and the same option with different value to a sub command.
|
3924
4172
|
|
3925
|
-
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
|
3926
|
-
As a
|
4173
|
+
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
|
4174
|
+
As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
|
3927
4175
|
|
3928
4176
|
This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
|
3929
4177
|
|
@@ -3955,7 +4203,7 @@ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or
|
|
3955
4203
|
|
3956
4204
|
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
3957
4205
|
|
3958
|
-
##
|
4206
|
+
## Miscellaneous
|
3959
4207
|
|
3960
4208
|
* remove rest and oauth classes and use ruby standard gems:
|
3961
4209
|
|