aspera-cli 4.4.0 → 4.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +1042 -787
- data/bin/ascli +1 -1
- data/bin/asession +3 -5
- data/docs/Makefile +4 -7
- data/docs/README.erb.md +988 -740
- data/examples/faspex4.rb +4 -6
- data/examples/transfer.rb +2 -2
- data/lib/aspera/aoc.rb +139 -118
- data/lib/aspera/cli/listener/progress_multi.rb +5 -5
- data/lib/aspera/cli/main.rb +64 -34
- data/lib/aspera/cli/manager.rb +19 -20
- data/lib/aspera/cli/plugin.rb +9 -1
- data/lib/aspera/cli/plugins/aoc.rb +156 -143
- data/lib/aspera/cli/plugins/ats.rb +11 -10
- data/lib/aspera/cli/plugins/bss.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +236 -112
- data/lib/aspera/cli/plugins/faspex.rb +29 -7
- data/lib/aspera/cli/plugins/faspex5.rb +21 -8
- data/lib/aspera/cli/plugins/node.rb +21 -9
- data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
- data/lib/aspera/cli/plugins/preview.rb +2 -2
- data/lib/aspera/cli/plugins/server.rb +3 -3
- data/lib/aspera/cli/plugins/shares.rb +17 -0
- data/lib/aspera/cli/transfer_agent.rb +47 -85
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/environment.rb +4 -4
- data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
- data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
- data/lib/aspera/fasp/{local.rb → agent_direct.rb} +14 -17
- data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +4 -4
- data/lib/aspera/fasp/{node.rb → agent_node.rb} +25 -8
- data/lib/aspera/fasp/agent_trsdk.rb +106 -0
- data/lib/aspera/fasp/default.rb +17 -0
- data/lib/aspera/fasp/installation.rb +64 -48
- data/lib/aspera/fasp/parameters.rb +7 -3
- data/lib/aspera/faspex_gw.rb +6 -6
- data/lib/aspera/keychain/encrypted_hash.rb +120 -0
- data/lib/aspera/keychain/macos_security.rb +94 -0
- data/lib/aspera/log.rb +45 -32
- data/lib/aspera/node.rb +3 -6
- data/lib/aspera/rest.rb +65 -49
- metadata +68 -27
- data/lib/aspera/api_detector.rb +0 -60
- data/lib/aspera/secrets.rb +0 -20
data/README.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
2
2
|
<font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
|
3
3
|
|
4
|
-
Version : 4.
|
4
|
+
Version : 4.5.0
|
5
5
|
|
6
6
|
_Laurent/2016-2021_
|
7
7
|
|
@@ -15,7 +15,9 @@ Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/ge
|
|
15
15
|
|
16
16
|
Required Ruby version: > 2.4
|
17
17
|
|
18
|
-
|
18
|
+
[Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
19
|
+
|
20
|
+
# <a id="when_to_use"></a>When to use and when not to use
|
19
21
|
|
20
22
|
`ascli` is designed to be used as a command line tool to:
|
21
23
|
|
@@ -38,21 +40,18 @@ One might be tempted to use it as an integration element, e.g. by building a com
|
|
38
40
|
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
39
41
|
|
40
42
|
* Product APIs (REST) : e.g. AoC, Faspex, node
|
41
|
-
* Transfer SDK : with gRPC interface and
|
43
|
+
* Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
42
44
|
|
43
45
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
44
46
|
|
45
47
|
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
46
48
|
|
47
|
-
# Notations
|
49
|
+
# <a id="parsing"></a>Notations, Shell and Command line parsing
|
48
50
|
|
49
|
-
In examples, command line operations
|
50
|
-
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
51
|
+
In examples, command line operations are shown using a shell such: `bash` or `zsh`.
|
51
52
|
|
52
53
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
53
54
|
|
54
|
-
# <a name="parsing"></a>Shell and Command line parsing
|
55
|
-
|
56
55
|
`ascli` is typically executed in a shell, either interactively or in a script. `ascli` receives its arguments from this shell.
|
57
56
|
|
58
57
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
@@ -65,12 +64,15 @@ On Windows, `cmd.exe` is typically used. Windows process creation does not recei
|
|
65
64
|
In case of doubt of argument values after parsing test like this:
|
66
65
|
|
67
66
|
```
|
68
|
-
|
67
|
+
ascli conf echo "Hello World" arg2 3
|
68
|
+
```
|
69
|
+
|
70
|
+
```
|
69
71
|
"Hello World"
|
70
72
|
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
71
73
|
```
|
72
74
|
|
73
|
-
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
|
75
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (`ascli`), next command line arguments are shown in the error message.
|
74
76
|
|
75
77
|
# Quick Start
|
76
78
|
|
@@ -81,8 +83,11 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
|
|
81
83
|
Once the gem is installed, `ascli` shall be accessible:
|
82
84
|
|
83
85
|
```
|
84
|
-
|
85
|
-
|
86
|
+
ascli --version
|
87
|
+
```
|
88
|
+
|
89
|
+
```
|
90
|
+
4.5.0
|
86
91
|
```
|
87
92
|
|
88
93
|
## First use
|
@@ -94,8 +99,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
94
99
|
To test with Aspera demo transfer server, setup the environment and then test:
|
95
100
|
|
96
101
|
```
|
97
|
-
|
98
|
-
|
102
|
+
ascli config initdemo
|
103
|
+
```
|
104
|
+
|
105
|
+
```
|
106
|
+
ascli server browse /
|
107
|
+
```
|
108
|
+
|
109
|
+
```
|
99
110
|
:............:...........:......:........:...........................:.......................:
|
100
111
|
: zmode : zuid : zgid : size : mtime : name :
|
101
112
|
:............:...........:......:........:...........................:.......................:
|
@@ -114,11 +125,26 @@ If you want to use `ascli` with another server, and in order to make further cal
|
|
114
125
|
* download a file
|
115
126
|
|
116
127
|
```
|
117
|
-
|
128
|
+
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
129
|
+
```
|
130
|
+
|
131
|
+
```
|
118
132
|
updated: myserver
|
119
|
-
|
133
|
+
```
|
134
|
+
|
135
|
+
```
|
136
|
+
ascli config preset set default server myserver
|
137
|
+
```
|
138
|
+
|
139
|
+
```
|
120
140
|
updated: default→server to myserver
|
121
|
-
|
141
|
+
```
|
142
|
+
|
143
|
+
```
|
144
|
+
ascli server browse /aspera-test-dir-large
|
145
|
+
```
|
146
|
+
|
147
|
+
```
|
122
148
|
:............:...........:......:..............:...........................:............................:
|
123
149
|
: zmode : zuid : zgid : size : mtime : name :
|
124
150
|
:............:...........:......:..............:...........................:............................:
|
@@ -135,7 +161,13 @@ $ ascli server browse /aspera-test-dir-large
|
|
135
161
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
136
162
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
137
163
|
:............:...........:......:..............:...........................:............................:
|
138
|
-
|
164
|
+
```
|
165
|
+
|
166
|
+
```
|
167
|
+
ascli server download /aspera-test-dir-large/200MB
|
168
|
+
```
|
169
|
+
|
170
|
+
```
|
139
171
|
Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
|
140
172
|
complete
|
141
173
|
```
|
@@ -146,9 +178,9 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
|
|
146
178
|
|
147
179
|
Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
148
180
|
|
149
|
-
# <a
|
181
|
+
# <a id="installation"></a>Installation
|
150
182
|
|
151
|
-
It is possible to install *either* directly on the host operating system (Linux, Windows,
|
183
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
|
152
184
|
|
153
185
|
The direct installation is recommended and consists in installing:
|
154
186
|
|
@@ -158,33 +190,36 @@ The direct installation is recommended and consists in installing:
|
|
158
190
|
|
159
191
|
The following sections provide information on the various installation methods.
|
160
192
|
|
161
|
-
An internet connection is required for the installation. If you
|
193
|
+
An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
162
194
|
|
163
195
|
## Docker container
|
164
196
|
|
165
197
|
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
166
198
|
|
167
|
-
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
199
|
+
This method installs a docker image that contains: Ruby, `ascli` and the FASP sdk.
|
168
200
|
|
169
201
|
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
170
202
|
|
171
203
|
Ensure that you have Docker installed.
|
172
204
|
|
173
205
|
```
|
174
|
-
|
206
|
+
docker --version
|
175
207
|
```
|
176
208
|
|
177
209
|
Download the wrapping script:
|
178
210
|
|
179
211
|
```
|
180
|
-
|
181
|
-
|
212
|
+
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
213
|
+
```
|
214
|
+
|
215
|
+
```
|
216
|
+
chmod a+x ascli
|
182
217
|
```
|
183
218
|
|
184
219
|
Install the container image:
|
185
220
|
|
186
221
|
```
|
187
|
-
|
222
|
+
./ascli install
|
188
223
|
```
|
189
224
|
|
190
225
|
Start using it !
|
@@ -194,9 +229,9 @@ Note that the tool is run in the container, so transfers are also executed in th
|
|
194
229
|
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/ascli` on host.
|
195
230
|
|
196
231
|
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
197
|
-
To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
|
232
|
+
To add local storage as a volume edit the script: `ascli` and add a `--volume` stanza.
|
198
233
|
|
199
|
-
## <a
|
234
|
+
## <a id="ruby"></a>Ruby
|
200
235
|
|
201
236
|
Use this method to install on the native host.
|
202
237
|
|
@@ -208,7 +243,7 @@ Required Ruby version: > 2.4. Ruby version 3 is also supported.
|
|
208
243
|
|
209
244
|
Refer to the following sections for a proposed method for specific operating systems.
|
210
245
|
|
211
|
-
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux,
|
246
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
212
247
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
213
248
|
If you have a simpler better way to install Ruby version > 2.4 : use it !
|
214
249
|
|
@@ -221,31 +256,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
|
221
256
|
Install the 2 keys
|
222
257
|
|
223
258
|
```
|
224
|
-
|
259
|
+
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
225
260
|
```
|
226
261
|
|
227
262
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
228
263
|
|
229
264
|
```
|
230
|
-
|
265
|
+
\curl -sSL https://get.rvm.io | bash -s stable
|
231
266
|
```
|
232
267
|
|
233
|
-
If you keep the same terminal (
|
268
|
+
If you keep the same terminal (not needed if re-login):
|
234
269
|
|
235
270
|
```
|
236
|
-
|
271
|
+
source ~/.rvm/scripts/rvm
|
237
272
|
```
|
238
273
|
|
239
274
|
It is advised to get one of the pre-compiled ruby version, you can list with:
|
240
275
|
|
241
276
|
```
|
242
|
-
|
277
|
+
rvm list --remote
|
243
278
|
```
|
244
279
|
|
245
280
|
Install the chosen pre-compiled Ruby version:
|
246
281
|
|
247
282
|
```
|
248
|
-
|
283
|
+
rvm install 2.7.2 --binary
|
249
284
|
```
|
250
285
|
|
251
286
|
Ruby is now installed for the user, go on to Gem installation.
|
@@ -258,7 +293,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
|
|
258
293
|
One can install in another location with :
|
259
294
|
|
260
295
|
```
|
261
|
-
|
296
|
+
curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
262
297
|
```
|
263
298
|
|
264
299
|
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
@@ -266,8 +301,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
|
|
266
301
|
To activate ruby (and ascli) later, source it:
|
267
302
|
|
268
303
|
```
|
269
|
-
|
270
|
-
|
304
|
+
source /etc/profile.d/rvm.sh.ok
|
305
|
+
```
|
306
|
+
|
307
|
+
```
|
308
|
+
rvm version
|
271
309
|
```
|
272
310
|
|
273
311
|
### Windows: Installer
|
@@ -280,16 +318,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
|
|
280
318
|
|
281
319
|
### macOS: pre-installed or `brew`
|
282
320
|
|
283
|
-
|
321
|
+
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
284
322
|
|
285
323
|
```
|
286
|
-
|
324
|
+
sudo gem install aspera-cli
|
287
325
|
```
|
288
326
|
|
289
327
|
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
290
328
|
|
291
329
|
```
|
292
|
-
|
330
|
+
brew install ruby
|
293
331
|
```
|
294
332
|
|
295
333
|
### Linux: package
|
@@ -299,90 +337,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
|
|
299
337
|
Example:
|
300
338
|
|
301
339
|
```
|
302
|
-
|
340
|
+
yum install -y ruby rubygems ruby-json
|
303
341
|
```
|
304
342
|
|
305
343
|
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
306
344
|
|
307
345
|
```
|
308
346
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
347
|
+
```
|
348
|
+
|
349
|
+
```
|
309
350
|
yum remove -y ruby ruby-libs
|
310
351
|
```
|
311
352
|
|
312
353
|
### Other Unixes: Aix, etc...
|
313
354
|
|
314
|
-
If your
|
355
|
+
If your Unix does not provide a pre-built ruby, you can get it using one of those
|
315
356
|
[methods](https://www.ruby-lang.org/en/documentation/installation/)
|
316
357
|
|
317
358
|
For instance to build from source, and install in `/opt/ruby` :
|
318
359
|
|
319
360
|
```
|
320
|
-
|
321
|
-
# gzip -d ruby-2.7.2.tar.gz
|
322
|
-
# tar xvf ruby-2.7.2.tar
|
323
|
-
# cd ruby-2.7.2
|
324
|
-
# ./configure --prefix=/opt/ruby
|
325
|
-
# make ruby.imp
|
326
|
-
# make
|
327
|
-
# make install
|
328
|
-
```
|
329
|
-
|
330
|
-
### <a name="offline_install"></a>Installation without internet access
|
331
|
-
|
332
|
-
Note that currently no pre-packaged version exist yet.
|
333
|
-
A method to build one provided here:
|
334
|
-
|
335
|
-
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
336
|
-
|
337
|
-
Then create an archive:
|
338
|
-
|
339
|
-
```
|
340
|
-
$ cd
|
341
|
-
$ tar zcvf rvm-ascli.tgz .rvm
|
342
|
-
```
|
343
|
-
|
344
|
-
Get the Aspera SDK. Execute:
|
345
|
-
|
346
|
-
```
|
347
|
-
$ ascli conf --show-config|grep sdk_url
|
348
|
-
```
|
349
|
-
|
350
|
-
Then download the SDK archive from that URL.
|
361
|
+
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
|
351
362
|
|
352
|
-
|
363
|
+
gzip -d ruby-2.7.2.tar.gz
|
353
364
|
|
354
|
-
|
365
|
+
tar xvf ruby-2.7.2.tar
|
355
366
|
|
356
|
-
|
367
|
+
cd ruby-2.7.2
|
357
368
|
|
358
|
-
|
359
|
-
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
369
|
+
./configure --prefix=/opt/ruby
|
360
370
|
|
361
|
-
|
362
|
-
source path_to_rvm_root/scripts/rvm
|
363
|
-
rvm use 2.7.2
|
364
|
-
```
|
371
|
+
make ruby.imp
|
365
372
|
|
366
|
-
|
373
|
+
make
|
367
374
|
|
375
|
+
make install
|
368
376
|
```
|
369
|
-
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
370
|
-
```
|
371
|
-
|
372
|
-
or restore the `$HOME/.aspera` folder for the user.
|
373
377
|
|
374
|
-
## <a
|
378
|
+
## <a id="the_gem"></a>`aspera-cli` gem
|
375
379
|
|
376
380
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
377
381
|
|
378
382
|
```
|
379
|
-
|
383
|
+
gem install aspera-cli
|
380
384
|
```
|
381
385
|
|
382
386
|
To upgrade to the latest version:
|
383
387
|
|
384
388
|
```
|
385
|
-
|
389
|
+
gem update aspera-cli
|
386
390
|
```
|
387
391
|
|
388
392
|
`ascli` checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
@@ -390,12 +394,10 @@ To upgrade to the latest version:
|
|
390
394
|
To check manually:
|
391
395
|
|
392
396
|
```
|
393
|
-
|
397
|
+
ascli conf check_update
|
394
398
|
```
|
395
399
|
|
396
|
-
|
397
|
-
|
398
|
-
## <a name="fasp_prot"></a>FASP Protocol
|
400
|
+
## <a id="fasp_prot"></a>FASP Protocol
|
399
401
|
|
400
402
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
401
403
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
@@ -403,17 +405,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
|
|
403
405
|
* ascp
|
404
406
|
* aspera-license (in same folder, or ../etc)
|
405
407
|
|
406
|
-
This can be installed either be installing an Aspera transfer
|
408
|
+
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
407
409
|
|
408
410
|
```
|
409
|
-
|
411
|
+
ascli conf ascp install
|
410
412
|
```
|
411
413
|
|
412
|
-
If a local SDK installation is
|
414
|
+
If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
|
413
415
|
|
414
416
|
```
|
415
|
-
|
416
|
-
|
417
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
418
|
+
```
|
419
|
+
|
420
|
+
```
|
421
|
+
ascli conf ascp install --sdk-url=file:///SDK.zip
|
417
422
|
```
|
418
423
|
|
419
424
|
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
@@ -433,36 +438,55 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
433
438
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
434
439
|
|
435
440
|
Several methods are provided to start a transfer.
|
436
|
-
Use of a local client ([`direct`](#
|
441
|
+
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
437
442
|
|
438
|
-
## <a
|
443
|
+
## <a id="offline_install"></a>Installation in air gapped environment
|
439
444
|
|
440
|
-
|
445
|
+
Note that currently no pre-packaged version exist yet.
|
446
|
+
A method to build one is provided here:
|
447
|
+
|
448
|
+
The procedure:
|
441
449
|
|
442
450
|
* Follow the non-root installation procedure with RVM, including gem
|
443
|
-
*
|
451
|
+
* Archive (zip, tar) the main RVM folder (includes ascli):
|
444
452
|
|
445
453
|
```
|
446
|
-
$
|
447
|
-
$ tar zcvf rvm_ascli.tgz .rvm
|
454
|
+
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
448
455
|
```
|
449
456
|
|
450
|
-
*
|
457
|
+
* Get the Aspera SDK.
|
451
458
|
|
452
459
|
```
|
453
|
-
|
460
|
+
ascli conf --show-config --fields=sdk_url
|
454
461
|
```
|
455
462
|
|
456
|
-
*
|
463
|
+
* Download the SDK archive from that URL.
|
457
464
|
|
458
465
|
```
|
459
|
-
|
460
|
-
$ tar zxvf rvm_ascli.tgz
|
461
|
-
$ source ~/.rvm/scripts/rvm
|
462
|
-
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
466
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
463
467
|
```
|
464
468
|
|
465
|
-
|
469
|
+
* Transfer those 2 files to the target system
|
470
|
+
|
471
|
+
* On target system
|
472
|
+
|
473
|
+
```
|
474
|
+
cd $HOME
|
475
|
+
|
476
|
+
tar zxvf rvm-ascli.tgz
|
477
|
+
|
478
|
+
source ~/.rvm/scripts/rvm
|
479
|
+
|
480
|
+
ascli conf ascp install --sdk-url=file:///SDK.zip
|
481
|
+
```
|
482
|
+
|
483
|
+
* Add those lines to shell init (`.profile`)
|
484
|
+
|
485
|
+
```
|
486
|
+
source ~/.rvm/scripts/rvm
|
487
|
+
```
|
488
|
+
|
489
|
+
# <a id="cli"></a>Command Line Interface: `ascli`
|
466
490
|
|
467
491
|
The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
468
492
|
|
@@ -491,7 +515,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
491
515
|
Basic usage is displayed by executing:
|
492
516
|
|
493
517
|
```
|
494
|
-
|
518
|
+
ascli -h
|
495
519
|
```
|
496
520
|
|
497
521
|
Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
|
@@ -505,7 +529,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
|
|
505
529
|
There are two types of arguments: Commands and Options. Example :
|
506
530
|
|
507
531
|
```
|
508
|
-
|
532
|
+
ascli command --option-name=VAL1 VAL2
|
509
533
|
```
|
510
534
|
|
511
535
|
* executes _command_: `command`
|
@@ -533,13 +557,16 @@ Exceptions:
|
|
533
557
|
* the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
534
558
|
|
535
559
|
```
|
536
|
-
|
560
|
+
ascli config echo -- --sample
|
561
|
+
```
|
562
|
+
|
563
|
+
```
|
537
564
|
"--sample"
|
538
565
|
```
|
539
566
|
|
540
567
|
Note that `--sample` is taken as an argument, and not option.
|
541
568
|
|
542
|
-
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on
|
569
|
+
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
543
570
|
|
544
571
|
The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
|
545
572
|
|
@@ -560,7 +587,7 @@ Command line arguments that are not options are either commands or arguments. If
|
|
560
587
|
|
561
588
|
Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
|
562
589
|
|
563
|
-
The
|
590
|
+
The behavior can be controlled with:
|
564
591
|
|
565
592
|
* --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
566
593
|
* yes : missing mandatory parameters/options are asked to the user
|
@@ -577,8 +604,8 @@ The information displayed depends on the action.
|
|
577
604
|
|
578
605
|
Depending on action, the output will contain:
|
579
606
|
|
580
|
-
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is
|
581
|
-
* `object_list` : displayed as a 2 dimensional table: one line per item, one
|
607
|
+
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
608
|
+
* `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
582
609
|
* `value_list` : a table with one column.
|
583
610
|
* `empty` : nothing
|
584
611
|
* `status` : a message
|
@@ -603,12 +630,15 @@ The style of output can be set using the `format` parameter, supporting:
|
|
603
630
|
* `yaml` : YAML
|
604
631
|
* `csv` : Comma Separated Values
|
605
632
|
|
606
|
-
### <a
|
633
|
+
### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
607
634
|
|
608
635
|
Table output can be filtered using the `select` parameter. Example:
|
609
636
|
|
610
637
|
```
|
611
|
-
|
638
|
+
ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
639
|
+
```
|
640
|
+
|
641
|
+
```
|
612
642
|
:...............................:..................................:...........:
|
613
643
|
: name : email : ats_admin :
|
614
644
|
:...............................:..................................:...........:
|
@@ -621,7 +651,7 @@ Note that `select` filters selected elements from the result of API calls, while
|
|
621
651
|
|
622
652
|
### Verbosity of output
|
623
653
|
|
624
|
-
|
654
|
+
Output messages are categorized in 3 types:
|
625
655
|
|
626
656
|
* `info` output contain additional information, such as number of elements in a table
|
627
657
|
* `data` output contain the actual output of the command (object, or list of objects)
|
@@ -635,7 +665,7 @@ The option `display` controls the level of output:
|
|
635
665
|
|
636
666
|
### Selection of output object properties
|
637
667
|
|
638
|
-
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific
|
668
|
+
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
639
669
|
|
640
670
|
* DEF : default display of columns (that's the default, when not set)
|
641
671
|
* ALL : all columns available
|
@@ -644,7 +674,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
644
674
|
* +a,b,c : add selected properties to the default selection.
|
645
675
|
* -a,b,c : remove selected properties from the default selection.
|
646
676
|
|
647
|
-
## <a
|
677
|
+
## <a id="extended"></a>Extended Value Syntax
|
648
678
|
|
649
679
|
Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
|
650
680
|
|
@@ -681,23 +711,32 @@ To display the result of an extended value, use the `config echo` command.
|
|
681
711
|
Example: read the content of the specified file, then, base64 decode, then unzip:
|
682
712
|
|
683
713
|
```
|
684
|
-
|
714
|
+
ascli config echo @zlib:@base64:@file:myfile.dat
|
685
715
|
```
|
686
716
|
|
687
717
|
Example: create a value as a hash, with one key and the value is read from a file:
|
688
718
|
|
689
719
|
```
|
690
|
-
|
720
|
+
ascli config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
|
691
721
|
```
|
692
722
|
|
693
723
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
694
724
|
|
695
725
|
```
|
696
|
-
|
726
|
+
cat test.csv
|
727
|
+
```
|
728
|
+
|
729
|
+
```
|
697
730
|
name,email
|
698
731
|
lolo,laurent@example.com
|
699
732
|
toto,titi@tutu.tata
|
700
|
-
|
733
|
+
```
|
734
|
+
|
735
|
+
```
|
736
|
+
ascli config echo @csvt:@file:test.csv
|
737
|
+
```
|
738
|
+
|
739
|
+
```
|
701
740
|
:......:.....................:
|
702
741
|
: name : email :
|
703
742
|
:......:.....................:
|
@@ -709,13 +748,16 @@ $ ascli config echo @csvt:@file:test.csv
|
|
709
748
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
710
749
|
|
711
750
|
```
|
712
|
-
|
751
|
+
ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
752
|
+
```
|
753
|
+
|
754
|
+
```
|
713
755
|
{"version"=>"0.9", "hello"=>true}
|
714
756
|
```
|
715
757
|
|
716
758
|
Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
|
717
759
|
|
718
|
-
## <a
|
760
|
+
## <a id="native"></a>Structured Value
|
719
761
|
|
720
762
|
Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
|
721
763
|
|
@@ -726,7 +768,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
|
|
726
768
|
|
727
769
|
It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
|
728
770
|
|
729
|
-
## <a
|
771
|
+
## <a id="conffolder"></a>Configuration and Persistency Folder
|
730
772
|
|
731
773
|
`ascli` configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/ascli`.
|
732
774
|
|
@@ -736,49 +778,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
|
|
736
778
|
The main folder can be displayed using :
|
737
779
|
|
738
780
|
```
|
739
|
-
|
781
|
+
ascli config folder
|
782
|
+
```
|
783
|
+
|
784
|
+
```
|
740
785
|
/Users/kenji/.aspera/ascli
|
741
786
|
```
|
742
787
|
|
743
|
-
It can be
|
788
|
+
It can be overridden using the environment variable `ASCLI_HOME`.
|
744
789
|
|
745
790
|
Example (Windows):
|
746
791
|
|
747
792
|
```
|
748
|
-
|
749
|
-
|
793
|
+
set ASCLI_HOME=C:\Users\Kenji\.aspera\ascli
|
794
|
+
ascli config folder
|
750
795
|
C:\Users\Kenji\.aspera\ascli
|
751
796
|
```
|
752
797
|
|
753
|
-
## <a
|
798
|
+
## <a id="configfile"></a>Configuration file
|
754
799
|
|
755
800
|
On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
|
756
801
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
757
802
|
|
758
803
|
Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
|
759
804
|
|
760
|
-
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the
|
805
|
+
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuration file.
|
761
806
|
|
762
807
|
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
763
808
|
|
764
|
-
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be
|
809
|
+
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
|
765
810
|
|
766
811
|
The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
767
812
|
|
768
|
-
### <a
|
813
|
+
### <a id="lprt"></a>Option preset
|
769
814
|
|
770
815
|
A [option preset](#lprt) is simply a collection of parameters and their associated values in a named section in the configuration file.
|
771
816
|
|
772
817
|
A named [option preset](#lprt) can be modified directly using `ascli`, which will update the configuration file :
|
773
818
|
|
774
819
|
```
|
775
|
-
|
820
|
+
ascli config preset set|delete|show|initialize|update <option preset>
|
776
821
|
```
|
777
822
|
|
778
823
|
The command `update` allows the easy creation of [option preset](#lprt) by simply providing the options in their command line format, e.g. :
|
779
824
|
|
780
825
|
```
|
781
|
-
|
826
|
+
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
782
827
|
```
|
783
828
|
|
784
829
|
* This creates a [option preset](#lprt) `demo_server` with all provided options.
|
@@ -786,39 +831,48 @@ $ ascli config id demo_server update --url=ssh://demo.asperasoft.com:33001 --use
|
|
786
831
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
787
832
|
|
788
833
|
```
|
789
|
-
|
834
|
+
ascli config preset set demo_server password _demo_pass_
|
790
835
|
```
|
791
836
|
|
792
837
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
793
838
|
|
794
839
|
```
|
795
|
-
|
840
|
+
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
796
841
|
```
|
797
842
|
|
798
|
-
A
|
799
|
-
If necessary, the configuration file can be edited (or simply consulted) with:
|
843
|
+
A full terminal based overview of the configuration can be displayed using:
|
800
844
|
|
801
845
|
```
|
802
|
-
|
846
|
+
ascli config preset over
|
803
847
|
```
|
804
848
|
|
805
|
-
A
|
849
|
+
A list of [option preset](#lprt) can be displayed using:
|
806
850
|
|
807
851
|
```
|
808
|
-
|
852
|
+
ascli config preset list
|
809
853
|
```
|
810
854
|
|
811
|
-
A
|
855
|
+
A good practice is to not manually edit the configuration file and use modification commands instead.
|
856
|
+
If necessary, the configuration file can opened in a text editor with:
|
857
|
+
|
858
|
+
```
|
859
|
+
ascli config open
|
860
|
+
```
|
861
|
+
|
862
|
+
Older format for commands are still supported:
|
812
863
|
|
813
864
|
```
|
814
|
-
|
865
|
+
ascli config id <name> set|delete|show|initialize|update
|
866
|
+
ascli config over
|
867
|
+
ascli config list
|
815
868
|
```
|
816
869
|
|
817
|
-
|
870
|
+
|
871
|
+
### <a id="lprtconf"></a>Special Option preset: config
|
818
872
|
|
819
873
|
This preset name is reserved and contains a single key: `version`. This is the version of `ascli` which created the file.
|
820
874
|
|
821
|
-
### <a
|
875
|
+
### <a id="lprtdef"></a>Special Option preset: default
|
822
876
|
|
823
877
|
This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
|
824
878
|
|
@@ -829,16 +883,16 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
829
883
|
Operations on this preset are done using regular `config` operations:
|
830
884
|
|
831
885
|
```
|
832
|
-
|
833
|
-
|
886
|
+
ascli config preset set default _plugin_name_ _default_preset_for_plugin_
|
887
|
+
ascli config preset get default _plugin_name_
|
834
888
|
"_default_preset_for_plugin_"
|
835
889
|
```
|
836
890
|
|
837
|
-
### <a
|
891
|
+
### <a id="lplugconf"></a>Special Plugin: config
|
838
892
|
|
839
893
|
Plugin `config` (not to be confused with Option preset config) is used to configure `ascli` but it also contains global options.
|
840
894
|
|
841
|
-
When `ascli` starts, it
|
895
|
+
When `ascli` starts, it looks for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
842
896
|
|
843
897
|
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
844
898
|
|
@@ -877,10 +931,10 @@ The user may create as many [option presets](#lprt) as needed. For instance, a p
|
|
877
931
|
|
878
932
|
Values in the configuration also follow the [Extended Value Syntax](#extended).
|
879
933
|
|
880
|
-
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config
|
934
|
+
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
|
881
935
|
|
882
936
|
```
|
883
|
-
|
937
|
+
ascli config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/ascli/aocapikey"
|
884
938
|
```
|
885
939
|
|
886
940
|
This creates the [option preset](#lprt):
|
@@ -916,26 +970,29 @@ is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration
|
|
916
970
|
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
917
971
|
|
918
972
|
```
|
919
|
-
|
920
|
-
|
973
|
+
ascli config preset set cli_default interactive no
|
974
|
+
```
|
975
|
+
|
976
|
+
```
|
977
|
+
ascli config preset set default config cli_default
|
921
978
|
```
|
922
979
|
|
923
980
|
A [option preset](#lprt) value can be removed with `unset`:
|
924
981
|
|
925
982
|
```
|
926
|
-
|
983
|
+
ascli config preset unset cli_default interactive
|
927
984
|
```
|
928
985
|
|
929
986
|
Example: Define options using command line:
|
930
987
|
|
931
988
|
```
|
932
|
-
|
989
|
+
ascli -N --url=x --password=y --username=y node --show-config
|
933
990
|
```
|
934
991
|
|
935
992
|
Example: Define options using a hash:
|
936
993
|
|
937
994
|
```
|
938
|
-
|
995
|
+
ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
939
996
|
```
|
940
997
|
|
941
998
|
### Examples
|
@@ -945,44 +1002,120 @@ only username/password and url are required (either on command line, or from con
|
|
945
1002
|
Those can usually be provided on the command line:
|
946
1003
|
|
947
1004
|
```
|
948
|
-
|
1005
|
+
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
|
949
1006
|
```
|
950
1007
|
|
951
1008
|
This can also be provisioned in a config file:
|
952
1009
|
|
1010
|
+
* Build [option preset](#lprt)
|
1011
|
+
|
953
1012
|
```
|
954
|
-
|
955
|
-
|
956
|
-
|
957
|
-
4$ ascli config id default set shares shares06
|
958
|
-
5$ ascli config overview
|
959
|
-
6$ ascli shares repo browse /
|
1013
|
+
ascli config preset set shares06 url https://10.25.0.6
|
1014
|
+
ascli config preset set shares06 username john
|
1015
|
+
ascli config preset set shares06 password 4sp3ra
|
960
1016
|
```
|
961
1017
|
|
962
|
-
The three first commands build a [option preset](#lprt).
|
963
1018
|
Note that this can also be done with one single command:
|
964
1019
|
|
965
1020
|
```
|
966
|
-
|
1021
|
+
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
|
967
1022
|
```
|
968
1023
|
|
969
|
-
|
970
|
-
|
971
|
-
|
972
|
-
|
1024
|
+
or
|
1025
|
+
|
1026
|
+
```
|
1027
|
+
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
|
1028
|
+
```
|
1029
|
+
|
1030
|
+
* Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
1031
|
+
|
1032
|
+
```
|
1033
|
+
ascli config preset set default shares shares06
|
1034
|
+
```
|
1035
|
+
|
1036
|
+
* Display the content of configuration file in table format
|
1037
|
+
|
1038
|
+
```
|
1039
|
+
ascli config overview
|
1040
|
+
```
|
1041
|
+
|
1042
|
+
* Execute a command on the shares application using default parameters
|
1043
|
+
|
1044
|
+
```
|
1045
|
+
ascli shares repo browse /
|
1046
|
+
```
|
1047
|
+
|
1048
|
+
## <a id="vault"></a>Secret Vault
|
1049
|
+
|
1050
|
+
When a secret or password is needed, it is possible to store in the secret vault.
|
1051
|
+
|
1052
|
+
By default the vault is defined using option `secrets`.
|
1053
|
+
|
1054
|
+
### Using system keychain
|
1055
|
+
|
1056
|
+
Only on macOS.
|
1057
|
+
|
1058
|
+
It is possible to store secrets in macOS keychain (only read supported currently).
|
1059
|
+
|
1060
|
+
Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
|
1061
|
+
|
1062
|
+
### Modern config file format: encrypted in config file
|
973
1063
|
|
974
|
-
|
1064
|
+
It is possible to store and use secrets encrypted.
|
1065
|
+
For this use the `config vault` command.
|
975
1066
|
|
1067
|
+
The vault can be initialized with `config vault init`
|
1068
|
+
|
1069
|
+
Then secrets can be manipulated using commands:
|
1070
|
+
|
1071
|
+
* `set`
|
1072
|
+
* `get`
|
1073
|
+
* `list`
|
1074
|
+
* `delete`
|
1075
|
+
|
1076
|
+
Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
|
1077
|
+
|
1078
|
+
### Legacy config file format
|
1079
|
+
|
1080
|
+
The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
|
1081
|
+
|
1082
|
+
For example, choose a repository name, for example `my_secrets`, and populate it like this:
|
1083
|
+
|
1084
|
+
```
|
1085
|
+
ascli conf id my_secrets set 'access_key1' 'secret1'
|
1086
|
+
|
1087
|
+
ascli conf id my_secrets set 'access_key2' 'secret2'
|
1088
|
+
|
1089
|
+
ascli conf id default get config
|
1090
|
+
|
1091
|
+
cli_default
|
1092
|
+
```
|
1093
|
+
|
1094
|
+
Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
|
1095
|
+
So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
|
1096
|
+
|
1097
|
+
```
|
1098
|
+
ascli conf id cli_default set secrets @val:@preset:my_secrets
|
1099
|
+
```
|
1100
|
+
|
1101
|
+
A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
|
1102
|
+
|
1103
|
+
To test if a secret can be found use:
|
1104
|
+
|
1105
|
+
```
|
1106
|
+
ascli conf vault get --username=access_key1
|
1107
|
+
```
|
976
1108
|
|
977
1109
|
## Plugins
|
978
1110
|
|
979
1111
|
The CLI tool uses a plugin mechanism. The first level command (just after `ascli` on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
|
980
1112
|
For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
|
981
1113
|
|
982
|
-
### Create your own plugin
|
1114
|
+
### <a id="createownplugin"></a>Create your own plugin
|
1115
|
+
|
983
1116
|
```
|
984
|
-
|
985
|
-
|
1117
|
+
mkdir -p ~/.aspera/ascli/plugins
|
1118
|
+
cat<<EOF>~/.aspera/ascli/plugins/test.rb
|
986
1119
|
require 'aspera/cli/plugin'
|
987
1120
|
module Aspera
|
988
1121
|
module Cli
|
@@ -997,39 +1130,78 @@ end # Aspera
|
|
997
1130
|
EOF
|
998
1131
|
```
|
999
1132
|
|
1000
|
-
|
1133
|
+
### <a id="plugins"></a>Plugins: Application URL and Authentication
|
1001
1134
|
|
1002
|
-
|
1135
|
+
`ascli` comes with several Aspera application plugins.
|
1003
1136
|
|
1004
|
-
|
1137
|
+
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
1138
|
+
|
1139
|
+
Those are using options:
|
1140
|
+
|
1141
|
+
* url
|
1142
|
+
* username
|
1143
|
+
* password
|
1144
|
+
|
1145
|
+
Those can be provided using command line, parameter set, env var, see section above.
|
1146
|
+
|
1147
|
+
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
1148
|
+
|
1149
|
+
## Logging, Debugging
|
1150
|
+
|
1151
|
+
The gem is equipped with traces. By default logging level is `warn`.
|
1152
|
+
To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `ASCLI_LOG_LEVEL`, or parameter in con file).
|
1153
|
+
|
1154
|
+
It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
|
1155
|
+
|
1156
|
+
By default passwords and secrets are removed from logs.
|
1157
|
+
Use option `log_passwords` to change this behaviour.
|
1005
1158
|
|
1006
1159
|
## Learning Aspera Product APIs (REST)
|
1007
1160
|
|
1008
1161
|
This CLI uses REST APIs.
|
1009
|
-
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
|
1010
|
-
exact content or HTTP requests and responses.
|
1162
|
+
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
|
1011
1163
|
|
1012
1164
|
In order to get traces of execution, use argument : `--log-level=debug`
|
1013
1165
|
|
1014
|
-
## <a
|
1166
|
+
## <a id="http_options"></a>HTTP socket parameters
|
1167
|
+
|
1168
|
+
If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
|
1169
|
+
|
1170
|
+
Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
|
1171
|
+
|
1172
|
+
* `read_timeout` 60 sec
|
1173
|
+
* `write_timeout` 60 sec
|
1174
|
+
|
1175
|
+
Default values are the ones of Ruby.
|
1176
|
+
|
1177
|
+
Example:
|
1178
|
+
|
1179
|
+
```
|
1180
|
+
ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
1181
|
+
```
|
1182
|
+
|
1183
|
+
## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
|
1015
1184
|
|
1016
1185
|
Some actions may require the use of a graphical tool:
|
1017
1186
|
|
1018
1187
|
* a browser for Aspera on Cloud authentication (web auth method)
|
1019
1188
|
* a text editor for configuration file edition
|
1020
1189
|
|
1021
|
-
By default the CLI will assume that a graphical environment is available on windows,
|
1022
|
-
and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1190
|
+
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
1023
1191
|
It is also possible to force the graphical mode with option --ui :
|
1024
1192
|
|
1025
|
-
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
|
1026
|
-
a text
|
1027
|
-
* `--ui=text` forces a text environment, the URL or file path to open is displayed on
|
1028
|
-
terminal.
|
1193
|
+
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
1194
|
+
* `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
1029
1195
|
|
1030
1196
|
## HTTP proxy for REST
|
1031
1197
|
|
1032
|
-
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are
|
1198
|
+
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
|
1199
|
+
|
1200
|
+
## <a id="certificates"></a>SSL CA certificate bundle
|
1201
|
+
|
1202
|
+
`ascli` uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
|
1203
|
+
|
1204
|
+
`ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). `ascli` overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
|
1033
1205
|
|
1034
1206
|
## Proxy auto config
|
1035
1207
|
|
@@ -1038,13 +1210,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
|
|
1038
1210
|
The PAC file can be tested with command: `config proxy_check` , example:
|
1039
1211
|
|
1040
1212
|
```
|
1041
|
-
|
1213
|
+
ascli config proxy_check --fpac=file:///./proxy.pac http://www.example.com
|
1042
1214
|
PROXY proxy.example.com:8080
|
1043
1215
|
```
|
1044
1216
|
|
1045
1217
|
This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
|
1046
1218
|
|
1047
|
-
## <a
|
1219
|
+
## <a id="client"></a>FASP configuration
|
1048
1220
|
|
1049
1221
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
1050
1222
|
|
@@ -1056,9 +1228,9 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
1056
1228
|
### Show path of currently used `ascp`
|
1057
1229
|
|
1058
1230
|
```
|
1059
|
-
|
1231
|
+
ascli config ascp show
|
1060
1232
|
/Users/laurent/.aspera/ascli/sdk/ascp
|
1061
|
-
|
1233
|
+
ascli config ascp info
|
1062
1234
|
+--------------------+-----------------------------------------------------------+
|
1063
1235
|
| key | value |
|
1064
1236
|
+--------------------+-----------------------------------------------------------+
|
@@ -1066,7 +1238,7 @@ $ ascli config ascp info
|
|
1066
1238
|
...
|
1067
1239
|
```
|
1068
1240
|
|
1069
|
-
### Selection of `ascp` location for [`direct`](#
|
1241
|
+
### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
1070
1242
|
|
1071
1243
|
By default, `ascli` uses any found local product with ascp, including SDK.
|
1072
1244
|
|
@@ -1077,7 +1249,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
|
|
1077
1249
|
Using a POSIX shell:
|
1078
1250
|
|
1079
1251
|
```
|
1080
|
-
|
1252
|
+
ascli config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
1081
1253
|
ascp version: 4.0.0.182279
|
1082
1254
|
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
1083
1255
|
Saved to default global preset global_common_defaults
|
@@ -1086,7 +1258,7 @@ Saved to default global preset global_common_defaults
|
|
1086
1258
|
Windows:
|
1087
1259
|
|
1088
1260
|
```
|
1089
|
-
|
1261
|
+
ascli config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1090
1262
|
ascp version: 4.0.0.182279
|
1091
1263
|
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
1092
1264
|
Saved to default global preset global_common_defaults
|
@@ -1099,7 +1271,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
|
|
1099
1271
|
Locally installed Aspera products can be listed with:
|
1100
1272
|
|
1101
1273
|
```
|
1102
|
-
|
1274
|
+
ascli config ascp products list
|
1103
1275
|
:.........................................:................................................:
|
1104
1276
|
: name : app_root :
|
1105
1277
|
:.........................................:................................................:
|
@@ -1110,7 +1282,7 @@ $ ascli config ascp products list
|
|
1110
1282
|
:.........................................:................................................:
|
1111
1283
|
```
|
1112
1284
|
|
1113
|
-
### Selection of local client for `ascp` for [`direct`](#
|
1285
|
+
### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
1114
1286
|
|
1115
1287
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
1116
1288
|
|
@@ -1119,14 +1291,14 @@ Using the option use_product finds the ascp binary of the selected product.
|
|
1119
1291
|
To permanently use the ascp of a product:
|
1120
1292
|
|
1121
1293
|
```
|
1122
|
-
|
1294
|
+
ascli config ascp products use 'Aspera Connect'
|
1123
1295
|
saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
|
1124
1296
|
```
|
1125
1297
|
|
1126
1298
|
### Installation of Connect Client on command line
|
1127
1299
|
|
1128
1300
|
```
|
1129
|
-
|
1301
|
+
ascli config ascp connect list
|
1130
1302
|
:...............................................:......................................:..............:
|
1131
1303
|
: id : title : version :
|
1132
1304
|
:...............................................:......................................:..............:
|
@@ -1139,7 +1311,7 @@ $ ascli config ascp connect list
|
|
1139
1311
|
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
|
1140
1312
|
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
|
1141
1313
|
:...............................................:......................................:..............:
|
1142
|
-
|
1314
|
+
ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
1143
1315
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1144
1316
|
: title : type : href : hreflang : rel :
|
1145
1317
|
:.............................................:..........................:.......................................................................:..........:...............:
|
@@ -1152,11 +1324,11 @@ $ ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
|
1152
1324
|
: Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
|
1153
1325
|
: Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
|
1154
1326
|
:.............................................:..........................:.......................................................................:..........:...............:
|
1155
|
-
|
1327
|
+
ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
|
1156
1328
|
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
|
1157
1329
|
```
|
1158
1330
|
|
1159
|
-
## <a
|
1331
|
+
## <a id="agents"></a>Transfer Agents
|
1160
1332
|
|
1161
1333
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
1162
1334
|
|
@@ -1165,102 +1337,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
1165
1337
|
|
1166
1338
|
There are currently 3 agents:
|
1167
1339
|
|
1168
|
-
* [`direct`](#
|
1169
|
-
* `connect` : use of a local Connect Client
|
1170
|
-
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
1171
|
-
* `httpgw` : use of an Aspera HTTP Gateway
|
1340
|
+
* [`direct`](#agt_direct) : a local execution of `ascp`
|
1341
|
+
* [`connect`](#agt_connect) : use of a local Connect Client
|
1342
|
+
* [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
|
1343
|
+
* [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
1344
|
+
* [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
1172
1345
|
|
1173
1346
|
Note that all transfer operation are seen from the point of view of the agent.
|
1174
1347
|
For instance, a node agent making an "upload", or "package send" operation,
|
1175
1348
|
will effectively push files to the related server from the agent node.
|
1176
1349
|
|
1177
|
-
`ascli`
|
1350
|
+
`ascli` standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
1178
1351
|
|
1179
1352
|
|
1180
|
-
### <a
|
1353
|
+
### <a id="agt_direct"></a>Direct
|
1181
1354
|
|
1182
|
-
|
1183
|
-
|
1355
|
+
The `direct` agent directly executes a local ascp.
|
1356
|
+
This is the default for `ascli`.
|
1357
|
+
This is equivalent to specifying `--transfer=direct`.
|
1358
|
+
`ascli` will detect locally installed Aspera products, including SDK.
|
1184
1359
|
Refer to section [FASP](#client).
|
1185
1360
|
|
1186
|
-
|
1187
|
-
|
1188
|
-
* `EX_fasp_proxy_url`
|
1189
|
-
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1190
|
-
* `EX_ascp_args`
|
1191
|
-
|
1192
|
-
The `transfer-info` accepts the following optional parameters:
|
1361
|
+
The `transfer-info` accepts the following optional parameters to control multi-session, WSS
|
1193
1362
|
|
1194
1363
|
<table>
|
1195
|
-
<tr><th>Name</th><th>Type</th><th>
|
1196
|
-
<tr><td>
|
1197
|
-
<tr><td>
|
1198
|
-
<tr><td>
|
1199
|
-
<tr><td>multi_incr_udp</td><td>Bool</td><td>
|
1200
|
-
<tr><td>resume</td><td>Hash</td><td>
|
1364
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1365
|
+
<tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
|
1366
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
|
1367
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
|
1368
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
|
1369
|
+
<tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
|
1370
|
+
<tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
|
1371
|
+
<tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
|
1372
|
+
<tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
|
1373
|
+
<tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
|
1201
1374
|
</table>
|
1202
1375
|
|
1203
|
-
Resume
|
1204
|
-
|
1205
|
-
In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
1206
|
-
Sleep between iteration is:
|
1376
|
+
Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
1377
|
+
Sleep between iterations is:
|
1207
1378
|
|
1208
1379
|
```
|
1209
1380
|
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
1210
1381
|
```
|
1211
1382
|
|
1212
|
-
|
1213
|
-
<tr><th>Name</th><th>Type</th><th>Default</th><th>Feature</th><th>Description</th></tr>
|
1214
|
-
<tr><td>iter_max</td><td>int</td><td>7</td><td>Resume</td><td>Max number of retry on error</td></tr>
|
1215
|
-
<tr><td>sleep_initial</td><td>int</td><td>2</td><td>Resume</td><td>First Sleep before retry</td></tr>
|
1216
|
-
<tr><td>sleep_factor</td><td>int</td><td>2</td><td>Resume</td><td>Multiplier of Sleep</td></tr>
|
1217
|
-
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
1218
|
-
</table>
|
1383
|
+
Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
|
1219
1384
|
|
1220
1385
|
Examples:
|
1221
1386
|
|
1222
1387
|
```
|
1223
|
-
|
1224
|
-
|
1388
|
+
ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
1389
|
+
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
1225
1390
|
```
|
1226
1391
|
|
1227
|
-
|
1392
|
+
To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
|
1393
|
+
|
1394
|
+
* `EX_fasp_proxy_url`
|
1395
|
+
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
1396
|
+
* `EX_ascp_args`
|
1397
|
+
|
1398
|
+
### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
1228
1399
|
|
1229
|
-
By specifying option: `--transfer=connect`, `ascli` will start transfers
|
1230
|
-
using the locally installed Aspera Connect Client.
|
1400
|
+
By specifying option: `--transfer=connect`, `ascli` will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
|
1231
1401
|
|
1232
|
-
### Aspera Node API : Node to node transfers
|
1402
|
+
### <a id="agt_node"></a>Aspera Node API : Node to node transfers
|
1233
1403
|
|
1234
1404
|
By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
|
1235
1405
|
Transfer Server using the Node API, either on a local or remote node.
|
1406
|
+
Parameters provided in option `transfer_info` are:
|
1236
1407
|
|
1237
|
-
|
1238
|
-
|
1239
|
-
|
1240
|
-
|
1241
|
-
|
1408
|
+
<table>
|
1409
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
1410
|
+
<tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
|
1411
|
+
<tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
|
1412
|
+
<tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
|
1413
|
+
<tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
|
1414
|
+
</table>
|
1242
1415
|
|
1243
|
-
|
1244
|
-
`--transfer-info=@preset:<psetname>` or specified using the
|
1416
|
+
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
1417
|
+
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
1245
1418
|
`--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
|
1246
1419
|
|
1247
|
-
|
1420
|
+
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
1248
1421
|
|
1249
|
-
|
1422
|
+
If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
|
1250
1423
|
|
1251
|
-
### <a
|
1424
|
+
### <a id="agt_httpgw"></a>HTTP Gateway
|
1252
1425
|
|
1253
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
1426
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
|
1254
1427
|
|
1255
1428
|
Example:
|
1256
1429
|
|
1257
1430
|
```
|
1258
|
-
|
1431
|
+
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
1259
1432
|
```
|
1260
1433
|
|
1261
1434
|
Note that the gateway only supports transfers authorized with a token.
|
1262
1435
|
|
1263
|
-
|
1436
|
+
### <a id="agt_trsdk"></a>Transfer SDK
|
1437
|
+
|
1438
|
+
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
1439
|
+
|
1440
|
+
By default it will listen on local port `55002` on `127.0.0.1`.
|
1441
|
+
|
1442
|
+
## <a id="transferspec"></a>Transfer Specification
|
1264
1443
|
|
1265
1444
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
1266
1445
|
is described in a _transfer-spec_ (Transfer Specification), such as:
|
@@ -1273,9 +1452,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
1273
1452
|
|
1274
1453
|
`ascli` builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
1275
1454
|
|
1276
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are
|
1455
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
|
1277
1456
|
|
1278
|
-
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#
|
1457
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
1279
1458
|
|
1280
1459
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
1281
1460
|
|
@@ -1284,21 +1463,22 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
1284
1463
|
|
1285
1464
|
A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
1286
1465
|
|
1287
|
-
## <a
|
1466
|
+
## <a id="transferparams"></a>Transfer Parameters
|
1288
1467
|
|
1289
|
-
All standard _transfer-spec_ parameters can be
|
1468
|
+
All standard _transfer-spec_ parameters can be specified.
|
1290
1469
|
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
1291
1470
|
|
1292
1471
|
References:
|
1293
1472
|
|
1294
1473
|
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
1295
1474
|
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
1475
|
+
* [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
1296
1476
|
|
1297
1477
|
Parameters can be displayed with commands:
|
1298
1478
|
|
1299
1479
|
```
|
1300
|
-
|
1301
|
-
|
1480
|
+
ascli config ascp spec
|
1481
|
+
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
1302
1482
|
```
|
1303
1483
|
|
1304
1484
|
Columns:
|
@@ -1306,9 +1486,10 @@ Columns:
|
|
1306
1486
|
* D=Direct (local `ascp` execution)
|
1307
1487
|
* N=Node API
|
1308
1488
|
* C=Connect Client
|
1309
|
-
* arg=`ascp` argument or environment variable
|
1310
1489
|
|
1311
|
-
|
1490
|
+
`ascp` argument or environment variable is provided in description.
|
1491
|
+
|
1492
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in `ascli`).
|
1312
1493
|
|
1313
1494
|
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
1314
1495
|
|
@@ -1326,17 +1507,17 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
1326
1507
|
|
1327
1508
|
### List of files for transfers
|
1328
1509
|
|
1329
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files
|
1510
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
1330
1511
|
So, by default, the list of files to transfer will be simply specified on the command line:
|
1331
1512
|
|
1332
1513
|
```
|
1333
|
-
|
1514
|
+
ascli server upload ~/mysample.file secondfile
|
1334
1515
|
```
|
1335
1516
|
|
1336
1517
|
This is equivalent to:
|
1337
1518
|
|
1338
1519
|
```
|
1339
|
-
|
1520
|
+
ascli server upload --sources=@args ~/mysample.file secondfile
|
1340
1521
|
```
|
1341
1522
|
|
1342
1523
|
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
@@ -1362,13 +1543,19 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
1362
1543
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
1363
1544
|
```
|
1364
1545
|
|
1546
|
+
providing a file list directly to ascp:
|
1547
|
+
|
1548
|
+
```
|
1549
|
+
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
|
1550
|
+
```
|
1551
|
+
|
1365
1552
|
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
1366
1553
|
|
1367
1554
|
```
|
1368
1555
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
1369
1556
|
```
|
1370
1557
|
|
1371
|
-
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#
|
1558
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
1372
1559
|
|
1373
1560
|
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
1374
1561
|
|
@@ -1378,10 +1565,10 @@ In case the file list is provided on the command line i.e. using `--sources=@arg
|
|
1378
1565
|
Example:
|
1379
1566
|
|
1380
1567
|
```
|
1381
|
-
|
1568
|
+
ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
1382
1569
|
```
|
1383
1570
|
|
1384
|
-
Internally, when transfer agent [`direct`](#
|
1571
|
+
Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
|
1385
1572
|
|
1386
1573
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
1387
1574
|
|
@@ -1394,7 +1581,7 @@ Source files are located on "Aspera on cloud", when :
|
|
1394
1581
|
* the server is Aspera on Cloud, and making a download / recv
|
1395
1582
|
* the agent is Aspera on Cloud, and making an upload / send
|
1396
1583
|
|
1397
|
-
### <a
|
1584
|
+
### <a id="multisession"></a>Support of multi-session
|
1398
1585
|
|
1399
1586
|
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
1400
1587
|
|
@@ -1445,16 +1632,14 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
1445
1632
|
--ts=@json:'{"precalculate_job_size":true}'
|
1446
1633
|
```
|
1447
1634
|
|
1448
|
-
|
1449
|
-
|
1450
|
-
## <a name="scheduling"></a>Lock for exclusive execution
|
1635
|
+
## <a id="scheduling"></a>Lock for exclusive execution
|
1451
1636
|
|
1452
1637
|
In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
|
1453
1638
|
|
1454
1639
|
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
1455
1640
|
|
1456
1641
|
* Executing instances may pile-up and kill the system
|
1457
|
-
* The same file may be
|
1642
|
+
* The same file may be transferred by multiple instances at the same time.
|
1458
1643
|
* `preview` may generate the same files in multiple instances.
|
1459
1644
|
|
1460
1645
|
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
@@ -1488,14 +1673,14 @@ WARN -- : Another instance is already running (Address already in use - bind(2)
|
|
1488
1673
|
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
1489
1674
|
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
1490
1675
|
|
1491
|
-
The list of supported *PVCL* adapters can be
|
1676
|
+
The list of supported *PVCL* adapters can be retrieved with command:
|
1492
1677
|
|
1493
1678
|
```
|
1494
|
-
|
1679
|
+
ascli conf ascp info
|
1495
1680
|
+--------------------+-----------------------------------------------------------+
|
1496
1681
|
| key | value |
|
1497
1682
|
+--------------------+-----------------------------------------------------------+
|
1498
|
-
-----8
|
1683
|
+
-----8<-----snip-----8<-----
|
1499
1684
|
| product_name | IBM Aspera SDK |
|
1500
1685
|
| product_version | 4.0.1.182389 |
|
1501
1686
|
| process | pvcl |
|
@@ -1520,20 +1705,26 @@ The simplified format is:
|
|
1520
1705
|
|
1521
1706
|
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
1522
1707
|
|
1523
|
-
## <a
|
1708
|
+
## <a id="faux_testing"></a>`faux:` for testing
|
1524
1709
|
|
1525
|
-
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli
|
1710
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`.
|
1526
1711
|
|
1527
1712
|
This adapter can be used to simulate a file or a directory.
|
1528
1713
|
|
1529
|
-
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form
|
1714
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
|
1530
1715
|
|
1531
|
-
|
1532
|
-
|
1716
|
+
```
|
1717
|
+
faux:///filename?filesize
|
1718
|
+
```
|
1719
|
+
|
1720
|
+
where:
|
1721
|
+
|
1722
|
+
* `filename` is the name that will be assigned to the file on the destination
|
1723
|
+
* `filesize` is the number of bytes that will be sent (in decimal).
|
1533
1724
|
|
1534
|
-
Note
|
1725
|
+
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
1535
1726
|
|
1536
|
-
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2
|
1727
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
1537
1728
|
|
1538
1729
|
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
1539
1730
|
|
@@ -1541,21 +1732,21 @@ To send uninitialized data in place of a source directory, the source argument i
|
|
1541
1732
|
faux:///dirname?<arg1>=<val1>&...
|
1542
1733
|
```
|
1543
1734
|
|
1544
|
-
|
1735
|
+
where:
|
1545
1736
|
|
1546
|
-
|
1737
|
+
* `dirname` is the folder name and can contain `/` to specify a subfolder.
|
1738
|
+
* supported arguments are:
|
1547
1739
|
|
1548
1740
|
<table>
|
1549
1741
|
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
1550
|
-
<tr><td>count</td><td>int</td><td>mandatory</td><td>
|
1551
|
-
<tr><td>file</td><td>string</td><td>file</td><td>
|
1552
|
-
<tr><td>size</td><td>int</td><td>0</td><td>
|
1553
|
-
<tr><td>inc</td><td>int</td><td>0</td><td>
|
1554
|
-
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>
|
1555
|
-
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>
|
1742
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
|
1743
|
+
<tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
|
1744
|
+
<tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
|
1745
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
|
1746
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
|
1747
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
|
1556
1748
|
</table>
|
1557
1749
|
|
1558
|
-
|
1559
1750
|
The sequence parameter is applied as follows:
|
1560
1751
|
|
1561
1752
|
* If `seq` is `random` then each file size is:
|
@@ -1563,40 +1754,40 @@ The sequence parameter is applied as follows:
|
|
1563
1754
|
* size +/- (inc * rand())
|
1564
1755
|
* Where rand is a random number between 0 and 1
|
1565
1756
|
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
1566
|
-
* Similarly, overall file size must be less than 8
|
1757
|
+
* Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
1567
1758
|
|
1568
1759
|
* If `seq` is `sequential` then each file size is:
|
1569
1760
|
|
1570
|
-
* size + ((fileindex - 1) * inc)
|
1761
|
+
* `size + ((fileindex - 1) * inc)`
|
1571
1762
|
* Where first file is index 1
|
1572
|
-
* So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
|
1573
|
-
* As with random
|
1763
|
+
* So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
1764
|
+
* As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
1574
1765
|
|
1575
|
-
Filenames generated are of the form: `<file>_<00000
|
1766
|
+
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
1576
1767
|
|
1577
1768
|
To discard data at the destination, the destination argument is set to `faux://` .
|
1578
1769
|
|
1579
1770
|
Examples:
|
1580
1771
|
|
1581
|
-
* Upload 20
|
1772
|
+
* Upload 20 gibibytes of random data to file myfile to directory /Upload
|
1582
1773
|
|
1583
1774
|
```
|
1584
|
-
|
1775
|
+
ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
1585
1776
|
```
|
1586
1777
|
|
1587
1778
|
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
1588
1779
|
|
1589
1780
|
```
|
1590
|
-
|
1781
|
+
ascli server upload /tmp/sample --to-folder=faux://
|
1591
1782
|
```
|
1592
1783
|
|
1593
|
-
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2
|
1784
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
1594
1785
|
|
1595
1786
|
```
|
1596
|
-
|
1787
|
+
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
1597
1788
|
```
|
1598
1789
|
|
1599
|
-
## <a
|
1790
|
+
## <a id="commands"></a>Sample Commands
|
1600
1791
|
|
1601
1792
|
A non complete list of commands used in unit tests:
|
1602
1793
|
|
@@ -1606,20 +1797,20 @@ ascli -h
|
|
1606
1797
|
ascli aoc -N remind --username=my_aoc_user_email
|
1607
1798
|
ascli aoc -N servers
|
1608
1799
|
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
1609
|
-
ascli aoc admin ats access_key --id=akibmcloud --secret=somesecret node browse /
|
1610
|
-
ascli aoc admin ats access_key --id=akibmcloud delete
|
1611
1800
|
ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
1612
1801
|
ascli aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
1802
|
+
ascli aoc admin ats access_key delete akibmcloud
|
1613
1803
|
ascli aoc admin ats access_key list --fields=name,id
|
1804
|
+
ascli aoc admin ats access_key node akibmcloud --secret=somesecret browse /
|
1614
1805
|
ascli aoc admin ats cluster clouds
|
1615
1806
|
ascli aoc admin ats cluster list
|
1616
1807
|
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
1617
|
-
ascli aoc admin ats cluster show
|
1808
|
+
ascli aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
1618
1809
|
ascli aoc admin res application list
|
1619
1810
|
ascli aoc admin res client list
|
1620
1811
|
ascli aoc admin res client_access_key list
|
1621
|
-
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
1622
1812
|
ascli aoc admin res client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}'
|
1813
|
+
ascli aoc admin res client_registration_token delete my_clt_reg_id
|
1623
1814
|
ascli aoc admin res client_registration_token list
|
1624
1815
|
ascli aoc admin res contact list
|
1625
1816
|
ascli aoc admin res dropbox list
|
@@ -1629,23 +1820,23 @@ ascli aoc admin res kms_profile list
|
|
1629
1820
|
ascli aoc admin res node list
|
1630
1821
|
ascli aoc admin res operation list
|
1631
1822
|
ascli aoc admin res organization show
|
1632
|
-
ascli aoc admin res package list
|
1823
|
+
ascli aoc admin res package list --http-options=@json:'{"read_timeout":120.0}'
|
1633
1824
|
ascli aoc admin res saml_configuration list
|
1634
1825
|
ascli aoc admin res self show
|
1635
1826
|
ascli aoc admin res short_link list
|
1636
1827
|
ascli aoc admin res user list
|
1637
1828
|
ascli aoc admin res workspace_membership list
|
1638
1829
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
|
1639
|
-
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 access_key delete --id=testsub1
|
1640
1830
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 events
|
1641
1831
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 browse /
|
1642
1832
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 delete /folder1
|
1643
1833
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 mkdir /folder1
|
1834
|
+
ascli aoc admin resource node v3 name AOC_NODE1_NAME --secret=AOC_NODE1_SECRET access_key delete testsub1
|
1644
1835
|
ascli aoc admin resource workspace list
|
1645
1836
|
ascli aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
1646
|
-
ascli aoc automation workflow
|
1837
|
+
ascli aoc automation workflow "my_wf_id" action create --value=@json:'{"name":"toto"}' | tee action.info
|
1647
1838
|
ascli aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
1648
|
-
ascli aoc automation workflow delete
|
1839
|
+
ascli aoc automation workflow delete "my_wf_id"
|
1649
1840
|
ascli aoc automation workflow list
|
1650
1841
|
ascli aoc automation workflow list --select=@json:'{"name":"test_workflow"}' --fields=id --format=csv --display=data > test
|
1651
1842
|
ascli aoc automation workflow list --value=@json:'{"show_org_workflows":"true"}' --scope=admin:all
|
@@ -1657,7 +1848,7 @@ ascli aoc files browse /
|
|
1657
1848
|
ascli aoc files browse / -N --link=my_aoc_publink_folder
|
1658
1849
|
ascli aoc files delete /testsrc
|
1659
1850
|
ascli aoc files download --transfer=connect /200KB.1
|
1660
|
-
ascli aoc files file
|
1851
|
+
ascli aoc files file show my_file_id
|
1661
1852
|
ascli aoc files find / --value='\.partial$'
|
1662
1853
|
ascli aoc files http_node_download --to-folder=. /200KB.1
|
1663
1854
|
ascli aoc files mkdir /testsrc
|
@@ -1674,8 +1865,8 @@ ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
|
|
1674
1865
|
ascli aoc organization
|
1675
1866
|
ascli aoc packages list
|
1676
1867
|
ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
1677
|
-
ascli aoc packages recv
|
1678
|
-
ascli aoc packages recv
|
1868
|
+
ascli aoc packages recv "my_package_id" --to-folder=.
|
1869
|
+
ascli aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
1679
1870
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
1680
1871
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["internal.user@example.com"],"note":"my note"}' testfile.bin
|
1681
1872
|
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
@@ -1686,20 +1877,20 @@ ascli aoc user info show
|
|
1686
1877
|
ascli aoc user shared_inboxes
|
1687
1878
|
ascli aoc user workspaces
|
1688
1879
|
ascli aoc workspace
|
1689
|
-
ascli ats access_key --
|
1690
|
-
ascli ats access_key --id=akibmcloud --secret=somesecret cluster
|
1691
|
-
ascli ats access_key --id=akibmcloud --secret=somesecret node browse /
|
1692
|
-
ascli ats access_key --id=akibmcloud delete
|
1880
|
+
ascli ats access_key cluster akibmcloud --secret=somesecret
|
1693
1881
|
ascli ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
1694
1882
|
ascli ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
1883
|
+
ascli ats access_key delete ak_aws
|
1884
|
+
ascli ats access_key delete akibmcloud
|
1695
1885
|
ascli ats access_key list --fields=name,id
|
1886
|
+
ascli ats access_key node akibmcloud browse / --secret=somesecret
|
1696
1887
|
ascli ats api_key create
|
1697
1888
|
ascli ats api_key instances
|
1698
1889
|
ascli ats api_key list
|
1699
1890
|
ascli ats cluster clouds
|
1700
1891
|
ascli ats cluster list
|
1701
1892
|
ascli ats cluster show --cloud=aws --region=eu-west-1
|
1702
|
-
ascli ats cluster show
|
1893
|
+
ascli ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
1703
1894
|
ascli conf flush_tokens
|
1704
1895
|
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes
|
1705
1896
|
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes --use-generic-client=yes
|
@@ -1713,6 +1904,8 @@ ascli config ascp products list
|
|
1713
1904
|
ascli config ascp show
|
1714
1905
|
ascli config ascp spec
|
1715
1906
|
ascli config check_update
|
1907
|
+
ascli config detect --url=https://my_aoc_org.ibmaspera.com;\
|
1908
|
+
ascli config detect --url=my_faspex_url
|
1716
1909
|
ascli config doc
|
1717
1910
|
ascli config doc transfer-parameters
|
1718
1911
|
ascli config email_test --notif-to=my_recipient_email
|
@@ -1725,34 +1918,35 @@ ascli console transfer smart list
|
|
1725
1918
|
ascli console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
1726
1919
|
ascli cos -N --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
1727
1920
|
ascli cos -N --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
1728
|
-
ascli cos node access_key
|
1921
|
+
ascli cos node access_key show self
|
1729
1922
|
ascli cos node download testfile.bin --to-folder=.
|
1730
1923
|
ascli cos node info
|
1731
1924
|
ascli cos node upload testfile.bin
|
1732
1925
|
ascli faspex health
|
1733
1926
|
ascli faspex package list
|
1734
|
-
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data
|
1735
|
-
ascli faspex package list --fields=package_id --format=csv --display=data
|
1736
|
-
ascli faspex package recv --to-folder=.
|
1737
|
-
ascli faspex package recv --to-folder=. --
|
1738
|
-
ascli faspex package recv --to-folder=.
|
1927
|
+
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data --query=@json:'{"max":1}');\
|
1928
|
+
ascli faspex package list --fields=package_id --format=csv --display=data --query=@json:'{"max":1}');\
|
1929
|
+
ascli faspex package recv "my_package_id" --to-folder=.
|
1930
|
+
ascli faspex package recv "my_package_id" --to-folder=. --box=sent
|
1931
|
+
ascli faspex package recv --to-folder=. "my_package_id"
|
1739
1932
|
ascli faspex package recv --to-folder=. --link="my_faspex_publink_recv_from_fxuser"
|
1933
|
+
ascli faspex package recv ALL --to-folder=. --once-only=yes
|
1740
1934
|
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com","FASPEX_USERNAME"]}' testfile.bin
|
1741
1935
|
ascli faspex package send --link="my_faspex_publink_send_to_dropbox" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
1742
1936
|
ascli faspex package send --link="my_faspex_publink_send_to_fxuser" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
1743
1937
|
ascli faspex source name "Server Files" node br /
|
1744
1938
|
ascli faspex5 node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
1745
1939
|
ascli faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
1746
|
-
ascli faspex5 package receive
|
1940
|
+
ascli faspex5 package receive "my_package_id" --to-folder=.
|
1747
1941
|
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"${f5_user}"}]}' testfile.bin
|
1748
1942
|
ascli node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
1749
|
-
ascli node -N -Ptst_node_preview access_key delete
|
1750
|
-
ascli node async
|
1751
|
-
ascli node async
|
1752
|
-
ascli node async
|
1943
|
+
ascli node -N -Ptst_node_preview access_key delete aoc_1
|
1944
|
+
ascli node async bandwidth 1
|
1945
|
+
ascli node async counters 1
|
1946
|
+
ascli node async files 1
|
1753
1947
|
ascli node async list
|
1754
|
-
ascli node async show
|
1755
|
-
ascli node async show
|
1948
|
+
ascli node async show 1
|
1949
|
+
ascli node async show ALL
|
1756
1950
|
ascli node basic_token
|
1757
1951
|
ascli node browse / -r
|
1758
1952
|
ascli node delete folder_1/10MB.1
|
@@ -1761,8 +1955,8 @@ ascli node download --to-folder=. folder_1/testfile.bin
|
|
1761
1955
|
ascli node health
|
1762
1956
|
ascli node info
|
1763
1957
|
ascli node search / --value=@json:'{"sort":"mtime"}'
|
1764
|
-
ascli node service --id=service1 delete
|
1765
1958
|
ascli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
1959
|
+
ascli node service delete service1
|
1766
1960
|
ascli node service list
|
1767
1961
|
ascli node transfer list --value=@json:'{"active_only":true}'
|
1768
1962
|
ascli node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass"}'
|
@@ -1770,12 +1964,12 @@ ascli node upload --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000
|
|
1770
1964
|
ascli orchestrator info
|
1771
1965
|
ascli orchestrator plugins
|
1772
1966
|
ascli orchestrator processes
|
1773
|
-
ascli orchestrator workflow
|
1774
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID start --params=@json:'{"Param":"world !"}'
|
1775
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID start --params=@json:'{"Param":"world !"}' --result=ResultStep:Complete_status_message
|
1776
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID status
|
1967
|
+
ascli orchestrator workflow inputs ORCH_WORKFLOW_ID
|
1777
1968
|
ascli orchestrator workflow list
|
1778
|
-
ascli orchestrator workflow
|
1969
|
+
ascli orchestrator workflow start ORCH_WORKFLOW_ID --params=@json:'{"Param":"world !"}'
|
1970
|
+
ascli orchestrator workflow start ORCH_WORKFLOW_ID --params=@json:'{"Param":"world !"}' --result=ResultStep:Complete_status_message
|
1971
|
+
ascli orchestrator workflow status ALL
|
1972
|
+
ascli orchestrator workflow status ORCH_WORKFLOW_ID
|
1779
1973
|
ascli preview check --skip-types=office
|
1780
1974
|
ascli preview folder 1 --skip-types=office --log-level=info --file-access=remote --ts=@json:'{"target_rate_kbps":1000000}'
|
1781
1975
|
ascli preview scan --skip-types=office --log-level=info
|
@@ -1831,12 +2025,12 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
1831
2025
|
...and more
|
1832
2026
|
```
|
1833
2027
|
|
1834
|
-
## <a
|
2028
|
+
## <a id="usage"></a>Usage
|
1835
2029
|
|
1836
2030
|
```
|
1837
|
-
|
2031
|
+
ascli -h
|
1838
2032
|
NAME
|
1839
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
2033
|
+
ascli -- a command line tool for Aspera Applications (v4.5.0)
|
1840
2034
|
|
1841
2035
|
SYNOPSIS
|
1842
2036
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -1846,10 +2040,11 @@ DESCRIPTION
|
|
1846
2040
|
Documentation and examples: https://rubygems.org/gems/aspera-cli
|
1847
2041
|
execute: ascli conf doc
|
1848
2042
|
or visit: http://www.rubydoc.info/gems/aspera-cli
|
2043
|
+
source repo: https://github.com/IBM/aspera-cli
|
1849
2044
|
|
1850
2045
|
ENVIRONMENT VARIABLES
|
1851
2046
|
ASCLI_HOME config folder, default: $HOME/.aspera/ascli
|
1852
|
-
|
2047
|
+
any option can be set as an environment variable, refer to the manual
|
1853
2048
|
|
1854
2049
|
COMMANDS
|
1855
2050
|
To list first level commands, execute: ascli
|
@@ -1857,7 +2052,8 @@ COMMANDS
|
|
1857
2052
|
|
1858
2053
|
OPTIONS
|
1859
2054
|
Options begin with a '-' (minus), and value is provided on command line.
|
1860
|
-
Special values are supported beginning with special prefix
|
2055
|
+
Special values are supported beginning with special prefix @pfx:, where pfx is one of:
|
2056
|
+
base64, json, zlib, ruby, csvt, lines, list, incps, val, file, path, env, stdin, preset
|
1861
2057
|
Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'
|
1862
2058
|
|
1863
2059
|
ARGS
|
@@ -1883,11 +2079,13 @@ OPTIONS: global
|
|
1883
2079
|
--logger=ENUM log method: stderr, stdout, syslog
|
1884
2080
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
1885
2081
|
--query=VALUE additional filter for API calls (extended value) (some commands)
|
2082
|
+
--http-options=VALUE options for http socket (extended value)
|
1886
2083
|
--insecure=ENUM do not validate HTTPS certificate: yes, no
|
1887
2084
|
--once-only=ENUM process only new items (some commands): yes, no
|
2085
|
+
--log-passwords=ENUM show passwords in logs: yes, no
|
1888
2086
|
|
1889
2087
|
COMMAND: config
|
1890
|
-
SUBCOMMANDS:
|
2088
|
+
SUBCOMMANDS: list overview id preset open documentation genkey gem_path plugins flush_tokens echo wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update initdemo vault
|
1891
2089
|
OPTIONS:
|
1892
2090
|
--value=VALUE extended value for create, update, list filter
|
1893
2091
|
--property=VALUE name of property to set
|
@@ -1898,14 +2096,14 @@ OPTIONS:
|
|
1898
2096
|
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
|
1899
2097
|
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
|
1900
2098
|
--test-mode=ENUM Wizard: skip private key check step: yes, no
|
2099
|
+
-P, --presetVALUE load the named option preset from current config file
|
1901
2100
|
--pkeypath=VALUE Wizard: path to private key for JWT
|
1902
2101
|
--ascp-path=VALUE path to ascp
|
1903
2102
|
--use-product=VALUE use ascp from specified product
|
1904
2103
|
--smtp=VALUE smtp configuration (extended value: hash)
|
1905
2104
|
--fpac=VALUE proxy auto configuration URL
|
1906
|
-
-P, --presetVALUE load the named option preset from current config file
|
1907
2105
|
--secret=VALUE default secret
|
1908
|
-
--secrets=VALUE secret
|
2106
|
+
--secrets=VALUE secret vault
|
1909
2107
|
--sdk-url=VALUE URL to get SDK
|
1910
2108
|
--sdk-folder=VALUE SDK folder path
|
1911
2109
|
--notif-to=VALUE email recipient for notification of transfers
|
@@ -1917,7 +2115,7 @@ OPTIONS:
|
|
1917
2115
|
--sources=VALUE list of source files (see doc)
|
1918
2116
|
--transfer-info=VALUE parameters for transfer agent
|
1919
2117
|
--src-type=ENUM type of file list: list, pair
|
1920
|
-
--transfer=ENUM type of transfer agent: direct,
|
2118
|
+
--transfer=ENUM type of transfer agent: direct, node, connect, httpgw, trsdk
|
1921
2119
|
--progress=ENUM type of progress bar: none, native, multi
|
1922
2120
|
|
1923
2121
|
|
@@ -1988,11 +2186,11 @@ OPTIONS:
|
|
1988
2186
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
1989
2187
|
--username=VALUE username to log in
|
1990
2188
|
--password=VALUE user's password
|
1991
|
-
--client-id=VALUE
|
1992
|
-
--client-secret=VALUE
|
1993
|
-
--redirect-uri=VALUE
|
1994
|
-
--auth=ENUM type of
|
1995
|
-
--private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
|
2189
|
+
--client-id=VALUE OAuth client identifier
|
2190
|
+
--client-secret=VALUE OAuth client secret
|
2191
|
+
--redirect-uri=VALUE OAuth redirect URI
|
2192
|
+
--auth=ENUM OAuth type of authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey, boot
|
2193
|
+
--private-key=VALUE Oauth RSA private key PEM value for JWT (prefix file path with @val:@file:)
|
1996
2194
|
|
1997
2195
|
|
1998
2196
|
COMMAND: cos
|
@@ -2120,34 +2318,18 @@ OPTIONS:
|
|
2120
2318
|
|
2121
2319
|
Note that actions and parameter values can be written in short form.
|
2122
2320
|
|
2123
|
-
# <a
|
2124
|
-
|
2125
|
-
`ascli` comes with several Aspera application plugins.
|
2126
|
-
|
2127
|
-
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
2128
|
-
|
2129
|
-
Those are using options:
|
2130
|
-
|
2131
|
-
* url
|
2132
|
-
* username
|
2133
|
-
* password
|
2134
|
-
|
2135
|
-
Those can be provided using command line, parameter set, env var, see section above.
|
2136
|
-
|
2137
|
-
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
2138
|
-
|
2139
|
-
# <a name="aoc"></a>Plugin: Aspera on Cloud
|
2321
|
+
# <a id="aoc"></a>Plugin: Aspera on Cloud
|
2140
2322
|
|
2141
2323
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
2142
2324
|
|
2143
2325
|
It is recommended to use the wizard to set it up, but manual configuration is also possible.
|
2144
2326
|
|
2145
|
-
## <a
|
2327
|
+
## <a id="aocwizard"></a>Configuration: using Wizard
|
2146
2328
|
|
2147
2329
|
`ascli` provides a configuration wizard. Here is a sample invocation :
|
2148
2330
|
|
2149
2331
|
```
|
2150
|
-
|
2332
|
+
ascli config wizard
|
2151
2333
|
option: url> https://myorg.ibmaspera.com
|
2152
2334
|
Detected: Aspera on Cloud
|
2153
2335
|
Preparing preset: aoc_myorg
|
@@ -2163,7 +2345,7 @@ Setting config preset as default for aspera
|
|
2163
2345
|
saving config file
|
2164
2346
|
Done.
|
2165
2347
|
You can test with:
|
2166
|
-
|
2348
|
+
ascli aoc user info show
|
2167
2349
|
```
|
2168
2350
|
|
2169
2351
|
Optionally, it is possible to create a new organization-specific "integration".
|
@@ -2171,7 +2353,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
2171
2353
|
|
2172
2354
|
This will guide you through the steps to create.
|
2173
2355
|
|
2174
|
-
## <a
|
2356
|
+
## <a id="aocmanual"></a>Configuration: using manual setup
|
2175
2357
|
|
2176
2358
|
If you used the wizard (recommended): skip this section.
|
2177
2359
|
|
@@ -2183,7 +2365,7 @@ Several types of OAuth authentication are supported:
|
|
2183
2365
|
* Web based authentication : authentication is made by user using a browser
|
2184
2366
|
* URL Token : external users authentication with url tokens (public links)
|
2185
2367
|
|
2186
|
-
The authentication method is
|
2368
|
+
The authentication method is controlled by option `auth`.
|
2187
2369
|
|
2188
2370
|
For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [[option preset](#lprt) for Aspera on Cloud](#aocpreset).
|
2189
2371
|
|
@@ -2191,7 +2373,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
|
|
2191
2373
|
|
2192
2374
|
In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.`ascli` saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
2193
2375
|
|
2194
|
-
### <a
|
2376
|
+
### <a id="clientreg"></a>Optional: API Client Registration
|
2195
2377
|
|
2196
2378
|
If you use the built-in client_id and client_secret, skip this and do not set them in next section.
|
2197
2379
|
|
@@ -2215,31 +2397,31 @@ Note: for web based authentication, `ascli` listens on a local port (e.g. specif
|
|
2215
2397
|
|
2216
2398
|
Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
|
2217
2399
|
|
2218
|
-
### <a
|
2400
|
+
### <a id="aocpreset"></a>[option preset](#lprt) for Aspera on Cloud
|
2219
2401
|
|
2220
2402
|
If you did not use the wizard, you can also manually create a [option preset](#lprt) for `ascli` in its configuration file.
|
2221
2403
|
|
2222
2404
|
Lets create an [option preset](#lprt) called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
2223
2405
|
|
2224
2406
|
```
|
2225
|
-
|
2407
|
+
ascli config preset ask my_aoc_org url client_id client_secret
|
2226
2408
|
option: url> https://myorg.ibmaspera.com/
|
2227
2409
|
option: client_id> BJLPObQiFw
|
2228
2410
|
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
|
2229
2411
|
updated: my_aoc_org
|
2230
2412
|
```
|
2231
2413
|
|
2232
|
-
(This can also be done in one line using the command `config
|
2414
|
+
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
2233
2415
|
|
2234
2416
|
Define this [option preset](#lprt) as default configuration for the `aspera` plugin:
|
2235
2417
|
|
2236
2418
|
```
|
2237
|
-
|
2419
|
+
ascli config preset set default aoc my_aoc_org
|
2238
2420
|
```
|
2239
2421
|
|
2240
2422
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
2241
2423
|
|
2242
|
-
### <a
|
2424
|
+
### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
|
2243
2425
|
|
2244
2426
|
For a Browser-less, Private Key-based authentication, use the following steps.
|
2245
2427
|
|
@@ -2254,13 +2436,13 @@ This can be done using any of the following method:
|
|
2254
2436
|
* using the CLI:
|
2255
2437
|
|
2256
2438
|
```
|
2257
|
-
|
2439
|
+
ascli config genkey ~/.aspera/ascli/aocapikey
|
2258
2440
|
```
|
2259
2441
|
|
2260
2442
|
* `ssh-keygen`:
|
2261
2443
|
|
2262
2444
|
```
|
2263
|
-
|
2445
|
+
ssh-keygen -t rsa -f ~/.aspera/ascli/aocapikey -N ''
|
2264
2446
|
```
|
2265
2447
|
|
2266
2448
|
* `openssl`
|
@@ -2268,11 +2450,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/ascli/aocapikey -N ''
|
|
2268
2450
|
(on some openssl implementation (mac) there is option: -nodes (no DES))
|
2269
2451
|
|
2270
2452
|
```
|
2271
|
-
|
2272
|
-
|
2273
|
-
|
2274
|
-
|
2275
|
-
|
2453
|
+
APIKEY=~/.aspera/ascli/aocapikey
|
2454
|
+
openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
|
2455
|
+
openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
|
2456
|
+
openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
|
2457
|
+
rm -f ${APIKEY}.protected
|
2276
2458
|
```
|
2277
2459
|
|
2278
2460
|
#### API Client JWT activation
|
@@ -2291,13 +2473,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
2291
2473
|
* Using command line
|
2292
2474
|
|
2293
2475
|
```
|
2294
|
-
|
2476
|
+
ascli aoc admin res client list
|
2295
2477
|
:............:.........:
|
2296
2478
|
: id : name :
|
2297
2479
|
:............:.........:
|
2298
2480
|
: BJLPObQiFw : ascli :
|
2299
2481
|
:............:.........:
|
2300
|
-
|
2482
|
+
ascli aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
2301
2483
|
modified
|
2302
2484
|
```
|
2303
2485
|
|
@@ -2318,14 +2500,14 @@ open the previously generated public key located here: `$HOME/.aspera/ascli/aoca
|
|
2318
2500
|
* Using command line
|
2319
2501
|
|
2320
2502
|
```
|
2321
|
-
|
2503
|
+
ascli aoc admin res user list
|
2322
2504
|
:........:................:
|
2323
2505
|
: id : name :
|
2324
2506
|
:........:................:
|
2325
2507
|
: 109952 : Tech Support :
|
2326
2508
|
: 109951 : LAURENT MARTIN :
|
2327
2509
|
:........:................:
|
2328
|
-
|
2510
|
+
ascli aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/ascli/aocapikey.pub"))}'
|
2329
2511
|
modified
|
2330
2512
|
```
|
2331
2513
|
|
@@ -2333,7 +2515,7 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
2333
2515
|
|
2334
2516
|
### [option preset](#lprt) modification for JWT
|
2335
2517
|
|
2336
|
-
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the
|
2518
|
+
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the following:
|
2337
2519
|
|
2338
2520
|
* change auth method to JWT
|
2339
2521
|
* provide location of private key
|
@@ -2342,7 +2524,7 @@ To activate default use of JWT authentication for `ascli` using the [option pres
|
|
2342
2524
|
Execute:
|
2343
2525
|
|
2344
2526
|
```
|
2345
|
-
|
2527
|
+
ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/ascli/aocapikey --username=laurent.martin.aspera@fr.ibm.com
|
2346
2528
|
```
|
2347
2529
|
|
2348
2530
|
Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
|
@@ -2350,26 +2532,29 @@ Note: the private key argument represents the actual PEM string. In order to rea
|
|
2350
2532
|
After this last step, commands do not require web login anymore.
|
2351
2533
|
|
2352
2534
|
|
2353
|
-
### <a
|
2535
|
+
### <a id="aocfirst"></a>First Use
|
2354
2536
|
|
2355
2537
|
Once client has been registered and [option preset](#lprt) created: `ascli` can be used:
|
2356
2538
|
|
2357
2539
|
```
|
2358
|
-
|
2540
|
+
ascli aoc files br /
|
2359
2541
|
Current Workspace: Default Workspace (default)
|
2360
2542
|
empty
|
2361
2543
|
```
|
2362
2544
|
|
2363
2545
|
|
2364
|
-
|
2546
|
+
## Administration
|
2365
2547
|
|
2366
2548
|
The `admin` command allows several administrative tasks (and require admin privilege).
|
2367
2549
|
|
2368
2550
|
It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
|
2369
2551
|
|
2370
|
-
Bulk
|
2552
|
+
### Bulk creation and deletion of resource
|
2371
2553
|
|
2372
|
-
|
2554
|
+
Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
|
2555
|
+
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
2556
|
+
|
2557
|
+
### Listing resources
|
2373
2558
|
|
2374
2559
|
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
2375
2560
|
|
@@ -2391,7 +2576,7 @@ Other parameters are directly sent as parameters to the GET request on API.
|
|
2391
2576
|
|
2392
2577
|
`page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
2393
2578
|
|
2394
|
-
`q` and `sort` are available on most
|
2579
|
+
`q` and `sort` are available on most resource types.
|
2395
2580
|
|
2396
2581
|
Other parameters depend on the type of entity (refer to AoC API).
|
2397
2582
|
|
@@ -2403,7 +2588,7 @@ Examples:
|
|
2403
2588
|
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
2404
2589
|
```
|
2405
2590
|
|
2406
|
-
* List users who
|
2591
|
+
* List users who logged-in before a date:
|
2407
2592
|
|
2408
2593
|
```
|
2409
2594
|
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
@@ -2419,40 +2604,85 @@ Refer to the AoC API for full list of query parameters, or use the browser in de
|
|
2419
2604
|
|
2420
2605
|
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
2421
2606
|
|
2422
|
-
|
2607
|
+
### <a id="res_select"></a>Selecting a resource
|
2608
|
+
|
2609
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
|
2610
|
+
|
2611
|
+
To execute an action on a specific resource, select it using one of those methods:
|
2423
2612
|
|
2424
|
-
|
2425
|
-
|
2613
|
+
* *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
|
2614
|
+
* give name on command line *after the action*: `aoc admin res node show name abc`
|
2615
|
+
* provide option `id` : `aoc admin res node show --id=123`
|
2616
|
+
* provide option `name` : `aoc admin res node show --name=abc`
|
2617
|
+
|
2618
|
+
### Access Key secrets
|
2619
|
+
|
2620
|
+
In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
|
2621
|
+
It is usually provided using the `secret` option.
|
2622
|
+
For example in a command like:
|
2426
2623
|
|
2427
2624
|
```
|
2428
|
-
|
2625
|
+
ascli aoc admin res node --id=123 --secret="secret1" v3 info
|
2429
2626
|
```
|
2430
2627
|
|
2431
|
-
It is also possible to provide a set of secrets used on a regular basis
|
2628
|
+
It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
|
2432
2629
|
|
2433
|
-
|
2630
|
+
### Activity
|
2631
|
+
|
2632
|
+
The activity app can be queried with:
|
2434
2633
|
|
2435
2634
|
```
|
2436
|
-
|
2437
|
-
$ ascli conf id my_secrets set 'access_key2' 'secret2'
|
2438
|
-
$ ascli conf id default get config
|
2439
|
-
"cli_default"
|
2635
|
+
ascli aoc admin analytics transfers
|
2440
2636
|
```
|
2441
2637
|
|
2442
|
-
|
2638
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
2639
|
+
|
2640
|
+
`mytemplate.erb`:
|
2443
2641
|
|
2444
2642
|
```
|
2445
|
-
|
2643
|
+
From: <%=from_name%> <<%=from_email%>>
|
2644
|
+
To: <<%=ev['user_email']%>>
|
2645
|
+
Subject: <%=ev['files_completed']%> files received
|
2646
|
+
|
2647
|
+
Dear <%=ev[:user_email.to_s]%>,
|
2648
|
+
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
|
2649
|
+
<%=ev['content']%>
|
2650
|
+
|
2651
|
+
Thank you.
|
2446
2652
|
```
|
2653
|
+
The environment provided contains the following additional variable:
|
2447
2654
|
|
2448
|
-
|
2655
|
+
* ev : all details on the transfer event
|
2656
|
+
|
2657
|
+
Example:
|
2449
2658
|
|
2450
|
-
|
2659
|
+
```
|
2660
|
+
ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
2661
|
+
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2662
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
2663
|
+
```
|
2451
2664
|
|
2452
|
-
|
2665
|
+
Options:
|
2666
|
+
|
2667
|
+
* `once_only` keep track of last date it was called, so next call will get only new events
|
2668
|
+
* `query` filter (on API call)
|
2669
|
+
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
2670
|
+
|
2671
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
2672
|
+
|
2673
|
+
### Transfer: Using specific transfer ports
|
2674
|
+
|
2675
|
+
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
2676
|
+
The option `default_ports` ([yes]/no) allows ascli to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
2677
|
+
|
2678
|
+
### Using ATS
|
2679
|
+
|
2680
|
+
Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
|
2681
|
+
|
2682
|
+
### Example: Bulk creation of users
|
2453
2683
|
|
2454
2684
|
```
|
2455
|
-
|
2685
|
+
ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
2456
2686
|
:.......:.........:
|
2457
2687
|
: id : status :
|
2458
2688
|
:.......:.........:
|
@@ -2461,20 +2691,20 @@ $ ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exampl
|
|
2461
2691
|
:.......:.........:
|
2462
2692
|
```
|
2463
2693
|
|
2464
|
-
|
2694
|
+
### Example: Find with filter and delete
|
2465
2695
|
|
2466
2696
|
```
|
2467
|
-
|
2697
|
+
ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
2468
2698
|
:.......:........................:
|
2469
2699
|
: id : email :
|
2470
2700
|
:.......:........................:
|
2471
2701
|
: 98398 : dummyuser1@example.com :
|
2472
2702
|
: 98399 : dummyuser2@example.com :
|
2473
2703
|
:.......:........................:
|
2474
|
-
|
2475
|
-
|
2704
|
+
thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
2705
|
+
echo $thelist
|
2476
2706
|
["113501","354061"]
|
2477
|
-
|
2707
|
+
ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
2478
2708
|
:.......:.........:
|
2479
2709
|
: id : status :
|
2480
2710
|
:.......:.........:
|
@@ -2483,7 +2713,7 @@ $ ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
2483
2713
|
:.......:.........:
|
2484
2714
|
```
|
2485
2715
|
|
2486
|
-
|
2716
|
+
### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
2487
2717
|
|
2488
2718
|
```
|
2489
2719
|
ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
@@ -2491,10 +2721,10 @@ ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_log
|
|
2491
2721
|
|
2492
2722
|
To delete them use the same method as before
|
2493
2723
|
|
2494
|
-
|
2724
|
+
### Example: Display current user's workspaces
|
2495
2725
|
|
2496
2726
|
```
|
2497
|
-
|
2727
|
+
ascli aoc user workspaces
|
2498
2728
|
:......:............................:
|
2499
2729
|
: id : name :
|
2500
2730
|
:......:............................:
|
@@ -2504,18 +2734,18 @@ $ ascli aoc user workspaces
|
|
2504
2734
|
:......:............................:
|
2505
2735
|
```
|
2506
2736
|
|
2507
|
-
|
2737
|
+
### Example: Create a sub access key in a "node"
|
2508
2738
|
|
2509
2739
|
Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
2510
2740
|
|
2511
2741
|
```
|
2512
|
-
|
2742
|
+
ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
|
2513
2743
|
```
|
2514
2744
|
|
2515
|
-
|
2745
|
+
### Example: Display transfer events (ops/transfer)
|
2516
2746
|
|
2517
2747
|
```
|
2518
|
-
|
2748
|
+
ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
2519
2749
|
```
|
2520
2750
|
|
2521
2751
|
Examples of query (TODO: cleanup):
|
@@ -2533,16 +2763,16 @@ Examples of query (TODO: cleanup):
|
|
2533
2763
|
# active_only=true|false
|
2534
2764
|
```
|
2535
2765
|
|
2536
|
-
|
2766
|
+
### Example: Display node events (events)
|
2537
2767
|
|
2538
2768
|
```
|
2539
|
-
|
2769
|
+
ascli aoc admin res node --secret=_secret_ v3 events
|
2540
2770
|
```
|
2541
2771
|
|
2542
|
-
|
2772
|
+
### Example: Display members of a workspace
|
2543
2773
|
|
2544
2774
|
```
|
2545
|
-
|
2775
|
+
ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
2546
2776
|
:.............:.........:..................................:
|
2547
2777
|
: member_type : manager : member.email :
|
2548
2778
|
:.............:.........:..................................:
|
@@ -2561,7 +2791,7 @@ other query parameters:
|
|
2561
2791
|
{"workspace_membership_through":true,"include_indirect":true}
|
2562
2792
|
```
|
2563
2793
|
|
2564
|
-
|
2794
|
+
### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
|
2565
2795
|
|
2566
2796
|
a- Get id of first workspace
|
2567
2797
|
|
@@ -2580,7 +2810,7 @@ WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --se
|
|
2580
2810
|
c- Extract membership information
|
2581
2811
|
|
2582
2812
|
```
|
2583
|
-
|
2813
|
+
ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
2584
2814
|
```
|
2585
2815
|
|
2586
2816
|
d- Convert to creation data for second workspace:
|
@@ -2598,13 +2828,13 @@ jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"
|
|
2598
2828
|
e- Add members to second workspace
|
2599
2829
|
|
2600
2830
|
```
|
2601
|
-
|
2831
|
+
ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
2602
2832
|
```
|
2603
2833
|
|
2604
|
-
|
2834
|
+
### Example: Get users who did not log since a date
|
2605
2835
|
|
2606
2836
|
```
|
2607
|
-
|
2837
|
+
ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
2608
2838
|
:...............................:
|
2609
2839
|
: email :
|
2610
2840
|
:...............................:
|
@@ -2613,20 +2843,20 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_a
|
|
2613
2843
|
:...............................:
|
2614
2844
|
```
|
2615
2845
|
|
2616
|
-
|
2846
|
+
### Example: List "Limited" users
|
2617
2847
|
|
2618
2848
|
```
|
2619
|
-
|
2849
|
+
ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
2620
2850
|
```
|
2621
2851
|
|
2622
|
-
|
2852
|
+
### Example: Perform a multi Gbps transfer between two remote shared folders
|
2623
2853
|
|
2624
|
-
In this example, a user has access to a workspace where two shared folders are located on
|
2854
|
+
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
2625
2855
|
|
2626
2856
|
First, setup the environment (skip if already done)
|
2627
2857
|
|
2628
2858
|
```
|
2629
|
-
|
2859
|
+
ascli conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
|
2630
2860
|
Detected: Aspera on Cloud
|
2631
2861
|
Preparing preset: aoc_sedemo
|
2632
2862
|
Using existing key:
|
@@ -2645,7 +2875,7 @@ Setting config preset as default for aspera
|
|
2645
2875
|
saving config file
|
2646
2876
|
Done.
|
2647
2877
|
You can test with:
|
2648
|
-
|
2878
|
+
ascli aoc user info show
|
2649
2879
|
```
|
2650
2880
|
|
2651
2881
|
This creates the option preset "aoc_<org name>" to allow seamless command line access and sets it as default for aspera on cloud.
|
@@ -2655,20 +2885,20 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
2655
2885
|
Then, transfer between those:
|
2656
2886
|
|
2657
2887
|
```
|
2658
|
-
|
2888
|
+
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
2659
2889
|
```
|
2660
2890
|
|
2661
|
-
|
2891
|
+
### Example: create registration key to register a node
|
2662
2892
|
|
2663
2893
|
```
|
2664
|
-
|
2894
|
+
ascli aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
2665
2895
|
jfqslfdjlfdjfhdjklqfhdkl
|
2666
2896
|
```
|
2667
2897
|
|
2668
|
-
|
2898
|
+
### Example: delete all registration keys
|
2669
2899
|
|
2670
2900
|
```
|
2671
|
-
|
2901
|
+
ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res client delete --bulk=yes --id=@lines:@stdin:
|
2672
2902
|
+-----+---------+
|
2673
2903
|
| id | status |
|
2674
2904
|
+-----+---------+
|
@@ -2679,81 +2909,107 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
|
|
2679
2909
|
+-----+---------+
|
2680
2910
|
```
|
2681
2911
|
|
2682
|
-
|
2912
|
+
### Example: Create a node
|
2683
2913
|
|
2684
|
-
|
2685
|
-
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2914
|
+
AoC nodes as actually composed with two related entities:
|
2686
2915
|
|
2687
|
-
|
2916
|
+
* An access key created on the Transfer Server (HSTS/ATS)
|
2917
|
+
* a `node` resource in the AoC application.
|
2688
2918
|
|
2689
|
-
|
2690
|
-
|
2919
|
+
The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
|
2920
|
+
Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
|
2691
2921
|
|
2692
|
-
|
2922
|
+
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
2923
|
+
|
2924
|
+
* create the access key on ATS
|
2925
|
+
|
2926
|
+
```
|
2927
|
+
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
2693
2928
|
```
|
2694
2929
|
|
2695
|
-
|
2930
|
+
Take a note of the randomly generated `id` and `secret`.
|
2696
2931
|
|
2697
|
-
*
|
2932
|
+
* Retrieve the ATS node address
|
2698
2933
|
|
2699
2934
|
```
|
2700
|
-
|
2935
|
+
ascli aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
|
2701
2936
|
```
|
2702
2937
|
|
2703
|
-
*
|
2938
|
+
* Create the node entity
|
2704
2939
|
|
2705
2940
|
```
|
2706
|
-
|
2941
|
+
ascli aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
2707
2942
|
```
|
2708
2943
|
|
2709
|
-
|
2944
|
+
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
2945
|
+
|
2946
|
+
### Example: List packages in a given shared inbox
|
2947
|
+
|
2948
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
2949
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
2950
|
+
|
2951
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
2710
2952
|
|
2711
2953
|
```
|
2712
|
-
|
2954
|
+
shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
2955
|
+
|
2956
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
2713
2957
|
```
|
2714
2958
|
|
2715
|
-
##
|
2959
|
+
## Packages
|
2960
|
+
|
2961
|
+
The webmail-like application.
|
2962
|
+
|
2963
|
+
### Send a Package
|
2716
2964
|
|
2717
2965
|
Send a package:
|
2718
2966
|
|
2719
2967
|
```
|
2720
|
-
|
2968
|
+
ascli aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
|
2721
2969
|
```
|
2722
2970
|
|
2723
2971
|
Notes:
|
2724
2972
|
|
2725
|
-
*
|
2726
|
-
*
|
2727
|
-
*
|
2728
|
-
*
|
2729
|
-
*
|
2973
|
+
* The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
2974
|
+
* List allowed shared inbox destinations with: `ascli aoc user shared_inboxes`
|
2975
|
+
* Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
2976
|
+
* Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
2977
|
+
* or just names: `"recipients":[{"The Dest"}]` . ascli will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
2978
|
+
* If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
2979
|
+
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
2730
2980
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
2731
2981
|
|
2732
2982
|
Examples:
|
2733
2983
|
|
2984
|
+
* Send a package with one file to two users, using their email
|
2985
|
+
|
2734
2986
|
```
|
2735
|
-
|
2987
|
+
ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
2736
2988
|
```
|
2737
2989
|
|
2990
|
+
* Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
|
2991
|
+
|
2738
2992
|
```
|
2739
|
-
|
2993
|
+
ascli aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
|
2740
2994
|
```
|
2741
2995
|
|
2996
|
+
* Send a package with one file to a shared inbox (by name) with metadata
|
2997
|
+
|
2742
2998
|
```
|
2743
|
-
|
2999
|
+
ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
2744
3000
|
```
|
2745
3001
|
|
2746
|
-
|
3002
|
+
### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
2747
3003
|
|
2748
3004
|
It is possible to automatically download new packages, like using Aspera Cargo:
|
2749
3005
|
|
2750
3006
|
```
|
2751
|
-
|
3007
|
+
ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
2752
3008
|
```
|
2753
3009
|
|
2754
3010
|
* `--id=ALL` (case sensitive) will download all packages
|
2755
|
-
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
2756
|
-
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid
|
3011
|
+
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
3012
|
+
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
2757
3013
|
|
2758
3014
|
Typically, one would execute this command on a regular basis, using the method of your choice:
|
2759
3015
|
|
@@ -2761,101 +3017,107 @@ Typically, one would execute this command on a regular basis, using the method o
|
|
2761
3017
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
2762
3018
|
* etc...
|
2763
3019
|
|
2764
|
-
##
|
3020
|
+
## Files
|
3021
|
+
|
3022
|
+
Folder sharing app.
|
3023
|
+
|
3024
|
+
### Download Files
|
2765
3025
|
|
2766
3026
|
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
2767
3027
|
|
2768
3028
|
* if only one source is provided, it is downloaded
|
2769
3029
|
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
2770
3030
|
|
2771
|
-
|
2772
|
-
|
2773
|
-
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2774
|
-
|
2775
|
-
The expression can be of 3 formats:
|
2776
|
-
|
2777
|
-
* empty (default) : all files, equivalent to value: `exec:true`
|
2778
|
-
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2779
|
-
|
2780
|
-
For instance, to find files with a special extension, use `--value='\.myext$'`
|
2781
|
-
|
2782
|
-
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
2783
|
-
|
2784
|
-
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
3031
|
+
### Shared folders
|
2785
3032
|
|
2786
|
-
*
|
3033
|
+
* list shared folders in node
|
2787
3034
|
|
2788
3035
|
```
|
2789
|
-
|
3036
|
+
ascli aoc admin res node --id=8669 shared_folders
|
2790
3037
|
```
|
2791
3038
|
|
2792
|
-
*
|
3039
|
+
* list shared folders in workspace
|
2793
3040
|
|
2794
3041
|
```
|
2795
|
-
|
3042
|
+
ascli aoc admin res workspace --id=10818 shared_folders
|
2796
3043
|
```
|
2797
3044
|
|
2798
|
-
*
|
3045
|
+
* list members of shared folder
|
2799
3046
|
|
2800
3047
|
```
|
2801
|
-
|
3048
|
+
ascli aoc admin res node --id=8669 v4 perm 82 show
|
2802
3049
|
```
|
2803
3050
|
|
2804
|
-
|
3051
|
+
### Cross Organization transfers
|
2805
3052
|
|
2806
|
-
|
2807
|
-
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
2808
|
-
```
|
3053
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
2809
3054
|
|
2810
|
-
|
3055
|
+
Although optional, the creation of [option preset](#lprt) is recommended to avoid placing all parameters in the command line.
|
2811
3056
|
|
2812
|
-
|
3057
|
+
Procedure to send a file from org1 to org2:
|
3058
|
+
|
3059
|
+
* Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
3060
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
3061
|
+
* Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
3062
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
3063
|
+
* execute the following:
|
2813
3064
|
|
2814
3065
|
```
|
2815
|
-
|
3066
|
+
ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
2816
3067
|
```
|
2817
3068
|
|
2818
|
-
|
3069
|
+
Explanation:
|
2819
3070
|
|
2820
|
-
`
|
3071
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
3072
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
3073
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
3074
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
3075
|
+
* `|` the standard output of the first command is fed into the second one
|
3076
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
3077
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
3078
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
3079
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2821
3080
|
|
2822
|
-
|
2823
|
-
From: <%=from_name%> <<%=from_email%>>
|
2824
|
-
To: <<%=ev['user_email']%>>
|
2825
|
-
Subject: <%=ev['files_completed']%> files received
|
3081
|
+
### Find Files
|
2826
3082
|
|
2827
|
-
|
2828
|
-
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
|
2829
|
-
<%=ev['content']%>
|
3083
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
2830
3084
|
|
2831
|
-
|
2832
|
-
```
|
2833
|
-
The environment provided contains the following additional variable:
|
3085
|
+
The expression can be of 3 formats:
|
2834
3086
|
|
2835
|
-
*
|
3087
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
3088
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
2836
3089
|
|
2837
|
-
|
3090
|
+
For instance, to find files with a special extension, use `--value='\.myext$'`
|
3091
|
+
|
3092
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
3093
|
+
|
3094
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
3095
|
+
|
3096
|
+
* Find files more recent than 100 days
|
2838
3097
|
|
2839
3098
|
```
|
2840
|
-
|
2841
|
-
--query=@json:'{"status":"completed","direction":"receive"}' \
|
2842
|
-
--notif-to=active --notif-template=@file:mytemplate.erb
|
3099
|
+
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
2843
3100
|
```
|
2844
3101
|
|
2845
|
-
|
3102
|
+
* Find files older than 1 year on a given node and store in file list
|
2846
3103
|
|
2847
|
-
|
2848
|
-
|
2849
|
-
|
3104
|
+
```
|
3105
|
+
ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
3106
|
+
```
|
2850
3107
|
|
2851
|
-
|
3108
|
+
* Delete the files, one by one
|
2852
3109
|
|
2853
|
-
|
3110
|
+
```
|
3111
|
+
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
3112
|
+
```
|
2854
3113
|
|
2855
|
-
|
3114
|
+
* Delete the files in bulk
|
2856
3115
|
|
3116
|
+
```
|
3117
|
+
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
3118
|
+
```
|
2857
3119
|
|
2858
|
-
# Plugin: Aspera Transfer Service
|
3120
|
+
# <a id="ats"></a>Plugin: Aspera Transfer Service
|
2859
3121
|
|
2860
3122
|
ATS is usable either :
|
2861
3123
|
|
@@ -2865,10 +3127,13 @@ ATS is usable either :
|
|
2865
3127
|
|
2866
3128
|
## IBM Cloud ATS : creation of api key
|
2867
3129
|
|
3130
|
+
This section is about using ATS with an IBM cloud subscription.
|
3131
|
+
If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
|
3132
|
+
|
2868
3133
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
2869
3134
|
|
2870
3135
|
```
|
2871
|
-
|
3136
|
+
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
2872
3137
|
OK
|
2873
3138
|
API key mykeyname was created
|
2874
3139
|
|
@@ -2887,78 +3152,47 @@ References:
|
|
2887
3152
|
* [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
2888
3153
|
* [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
2889
3154
|
|
2890
|
-
|
2891
3155
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
2892
3156
|
|
2893
3157
|
```
|
2894
|
-
|
2895
|
-
|
2896
|
-
|
3158
|
+
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
|
3159
|
+
ascli config preset set default ats my_ibm_ats
|
3160
|
+
ascli ats api_key instances
|
2897
3161
|
+--------------------------------------+
|
2898
3162
|
| instance |
|
2899
3163
|
+--------------------------------------+
|
2900
3164
|
| aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
|
2901
3165
|
+--------------------------------------+
|
2902
|
-
|
2903
|
-
|
3166
|
+
ascli config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
|
3167
|
+
ascli ats api_key create
|
2904
3168
|
+--------+----------------------------------------------+
|
2905
3169
|
| key | value |
|
2906
3170
|
+--------+----------------------------------------------+
|
2907
3171
|
| id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
|
2908
3172
|
| secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
|
2909
3173
|
+--------+----------------------------------------------+
|
2910
|
-
|
2911
|
-
```
|
2912
|
-
## Cross Organization transfers
|
2913
|
-
|
2914
|
-
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
2915
|
-
|
2916
|
-
Although optional, the creation of [option preset](#lprt) is recommended to avoid placing all parameters in the command line.
|
2917
|
-
|
2918
|
-
Procedure to send a file from org1 to org2:
|
2919
|
-
|
2920
|
-
* Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
2921
|
-
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
2922
|
-
* Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
2923
|
-
* Check that access works and locate the destination folder `mydestfolder`
|
2924
|
-
* execute the following:
|
2925
|
-
|
3174
|
+
ascli config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
2926
3175
|
```
|
2927
|
-
$ ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
2928
|
-
```
|
2929
|
-
|
2930
|
-
Explanation:
|
2931
|
-
|
2932
|
-
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
2933
|
-
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
2934
|
-
* `--format=json` format the output in JSON (instead of default text table)
|
2935
|
-
* `--display=data` display only the result, and remove other information, such as workspace name
|
2936
|
-
* `|` the standard output of the first command is fed into the second one
|
2937
|
-
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
2938
|
-
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
2939
|
-
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#direct)
|
2940
|
-
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
2941
3176
|
|
2942
|
-
Note that when using a POSIX shell, another possibility to write `cmd1 | cmd2 --transfer-info=@json:stdin:` is `cmd2 --transfer-info=@json:$(cmd1)` instead of ``
|
2943
3177
|
## Examples
|
2944
3178
|
|
2945
|
-
Example: create access key on softlayer:
|
3179
|
+
Example: create access key on IBM Cloud (softlayer):
|
2946
3180
|
|
2947
3181
|
```
|
2948
|
-
|
3182
|
+
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
2949
3183
|
```
|
2950
3184
|
|
2951
3185
|
Example: create access key on AWS:
|
2952
3186
|
|
2953
3187
|
```
|
2954
|
-
|
3188
|
+
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
|
2955
3189
|
|
2956
3190
|
```
|
2957
3191
|
|
2958
3192
|
Example: create access key on Azure SAS:
|
2959
3193
|
|
2960
3194
|
```
|
2961
|
-
|
3195
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
2962
3196
|
|
2963
3197
|
```
|
2964
3198
|
|
@@ -2967,7 +3201,7 @@ $ ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id
|
|
2967
3201
|
Example: create access key on Azure:
|
2968
3202
|
|
2969
3203
|
```
|
2970
|
-
|
3204
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
2971
3205
|
|
2972
3206
|
```
|
2973
3207
|
|
@@ -2977,74 +3211,77 @@ delete all my access keys:
|
|
2977
3211
|
for k in $(ascli ats access_key list --field=id --format=csv);do ascli ats access_key id $k delete;done
|
2978
3212
|
```
|
2979
3213
|
|
3214
|
+
The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
|
3215
|
+
|
2980
3216
|
# Plugin: IBM Aspera High Speed Transfer Server (transfer)
|
2981
3217
|
|
2982
|
-
This plugin
|
3218
|
+
This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
|
3219
|
+
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
3220
|
+
Modern mode is to use the node API and transfer tokens.
|
2983
3221
|
|
2984
3222
|
## Authentication
|
2985
3223
|
|
2986
3224
|
Both password and SSH keys auth are supported.
|
2987
3225
|
|
2988
|
-
If
|
3226
|
+
If username is not provided, the default transfer user `xfer` is used.
|
2989
3227
|
|
2990
|
-
If no
|
3228
|
+
If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
|
2991
3229
|
|
2992
3230
|
```
|
2993
|
-
|
3231
|
+
ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
2994
3232
|
```
|
2995
3233
|
|
2996
|
-
Multiple SSH key paths can be provided.
|
3234
|
+
Multiple SSH key paths can be provided.
|
3235
|
+
The value of the parameter `ssh_keys` can be a single value or an array.
|
3236
|
+
Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
|
2997
3237
|
|
2998
3238
|
Examples:
|
2999
3239
|
|
3000
3240
|
```
|
3001
|
-
|
3002
|
-
|
3003
|
-
|
3241
|
+
ascli server --ssh-keys=~/.ssh/id_rsa
|
3242
|
+
ascli server --ssh-keys=@list:,~/.ssh/id_rsa
|
3243
|
+
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
3004
3244
|
```
|
3005
3245
|
|
3006
|
-
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
3246
|
+
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
3247
|
+
By default the ssh library expect that an ssh-agent is running.
|
3007
3248
|
|
3008
|
-
|
3249
|
+
On Linux, if you get an error message such as:
|
3009
3250
|
|
3010
3251
|
```
|
3011
|
-
[Linux]
|
3012
3252
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
|
3013
3253
|
```
|
3014
3254
|
|
3015
|
-
or
|
3255
|
+
or on Windows:
|
3016
3256
|
|
3017
3257
|
```
|
3018
|
-
[Windows]
|
3019
3258
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
3020
3259
|
```
|
3021
3260
|
|
3022
|
-
This means that you
|
3261
|
+
This means that you don't have such an ssh agent running, then:
|
3023
3262
|
|
3024
3263
|
* check env var: `SSH_AGENT_SOCK`
|
3025
|
-
* check if the key is protected with a passphrase
|
3264
|
+
* check if the ssh key is protected with a passphrase
|
3026
3265
|
* [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
3027
|
-
* To
|
3266
|
+
* To disable use of `ssh-agent`, use the option `ssh_option` like this:
|
3028
3267
|
|
3029
3268
|
```
|
3030
|
-
|
3269
|
+
ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
3031
3270
|
```
|
3032
3271
|
|
3033
|
-
This can also be set as default using a preset.
|
3272
|
+
This can also be set as default using a global preset.
|
3034
3273
|
|
3035
3274
|
## Example
|
3036
3275
|
|
3037
|
-
One can test the
|
3276
|
+
One can test the `server` application using the well known demo server:
|
3038
3277
|
|
3039
3278
|
```
|
3040
|
-
|
3041
|
-
|
3042
|
-
|
3043
|
-
$ ascli server download /aspera-test-dir-large/200MB
|
3279
|
+
ascli config initdemo
|
3280
|
+
ascli server browse /aspera-test-dir-large
|
3281
|
+
ascli server download /aspera-test-dir-large/200MB
|
3044
3282
|
```
|
3045
3283
|
|
3046
|
-
|
3047
|
-
|
3284
|
+
`initdemo` creates a [option preset](#lprt) `demoserver` and set it as default for plugin `server`.
|
3048
3285
|
|
3049
3286
|
# Plugin: IBM Aspera High Speed Transfer Server (node)
|
3050
3287
|
|
@@ -3076,12 +3313,12 @@ For transfers, it is possible to control how transfer is authorized using option
|
|
3076
3313
|
|
3077
3314
|
## Central
|
3078
3315
|
|
3079
|
-
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and
|
3316
|
+
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
|
3080
3317
|
|
3081
3318
|
Filtering can be applied:
|
3082
3319
|
|
3083
3320
|
```
|
3084
|
-
|
3321
|
+
ascli node central file list
|
3085
3322
|
```
|
3086
3323
|
|
3087
3324
|
by providing the `validator` option, offline transfer validation can be done.
|
@@ -3093,7 +3330,7 @@ It is possible to start a FASPStream session using the node API:
|
|
3093
3330
|
Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
|
3094
3331
|
|
3095
3332
|
```
|
3096
|
-
|
3333
|
+
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
|
3097
3334
|
```
|
3098
3335
|
|
3099
3336
|
## Watchfolder
|
@@ -3106,9 +3343,9 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
3106
3343
|
* configure a watchfolder to define automated transfers
|
3107
3344
|
|
3108
3345
|
```
|
3109
|
-
|
3110
|
-
|
3111
|
-
|
3346
|
+
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
3347
|
+
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
3348
|
+
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
3112
3349
|
```
|
3113
3350
|
|
3114
3351
|
## Out of Transfer File Validation
|
@@ -3116,13 +3353,13 @@ $ ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1"
|
|
3116
3353
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
3117
3354
|
|
3118
3355
|
```
|
3119
|
-
|
3356
|
+
ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
3120
3357
|
:..............:..............:............:......................................:
|
3121
3358
|
: session_uuid : file_id : status : path :
|
3122
3359
|
:..............:..............:............:......................................:
|
3123
3360
|
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
|
3124
3361
|
:..............:..............:............:......................................:
|
3125
|
-
|
3362
|
+
ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
3126
3363
|
updated
|
3127
3364
|
```
|
3128
3365
|
|
@@ -3138,7 +3375,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
|
|
3138
3375
|
Then execute the following command:
|
3139
3376
|
|
3140
3377
|
```
|
3141
|
-
|
3378
|
+
ascli node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
|
3142
3379
|
```
|
3143
3380
|
|
3144
3381
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance
|
@@ -3147,7 +3384,7 @@ to download files.
|
|
3147
3384
|
## Create access key
|
3148
3385
|
|
3149
3386
|
```
|
3150
|
-
|
3387
|
+
ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
3151
3388
|
```
|
3152
3389
|
|
3153
3390
|
# Plugin: IBM Aspera Faspex5
|
@@ -3172,7 +3409,7 @@ For boot method: (will be removed in future)
|
|
3172
3409
|
Use it as password and use `--auth=boot`.
|
3173
3410
|
|
3174
3411
|
```
|
3175
|
-
|
3412
|
+
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
3176
3413
|
```
|
3177
3414
|
|
3178
3415
|
Ready to use Faspex5 with CLI.
|
@@ -3218,7 +3455,7 @@ If no parameter `max` or `pmax` is provided, then all packages will be listed in
|
|
3218
3455
|
### Example
|
3219
3456
|
|
3220
3457
|
```
|
3221
|
-
|
3458
|
+
ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
3222
3459
|
```
|
3223
3460
|
|
3224
3461
|
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
@@ -3232,14 +3469,14 @@ The command is `package recv`, possible methods are:
|
|
3232
3469
|
* provide a `faspe:` URI with option `link`
|
3233
3470
|
|
3234
3471
|
```
|
3235
|
-
|
3236
|
-
|
3472
|
+
ascli faspex package recv --id=12345
|
3473
|
+
ascli faspex package recv --link=faspe://...
|
3237
3474
|
```
|
3238
3475
|
|
3239
3476
|
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
3240
3477
|
|
3241
3478
|
```
|
3242
|
-
|
3479
|
+
ascli faspex package list --recipient='*thedropboxname'
|
3243
3480
|
```
|
3244
3481
|
|
3245
3482
|
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
@@ -3251,7 +3488,7 @@ The command is `faspex package send`. Package information (title, note, metadata
|
|
3251
3488
|
Example:
|
3252
3489
|
|
3253
3490
|
```
|
3254
|
-
|
3491
|
+
ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
3255
3492
|
```
|
3256
3493
|
|
3257
3494
|
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
@@ -3268,7 +3505,7 @@ Like for any transfer, a notification can be sent by email using parameters: `no
|
|
3268
3505
|
Example:
|
3269
3506
|
|
3270
3507
|
```
|
3271
|
-
|
3508
|
+
ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
3272
3509
|
```
|
3273
3510
|
|
3274
3511
|
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
@@ -3278,9 +3515,9 @@ In this example the notification template is directly provided on command line.
|
|
3278
3515
|
Example:
|
3279
3516
|
|
3280
3517
|
```
|
3281
|
-
|
3282
|
-
|
3283
|
-
|
3518
|
+
ascli faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
3519
|
+
ascli faspex v4 dropbox list
|
3520
|
+
ascli faspex v4 dropbox delete --id=36
|
3284
3521
|
```
|
3285
3522
|
|
3286
3523
|
## Remote sources
|
@@ -3318,7 +3555,7 @@ It is possible to tell `ascli` to download newly received packages, much like th
|
|
3318
3555
|
cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
|
3319
3556
|
|
3320
3557
|
```
|
3321
|
-
|
3558
|
+
ascli faspex packages recv --id=ALL --once-only=yes --lock-port=12345
|
3322
3559
|
```
|
3323
3560
|
|
3324
3561
|
# Plugin: IBM Aspera Shares
|
@@ -3328,7 +3565,7 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
3328
3565
|
In Shares2, users, groups listing are paged, to display sequential pages:
|
3329
3566
|
|
3330
3567
|
```
|
3331
|
-
|
3568
|
+
for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}';done
|
3332
3569
|
```
|
3333
3570
|
|
3334
3571
|
# Plugin: IBM Cloud Object Storage
|
@@ -3337,9 +3574,9 @@ The IBM Cloud Object Storage provides the possibility to execute transfers using
|
|
3337
3574
|
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
3338
3575
|
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
3339
3576
|
|
3340
|
-
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the
|
3577
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
3341
3578
|
|
3342
|
-
## Using endpoint, apikey and
|
3579
|
+
## Using endpoint, apikey and Resource Instance ID (CRN)
|
3343
3580
|
|
3344
3581
|
If you have those parameters already, then following options shall be provided:
|
3345
3582
|
|
@@ -3351,15 +3588,15 @@ If you have those parameters already, then following options shall be provided:
|
|
3351
3588
|
For example, let us create a default configuration:
|
3352
3589
|
|
3353
3590
|
```
|
3354
|
-
|
3355
|
-
|
3591
|
+
ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
3592
|
+
ascli conf id default set cos mycos
|
3356
3593
|
```
|
3357
3594
|
|
3358
3595
|
Then, jump to the transfer example.
|
3359
3596
|
|
3360
3597
|
## Using service credential file
|
3361
3598
|
|
3362
|
-
If you are the COS administrator and
|
3599
|
+
If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
3363
3600
|
|
3364
3601
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
3365
3602
|
|
@@ -3368,11 +3605,11 @@ Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
3368
3605
|
or using the IBM Cloud CLI:
|
3369
3606
|
|
3370
3607
|
```
|
3371
|
-
|
3372
|
-
|
3608
|
+
ibmcloud resource service-keys
|
3609
|
+
ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
3373
3610
|
```
|
3374
3611
|
|
3375
|
-
(if you
|
3612
|
+
(if you don't have `jq` installed, extract the structure as follows)
|
3376
3613
|
|
3377
3614
|
It consists in the following structure:
|
3378
3615
|
|
@@ -3407,8 +3644,8 @@ The required options for this method are:
|
|
3407
3644
|
For example, let us create a default configuration:
|
3408
3645
|
|
3409
3646
|
```
|
3410
|
-
|
3411
|
-
|
3647
|
+
ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
3648
|
+
ascli conf id default set cos mycos
|
3412
3649
|
```
|
3413
3650
|
|
3414
3651
|
## Operations, transfers
|
@@ -3418,8 +3655,8 @@ Let's assume you created a default configuration from once of the two previous s
|
|
3418
3655
|
A subset of `node` plugin operations are supported, basically node API:
|
3419
3656
|
|
3420
3657
|
```
|
3421
|
-
|
3422
|
-
|
3658
|
+
ascli cos node info
|
3659
|
+
ascli cos node upload 'faux:///sample1G?1g'
|
3423
3660
|
```
|
3424
3661
|
|
3425
3662
|
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
@@ -3448,8 +3685,10 @@ Specify the previews folder as shown in:
|
|
3448
3685
|
By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
|
3449
3686
|
|
3450
3687
|
```
|
3451
|
-
|
3452
|
-
|
3688
|
+
PATH=/opt/aspera/bin:$PATH
|
3689
|
+
|
3690
|
+
asconfigurator -x "server;preview_dir,previews"
|
3691
|
+
asnodeadmin --reload
|
3453
3692
|
```
|
3454
3693
|
|
3455
3694
|
Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
|
@@ -3462,16 +3701,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
|
|
3462
3701
|
To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
|
3463
3702
|
|
3464
3703
|
```
|
3465
|
-
|
3704
|
+
asuserdata -a | grep max_request_file_create_size_kb
|
3705
|
+
|
3466
3706
|
max_request_file_create_size_kb: "1024"
|
3467
|
-
|
3707
|
+
|
3708
|
+
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
3468
3709
|
```
|
3469
3710
|
|
3470
3711
|
If you use a value different than 16777216, then specify it using option `max_size`.
|
3471
3712
|
|
3472
3713
|
Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
|
3473
3714
|
|
3474
|
-
## <a
|
3715
|
+
## <a id="prev_ext"></a>External tools: Linux
|
3475
3716
|
|
3476
3717
|
The tool requires the following external tools available in the `PATH`:
|
3477
3718
|
|
@@ -3488,7 +3729,7 @@ Other OSes should work as well, but are note tested.
|
|
3488
3729
|
To check if all tools are found properly, execute:
|
3489
3730
|
|
3490
3731
|
```
|
3491
|
-
|
3732
|
+
ascli preview check
|
3492
3733
|
```
|
3493
3734
|
|
3494
3735
|
### mimemagic
|
@@ -3496,22 +3737,22 @@ $ ascli preview check
|
|
3496
3737
|
To benefit from extra mime type detection install gem mimemagic:
|
3497
3738
|
|
3498
3739
|
```
|
3499
|
-
|
3740
|
+
gem install mimemagic
|
3500
3741
|
```
|
3501
3742
|
|
3502
3743
|
or to install an earlier version if any problem:
|
3503
3744
|
|
3504
3745
|
```
|
3505
|
-
|
3746
|
+
gem install mimemagic -v '~> 0.3.0'
|
3506
3747
|
```
|
3507
3748
|
|
3508
3749
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
3509
3750
|
|
3510
3751
|
If not used, Mime type used for conversion is the one provided by the node API.
|
3511
3752
|
|
3512
|
-
If used, it the `preview` command will first
|
3753
|
+
If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
|
3513
3754
|
|
3514
|
-
### Image:
|
3755
|
+
### Image: ImageMagick and optipng
|
3515
3756
|
|
3516
3757
|
```
|
3517
3758
|
yum install -y ImageMagick optipng
|
@@ -3527,25 +3768,25 @@ curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.ta
|
|
3527
3768
|
|
3528
3769
|
### Office: Unoconv and Libreoffice
|
3529
3770
|
|
3530
|
-
If you
|
3771
|
+
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
3531
3772
|
|
3532
3773
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
3533
3774
|
|
3534
|
-
*
|
3775
|
+
* CentOS 8
|
3535
3776
|
|
3536
3777
|
```
|
3537
|
-
|
3778
|
+
dnf install unoconv
|
3538
3779
|
```
|
3539
3780
|
|
3540
3781
|
* Amazon Linux
|
3541
3782
|
|
3542
3783
|
```
|
3543
|
-
|
3544
|
-
|
3545
|
-
|
3546
|
-
|
3547
|
-
|
3548
|
-
|
3784
|
+
amazon-linux-extras enable libreoffice
|
3785
|
+
yum clean metadata
|
3786
|
+
yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
3787
|
+
wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
3788
|
+
mv unoconv /usr/bin
|
3789
|
+
chmod a+x /usr/bin/unoconv
|
3549
3790
|
```
|
3550
3791
|
|
3551
3792
|
## Configuration
|
@@ -3557,9 +3798,11 @@ Like any `ascli` commands, parameters can be passed on command line or using a c
|
|
3557
3798
|
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
3558
3799
|
|
3559
3800
|
```
|
3560
|
-
|
3561
|
-
|
3562
|
-
|
3801
|
+
su -s /bin/bash - xfer
|
3802
|
+
|
3803
|
+
ascli config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
3804
|
+
|
3805
|
+
ascli config preset set default preview previewconf
|
3563
3806
|
```
|
3564
3807
|
|
3565
3808
|
Here we assume that Office file generation is disabled, else remove this option.
|
@@ -3568,7 +3811,7 @@ Here we assume that Office file generation is disabled, else remove this option.
|
|
3568
3811
|
One can check if the access key is well configured using:
|
3569
3812
|
|
3570
3813
|
```
|
3571
|
-
|
3814
|
+
ascli -Ppreviewconf node browse /
|
3572
3815
|
```
|
3573
3816
|
|
3574
3817
|
This shall list the contents of the storage root of the access key.
|
@@ -3584,8 +3827,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
|
|
3584
3827
|
Lets do a one shot test, using the configuration previously created:
|
3585
3828
|
|
3586
3829
|
```
|
3587
|
-
|
3588
|
-
|
3830
|
+
su -s /bin/bash - xfer
|
3831
|
+
|
3832
|
+
ascli preview scan --overwrite=always
|
3589
3833
|
```
|
3590
3834
|
|
3591
3835
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
@@ -3599,13 +3843,13 @@ Adapt the scripts to your own preference.
|
|
3599
3843
|
|
3600
3844
|
We assume here that a configuration preset was created as shown previously.
|
3601
3845
|
|
3602
|
-
Lets first setup a script that will be used in the
|
3846
|
+
Lets first setup a script that will be used in the scheduler and sets up the environment.
|
3603
3847
|
|
3604
3848
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
3605
3849
|
|
3606
3850
|
```
|
3607
|
-
#!/bin/bash
|
3608
|
-
# set a timeout protection, just in case
|
3851
|
+
#!/bin/bash
|
3852
|
+
# set a timeout protection, just in case
|
3609
3853
|
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
3610
3854
|
. /etc/profile.d/rvm.sh
|
3611
3855
|
rvm use 2.6 --quiet
|
@@ -3615,13 +3859,13 @@ exec timeout ${tmout} ascli "${@}"
|
|
3615
3859
|
Here the cronjob is created for user `xfer`.
|
3616
3860
|
|
3617
3861
|
```
|
3618
|
-
|
3862
|
+
crontab<<EOF
|
3619
3863
|
0 * * * * /home/xfer/cron_ascli preview scan --logger=syslog --display=error
|
3620
3864
|
2-59 * * * * /home/xfer/cron_ascli preview trev --logger=syslog --display=error
|
3621
3865
|
EOF
|
3622
3866
|
```
|
3623
3867
|
|
3624
|
-
Note that the
|
3868
|
+
Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
3625
3869
|
|
3626
3870
|
## Candidate detection for creation or update (or deletion)
|
3627
3871
|
|
@@ -3646,12 +3890,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
|
|
3646
3890
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
3647
3891
|
|
3648
3892
|
```
|
3649
|
-
|
3893
|
+
ascli preview scan --skip-folders=@json:'["/not_here"]'
|
3650
3894
|
```
|
3651
3895
|
|
3652
3896
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
3653
3897
|
|
3654
|
-
When scanning the option `value` has the same
|
3898
|
+
When scanning the option `value` has the same behavior as for the `node find` command.
|
3655
3899
|
|
3656
3900
|
For instance to filter out files beginning with `._` do:
|
3657
3901
|
|
@@ -3670,7 +3914,7 @@ Use option `skip_format` to skip generation of a format.
|
|
3670
3914
|
|
3671
3915
|
## Supported input Files types
|
3672
3916
|
|
3673
|
-
The preview generator supports
|
3917
|
+
The preview generator supports rendering of those file categories:
|
3674
3918
|
|
3675
3919
|
* image
|
3676
3920
|
* pdf
|
@@ -3690,7 +3934,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
|
|
3690
3934
|
|
3691
3935
|
## Access to original files and preview creation
|
3692
3936
|
|
3693
|
-
Standard open source tools are used to create
|
3937
|
+
Standard open source tools are used to create thumbnails and video previews.
|
3694
3938
|
Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
|
3695
3939
|
The tool provides 2 ways to read and write files with the option: `file_access`
|
3696
3940
|
|
@@ -3720,28 +3964,28 @@ The `smtp` option is a hash table (extended value) with the following fields:
|
|
3720
3964
|
## Example of configuration:
|
3721
3965
|
|
3722
3966
|
```
|
3723
|
-
|
3724
|
-
|
3725
|
-
|
3967
|
+
ascli config preset set smtp_google server smtp.google.com
|
3968
|
+
ascli config preset set smtp_google username john@gmail.com
|
3969
|
+
ascli config preset set smtp_google password P@ssw0rd
|
3726
3970
|
```
|
3727
3971
|
|
3728
3972
|
or
|
3729
3973
|
|
3730
3974
|
```
|
3731
|
-
|
3975
|
+
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
|
3732
3976
|
```
|
3733
3977
|
|
3734
3978
|
or
|
3735
3979
|
|
3736
3980
|
```
|
3737
|
-
|
3981
|
+
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
|
3738
3982
|
```
|
3739
3983
|
|
3740
|
-
Set this
|
3984
|
+
Set this configuration as global default, for instance:
|
3741
3985
|
|
3742
3986
|
```
|
3743
|
-
|
3744
|
-
|
3987
|
+
ascli config preset set cli_default smtp @val:@preset:smtp_google
|
3988
|
+
ascli config preset set default config cli_default
|
3745
3989
|
```
|
3746
3990
|
|
3747
3991
|
## Email templates
|
@@ -3763,8 +4007,8 @@ Other variables are defined depending on context.
|
|
3763
4007
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
3764
4008
|
|
3765
4009
|
```
|
3766
|
-
|
3767
|
-
|
4010
|
+
ascli config --smtp=@preset:smtp_google smtp
|
4011
|
+
ascli config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
3768
4012
|
```
|
3769
4013
|
|
3770
4014
|
## Notifications for transfer status
|
@@ -3773,7 +4017,7 @@ An e-mail notification can be sent upon transfer success and failure (one email
|
|
3773
4017
|
|
3774
4018
|
To activate, use option `notif_to`.
|
3775
4019
|
|
3776
|
-
A default e-mail template is used, but it can be
|
4020
|
+
A default e-mail template is used, but it can be overridden with option `notif_template`.
|
3777
4021
|
|
3778
4022
|
The environment provided contains the following additional variables:
|
3779
4023
|
|
@@ -3797,7 +4041,7 @@ Transfer is: <%=global_transfer_status%>
|
|
3797
4041
|
This gem comes with a second executable tool providing a simplified standardized interface
|
3798
4042
|
to start a FASP session: `asession`.
|
3799
4043
|
|
3800
|
-
It aims at simplifying the startup of a FASP session from a programmatic stand point as
|
4044
|
+
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
|
3801
4045
|
|
3802
4046
|
* common to Aspera Node API (HTTP POST /ops/transfer)
|
3803
4047
|
* common to Aspera Connect API (browser javascript startTransfer)
|
@@ -3809,7 +4053,7 @@ This makes it easy to integrate with any language provided that one can spawn a
|
|
3809
4053
|
|
3810
4054
|
The tool expect one single argument: a [_transfer-spec_](#transferspec).
|
3811
4055
|
|
3812
|
-
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON
|
4056
|
+
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
|
3813
4057
|
|
3814
4058
|
Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
|
3815
4059
|
|
@@ -3820,7 +4064,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
3820
4064
|
* `EX_loglevel` to change log level of the tool
|
3821
4065
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
3822
4066
|
|
3823
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#
|
4067
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
3824
4068
|
|
3825
4069
|
## Comparison of interfaces
|
3826
4070
|
|
@@ -3847,7 +4091,7 @@ echo "${MY_TSPEC}"|asession
|
|
3847
4091
|
This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
|
3848
4092
|
|
3849
4093
|
```
|
3850
|
-
|
4094
|
+
asession
|
3851
4095
|
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
3852
4096
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
3853
4097
|
{"type":"DONE"}
|
@@ -3862,7 +4106,7 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
|
|
3862
4106
|
## Help
|
3863
4107
|
|
3864
4108
|
```
|
3865
|
-
|
4109
|
+
asession -h
|
3866
4110
|
USAGE
|
3867
4111
|
asession
|
3868
4112
|
asession -h|--help
|
@@ -3921,7 +4165,7 @@ Note that:
|
|
3921
4165
|
|
3922
4166
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
3923
4167
|
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
3924
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#
|
4168
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
|
3925
4169
|
|
3926
4170
|
### server side and configuration
|
3927
4171
|
|
@@ -3936,18 +4180,18 @@ Once `ascli` parameters are defined, run the command using the OS native schedul
|
|
3936
4180
|
## Example
|
3937
4181
|
|
3938
4182
|
```
|
3939
|
-
|
4183
|
+
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
|
3940
4184
|
|
3941
4185
|
```
|
3942
4186
|
|
3943
|
-
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they
|
4187
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
3944
4188
|
|
3945
|
-
#
|
4189
|
+
# Health check and Nagios
|
3946
4190
|
|
3947
|
-
|
4191
|
+
Most plugin provide a `health` command that will check the health status of the application. Example:
|
3948
4192
|
|
3949
4193
|
```
|
3950
|
-
|
4194
|
+
ascli console health
|
3951
4195
|
+--------+-------------+------------+
|
3952
4196
|
| status | component | message |
|
3953
4197
|
+--------+-------------+------------+
|
@@ -3964,13 +4208,13 @@ Typically, the health check uses the REST API of the application with the follow
|
|
3964
4208
|
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
3965
4209
|
|
3966
4210
|
```
|
3967
|
-
|
4211
|
+
ascli server health transfer --to-folder=/Upload --format=nagios --progress=none
|
3968
4212
|
OK - [transfer:ok]
|
3969
|
-
|
4213
|
+
ascli server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
3970
4214
|
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
3971
4215
|
```
|
3972
4216
|
|
3973
|
-
# Module: `Aspera`
|
4217
|
+
# Ruby Module: `Aspera`
|
3974
4218
|
|
3975
4219
|
Main components:
|
3976
4220
|
|
@@ -3981,8 +4225,8 @@ Main components:
|
|
3981
4225
|
A working example can be found in the gem, example:
|
3982
4226
|
|
3983
4227
|
```
|
3984
|
-
|
3985
|
-
|
4228
|
+
ascli config gem_path
|
4229
|
+
cat $(ascli config gem_path)/../examples/transfer.rb
|
3986
4230
|
```
|
3987
4231
|
|
3988
4232
|
This sample code shows some example of use of the API as well as
|
@@ -4017,132 +4261,143 @@ So, it evolved into `ascli`:
|
|
4017
4261
|
|
4018
4262
|
# Changes (Release notes)
|
4019
4263
|
|
4264
|
+
* 4.5.0
|
4265
|
+
|
4266
|
+
* new: support transfer agent: [Transfer SDK](#agt_trsdk)
|
4267
|
+
* new: support [http socket options](#http_options)
|
4268
|
+
* new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
|
4269
|
+
* new: `config vault` supports encrypted passwords, also macos keychain
|
4270
|
+
* new: `config preset` command for consistency with id
|
4271
|
+
* new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
|
4272
|
+
* change: when using wss, use [ruby's CA certs](#certificates)
|
4273
|
+
* change: unexpected parameter makes exit code not zero
|
4274
|
+
* change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
|
4275
|
+
* change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
|
4276
|
+
* change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
|
4277
|
+
* fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
|
4278
|
+
|
4020
4279
|
* 4.4.0
|
4021
4280
|
|
4022
|
-
|
4023
|
-
|
4024
|
-
|
4025
|
-
|
4026
|
-
|
4027
|
-
|
4028
|
-
|
4281
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
4282
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
4283
|
+
* new: `preset` option can specify name or hash value
|
4284
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
4285
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
4286
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
4287
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
4029
4288
|
|
4030
4289
|
* 4.3.0
|
4031
4290
|
|
4032
|
-
|
4033
|
-
|
4291
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
|
4292
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
4034
4293
|
|
4035
4294
|
* 4.2.2
|
4036
4295
|
|
4037
|
-
|
4038
|
-
|
4039
|
-
|
4040
|
-
|
4041
|
-
|
4042
|
-
|
4043
|
-
|
4044
|
-
|
4045
|
-
|
4296
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
4297
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
4298
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
4299
|
+
* new: `conf ascp spec` displays supported transfer spec
|
4300
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
4301
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
4302
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
4303
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then `ascli` exits with code 1 (instead of zero=success)
|
4304
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
4046
4305
|
|
4047
4306
|
* 4.2.1
|
4048
4307
|
|
4049
|
-
|
4050
|
-
|
4308
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
4309
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
4051
4310
|
|
4052
4311
|
* 4.2.0
|
4053
4312
|
|
4054
|
-
|
4055
|
-
|
4056
|
-
|
4057
|
-
|
4058
|
-
|
4059
|
-
|
4060
|
-
|
4061
|
-
|
4062
|
-
|
4063
|
-
|
4313
|
+
* new: command `aoc remind` to receive organization membership by email
|
4314
|
+
* new: in `preview` option `value` to filter out on file name
|
4315
|
+
* new: `initdemo` to initialize for demo server
|
4316
|
+
* new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
4317
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
4318
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
4319
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
4320
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
4321
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
4322
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
|
4064
4323
|
|
4065
4324
|
* 4.1.0
|
4066
4325
|
|
4067
|
-
|
4068
|
-
|
4069
|
-
|
4070
|
-
|
4071
|
-
|
4072
|
-
|
4073
|
-
|
4074
|
-
|
4075
|
-
|
4076
|
-
|
4077
|
-
|
4078
|
-
|
4326
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
4327
|
+
* new: update documentation with regard to offline and docker installation
|
4328
|
+
* new: renamed command `nagios_check` to `health`
|
4329
|
+
* new: agent `http_gw` now supports upload
|
4330
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
4331
|
+
* new: check new gem version periodically
|
4332
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
4333
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
4334
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
4335
|
+
* new: AoC a password can be provided for a public link
|
4336
|
+
* new: `conf doc` take an optional parameter to go to a section
|
4337
|
+
* new: initial support for Faspex 5 Beta 1
|
4079
4338
|
|
4080
4339
|
* 4.0.0
|
4081
4340
|
|
4082
|
-
|
4083
|
-
|
4084
|
-
|
4085
|
-
|
4086
|
-
|
4087
|
-
|
4088
|
-
|
4089
|
-
|
4341
|
+
* now available as open source at [https://github.com/IBM/aspera-cli](https://github.com/IBM/aspera-cli) with general cleanup
|
4342
|
+
* changed default tool name from `mlia` to `ascli`
|
4343
|
+
* changed `aspera` command to `aoc`
|
4344
|
+
* changed gem name from `asperalm` to `aspera-cli`
|
4345
|
+
* changed module name from `Asperalm` to `Aspera`
|
4346
|
+
* removed command `folder` in `preview`, merged to `scan`
|
4347
|
+
* persistency files go to sub folder instead of main folder
|
4348
|
+
* added possibility to install SDK: `config ascp install`
|
4090
4349
|
|
4091
4350
|
* 0.11.8
|
4092
4351
|
|
4093
|
-
|
4352
|
+
* Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
|
4094
4353
|
|
4095
4354
|
* 0.11.7
|
4096
4355
|
|
4097
|
-
|
4098
|
-
|
4099
|
-
|
4100
|
-
|
4101
|
-
|
4102
|
-
|
4356
|
+
* rework on rest call error handling
|
4357
|
+
* use option `display` with value `data` to remove out of extraneous information
|
4358
|
+
* fixed option `lock_port` not working
|
4359
|
+
* generate special icon if preview failed
|
4360
|
+
* possibility to choose transfer progress bar type with option `progress`
|
4361
|
+
* AoC package creation now output package id
|
4103
4362
|
|
4104
4363
|
* 0.11.6
|
4105
4364
|
|
4106
|
-
|
4107
|
-
|
4108
|
-
|
4109
|
-
|
4110
|
-
|
4111
|
-
|
4112
|
-
|
4365
|
+
* orchestrator : added more choice in auth type
|
4366
|
+
* preview: cleanup in generator (removed and renamed parameters)
|
4367
|
+
* preview: better documentation
|
4368
|
+
* preview: animated thumbnails for video (option: `video_png_conv=animated`)
|
4369
|
+
* preview: new event trigger: `trevents` (`events` seems broken)
|
4370
|
+
* preview: unique tmp folder to avoid clash of multiple instances
|
4371
|
+
* repo: added template for secrets used for testing
|
4113
4372
|
|
4114
4373
|
* 0.11.5
|
4115
4374
|
|
4116
|
-
|
4117
|
-
|
4118
|
-
|
4119
|
-
|
4120
|
-
|
4121
|
-
|
4375
|
+
* added option `default_ports` for AoC (see manual)
|
4376
|
+
* allow bulk delete in `aspera files` with option `bulk=yes`
|
4377
|
+
* fix getting connect versions
|
4378
|
+
* added section for Aix
|
4379
|
+
* support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
|
4380
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
|
4122
4381
|
|
4123
4382
|
* 0.11.4
|
4124
4383
|
|
4125
|
-
|
4384
|
+
* possibility to give shared inbox name when sending a package (else use id and type)
|
4126
4385
|
|
4127
4386
|
* 0.11.3
|
4128
4387
|
|
4129
|
-
|
4388
|
+
* minor fixes on multi-session: avoid exception on progress bar
|
4130
4389
|
|
4131
4390
|
* 0.11.2
|
4132
4391
|
|
4133
|
-
|
4392
|
+
* fixes on multi-session: progress bat and transfer spec param for "direct"
|
4134
4393
|
|
4135
4394
|
* 0.11.1
|
4136
4395
|
|
4137
|
-
|
4396
|
+
* enhanced short_link creation commands (see examples)
|
4138
4397
|
|
4139
4398
|
* 0.11
|
4140
4399
|
|
4141
|
-
|
4142
|
-
|
4143
|
-
```
|
4144
|
-
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
|
4145
|
-
```
|
4400
|
+
* add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
|
4146
4401
|
|
4147
4402
|
* 0.10.18
|
4148
4403
|
|
@@ -4151,7 +4406,7 @@ So, it evolved into `ascli`:
|
|
4151
4406
|
* 0.10.17
|
4152
4407
|
|
4153
4408
|
* fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
|
4154
|
-
* new modifier: `@list:<
|
4409
|
+
* new modifier: `@list:<separator>val1<separator>...`
|
4155
4410
|
|
4156
4411
|
* 0.10.16
|
4157
4412
|
|
@@ -4173,7 +4428,7 @@ So, it evolved into `ascli`:
|
|
4173
4428
|
* 0.10.12
|
4174
4429
|
|
4175
4430
|
* added support for AoC node registration keys
|
4176
|
-
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#
|
4431
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
|
4177
4432
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
4178
4433
|
* `@incps` : new extended value modifier
|
4179
4434
|
* ATS: no more provides access keys secrets: now user must provide it
|
@@ -4205,7 +4460,7 @@ So, it evolved into `ascli`:
|
|
4205
4460
|
|
4206
4461
|
* 0.10.6
|
4207
4462
|
|
4208
|
-
* FaspManager: transfer spec `authentication` no more needed for local
|
4463
|
+
* FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
4209
4464
|
* gem version requirements made more open
|
4210
4465
|
|
4211
4466
|
* 0.10.5
|
@@ -4355,10 +4610,10 @@ So, it evolved into `ascli`:
|
|
4355
4610
|
|
4356
4611
|
* 0.9.7
|
4357
4612
|
|
4358
|
-
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#
|
4613
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
|
4359
4614
|
* preview persistency goes to unique file by default
|
4360
4615
|
* catch mxf extension in preview as video
|
4361
|
-
* Faspex: possibility to download all
|
4616
|
+
* Faspex: possibility to download all packages by specifying id=ALL
|
4362
4617
|
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
4363
4618
|
|
4364
4619
|
* 0.9.6
|
@@ -4438,16 +4693,16 @@ For issues or feature requests use the Github repository and issues.
|
|
4438
4693
|
|
4439
4694
|
You can also contribute to this open source project.
|
4440
4695
|
|
4441
|
-
One can also create one's own
|
4696
|
+
One can also [create one's own plugin](#createownplugin).
|
4442
4697
|
|
4443
4698
|
## Only one value for any option
|
4444
4699
|
|
4445
4700
|
Some commands and sub commands may ask for the same option name.
|
4446
|
-
Currently, since option definition is position
|
4701
|
+
Currently, since option definition is position independent (last one wins), it is not possible
|
4447
4702
|
to give an option to a command and the same option with different value to a sub command.
|
4448
4703
|
|
4449
|
-
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
|
4450
|
-
As a
|
4704
|
+
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
|
4705
|
+
As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
|
4451
4706
|
|
4452
4707
|
This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
|
4453
4708
|
|
@@ -4479,7 +4734,7 @@ Workaround on server side: Either remove the fingerprint from `aspera.conf`, or
|
|
4479
4734
|
|
4480
4735
|
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
4481
4736
|
|
4482
|
-
##
|
4737
|
+
## Miscellaneous
|
4483
4738
|
|
4484
4739
|
* remove rest and oauth classes and use ruby standard gems:
|
4485
4740
|
|