aspera-cli 4.2.1 → 4.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +1580 -946
- data/bin/ascli +1 -1
- data/bin/asession +3 -5
- data/docs/Makefile +8 -11
- data/docs/README.erb.md +1521 -829
- data/docs/doc_tools.rb +58 -0
- data/docs/test_env.conf +3 -1
- data/examples/faspex4.rb +28 -19
- data/examples/transfer.rb +2 -2
- data/lib/aspera/aoc.rb +157 -134
- data/lib/aspera/cli/listener/progress_multi.rb +5 -5
- data/lib/aspera/cli/main.rb +106 -48
- data/lib/aspera/cli/manager.rb +19 -20
- data/lib/aspera/cli/plugin.rb +22 -7
- data/lib/aspera/cli/plugins/aoc.rb +260 -208
- data/lib/aspera/cli/plugins/ats.rb +11 -10
- data/lib/aspera/cli/plugins/bss.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +360 -189
- data/lib/aspera/cli/plugins/faspex.rb +119 -56
- data/lib/aspera/cli/plugins/faspex5.rb +32 -17
- data/lib/aspera/cli/plugins/node.rb +72 -31
- data/lib/aspera/cli/plugins/orchestrator.rb +5 -3
- data/lib/aspera/cli/plugins/preview.rb +94 -68
- data/lib/aspera/cli/plugins/server.rb +16 -5
- data/lib/aspera/cli/plugins/shares.rb +17 -0
- data/lib/aspera/cli/transfer_agent.rb +64 -82
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/command_line_builder.rb +48 -31
- data/lib/aspera/cos_node.rb +4 -3
- data/lib/aspera/environment.rb +4 -4
- data/lib/aspera/fasp/{manager.rb → agent_base.rb} +7 -6
- data/lib/aspera/fasp/{connect.rb → agent_connect.rb} +46 -39
- data/lib/aspera/fasp/{local.rb → agent_direct.rb} +42 -38
- data/lib/aspera/fasp/{http_gw.rb → agent_httpgw.rb} +50 -29
- data/lib/aspera/fasp/{node.rb → agent_node.rb} +43 -4
- data/lib/aspera/fasp/agent_trsdk.rb +106 -0
- data/lib/aspera/fasp/default.rb +17 -0
- data/lib/aspera/fasp/installation.rb +64 -48
- data/lib/aspera/fasp/parameters.rb +78 -91
- data/lib/aspera/fasp/parameters.yaml +531 -0
- data/lib/aspera/fasp/uri.rb +1 -1
- data/lib/aspera/faspex_gw.rb +12 -11
- data/lib/aspera/id_generator.rb +22 -0
- data/lib/aspera/keychain/encrypted_hash.rb +120 -0
- data/lib/aspera/keychain/macos_security.rb +94 -0
- data/lib/aspera/log.rb +45 -32
- data/lib/aspera/node.rb +9 -4
- data/lib/aspera/oauth.rb +116 -100
- data/lib/aspera/persistency_action_once.rb +11 -7
- data/lib/aspera/persistency_folder.rb +6 -26
- data/lib/aspera/rest.rb +66 -50
- data/lib/aspera/sync.rb +40 -35
- data/lib/aspera/timer_limiter.rb +22 -0
- metadata +86 -29
- data/docs/transfer_spec.html +0 -99
- data/lib/aspera/api_detector.rb +0 -60
- data/lib/aspera/fasp/aoc.rb +0 -24
- data/lib/aspera/secrets.rb +0 -20
data/README.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
|
2
|
-
|
|
2
|
+
<font size="+12"><center>`ascli` : Command Line Interface for IBM Aspera products</center></font>
|
|
3
3
|
|
|
4
|
-
Version : 4.
|
|
4
|
+
Version : 4.5.0
|
|
5
5
|
|
|
6
6
|
_Laurent/2016-2021_
|
|
7
7
|
|
|
8
|
-
This gem provides `ascli
|
|
8
|
+
This gem provides the `ascli` Command Line Interface to IBM Aspera software.
|
|
9
9
|
|
|
10
10
|
`ascli` is a also great tool to learn Aspera APIs.
|
|
11
11
|
|
|
@@ -13,9 +13,11 @@ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/asper
|
|
|
13
13
|
|
|
14
14
|
Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
|
|
15
15
|
|
|
16
|
-
Ruby version
|
|
16
|
+
Required Ruby version: > 2.4
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
[Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
|
|
19
|
+
|
|
20
|
+
# <a id="when_to_use"></a>When to use and when not to use
|
|
19
21
|
|
|
20
22
|
`ascli` is designed to be used as a command line tool to:
|
|
21
23
|
|
|
@@ -29,7 +31,8 @@ So it is designed for:
|
|
|
29
31
|
|
|
30
32
|
`ascli` can be seen as a command line tool integrating:
|
|
31
33
|
|
|
32
|
-
* a configuration file (config.yaml)
|
|
34
|
+
* a configuration file (config.yaml)
|
|
35
|
+
* advanced command line options
|
|
33
36
|
* cURL (for REST calls)
|
|
34
37
|
* Aspera transfer (ascp)
|
|
35
38
|
|
|
@@ -37,26 +40,23 @@ One might be tempted to use it as an integration element, e.g. by building a com
|
|
|
37
40
|
For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
|
|
38
41
|
|
|
39
42
|
* Product APIs (REST) : e.g. AoC, Faspex, node
|
|
40
|
-
* Transfer SDK : with gRPC interface and
|
|
43
|
+
* Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
|
|
41
44
|
|
|
42
45
|
Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
|
|
43
46
|
|
|
44
47
|
For scripting and ad'hoc command line operations, `ascli` is perfect.
|
|
45
48
|
|
|
46
|
-
# Notations
|
|
49
|
+
# <a id="parsing"></a>Notations, Shell and Command line parsing
|
|
47
50
|
|
|
48
|
-
In examples, command line operations
|
|
49
|
-
Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
|
|
51
|
+
In examples, command line operations are shown using a shell such: `bash` or `zsh`.
|
|
50
52
|
|
|
51
53
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
|
52
54
|
|
|
53
|
-
# <a name="parsing"></a>Shell and Command line parsing
|
|
54
|
-
|
|
55
55
|
`ascli` is typically executed in a shell, either interactively or in a script. `ascli` receives its arguments from this shell.
|
|
56
56
|
|
|
57
57
|
On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
|
|
58
58
|
|
|
59
|
-
On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
|
59
|
+
On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
|
|
60
60
|
|
|
61
61
|
* [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
|
|
62
62
|
* [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
|
|
@@ -64,12 +64,15 @@ On Windows, `cmd` is typically used. Windows process creation does not receive t
|
|
|
64
64
|
In case of doubt of argument values after parsing test like this:
|
|
65
65
|
|
|
66
66
|
```
|
|
67
|
-
|
|
67
|
+
ascli conf echo "Hello World" arg2 3
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
```
|
|
68
71
|
"Hello World"
|
|
69
72
|
ERROR: Argument: unprocessed values: ["arg2", "3"]
|
|
70
73
|
```
|
|
71
74
|
|
|
72
|
-
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
|
|
75
|
+
`echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (`ascli`), next command line arguments are shown in the error message.
|
|
73
76
|
|
|
74
77
|
# Quick Start
|
|
75
78
|
|
|
@@ -80,8 +83,11 @@ First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to st
|
|
|
80
83
|
Once the gem is installed, `ascli` shall be accessible:
|
|
81
84
|
|
|
82
85
|
```
|
|
83
|
-
|
|
84
|
-
|
|
86
|
+
ascli --version
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
4.5.0
|
|
85
91
|
```
|
|
86
92
|
|
|
87
93
|
## First use
|
|
@@ -93,8 +99,14 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
|
93
99
|
To test with Aspera demo transfer server, setup the environment and then test:
|
|
94
100
|
|
|
95
101
|
```
|
|
96
|
-
|
|
97
|
-
|
|
102
|
+
ascli config initdemo
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
ascli server browse /
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
```
|
|
98
110
|
:............:...........:......:........:...........................:.......................:
|
|
99
111
|
: zmode : zuid : zgid : size : mtime : name :
|
|
100
112
|
:............:...........:......:........:...........................:.......................:
|
|
@@ -108,16 +120,31 @@ $ ascli server browse /
|
|
|
108
120
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
|
109
121
|
|
|
110
122
|
* create a [option preset](#lprt)
|
|
111
|
-
* define it as default for
|
|
123
|
+
* define it as default for `server` plugin
|
|
112
124
|
* list files in a folder
|
|
113
125
|
* download a file
|
|
114
126
|
|
|
115
127
|
```
|
|
116
|
-
|
|
128
|
+
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
```
|
|
117
132
|
updated: myserver
|
|
118
|
-
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
ascli config preset set default server myserver
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
```
|
|
119
140
|
updated: default→server to myserver
|
|
120
|
-
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
```
|
|
144
|
+
ascli server browse /aspera-test-dir-large
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
```
|
|
121
148
|
:............:...........:......:..............:...........................:............................:
|
|
122
149
|
: zmode : zuid : zgid : size : mtime : name :
|
|
123
150
|
:............:...........:......:..............:...........................:............................:
|
|
@@ -134,7 +161,13 @@ $ ascli server browse /aspera-test-dir-large
|
|
|
134
161
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
|
135
162
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
|
136
163
|
:............:...........:......:..............:...........................:............................:
|
|
137
|
-
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
```
|
|
167
|
+
ascli server download /aspera-test-dir-large/200MB
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
```
|
|
138
171
|
Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
|
|
139
172
|
complete
|
|
140
173
|
```
|
|
@@ -145,43 +178,48 @@ Get familiar with configuration, options, commands : [Command Line Interface](#c
|
|
|
145
178
|
|
|
146
179
|
Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
|
|
147
180
|
|
|
148
|
-
# <a
|
|
181
|
+
# <a id="installation"></a>Installation
|
|
149
182
|
|
|
150
|
-
It is possible to install *either* directly on the host operating system (Linux, Windows,
|
|
183
|
+
It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
|
|
151
184
|
|
|
152
185
|
The direct installation is recommended and consists in installing:
|
|
153
186
|
|
|
154
|
-
* [Ruby](#ruby) version
|
|
187
|
+
* [Ruby](#ruby) version > 2.4
|
|
155
188
|
* [aspera-cli](#the_gem)
|
|
156
189
|
* [Aspera SDK (ascp)](#fasp_prot)
|
|
157
190
|
|
|
158
191
|
The following sections provide information on the various installation methods.
|
|
159
192
|
|
|
160
|
-
An internet connection is required for the installation. If you
|
|
193
|
+
An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
|
|
161
194
|
|
|
162
195
|
## Docker container
|
|
163
196
|
|
|
164
197
|
Use this method only if you know what you do, else use the standard recommended method as described here above.
|
|
165
198
|
|
|
166
|
-
This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
|
|
199
|
+
This method installs a docker image that contains: Ruby, `ascli` and the FASP sdk.
|
|
200
|
+
|
|
201
|
+
The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
|
|
167
202
|
|
|
168
203
|
Ensure that you have Docker installed.
|
|
169
204
|
|
|
170
205
|
```
|
|
171
|
-
|
|
206
|
+
docker --version
|
|
172
207
|
```
|
|
173
208
|
|
|
174
209
|
Download the wrapping script:
|
|
175
210
|
|
|
176
211
|
```
|
|
177
|
-
|
|
178
|
-
|
|
212
|
+
curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
```
|
|
216
|
+
chmod a+x ascli
|
|
179
217
|
```
|
|
180
218
|
|
|
181
219
|
Install the container image:
|
|
182
220
|
|
|
183
221
|
```
|
|
184
|
-
|
|
222
|
+
./ascli install
|
|
185
223
|
```
|
|
186
224
|
|
|
187
225
|
Start using it !
|
|
@@ -191,23 +229,23 @@ Note that the tool is run in the container, so transfers are also executed in th
|
|
|
191
229
|
The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/ascli` on host.
|
|
192
230
|
|
|
193
231
|
To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
|
|
194
|
-
To add local storage as a volume edit the script: ascli and add a `--volume` stanza.
|
|
232
|
+
To add local storage as a volume edit the script: `ascli` and add a `--volume` stanza.
|
|
195
233
|
|
|
196
|
-
## <a
|
|
234
|
+
## <a id="ruby"></a>Ruby
|
|
197
235
|
|
|
198
236
|
Use this method to install on the native host.
|
|
199
237
|
|
|
200
238
|
A ruby interpreter is required to run the tool or to use the gem and tool.
|
|
201
239
|
|
|
202
|
-
Ruby
|
|
240
|
+
Required Ruby version: > 2.4. Ruby version 3 is also supported.
|
|
203
241
|
|
|
204
242
|
*Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
|
|
205
243
|
|
|
206
244
|
Refer to the following sections for a proposed method for specific operating systems.
|
|
207
245
|
|
|
208
|
-
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux,
|
|
246
|
+
The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
|
|
209
247
|
If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
|
|
210
|
-
If you have a simpler better way to install Ruby version
|
|
248
|
+
If you have a simpler better way to install Ruby version > 2.4 : use it !
|
|
211
249
|
|
|
212
250
|
### Generic: RVM: single user installation (not root)
|
|
213
251
|
|
|
@@ -218,31 +256,31 @@ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
|
|
|
218
256
|
Install the 2 keys
|
|
219
257
|
|
|
220
258
|
```
|
|
221
|
-
|
|
259
|
+
gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
|
222
260
|
```
|
|
223
261
|
|
|
224
262
|
Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
|
|
225
263
|
|
|
226
264
|
```
|
|
227
|
-
|
|
265
|
+
\curl -sSL https://get.rvm.io | bash -s stable
|
|
228
266
|
```
|
|
229
267
|
|
|
230
|
-
If you keep the same terminal (
|
|
268
|
+
If you keep the same terminal (not needed if re-login):
|
|
231
269
|
|
|
232
270
|
```
|
|
233
|
-
|
|
271
|
+
source ~/.rvm/scripts/rvm
|
|
234
272
|
```
|
|
235
273
|
|
|
236
|
-
It is advised to get one of the pre-compiled ruby version, you can list with:
|
|
274
|
+
It is advised to get one of the pre-compiled ruby version, you can list with:
|
|
237
275
|
|
|
238
276
|
```
|
|
239
|
-
|
|
277
|
+
rvm list --remote
|
|
240
278
|
```
|
|
241
279
|
|
|
242
280
|
Install the chosen pre-compiled Ruby version:
|
|
243
281
|
|
|
244
282
|
```
|
|
245
|
-
|
|
283
|
+
rvm install 2.7.2 --binary
|
|
246
284
|
```
|
|
247
285
|
|
|
248
286
|
Ruby is now installed for the user, go on to Gem installation.
|
|
@@ -255,7 +293,7 @@ As root, it installs by default in /usr/local/rvm for all users and creates `/et
|
|
|
255
293
|
One can install in another location with :
|
|
256
294
|
|
|
257
295
|
```
|
|
258
|
-
|
|
296
|
+
curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
|
|
259
297
|
```
|
|
260
298
|
|
|
261
299
|
As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
|
|
@@ -263,8 +301,11 @@ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d
|
|
|
263
301
|
To activate ruby (and ascli) later, source it:
|
|
264
302
|
|
|
265
303
|
```
|
|
266
|
-
|
|
267
|
-
|
|
304
|
+
source /etc/profile.d/rvm.sh.ok
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
```
|
|
308
|
+
rvm version
|
|
268
309
|
```
|
|
269
310
|
|
|
270
311
|
### Windows: Installer
|
|
@@ -277,16 +318,16 @@ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstal
|
|
|
277
318
|
|
|
278
319
|
### macOS: pre-installed or `brew`
|
|
279
320
|
|
|
280
|
-
|
|
321
|
+
macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
|
|
281
322
|
|
|
282
323
|
```
|
|
283
|
-
|
|
324
|
+
sudo gem install aspera-cli
|
|
284
325
|
```
|
|
285
326
|
|
|
286
327
|
Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
|
|
287
328
|
|
|
288
329
|
```
|
|
289
|
-
|
|
330
|
+
brew install ruby
|
|
290
331
|
```
|
|
291
332
|
|
|
292
333
|
### Linux: package
|
|
@@ -296,90 +337,56 @@ If your Linux distribution provides a standard ruby package, you can use it prov
|
|
|
296
337
|
Example:
|
|
297
338
|
|
|
298
339
|
```
|
|
299
|
-
|
|
340
|
+
yum install -y ruby rubygems ruby-json
|
|
300
341
|
```
|
|
301
342
|
|
|
302
343
|
One can cleanup the whole yum-installed ruby environment like this to uninstall:
|
|
303
344
|
|
|
304
345
|
```
|
|
305
346
|
gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
```
|
|
306
350
|
yum remove -y ruby ruby-libs
|
|
307
351
|
```
|
|
308
352
|
|
|
309
353
|
### Other Unixes: Aix, etc...
|
|
310
354
|
|
|
311
|
-
If your
|
|
355
|
+
If your Unix does not provide a pre-built ruby, you can get it using one of those
|
|
312
356
|
[methods](https://www.ruby-lang.org/en/documentation/installation/)
|
|
313
357
|
|
|
314
358
|
For instance to build from source, and install in `/opt/ruby` :
|
|
315
359
|
|
|
316
360
|
```
|
|
317
|
-
|
|
318
|
-
# gzip -d ruby-2.7.2.tar.gz
|
|
319
|
-
# tar xvf ruby-2.7.2.tar
|
|
320
|
-
# cd ruby-2.7.2
|
|
321
|
-
# ./configure --prefix=/opt/ruby
|
|
322
|
-
# make ruby.imp
|
|
323
|
-
# make
|
|
324
|
-
# make install
|
|
325
|
-
```
|
|
326
|
-
|
|
327
|
-
### <a name="offline_install"></a>Installation without internet access
|
|
328
|
-
|
|
329
|
-
Note that currently no pre-packaged version exist yet.
|
|
330
|
-
A method to build one provided here:
|
|
331
|
-
|
|
332
|
-
On a server with the same OS version and with internet access follow the "Generic single user installation" method.
|
|
333
|
-
|
|
334
|
-
Then create an archive:
|
|
335
|
-
|
|
336
|
-
```
|
|
337
|
-
$ cd
|
|
338
|
-
$ tar zcvf rvm-ascli.tgz .rvm
|
|
339
|
-
```
|
|
361
|
+
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
|
|
340
362
|
|
|
341
|
-
|
|
363
|
+
gzip -d ruby-2.7.2.tar.gz
|
|
342
364
|
|
|
343
|
-
|
|
344
|
-
$ ascli conf --show-config|grep sdk_url
|
|
345
|
-
```
|
|
346
|
-
|
|
347
|
-
Then download the SDK archive from that URL.
|
|
365
|
+
tar xvf ruby-2.7.2.tar
|
|
348
366
|
|
|
349
|
-
|
|
367
|
+
cd ruby-2.7.2
|
|
350
368
|
|
|
351
|
-
|
|
369
|
+
./configure --prefix=/opt/ruby
|
|
352
370
|
|
|
353
|
-
|
|
371
|
+
make ruby.imp
|
|
354
372
|
|
|
355
|
-
|
|
356
|
-
* in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
|
|
373
|
+
make
|
|
357
374
|
|
|
358
|
-
|
|
359
|
-
source path_to_rvm_root/scripts/rvm
|
|
360
|
-
rvm use 2.7.2
|
|
375
|
+
make install
|
|
361
376
|
```
|
|
362
377
|
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
```
|
|
366
|
-
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
|
367
|
-
```
|
|
368
|
-
|
|
369
|
-
or restore the `$HOME/.aspera` folder for the user.
|
|
370
|
-
|
|
371
|
-
## <a name="the_gem"></a>`aspera-cli` gem
|
|
378
|
+
## <a id="the_gem"></a>`aspera-cli` gem
|
|
372
379
|
|
|
373
380
|
Once you have Ruby and rights to install gems: Install the gem and its dependencies:
|
|
374
381
|
|
|
375
382
|
```
|
|
376
|
-
|
|
383
|
+
gem install aspera-cli
|
|
377
384
|
```
|
|
378
385
|
|
|
379
386
|
To upgrade to the latest version:
|
|
380
387
|
|
|
381
388
|
```
|
|
382
|
-
|
|
389
|
+
gem update aspera-cli
|
|
383
390
|
```
|
|
384
391
|
|
|
385
392
|
`ascli` checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
|
|
@@ -387,12 +394,10 @@ To upgrade to the latest version:
|
|
|
387
394
|
To check manually:
|
|
388
395
|
|
|
389
396
|
```
|
|
390
|
-
|
|
397
|
+
ascli conf check_update
|
|
391
398
|
```
|
|
392
399
|
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
## <a name="fasp_prot"></a>FASP Protocol
|
|
400
|
+
## <a id="fasp_prot"></a>FASP Protocol
|
|
396
401
|
|
|
397
402
|
Most file transfers will be done using the FASP protocol, using `ascp`.
|
|
398
403
|
Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
|
|
@@ -400,17 +405,20 @@ Only two additional files are required to perform an Aspera Transfer, which are
|
|
|
400
405
|
* ascp
|
|
401
406
|
* aspera-license (in same folder, or ../etc)
|
|
402
407
|
|
|
403
|
-
This can be installed either be installing an Aspera transfer
|
|
408
|
+
This can be installed either be installing an Aspera transfer software, or using an embedded command:
|
|
404
409
|
|
|
405
410
|
```
|
|
406
|
-
|
|
411
|
+
ascli conf ascp install
|
|
407
412
|
```
|
|
408
413
|
|
|
409
|
-
If a local SDK installation is
|
|
414
|
+
If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
|
|
410
415
|
|
|
411
416
|
```
|
|
412
|
-
|
|
413
|
-
|
|
417
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
```
|
|
421
|
+
ascli conf ascp install --sdk-url=file:///SDK.zip
|
|
414
422
|
```
|
|
415
423
|
|
|
416
424
|
The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
|
|
@@ -429,37 +437,56 @@ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/
|
|
|
429
437
|
`ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
|
|
430
438
|
Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
|
|
431
439
|
|
|
432
|
-
Several methods are provided
|
|
433
|
-
other methods are available. Refer to section: [Transfer Agents](#agents)
|
|
440
|
+
Several methods are provided to start a transfer.
|
|
441
|
+
Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
|
|
434
442
|
|
|
435
|
-
## <a
|
|
443
|
+
## <a id="offline_install"></a>Installation in air gapped environment
|
|
436
444
|
|
|
437
|
-
|
|
445
|
+
Note that currently no pre-packaged version exist yet.
|
|
446
|
+
A method to build one is provided here:
|
|
447
|
+
|
|
448
|
+
The procedure:
|
|
438
449
|
|
|
439
450
|
* Follow the non-root installation procedure with RVM, including gem
|
|
440
|
-
*
|
|
451
|
+
* Archive (zip, tar) the main RVM folder (includes ascli):
|
|
452
|
+
|
|
453
|
+
```
|
|
454
|
+
cd $HOME && tar zcvf rvm-ascli.tgz .rvm
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
* Get the Aspera SDK.
|
|
458
|
+
|
|
459
|
+
```
|
|
460
|
+
ascli conf --show-config --fields=sdk_url
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
* Download the SDK archive from that URL.
|
|
441
464
|
|
|
442
465
|
```
|
|
443
|
-
|
|
444
|
-
$ tar zcvf rvm_ascli.tgz .rvm
|
|
466
|
+
curl -Lso SDK.zip https://ibm.biz/aspera_sdk
|
|
445
467
|
```
|
|
446
468
|
|
|
447
|
-
*
|
|
469
|
+
* Transfer those 2 files to the target system
|
|
470
|
+
|
|
471
|
+
* On target system
|
|
448
472
|
|
|
449
473
|
```
|
|
450
|
-
$
|
|
474
|
+
cd $HOME
|
|
475
|
+
|
|
476
|
+
tar zxvf rvm-ascli.tgz
|
|
477
|
+
|
|
478
|
+
source ~/.rvm/scripts/rvm
|
|
479
|
+
|
|
480
|
+
ascli conf ascp install --sdk-url=file:///SDK.zip
|
|
451
481
|
```
|
|
452
482
|
|
|
453
|
-
*
|
|
483
|
+
* Add those lines to shell init (`.profile`)
|
|
454
484
|
|
|
455
485
|
```
|
|
456
|
-
|
|
457
|
-
$ tar zxvf rvm_ascli.tgz
|
|
458
|
-
$ source ~/.rvm/scripts/rvm
|
|
459
|
-
$ ascli conf ascp install --sdk-url=file:///SDK.zip
|
|
486
|
+
source ~/.rvm/scripts/rvm
|
|
460
487
|
```
|
|
461
488
|
|
|
462
|
-
# <a
|
|
489
|
+
# <a id="cli"></a>Command Line Interface: `ascli`
|
|
463
490
|
|
|
464
491
|
The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
|
|
465
492
|
|
|
@@ -476,7 +503,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
|
476
503
|
* Supports most Aspera server products (on-premise and SaaS)
|
|
477
504
|
* Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
|
|
478
505
|
* Supports Commands, Option values and Parameters shortcuts
|
|
479
|
-
* FASP [Transfer Agents](#agents) can be:
|
|
506
|
+
* FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
|
|
480
507
|
* Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
|
|
481
508
|
* Allows transfers from products to products, essentially at node level (using the node transfer agent)
|
|
482
509
|
* Supports FaspStream creation (using Node API)
|
|
@@ -488,7 +515,7 @@ The `aspera-cli` Gem provides a command line interface (CLI) which interacts wit
|
|
|
488
515
|
Basic usage is displayed by executing:
|
|
489
516
|
|
|
490
517
|
```
|
|
491
|
-
|
|
518
|
+
ascli -h
|
|
492
519
|
```
|
|
493
520
|
|
|
494
521
|
Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
|
|
@@ -502,7 +529,7 @@ Arguments are the units of command line, as parsed by the shell, typically separ
|
|
|
502
529
|
There are two types of arguments: Commands and Options. Example :
|
|
503
530
|
|
|
504
531
|
```
|
|
505
|
-
|
|
532
|
+
ascli command --option-name=VAL1 VAL2
|
|
506
533
|
```
|
|
507
534
|
|
|
508
535
|
* executes _command_: `command`
|
|
@@ -530,13 +557,16 @@ Exceptions:
|
|
|
530
557
|
* the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
|
|
531
558
|
|
|
532
559
|
```
|
|
533
|
-
|
|
560
|
+
ascli config echo -- --sample
|
|
561
|
+
```
|
|
562
|
+
|
|
563
|
+
```
|
|
534
564
|
"--sample"
|
|
535
565
|
```
|
|
536
566
|
|
|
537
567
|
Note that `--sample` is taken as an argument, and not option.
|
|
538
568
|
|
|
539
|
-
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on
|
|
569
|
+
Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
|
|
540
570
|
|
|
541
571
|
The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
|
|
542
572
|
|
|
@@ -557,7 +587,7 @@ Command line arguments that are not options are either commands or arguments. If
|
|
|
557
587
|
|
|
558
588
|
Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
|
|
559
589
|
|
|
560
|
-
The
|
|
590
|
+
The behavior can be controlled with:
|
|
561
591
|
|
|
562
592
|
* --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
|
|
563
593
|
* yes : missing mandatory parameters/options are asked to the user
|
|
@@ -574,8 +604,8 @@ The information displayed depends on the action.
|
|
|
574
604
|
|
|
575
605
|
Depending on action, the output will contain:
|
|
576
606
|
|
|
577
|
-
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is
|
|
578
|
-
* `object_list` : displayed as a 2 dimensional table: one line per item, one
|
|
607
|
+
* `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
|
|
608
|
+
* `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
|
|
579
609
|
* `value_list` : a table with one column.
|
|
580
610
|
* `empty` : nothing
|
|
581
611
|
* `status` : a message
|
|
@@ -600,12 +630,15 @@ The style of output can be set using the `format` parameter, supporting:
|
|
|
600
630
|
* `yaml` : YAML
|
|
601
631
|
* `csv` : Comma Separated Values
|
|
602
632
|
|
|
603
|
-
###
|
|
633
|
+
### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
|
|
604
634
|
|
|
605
635
|
Table output can be filtered using the `select` parameter. Example:
|
|
606
636
|
|
|
607
637
|
```
|
|
608
|
-
|
|
638
|
+
ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
```
|
|
609
642
|
:...............................:..................................:...........:
|
|
610
643
|
: name : email : ats_admin :
|
|
611
644
|
:...............................:..................................:...........:
|
|
@@ -618,7 +651,7 @@ Note that `select` filters selected elements from the result of API calls, while
|
|
|
618
651
|
|
|
619
652
|
### Verbosity of output
|
|
620
653
|
|
|
621
|
-
|
|
654
|
+
Output messages are categorized in 3 types:
|
|
622
655
|
|
|
623
656
|
* `info` output contain additional information, such as number of elements in a table
|
|
624
657
|
* `data` output contain the actual output of the command (object, or list of objects)
|
|
@@ -632,7 +665,7 @@ The option `display` controls the level of output:
|
|
|
632
665
|
|
|
633
666
|
### Selection of output object properties
|
|
634
667
|
|
|
635
|
-
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific
|
|
668
|
+
By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
|
|
636
669
|
|
|
637
670
|
* DEF : default display of columns (that's the default, when not set)
|
|
638
671
|
* ALL : all columns available
|
|
@@ -641,7 +674,7 @@ By default, a table output will display one line per entry, and columns for each
|
|
|
641
674
|
* +a,b,c : add selected properties to the default selection.
|
|
642
675
|
* -a,b,c : remove selected properties from the default selection.
|
|
643
676
|
|
|
644
|
-
## <a
|
|
677
|
+
## <a id="extended"></a>Extended Value Syntax
|
|
645
678
|
|
|
646
679
|
Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
|
|
647
680
|
|
|
@@ -678,23 +711,32 @@ To display the result of an extended value, use the `config echo` command.
|
|
|
678
711
|
Example: read the content of the specified file, then, base64 decode, then unzip:
|
|
679
712
|
|
|
680
713
|
```
|
|
681
|
-
|
|
714
|
+
ascli config echo @zlib:@base64:@file:myfile.dat
|
|
682
715
|
```
|
|
683
716
|
|
|
684
717
|
Example: create a value as a hash, with one key and the value is read from a file:
|
|
685
718
|
|
|
686
719
|
```
|
|
687
|
-
|
|
720
|
+
ascli config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
|
|
688
721
|
```
|
|
689
722
|
|
|
690
723
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
|
691
724
|
|
|
692
725
|
```
|
|
693
|
-
|
|
726
|
+
cat test.csv
|
|
727
|
+
```
|
|
728
|
+
|
|
729
|
+
```
|
|
694
730
|
name,email
|
|
695
731
|
lolo,laurent@example.com
|
|
696
732
|
toto,titi@tutu.tata
|
|
697
|
-
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
```
|
|
736
|
+
ascli config echo @csvt:@file:test.csv
|
|
737
|
+
```
|
|
738
|
+
|
|
739
|
+
```
|
|
698
740
|
:......:.....................:
|
|
699
741
|
: name : email :
|
|
700
742
|
:......:.....................:
|
|
@@ -706,13 +748,16 @@ $ ascli config echo @csvt:@file:test.csv
|
|
|
706
748
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
|
707
749
|
|
|
708
750
|
```
|
|
709
|
-
|
|
751
|
+
ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
|
752
|
+
```
|
|
753
|
+
|
|
754
|
+
```
|
|
710
755
|
{"version"=>"0.9", "hello"=>true}
|
|
711
756
|
```
|
|
712
757
|
|
|
713
758
|
Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
|
|
714
759
|
|
|
715
|
-
## <a
|
|
760
|
+
## <a id="native"></a>Structured Value
|
|
716
761
|
|
|
717
762
|
Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
|
|
718
763
|
|
|
@@ -723,7 +768,7 @@ A convenient way to specify a _Structured Value_ is to use the `@json:` decoder,
|
|
|
723
768
|
|
|
724
769
|
It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
|
|
725
770
|
|
|
726
|
-
## <a
|
|
771
|
+
## <a id="conffolder"></a>Configuration and Persistency Folder
|
|
727
772
|
|
|
728
773
|
`ascli` configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/ascli`.
|
|
729
774
|
|
|
@@ -733,53 +778,52 @@ It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMED
|
|
|
733
778
|
The main folder can be displayed using :
|
|
734
779
|
|
|
735
780
|
```
|
|
736
|
-
|
|
781
|
+
ascli config folder
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
```
|
|
737
785
|
/Users/kenji/.aspera/ascli
|
|
738
786
|
```
|
|
739
787
|
|
|
740
|
-
It can be
|
|
788
|
+
It can be overridden using the environment variable `ASCLI_HOME`.
|
|
741
789
|
|
|
742
790
|
Example (Windows):
|
|
743
791
|
|
|
744
792
|
```
|
|
745
|
-
|
|
746
|
-
|
|
793
|
+
set ASCLI_HOME=C:\Users\Kenji\.aspera\ascli
|
|
794
|
+
ascli config folder
|
|
747
795
|
C:\Users\Kenji\.aspera\ascli
|
|
748
796
|
```
|
|
749
797
|
|
|
750
|
-
## <a
|
|
798
|
+
## <a id="configfile"></a>Configuration file
|
|
751
799
|
|
|
752
800
|
On the first execution of `ascli`, an empty configuration file is created in the configuration folder.
|
|
753
801
|
Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
|
|
754
802
|
|
|
755
|
-
Although the file is a standard YAML file, `ascli` provides commands to read and modify it
|
|
756
|
-
using the `config` command.
|
|
803
|
+
Although the file is a standard YAML file, `ascli` provides commands to read and modify it using the `config` command.
|
|
757
804
|
|
|
758
|
-
All options for `ascli`
|
|
805
|
+
All options for `ascli` can be set on command line, or by env vars, or using [option presets](#lprt) in the configuration file.
|
|
759
806
|
|
|
760
|
-
A configuration file provides a way to define default values, especially
|
|
761
|
-
for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
|
807
|
+
A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
|
|
762
808
|
|
|
763
|
-
The default configuration file is: `$HOME/.aspera/ascli/config.yaml`
|
|
764
|
-
(this can be overriden with option `--config-file=path` or equivalent env var).
|
|
809
|
+
The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
|
|
765
810
|
|
|
766
|
-
|
|
767
|
-
called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
|
811
|
+
The configuration file is simply a catalog of pre-defined lists of options, called: [option presets](#lprt). Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a [option preset](#lprt) (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
|
|
768
812
|
|
|
769
|
-
### <a
|
|
813
|
+
### <a id="lprt"></a>Option preset
|
|
770
814
|
|
|
771
815
|
A [option preset](#lprt) is simply a collection of parameters and their associated values in a named section in the configuration file.
|
|
772
816
|
|
|
773
817
|
A named [option preset](#lprt) can be modified directly using `ascli`, which will update the configuration file :
|
|
774
818
|
|
|
775
819
|
```
|
|
776
|
-
|
|
820
|
+
ascli config preset set|delete|show|initialize|update <option preset>
|
|
777
821
|
```
|
|
778
822
|
|
|
779
823
|
The command `update` allows the easy creation of [option preset](#lprt) by simply providing the options in their command line format, e.g. :
|
|
780
824
|
|
|
781
825
|
```
|
|
782
|
-
|
|
826
|
+
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
|
|
783
827
|
```
|
|
784
828
|
|
|
785
829
|
* This creates a [option preset](#lprt) `demo_server` with all provided options.
|
|
@@ -787,39 +831,48 @@ $ ascli config id demo_server update --url=ssh://demo.asperasoft.com:33001 --use
|
|
|
787
831
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
|
788
832
|
|
|
789
833
|
```
|
|
790
|
-
|
|
834
|
+
ascli config preset set demo_server password _demo_pass_
|
|
791
835
|
```
|
|
792
836
|
|
|
793
837
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
|
|
794
838
|
|
|
795
839
|
```
|
|
796
|
-
|
|
840
|
+
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
|
|
797
841
|
```
|
|
798
842
|
|
|
799
|
-
A
|
|
800
|
-
If necessary, the configuration file can be edited (or simply consulted) with:
|
|
843
|
+
A full terminal based overview of the configuration can be displayed using:
|
|
801
844
|
|
|
802
845
|
```
|
|
803
|
-
|
|
846
|
+
ascli config preset over
|
|
804
847
|
```
|
|
805
848
|
|
|
806
|
-
A
|
|
849
|
+
A list of [option preset](#lprt) can be displayed using:
|
|
807
850
|
|
|
808
851
|
```
|
|
809
|
-
|
|
852
|
+
ascli config preset list
|
|
810
853
|
```
|
|
811
854
|
|
|
812
|
-
A
|
|
855
|
+
A good practice is to not manually edit the configuration file and use modification commands instead.
|
|
856
|
+
If necessary, the configuration file can opened in a text editor with:
|
|
857
|
+
|
|
858
|
+
```
|
|
859
|
+
ascli config open
|
|
860
|
+
```
|
|
861
|
+
|
|
862
|
+
Older format for commands are still supported:
|
|
813
863
|
|
|
814
864
|
```
|
|
815
|
-
|
|
865
|
+
ascli config id <name> set|delete|show|initialize|update
|
|
866
|
+
ascli config over
|
|
867
|
+
ascli config list
|
|
816
868
|
```
|
|
817
869
|
|
|
818
|
-
|
|
870
|
+
|
|
871
|
+
### <a id="lprtconf"></a>Special Option preset: config
|
|
819
872
|
|
|
820
873
|
This preset name is reserved and contains a single key: `version`. This is the version of `ascli` which created the file.
|
|
821
874
|
|
|
822
|
-
### <a
|
|
875
|
+
### <a id="lprtdef"></a>Special Option preset: default
|
|
823
876
|
|
|
824
877
|
This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
|
|
825
878
|
|
|
@@ -830,16 +883,16 @@ Note that special plugin name: `config` can be associated with a preset that is
|
|
|
830
883
|
Operations on this preset are done using regular `config` operations:
|
|
831
884
|
|
|
832
885
|
```
|
|
833
|
-
|
|
834
|
-
|
|
886
|
+
ascli config preset set default _plugin_name_ _default_preset_for_plugin_
|
|
887
|
+
ascli config preset get default _plugin_name_
|
|
835
888
|
"_default_preset_for_plugin_"
|
|
836
889
|
```
|
|
837
890
|
|
|
838
|
-
### <a
|
|
891
|
+
### <a id="lplugconf"></a>Special Plugin: config
|
|
839
892
|
|
|
840
893
|
Plugin `config` (not to be confused with Option preset config) is used to configure `ascli` but it also contains global options.
|
|
841
894
|
|
|
842
|
-
When `ascli` starts, it
|
|
895
|
+
When `ascli` starts, it looks for the `default` Option preset and if there is a value for `config`, if so, it loads the option values for any plugin used.
|
|
843
896
|
|
|
844
897
|
If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
|
|
845
898
|
|
|
@@ -864,7 +917,7 @@ demo_server:
|
|
|
864
917
|
We can see here:
|
|
865
918
|
|
|
866
919
|
* The configuration was created with CLI version 0.3.7
|
|
867
|
-
* the default [option preset](#lprt) to load for plugin
|
|
920
|
+
* the default [option preset](#lprt) to load for `server` plugin is : `demo_server`
|
|
868
921
|
* the [option preset](#lprt) `demo_server` defines some parameters: the URL and credentials
|
|
869
922
|
* the default [option preset](#lprt) to load in any case is : `cli_default`
|
|
870
923
|
|
|
@@ -878,10 +931,10 @@ The user may create as many [option presets](#lprt) as needed. For instance, a p
|
|
|
878
931
|
|
|
879
932
|
Values in the configuration also follow the [Extended Value Syntax](#extended).
|
|
880
933
|
|
|
881
|
-
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config
|
|
934
|
+
Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
|
|
882
935
|
|
|
883
936
|
```
|
|
884
|
-
|
|
937
|
+
ascli config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/ascli/aocapikey"
|
|
885
938
|
```
|
|
886
939
|
|
|
887
940
|
This creates the [option preset](#lprt):
|
|
@@ -895,36 +948,51 @@ my_aoc_org:
|
|
|
895
948
|
|
|
896
949
|
So, the key file will be read only at execution time, but not be embedded in the configuration file.
|
|
897
950
|
|
|
951
|
+
### Options evaluation order
|
|
952
|
+
|
|
953
|
+
Some options are global, some options are available only for some plugins. (the plugin is the first level command).
|
|
954
|
+
|
|
898
955
|
Options are loaded using this algorithm:
|
|
899
956
|
|
|
900
|
-
*
|
|
901
|
-
|
|
902
|
-
|
|
903
|
-
*
|
|
904
|
-
*
|
|
957
|
+
* If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
|
|
958
|
+
* else it looks for the name of the plugin as key in section `default`, the value is the name of the default [option preset](#lprt) for it, and loads it.
|
|
959
|
+
* If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the [option preset](#lprt) specified from the configuration file, or of the value is a Hash, it uses it as options values.
|
|
960
|
+
* Environment variables are evaluated
|
|
961
|
+
* Command line options are evaluated
|
|
905
962
|
|
|
906
963
|
Parameters are evaluated in the order of command line.
|
|
907
964
|
|
|
908
|
-
To avoid loading the default [option preset](#lprt) for a plugin,
|
|
965
|
+
To avoid loading the default [option preset](#lprt) for a plugin, use: `-N`
|
|
909
966
|
|
|
910
967
|
On command line, words in parameter names are separated by a dash, in configuration file, separator
|
|
911
968
|
is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
|
|
912
969
|
|
|
913
|
-
|
|
914
|
-
convert olver versions, remove the leading ":" in front of keys.
|
|
970
|
+
The main plugin name is `config`, so it is possible to define a default [option preset](#lprt) for the main plugin with:
|
|
915
971
|
|
|
916
|
-
|
|
917
|
-
|
|
972
|
+
```
|
|
973
|
+
ascli config preset set cli_default interactive no
|
|
974
|
+
```
|
|
918
975
|
|
|
919
976
|
```
|
|
920
|
-
|
|
921
|
-
$ ascli config id default set config cli_default
|
|
977
|
+
ascli config preset set default config cli_default
|
|
922
978
|
```
|
|
923
979
|
|
|
924
980
|
A [option preset](#lprt) value can be removed with `unset`:
|
|
925
981
|
|
|
926
982
|
```
|
|
927
|
-
|
|
983
|
+
ascli config preset unset cli_default interactive
|
|
984
|
+
```
|
|
985
|
+
|
|
986
|
+
Example: Define options using command line:
|
|
987
|
+
|
|
988
|
+
```
|
|
989
|
+
ascli -N --url=x --password=y --username=y node --show-config
|
|
990
|
+
```
|
|
991
|
+
|
|
992
|
+
Example: Define options using a hash:
|
|
993
|
+
|
|
994
|
+
```
|
|
995
|
+
ascli -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
|
|
928
996
|
```
|
|
929
997
|
|
|
930
998
|
### Examples
|
|
@@ -934,44 +1002,120 @@ only username/password and url are required (either on command line, or from con
|
|
|
934
1002
|
Those can usually be provided on the command line:
|
|
935
1003
|
|
|
936
1004
|
```
|
|
937
|
-
|
|
1005
|
+
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
|
|
938
1006
|
```
|
|
939
1007
|
|
|
940
1008
|
This can also be provisioned in a config file:
|
|
941
1009
|
|
|
1010
|
+
* Build [option preset](#lprt)
|
|
1011
|
+
|
|
942
1012
|
```
|
|
943
|
-
|
|
944
|
-
|
|
945
|
-
|
|
946
|
-
4$ ascli config id default set shares shares06
|
|
947
|
-
5$ ascli config overview
|
|
948
|
-
6$ ascli shares repo browse /
|
|
1013
|
+
ascli config preset set shares06 url https://10.25.0.6
|
|
1014
|
+
ascli config preset set shares06 username john
|
|
1015
|
+
ascli config preset set shares06 password 4sp3ra
|
|
949
1016
|
```
|
|
950
1017
|
|
|
951
|
-
The three first commands build a [option preset](#lprt).
|
|
952
1018
|
Note that this can also be done with one single command:
|
|
953
1019
|
|
|
954
1020
|
```
|
|
955
|
-
|
|
1021
|
+
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
|
|
1022
|
+
```
|
|
1023
|
+
|
|
1024
|
+
or
|
|
1025
|
+
|
|
1026
|
+
```
|
|
1027
|
+
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
|
|
1028
|
+
```
|
|
1029
|
+
|
|
1030
|
+
* Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
|
1031
|
+
|
|
1032
|
+
```
|
|
1033
|
+
ascli config preset set default shares shares06
|
|
1034
|
+
```
|
|
1035
|
+
|
|
1036
|
+
* Display the content of configuration file in table format
|
|
1037
|
+
|
|
1038
|
+
```
|
|
1039
|
+
ascli config overview
|
|
1040
|
+
```
|
|
1041
|
+
|
|
1042
|
+
* Execute a command on the shares application using default parameters
|
|
1043
|
+
|
|
1044
|
+
```
|
|
1045
|
+
ascli shares repo browse /
|
|
1046
|
+
```
|
|
1047
|
+
|
|
1048
|
+
## <a id="vault"></a>Secret Vault
|
|
1049
|
+
|
|
1050
|
+
When a secret or password is needed, it is possible to store in the secret vault.
|
|
1051
|
+
|
|
1052
|
+
By default the vault is defined using option `secrets`.
|
|
1053
|
+
|
|
1054
|
+
### Using system keychain
|
|
1055
|
+
|
|
1056
|
+
Only on macOS.
|
|
1057
|
+
|
|
1058
|
+
It is possible to store secrets in macOS keychain (only read supported currently).
|
|
1059
|
+
|
|
1060
|
+
Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
|
|
1061
|
+
|
|
1062
|
+
### Modern config file format: encrypted in config file
|
|
1063
|
+
|
|
1064
|
+
It is possible to store and use secrets encrypted.
|
|
1065
|
+
For this use the `config vault` command.
|
|
1066
|
+
|
|
1067
|
+
The vault can be initialized with `config vault init`
|
|
1068
|
+
|
|
1069
|
+
Then secrets can be manipulated using commands:
|
|
1070
|
+
|
|
1071
|
+
* `set`
|
|
1072
|
+
* `get`
|
|
1073
|
+
* `list`
|
|
1074
|
+
* `delete`
|
|
1075
|
+
|
|
1076
|
+
Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
|
|
1077
|
+
|
|
1078
|
+
### Legacy config file format
|
|
1079
|
+
|
|
1080
|
+
The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
|
|
1081
|
+
|
|
1082
|
+
For example, choose a repository name, for example `my_secrets`, and populate it like this:
|
|
1083
|
+
|
|
1084
|
+
```
|
|
1085
|
+
ascli conf id my_secrets set 'access_key1' 'secret1'
|
|
1086
|
+
|
|
1087
|
+
ascli conf id my_secrets set 'access_key2' 'secret2'
|
|
1088
|
+
|
|
1089
|
+
ascli conf id default get config
|
|
1090
|
+
|
|
1091
|
+
cli_default
|
|
1092
|
+
```
|
|
1093
|
+
|
|
1094
|
+
Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
|
|
1095
|
+
So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
|
|
1096
|
+
|
|
1097
|
+
```
|
|
1098
|
+
ascli conf id cli_default set secrets @val:@preset:my_secrets
|
|
956
1099
|
```
|
|
957
1100
|
|
|
958
|
-
|
|
959
|
-
specified application ("shares"). The 5th command displays the content of configuration file in table format.
|
|
960
|
-
Alternative [option presets](#lprt) can be used with option "-P<[option preset](#lprt)>"
|
|
961
|
-
(or --preset=<[option preset](#lprt)>)
|
|
1101
|
+
A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
|
|
962
1102
|
|
|
963
|
-
|
|
1103
|
+
To test if a secret can be found use:
|
|
964
1104
|
|
|
1105
|
+
```
|
|
1106
|
+
ascli conf vault get --username=access_key1
|
|
1107
|
+
```
|
|
965
1108
|
|
|
966
1109
|
## Plugins
|
|
967
1110
|
|
|
968
1111
|
The CLI tool uses a plugin mechanism. The first level command (just after `ascli` on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application.
|
|
969
1112
|
For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".
|
|
970
1113
|
|
|
971
|
-
### Create your own plugin
|
|
1114
|
+
### <a id="createownplugin"></a>Create your own plugin
|
|
1115
|
+
|
|
972
1116
|
```
|
|
973
|
-
|
|
974
|
-
|
|
1117
|
+
mkdir -p ~/.aspera/ascli/plugins
|
|
1118
|
+
cat<<EOF>~/.aspera/ascli/plugins/test.rb
|
|
975
1119
|
require 'aspera/cli/plugin'
|
|
976
1120
|
module Aspera
|
|
977
1121
|
module Cli
|
|
@@ -986,39 +1130,78 @@ end # Aspera
|
|
|
986
1130
|
EOF
|
|
987
1131
|
```
|
|
988
1132
|
|
|
989
|
-
|
|
1133
|
+
### <a id="plugins"></a>Plugins: Application URL and Authentication
|
|
1134
|
+
|
|
1135
|
+
`ascli` comes with several Aspera application plugins.
|
|
1136
|
+
|
|
1137
|
+
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
|
1138
|
+
|
|
1139
|
+
Those are using options:
|
|
1140
|
+
|
|
1141
|
+
* url
|
|
1142
|
+
* username
|
|
1143
|
+
* password
|
|
1144
|
+
|
|
1145
|
+
Those can be provided using command line, parameter set, env var, see section above.
|
|
1146
|
+
|
|
1147
|
+
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
|
1148
|
+
|
|
1149
|
+
## Logging, Debugging
|
|
990
1150
|
|
|
991
|
-
The gem is equipped with traces. By default logging level is
|
|
1151
|
+
The gem is equipped with traces. By default logging level is `warn`.
|
|
1152
|
+
To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `ASCLI_LOG_LEVEL`, or parameter in con file).
|
|
992
1153
|
|
|
993
|
-
It is also possible to activate traces before
|
|
1154
|
+
It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
|
|
1155
|
+
|
|
1156
|
+
By default passwords and secrets are removed from logs.
|
|
1157
|
+
Use option `log_passwords` to change this behaviour.
|
|
994
1158
|
|
|
995
1159
|
## Learning Aspera Product APIs (REST)
|
|
996
1160
|
|
|
997
1161
|
This CLI uses REST APIs.
|
|
998
|
-
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display
|
|
999
|
-
exact content or HTTP requests and responses.
|
|
1162
|
+
To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
|
|
1000
1163
|
|
|
1001
1164
|
In order to get traces of execution, use argument : `--log-level=debug`
|
|
1002
1165
|
|
|
1003
|
-
## <a
|
|
1166
|
+
## <a id="http_options"></a>HTTP socket parameters
|
|
1167
|
+
|
|
1168
|
+
If the server does not provide a valid certificate, use parameter: `--insecure=yes`.
|
|
1169
|
+
|
|
1170
|
+
Some of HTTP socket parameters can be adjusted, those are the parameters of Ruby [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html), for example:
|
|
1171
|
+
|
|
1172
|
+
* `read_timeout` 60 sec
|
|
1173
|
+
* `write_timeout` 60 sec
|
|
1174
|
+
|
|
1175
|
+
Default values are the ones of Ruby.
|
|
1176
|
+
|
|
1177
|
+
Example:
|
|
1178
|
+
|
|
1179
|
+
```
|
|
1180
|
+
ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
|
1181
|
+
```
|
|
1182
|
+
|
|
1183
|
+
## <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
|
|
1004
1184
|
|
|
1005
1185
|
Some actions may require the use of a graphical tool:
|
|
1006
1186
|
|
|
1007
1187
|
* a browser for Aspera on Cloud authentication (web auth method)
|
|
1008
1188
|
* a text editor for configuration file edition
|
|
1009
1189
|
|
|
1010
|
-
By default the CLI will assume that a graphical environment is available on windows,
|
|
1011
|
-
and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
|
1190
|
+
By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
|
|
1012
1191
|
It is also possible to force the graphical mode with option --ui :
|
|
1013
1192
|
|
|
1014
|
-
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or
|
|
1015
|
-
a text
|
|
1016
|
-
* `--ui=text` forces a text environment, the URL or file path to open is displayed on
|
|
1017
|
-
terminal.
|
|
1193
|
+
* `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
|
|
1194
|
+
* `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
|
|
1018
1195
|
|
|
1019
1196
|
## HTTP proxy for REST
|
|
1020
1197
|
|
|
1021
|
-
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are
|
|
1198
|
+
To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
|
|
1199
|
+
|
|
1200
|
+
## <a id="certificates"></a>SSL CA certificate bundle
|
|
1201
|
+
|
|
1202
|
+
`ascli` uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
|
|
1203
|
+
|
|
1204
|
+
`ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). `ascli` overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
|
|
1022
1205
|
|
|
1023
1206
|
## Proxy auto config
|
|
1024
1207
|
|
|
@@ -1027,13 +1210,13 @@ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file,
|
|
|
1027
1210
|
The PAC file can be tested with command: `config proxy_check` , example:
|
|
1028
1211
|
|
|
1029
1212
|
```
|
|
1030
|
-
|
|
1213
|
+
ascli config proxy_check --fpac=file:///./proxy.pac http://www.example.com
|
|
1031
1214
|
PROXY proxy.example.com:8080
|
|
1032
1215
|
```
|
|
1033
1216
|
|
|
1034
1217
|
This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
|
|
1035
1218
|
|
|
1036
|
-
## <a
|
|
1219
|
+
## <a id="client"></a>FASP configuration
|
|
1037
1220
|
|
|
1038
1221
|
The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
|
|
1039
1222
|
|
|
@@ -1045,9 +1228,9 @@ The `config` plugin also allows specification for the use of a local FASP client
|
|
|
1045
1228
|
### Show path of currently used `ascp`
|
|
1046
1229
|
|
|
1047
1230
|
```
|
|
1048
|
-
|
|
1231
|
+
ascli config ascp show
|
|
1049
1232
|
/Users/laurent/.aspera/ascli/sdk/ascp
|
|
1050
|
-
|
|
1233
|
+
ascli config ascp info
|
|
1051
1234
|
+--------------------+-----------------------------------------------------------+
|
|
1052
1235
|
| key | value |
|
|
1053
1236
|
+--------------------+-----------------------------------------------------------+
|
|
@@ -1055,7 +1238,7 @@ $ ascli config ascp info
|
|
|
1055
1238
|
...
|
|
1056
1239
|
```
|
|
1057
1240
|
|
|
1058
|
-
### Selection of
|
|
1241
|
+
### Selection of `ascp` location for [`direct`](#agt_direct) agent
|
|
1059
1242
|
|
|
1060
1243
|
By default, `ascli` uses any found local product with ascp, including SDK.
|
|
1061
1244
|
|
|
@@ -1066,7 +1249,7 @@ For a permanent change, the command `config ascp use` sets the same parameter fo
|
|
|
1066
1249
|
Using a POSIX shell:
|
|
1067
1250
|
|
|
1068
1251
|
```
|
|
1069
|
-
|
|
1252
|
+
ascli config ascp use '/Users/laurent/Applications/Aspera CLI/bin/ascp'
|
|
1070
1253
|
ascp version: 4.0.0.182279
|
|
1071
1254
|
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
|
|
1072
1255
|
Saved to default global preset global_common_defaults
|
|
@@ -1075,7 +1258,7 @@ Saved to default global preset global_common_defaults
|
|
|
1075
1258
|
Windows:
|
|
1076
1259
|
|
|
1077
1260
|
```
|
|
1078
|
-
|
|
1261
|
+
ascli config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
|
1079
1262
|
ascp version: 4.0.0.182279
|
|
1080
1263
|
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
|
|
1081
1264
|
Saved to default global preset global_common_defaults
|
|
@@ -1088,7 +1271,7 @@ If the path has spaces, read section: [Shell and Command line parsing](#parsing)
|
|
|
1088
1271
|
Locally installed Aspera products can be listed with:
|
|
1089
1272
|
|
|
1090
1273
|
```
|
|
1091
|
-
|
|
1274
|
+
ascli config ascp products list
|
|
1092
1275
|
:.........................................:................................................:
|
|
1093
1276
|
: name : app_root :
|
|
1094
1277
|
:.........................................:................................................:
|
|
@@ -1099,7 +1282,7 @@ $ ascli config ascp products list
|
|
|
1099
1282
|
:.........................................:................................................:
|
|
1100
1283
|
```
|
|
1101
1284
|
|
|
1102
|
-
### Selection of local client
|
|
1285
|
+
### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
|
|
1103
1286
|
|
|
1104
1287
|
If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
|
|
1105
1288
|
|
|
@@ -1108,14 +1291,14 @@ Using the option use_product finds the ascp binary of the selected product.
|
|
|
1108
1291
|
To permanently use the ascp of a product:
|
|
1109
1292
|
|
|
1110
1293
|
```
|
|
1111
|
-
|
|
1294
|
+
ascli config ascp products use 'Aspera Connect'
|
|
1112
1295
|
saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
|
|
1113
1296
|
```
|
|
1114
1297
|
|
|
1115
1298
|
### Installation of Connect Client on command line
|
|
1116
1299
|
|
|
1117
1300
|
```
|
|
1118
|
-
|
|
1301
|
+
ascli config ascp connect list
|
|
1119
1302
|
:...............................................:......................................:..............:
|
|
1120
1303
|
: id : title : version :
|
|
1121
1304
|
:...............................................:......................................:..............:
|
|
@@ -1128,7 +1311,7 @@ $ ascli config ascp connect list
|
|
|
1128
1311
|
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32 : 3.6.2.117442 :
|
|
1129
1312
|
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64 : 3.7.2.141527 :
|
|
1130
1313
|
:...............................................:......................................:..............:
|
|
1131
|
-
|
|
1314
|
+
ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
|
1132
1315
|
:.............................................:..........................:.......................................................................:..........:...............:
|
|
1133
1316
|
: title : type : href : hreflang : rel :
|
|
1134
1317
|
:.............................................:..........................:.......................................................................:..........:...............:
|
|
@@ -1141,11 +1324,11 @@ $ ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links list
|
|
|
1141
1324
|
: Aspera Connect PDF Documentation for Mac OS : application/pdf : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf : zh-cn : documentation :
|
|
1142
1325
|
: Aspera Connect for Mac Release Notes : text/html : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en : release-notes :
|
|
1143
1326
|
:.............................................:..........................:.......................................................................:..........:...............:
|
|
1144
|
-
|
|
1327
|
+
ascli config ascp connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
|
|
1145
1328
|
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg
|
|
1146
1329
|
```
|
|
1147
1330
|
|
|
1148
|
-
## <a
|
|
1331
|
+
## <a id="agents"></a>Transfer Agents
|
|
1149
1332
|
|
|
1150
1333
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
|
1151
1334
|
|
|
@@ -1154,94 +1337,109 @@ This [_transfer-spec_](#transferspec) will be executed by a transfer client, her
|
|
|
1154
1337
|
|
|
1155
1338
|
There are currently 3 agents:
|
|
1156
1339
|
|
|
1157
|
-
* `direct` : a local execution of `ascp`
|
|
1158
|
-
* `connect` : use of a local Connect Client
|
|
1159
|
-
* `node` : use of an Aspera Transfer Node (potentially _remote_).
|
|
1160
|
-
* `httpgw` : use of an Aspera HTTP Gateway
|
|
1340
|
+
* [`direct`](#agt_direct) : a local execution of `ascp`
|
|
1341
|
+
* [`connect`](#agt_connect) : use of a local Connect Client
|
|
1342
|
+
* [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
|
|
1343
|
+
* [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
|
|
1344
|
+
* [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
|
|
1161
1345
|
|
|
1162
1346
|
Note that all transfer operation are seen from the point of view of the agent.
|
|
1163
1347
|
For instance, a node agent making an "upload", or "package send" operation,
|
|
1164
1348
|
will effectively push files to the related server from the agent node.
|
|
1165
1349
|
|
|
1166
|
-
`ascli`
|
|
1350
|
+
`ascli` standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
|
|
1167
1351
|
|
|
1168
1352
|
|
|
1169
|
-
### <a
|
|
1353
|
+
### <a id="agt_direct"></a>Direct
|
|
1170
1354
|
|
|
1171
|
-
|
|
1172
|
-
|
|
1355
|
+
The `direct` agent directly executes a local ascp.
|
|
1356
|
+
This is the default for `ascli`.
|
|
1357
|
+
This is equivalent to specifying `--transfer=direct`.
|
|
1358
|
+
`ascli` will detect locally installed Aspera products, including SDK.
|
|
1173
1359
|
Refer to section [FASP](#client).
|
|
1174
1360
|
|
|
1175
|
-
|
|
1176
|
-
|
|
1177
|
-
* `EX_fasp_proxy_url`
|
|
1178
|
-
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
|
1179
|
-
* `EX_ascp_args`
|
|
1180
|
-
|
|
1181
|
-
The `transfer-info` accepts the following optional parameters:
|
|
1361
|
+
The `transfer-info` accepts the following optional parameters to control multi-session, WSS
|
|
1182
1362
|
|
|
1183
1363
|
<table>
|
|
1184
|
-
<tr><th>Name</th><th>Type</th><th>
|
|
1185
|
-
<tr><td>
|
|
1186
|
-
<tr><td>
|
|
1187
|
-
<tr><td>
|
|
1188
|
-
<tr><td>
|
|
1364
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
|
1365
|
+
<tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
|
|
1366
|
+
<tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
|
|
1367
|
+
<tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
|
|
1368
|
+
<tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
|
|
1369
|
+
<tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
|
|
1370
|
+
<tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
|
|
1371
|
+
<tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
|
|
1372
|
+
<tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
|
|
1373
|
+
<tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
|
|
1189
1374
|
</table>
|
|
1190
1375
|
|
|
1191
|
-
Resume
|
|
1376
|
+
Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
|
|
1377
|
+
Sleep between iterations is:
|
|
1192
1378
|
|
|
1193
|
-
|
|
1194
|
-
|
|
1195
|
-
|
|
1196
|
-
|
|
1197
|
-
|
|
1198
|
-
<tr><td>sleep_max</td><td>int</td><td>60</td><td>Resume</td><td>Maximum sleep</td></tr>
|
|
1199
|
-
</table>
|
|
1379
|
+
```
|
|
1380
|
+
max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
|
|
1381
|
+
```
|
|
1382
|
+
|
|
1383
|
+
Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
|
|
1200
1384
|
|
|
1201
1385
|
Examples:
|
|
1202
1386
|
|
|
1203
1387
|
```
|
|
1204
|
-
|
|
1205
|
-
|
|
1388
|
+
ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
|
|
1389
|
+
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
|
1206
1390
|
```
|
|
1207
1391
|
|
|
1208
|
-
|
|
1392
|
+
To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
|
|
1393
|
+
|
|
1394
|
+
* `EX_fasp_proxy_url`
|
|
1395
|
+
* `EX_http_proxy_url` (proxy for legacy http fallback)
|
|
1396
|
+
* `EX_ascp_args`
|
|
1397
|
+
|
|
1398
|
+
### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
|
1209
1399
|
|
|
1210
|
-
By specifying option: `--transfer=connect`, `ascli` will start transfers
|
|
1211
|
-
using the locally installed Aspera Connect Client.
|
|
1400
|
+
By specifying option: `--transfer=connect`, `ascli` will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
|
|
1212
1401
|
|
|
1213
|
-
### Aspera Node API : Node to node transfers
|
|
1402
|
+
### <a id="agt_node"></a>Aspera Node API : Node to node transfers
|
|
1214
1403
|
|
|
1215
1404
|
By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
|
|
1216
1405
|
Transfer Server using the Node API, either on a local or remote node.
|
|
1406
|
+
Parameters provided in option `transfer_info` are:
|
|
1217
1407
|
|
|
1218
|
-
|
|
1219
|
-
|
|
1220
|
-
|
|
1221
|
-
|
|
1222
|
-
|
|
1408
|
+
<table>
|
|
1409
|
+
<tr><th>Name</th><th>Type</th><th>Description</th></tr>
|
|
1410
|
+
<tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
|
|
1411
|
+
<tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
|
|
1412
|
+
<tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
|
|
1413
|
+
<tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
|
|
1414
|
+
</table>
|
|
1223
1415
|
|
|
1224
|
-
|
|
1225
|
-
`--transfer-info=@preset:<psetname>` or specified using the
|
|
1416
|
+
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
|
1417
|
+
`--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
|
|
1226
1418
|
`--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
|
|
1227
1419
|
|
|
1228
|
-
|
|
1420
|
+
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
|
1229
1421
|
|
|
1230
|
-
|
|
1422
|
+
If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
|
|
1231
1423
|
|
|
1232
|
-
### <a
|
|
1424
|
+
### <a id="agt_httpgw"></a>HTTP Gateway
|
|
1233
1425
|
|
|
1234
|
-
If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
1426
|
+
If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
|
|
1235
1427
|
|
|
1236
1428
|
Example:
|
|
1237
1429
|
|
|
1238
1430
|
```
|
|
1239
|
-
|
|
1431
|
+
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
|
1240
1432
|
```
|
|
1241
1433
|
|
|
1242
1434
|
Note that the gateway only supports transfers authorized with a token.
|
|
1243
1435
|
|
|
1244
|
-
|
|
1436
|
+
### <a id="agt_trsdk"></a>Transfer SDK
|
|
1437
|
+
|
|
1438
|
+
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
|
1439
|
+
|
|
1440
|
+
By default it will listen on local port `55002` on `127.0.0.1`.
|
|
1441
|
+
|
|
1442
|
+
## <a id="transferspec"></a>Transfer Specification
|
|
1245
1443
|
|
|
1246
1444
|
Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
|
|
1247
1445
|
is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
@@ -1254,9 +1452,9 @@ is described in a _transfer-spec_ (Transfer Specification), such as:
|
|
|
1254
1452
|
|
|
1255
1453
|
`ascli` builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
|
|
1256
1454
|
|
|
1257
|
-
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are
|
|
1455
|
+
If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
|
|
1258
1456
|
|
|
1259
|
-
It is possible to specify ascp options when the `transfer` option is set to `direct` using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is
|
|
1457
|
+
It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
|
|
1260
1458
|
|
|
1261
1459
|
The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
1262
1460
|
|
|
@@ -1265,99 +1463,35 @@ The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
|
|
|
1265
1463
|
|
|
1266
1464
|
A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
|
|
1267
1465
|
|
|
1268
|
-
## <a
|
|
1466
|
+
## <a id="transferparams"></a>Transfer Parameters
|
|
1269
1467
|
|
|
1270
|
-
All standard _transfer-spec_ parameters can be
|
|
1271
|
-
|
|
1272
|
-
|
|
1468
|
+
All standard _transfer-spec_ parameters can be specified.
|
|
1469
|
+
[_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
|
|
1470
|
+
|
|
1471
|
+
References:
|
|
1273
1472
|
|
|
1473
|
+
* [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)→/opt/transfers
|
|
1474
|
+
* [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)→Guides→API Ref→Transfer Spec V1
|
|
1475
|
+
* [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.`
|
|
1476
|
+
|
|
1477
|
+
Parameters can be displayed with commands:
|
|
1478
|
+
|
|
1479
|
+
```
|
|
1480
|
+
ascli config ascp spec
|
|
1481
|
+
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
|
1482
|
+
```
|
|
1274
1483
|
|
|
1275
|
-
<p>
|
|
1276
1484
|
Columns:
|
|
1277
|
-
<ul>
|
|
1278
|
-
<li>F=Fasp Manager(local FASP execution)</li>
|
|
1279
|
-
<li>N=remote node(node API)</li>
|
|
1280
|
-
<li>C=Connect Client(web plugin)</li>
|
|
1281
|
-
</ul>
|
|
1282
|
-
</p>
|
|
1283
|
-
<p>
|
|
1284
|
-
Req/Def : Required or default value (- means emty)
|
|
1285
|
-
</p>
|
|
1286
|
-
<p>
|
|
1287
|
-
Fields with EX_ prefix are specific extensions to local mode.
|
|
1288
|
-
</p>
|
|
1289
|
-
<p>
|
|
1290
|
-
arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)
|
|
1291
|
-
</p>
|
|
1292
|
-
<p>
|
|
1293
|
-
UNDER CONSTRUCTION<br/>
|
|
1294
|
-
<a href="https://developer.ibm.com/apis/catalog/?search=aspera">Aspera API Documentation</a>→Node API→/opt/transfers<br/>
|
|
1295
|
-
</p>
|
|
1296
1485
|
|
|
1297
|
-
|
|
1298
|
-
|
|
1299
|
-
|
|
1300
|
-
|
|
1301
|
-
|
|
1302
|
-
<tr><td>destination_root</td><td>Required</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>last arg</td><td>Destination root directory.</td></tr>
|
|
1303
|
-
<tr><td>title</td><td>-</td><td>string</td><td class="no">N</td><td class="yes">Y</td><td class="yes">Y</td><td>-</td><td>Title of the transfer</td></tr>
|
|
1304
|
-
<tr><td>tags</td><td>-</td><td>hash</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--tags<br>--tags64</td><td>Metadata for transfer</td></tr>
|
|
1305
|
-
<tr><td>token</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>TOKEN<br/>-W</td><td>Authorization token: Bearer, Basic or ATM</td></tr>
|
|
1306
|
-
<tr><td>cookie</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>COOKIE</td><td>Metadata for transfer (older,string)</td></tr>
|
|
1307
|
-
<tr><td>remote_access_key</td><td>TODO</td><td>string</td><td></td><td></td><td></td><td>?</td><td>Node only?</td></tr>
|
|
1308
|
-
<tr><td>source_root</td><td>-</td><td>string</td><td></td><td></td><td></td><td>--source-prefix<br/>--source-prefix64</td><td>Source root directory.(TODO: verify option)</td></tr>
|
|
1309
|
-
<tr><td>fasp_port</td><td>33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-O</td><td>Specifies fasp (UDP) port.</td></tr>
|
|
1310
|
-
<tr><td>ssh_port</td><td>22 or 33001</td><td>integer</td></td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-P</td><td>Specifies ssh (TCP) port.</td></tr>
|
|
1311
|
-
<tr><td>rate_policy</td><td>server config</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--policy</td><td>Valid literals include "low","fair","high" and "fixed".</td></tr>
|
|
1312
|
-
<tr><td>symlink_policy</td><td>follow</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--symbolic-links</td><td>copy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).</td></tr>
|
|
1313
|
-
<tr><td>target_rate_kbps</td><td>-</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-l</td><td>Specifies desired speed for the transfer.</td></tr>
|
|
1314
|
-
<tr><td>min_rate_kbps</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-m</td><td>Set the minimum transfer rate in kilobits per second.</td></tr>
|
|
1315
|
-
<tr><td>cipher</td><td>none</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-c</td><td>in transit encryption type.<br/>none, aes-128, aes-256</td></tr>
|
|
1316
|
-
<tr><td>content_protection</td><td>encrypt<br/>decrypt</td><td>string</td><td></td><td></td><td></td><td>--file-crypt=</td><td>encryption at rest</td></tr>
|
|
1317
|
-
<tr><td>content_protection_password</td><td>-</td><td>string</td><td></td><td></td><td></td><td>PASS</td><td>Specifies a string password.</td></tr>
|
|
1318
|
-
<tr><td>overwrite</td><td>diff</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--overwrite</td><td>Overwrite destination files with the source files of the same name.<br/>never, always, diff, older, or diff+older</td></tr>
|
|
1319
|
-
<tr><td>retry_duration</td><td></td><td>string</td><td></td><td></td><td></td><td>TODO</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr>
|
|
1320
|
-
<tr><td>http_fallback</td><td></td><td>bool (node), integer</td><td></td><td></td><td></td><td>-y<br/>TODO</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.</td></tr>
|
|
1321
|
-
<tr><td>create_dir</td><td></td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-d</td><td>Specifies whether to create new directories.</td></tr>
|
|
1322
|
-
<tr><td>precalculate_job_size</td><td>srv. def.</td><td>boolean</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>--precalculate-job-size</td><td>Specifies whether to precalculate the job size.</td></tr>
|
|
1323
|
-
<tr><td>delete_source</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>?</td></tr>
|
|
1324
|
-
<tr><td>remove_after_transfer</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove file after transfer.</td></tr>
|
|
1325
|
-
<tr><td>remove_empty_directories</td><td></td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>?</td><td>Specifies whether to remove empty directories.</td></tr>
|
|
1326
|
-
<tr><td>multi_session</td><td>1</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-C</td><td>Specifies how many parts the transfer is in.</td></tr>
|
|
1327
|
-
<tr><td>multi_session_threshold</td><td>null</td><td>integer</td><td class="no">N</td><td class="yes">Y</td><td class="no">N</td><td>-</td><td>in bytes</td></tr>
|
|
1328
|
-
<tr><td>exclude_newer_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-newer-than</td><td>-</td></tr>
|
|
1329
|
-
<tr><td>exclude_older_than</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>--exclude-older-than</td><td>-</td></tr>
|
|
1330
|
-
<tr><td>preserve_acls</td><td></td><td>string</td><td class="yes">Y</td><td></td><td></td><td>--preserve-acls</td><td>-</td></tr>
|
|
1331
|
-
<tr><td>dgram_size</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-Z</td><td>in bytes</td></tr>
|
|
1332
|
-
<tr><td>compression</td><td></td><td>integer</td><td></td><td></td><td></td><td></td><td>ascp4 only, 0 / 1?</td></tr>
|
|
1333
|
-
<tr><td>read_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
|
1334
|
-
<tr><td>write_threads</td><td></td><td>integer</td><td></td><td></td><td></td><td>-</td><td>ascp4 only</td></tr>
|
|
1335
|
-
<tr><td>use_ascp4</td><td>false</td><td>boolean</td><td></td><td class="yes">Y</td><td></td><td>-</td><td>specify version of protocol</td></tr>
|
|
1336
|
-
<tr><td>paths</td><td>source files (dest)</td><td>array</td><td></td><td></td><td></td><td>positional<br/>--file-list<br/>--file-pair-list</td><td>Contains a path to the source (required) and a path to the destination.</td></tr>
|
|
1337
|
-
<tr><td>http_fallback_port</td><td></td><td>integer</td><td class="yes">Y</td><td></td><td></td><td>-t</td><td>Specifies http port.</td></tr>
|
|
1338
|
-
<tr><td>https_fallback_port</td><td></td><td>integer</td><td></td><td></td><td></td><td>todo</td><td>Specifies https port.</td></tr>
|
|
1339
|
-
<tr><td>cipher_allowed</td><td></td><td>string</td><td></td><td></td><td></td><td>-</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr>
|
|
1340
|
-
<tr><td>target_rate_cap_kbps</td><td></td><td></td><td class="no">N</td><td class="no">?</td><td class="yes">?</td><td>-</td><td>Returned by upload/download_setup node api.</td></tr>
|
|
1341
|
-
<tr><td>rate_policy_allowed</td><td></td><td></td><td></td><td></td><td></td><td>-</td><td>returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".</td></tr>
|
|
1342
|
-
<tr><td>ssh_private_key</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>KEY</td><td>Private key used for SSH authentication, Shall look like: `-----BEGIN RSA PRIVATE KEY-----\nMII`<br/>Note the JSON encoding `\` + `n` for newlines.</td></tr>
|
|
1343
|
-
<tr><td>remote_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>PASS</td><td>SSH session password</td></tr>
|
|
1344
|
-
<tr><td>resume_policy</td><td>faspmgr:<br/>none<br/>other:<br/>sparse_csum</td><td>string</td><td class="yes">Y</td><td class="yes">Y</td><td class="yes">Y</td><td>-k</td><td>none,attrs,sparse_csum,full_csum</td></tr>
|
|
1345
|
-
<tr><td>authentication</td><td>-</td><td class="no">N</td><td class="no">N</td><td class="yes">Y</td><td>-</td><td>token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.</td></tr>
|
|
1346
|
-
<tr><td>EX_ssh_key_paths</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-i</td><td>Use public key authentication and specify the private key file</td></tr>
|
|
1347
|
-
<tr><td>EX_at_rest_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>FILEPASS</td><td>Passphrase used for at rest encryption or decryption</td></tr>
|
|
1348
|
-
<tr><td>EX_proxy_password</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>PROXY_PASS</td><td>TODO</td></tr>
|
|
1349
|
-
<tr><td>EX_fasp_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--proxy</td><td>Specify the address of the Aspera high-speed proxy server</td></tr>
|
|
1350
|
-
<tr><td>EX_http_proxy_url</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-x</td><td>Specify the proxy server address used by HTTP Fallback</td></tr>
|
|
1351
|
-
<tr><td>EX_ascp_args</td><td>-</td><td>array</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>same</td><td>Add command line arguments to ascp</td></tr>
|
|
1352
|
-
<tr><td>EX_http_transfer_jpeg</td><td>0</td><td>integer</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-j</td><td>HTTP transfers as JPEG file</td></tr>
|
|
1353
|
-
<tr><td>EX_license_text</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>LICENSE</td><td>license file text</td></tr>
|
|
1354
|
-
<tr><td>EX_file_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-list</td><td>source file list</td></tr>
|
|
1355
|
-
<tr><td>EX_file_pair_list</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--file-pair-list</td><td>source file pair list</td></tr>
|
|
1356
|
-
<tr><td>EX_multi_session_part</td><td>-</td><td>string</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>-C</td><td>part for multisession</td></tr>
|
|
1357
|
-
<tr><td>EX_no_read</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-read</td><td>no read source</td></tr>
|
|
1358
|
-
<tr><td>EX_no_write</td><td>-</td><td>-</td><td class="yes">Y</td><td class="no">N</td><td class="no">N</td><td>--no-write</td><td>no write estination</td></tr>
|
|
1359
|
-
</table>
|
|
1486
|
+
* D=Direct (local `ascp` execution)
|
|
1487
|
+
* N=Node API
|
|
1488
|
+
* C=Connect Client
|
|
1489
|
+
|
|
1490
|
+
`ascp` argument or environment variable is provided in description.
|
|
1360
1491
|
|
|
1492
|
+
Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in `ascli`).
|
|
1493
|
+
|
|
1494
|
+
<table><tr><th>Field</th><th>Type</th><th>D</th><th>N</th><th>C</th><th>Description</th></tr><tr><td>cipher</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>In transit encryption type.<br/>none, aes-128, aes-256<br/>Allowed values: aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm<br/>(-c)</td></tr><tr><td>content_protection</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Enable client-side content protection. (encryption at rest)<br/>Allowed values: encrypt, decrypt</td></tr><tr><td>content_protection_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies CSEAR password.</td></tr><tr><td>cookie</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer (older,string)<br/>(env:ASPERA_SCP_COOKIE)</td></tr><tr><td>create_dir</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to create new directories.<br/>(-d)</td></tr><tr><td>delete_before_transfer</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--delete-before-transfer)</td></tr><tr><td>delete_source</td><td>bool</td><td> </td><td>Y</td><td> </td><td>Remove SRC files after transfer success</td></tr><tr><td>direction</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Direction of transfer (on client side)<br/>Allowed values: send, receive<br/>(--mode)</td></tr><tr><td>exclude_newer_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime > arg<br/>(--exclude-newer-than)</td></tr><tr><td>exclude_older_than</td><td>int</td><td>Y</td><td> </td><td> </td><td>skip src files with mtime < arg<br/>(--exclude-older-than)</td></tr><tr><td>fasp_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies fasp (UDP) port.<br/>(-O)</td></tr><tr><td>http_fallback</td><td>string<br/>bool</td><td>Y</td><td>Y</td><td>Y</td><td>When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.<br/>(-y)</td></tr><tr><td>http_fallback_port</td><td>int</td><td>Y</td><td> </td><td> </td><td>Specifies http port.<br/>(-t)</td></tr><tr><td>https_fallback_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies https port.</td></tr><tr><td>move_after_transfer</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>(--move-after-transfer)</td></tr><tr><td>multi_session</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Use multi-session transfer. max 128.<br/> Each participant on one host needs an independent UDP (-O) port.<br/> Large files are split between sessions only when transferring with resume_policy=none.</td></tr><tr><td>multi_session_threshold</td><td>int</td><td>Y</td><td>Y</td><td> </td><td>in bytes<br/>(--multi-session-threshold)</td></tr><tr><td>overwrite</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Overwrite destination files with the source files of the same name.<br/>Allowed values: never, always, diff, older, diff+older<br/>(--overwrite)</td></tr><tr><td>paths</td><td>array</td><td>Y</td><td>Y</td><td>Y</td><td>Required. Contains a path to the source (required) and a path to the destination.</td></tr><tr><td>precalculate_job_size</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies whether to precalculate the job size.<br/>(--precalculate-job-size)</td></tr><tr><td>preserve_access_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-access-time)</td></tr><tr><td>preserve_creation_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-creation-time)</td></tr><tr><td>preserve_modification_time</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-modification-time)</td></tr><tr><td>preserve_times</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td>(--preserve-times)</td></tr><tr><td>rate_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>The transfer rate policy to use when sharing bandwidth.<br/>Allowed values: low, fair, high, fixed<br/>(--policy)</td></tr><tr><td>remote_access_key</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Node only?</td></tr><tr><td>remote_host</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>IP or fully qualified domain name of the remote server<br/>(--host)</td></tr><tr><td>remote_user</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Remote user. Default value is "xfer" on node or connect.<br/>(--user)</td></tr><tr><td>remote_password</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>SSH session password<br/>(env:ASPERA_SCP_PASS)</td></tr><tr><td>remove_after_transfer</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Remove SRC files after transfer success<br/>(--remove-after-transfer)</td></tr><tr><td>remove_empty_directories</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>Specifies whether to remove empty directories.<br/>(--remove-empty-directories)</td></tr><tr><td>proxy</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the address of the Aspera high-speed proxy server.<br/> dnat(s)://[user[:password]@]server:port<br/> Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/> Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS.<br/>(--proxy)</td></tr><tr><td>resume_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>If a transfer is interrupted or fails to finish, resume without re-transferring the whole files.<br/>Allowed values: none, attrs, sparse_csum, full_csum<br/>(-k)</td></tr><tr><td>retry_duration</td><td>string<br/>int</td><td> </td><td>Y</td><td>Y</td><td>Specifies how long to wait before retrying transfer. (e.g. "5min")</td></tr><tr><td>ssh_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies ssh (TCP) port. Default: local:22, other:33001<br/>(-P)</td></tr><tr><td>ssh_private_key</td><td>string</td><td>Y</td><td> </td><td> </td><td>Private key used for SSH authentication.<br/> Shall look like: -----BEGIN RSA PRIVATE KEY-----\nMII...<br/> Note the JSON encoding: \n for newlines.<br/>(env:ASPERA_SCP_KEY)</td></tr><tr><td>symlink_policy</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Handle source side symbolic links by:<br/> following the link (follow),<br/> copying the link itself (copy),<br/> skipping (skip),<br/> or forcibly copying the link itself (copy+force).<br/> Default: follow<br/>Allowed values: follow, copy, copy+force, skip<br/>(--symbolic-links)</td></tr><tr><td>tags</td><td>hash</td><td>Y</td><td>Y</td><td>Y</td><td>Metadata for transfer<br/>(--tags64)</td></tr><tr><td>target_rate_cap_kbps</td><td>int</td><td> </td><td> </td><td>Y</td><td>Returned by upload/download_setup node api.</td></tr><tr><td>target_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Specifies desired speed for the transfer.<br/>(-l)</td></tr><tr><td>title</td><td>string</td><td> </td><td>Y</td><td>Y</td><td>Title of the transfer</td></tr><tr><td>token</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN)</td></tr><tr><td>use_ascp4</td><td>bool</td><td>Y</td><td>Y</td><td> </td><td>specify version of protocol</td></tr><tr><td>destination_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Destination root directory.</td></tr><tr><td>source_root</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Path to be prepended to each source path.<br/> This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(--source-prefix64)</td></tr><tr><td>min_rate_cap_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_rate_policy</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_min_rate_kbps</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>apply_local_docroot</td><td>bool</td><td>Y</td><td> </td><td> </td><td>(--apply-local-docroot)</td></tr><tr><td>preserve_acls</td><td>string</td><td>Y</td><td> </td><td> </td><td>Preserve access control lists.<br/>Allowable values: none, native, metafile<br/>(--preserve-acls)</td></tr><tr><td>remove_empty_source_directory</td><td>bool</td><td>Y</td><td> </td><td> </td><td>TODO: check node, sdk<br/>(--remove-empty-source-directory)</td></tr><tr><td>EX_at_rest_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Passphrase used for at rest encryption or decryption<br/>(env:ASPERA_SCP_FILEPASS)</td></tr><tr><td>EX_proxy_password</td><td>string</td><td>Y</td><td> </td><td> </td><td>Password used for Aspera proxy server authentication.<br/> May be overridden by password in URL EX_fasp_proxy_url.<br/>(env:ASPERA_PROXY_PASS)</td></tr><tr><td>EX_license_text</td><td>string</td><td>Y</td><td> </td><td> </td><td>License file text override.<br/>By default ascp looks for license file near executable.<br/>(env:ASPERA_SCP_LICENSE)</td></tr><tr><td>dgram_size</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>in bytes<br/>(-Z)</td></tr><tr><td>min_rate_kbps</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td>Set the minimum transfer rate in kilobits per second.<br/>(-m)</td></tr><tr><td>sshfp</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>Check it against server SSH host key fingerprint<br/>(--check-sshfp)</td></tr><tr><td>EX_http_proxy_url</td><td>string</td><td>Y</td><td> </td><td> </td><td>Specify the proxy server address used by HTTP Fallback<br/>(-x)</td></tr><tr><td>EX_ssh_key_paths</td><td>array</td><td>Y</td><td> </td><td> </td><td>Use public key authentication for SSH and specify the private key file paths<br/>(-i)</td></tr><tr><td>EX_http_transfer_jpeg</td><td>int</td><td>Y</td><td> </td><td> </td><td>HTTP transfers as JPEG file<br/>(-j)</td></tr><tr><td>EX_no_read</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no read source<br/>(--no-read)</td></tr><tr><td>EX_no_write</td><td>bool</td><td>Y</td><td> </td><td> </td><td>no write on destination<br/>(--no-write)</td></tr><tr><td>target_rate_percentage</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>rate_policy_allowed</td><td>string</td><td> </td><td> </td><td>Y</td><td>Specifies most aggressive rate policy that is allowed.<br/> Returned by node API.<br/>Allowed values: low, fair, high, fixed</td></tr><tr><td>lock_min_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>lock_target_rate</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>authentication</td><td>string</td><td> </td><td> </td><td>Y</td><td>value=token for SSH bypass keys, else password asked if not provided.</td></tr><tr><td>cipher_allowed</td><td>string</td><td>Y</td><td>Y</td><td>Y</td><td>returned by node API. Valid literals include "aes-128" and "none".</td></tr><tr><td>EX_file_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file list</td></tr><tr><td>EX_file_pair_list</td><td>string</td><td>Y</td><td> </td><td> </td><td>source file pair list</td></tr><tr><td>EX_ascp_args</td><td>array</td><td>Y</td><td> </td><td> </td><td>Add command line arguments to ascp</td></tr><tr><td>wss_enabled</td><td>bool</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr><tr><td>wss_port</td><td>int</td><td>Y</td><td>Y</td><td>Y</td><td> </td></tr></table>
|
|
1361
1495
|
|
|
1362
1496
|
### Destination folder for transfers
|
|
1363
1497
|
|
|
@@ -1373,20 +1507,20 @@ The option `to_folder` provides an equivalent and convenient way to change this
|
|
|
1373
1507
|
|
|
1374
1508
|
### List of files for transfers
|
|
1375
1509
|
|
|
1376
|
-
When uploading, downloading or sending files, the user must specify the list of files to transfer.
|
|
1377
|
-
|
|
1510
|
+
When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
|
|
1511
|
+
So, by default, the list of files to transfer will be simply specified on the command line:
|
|
1378
1512
|
|
|
1379
1513
|
```
|
|
1380
|
-
|
|
1514
|
+
ascli server upload ~/mysample.file secondfile
|
|
1381
1515
|
```
|
|
1382
1516
|
|
|
1383
1517
|
This is equivalent to:
|
|
1384
1518
|
|
|
1385
1519
|
```
|
|
1386
|
-
|
|
1520
|
+
ascli server upload --sources=@args ~/mysample.file secondfile
|
|
1387
1521
|
```
|
|
1388
1522
|
|
|
1389
|
-
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
|
1523
|
+
More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
|
|
1390
1524
|
|
|
1391
1525
|
Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
|
|
1392
1526
|
all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
|
|
@@ -1409,13 +1543,21 @@ For ease of use and flexibility, the list of files to transfer is specified by t
|
|
|
1409
1543
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
|
1410
1544
|
```
|
|
1411
1545
|
|
|
1412
|
-
|
|
1546
|
+
providing a file list directly to ascp:
|
|
1547
|
+
|
|
1548
|
+
```
|
|
1549
|
+
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
|
|
1550
|
+
```
|
|
1551
|
+
|
|
1552
|
+
* Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
|
|
1413
1553
|
|
|
1414
1554
|
```
|
|
1415
1555
|
--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
|
|
1416
1556
|
```
|
|
1417
1557
|
|
|
1418
|
-
|
|
1558
|
+
This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
|
|
1559
|
+
|
|
1560
|
+
In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
|
|
1419
1561
|
|
|
1420
1562
|
* `list` : (default) the path of destination is the same as source
|
|
1421
1563
|
* `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
|
|
@@ -1423,9 +1565,11 @@ In case the file list is provided on the command line (i.e. using `--sources=@ar
|
|
|
1423
1565
|
Example:
|
|
1424
1566
|
|
|
1425
1567
|
```
|
|
1426
|
-
|
|
1568
|
+
ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
|
|
1427
1569
|
```
|
|
1428
1570
|
|
|
1571
|
+
Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
|
|
1572
|
+
|
|
1429
1573
|
Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
|
|
1430
1574
|
|
|
1431
1575
|
* All files must be in the same source folder.
|
|
@@ -1437,9 +1581,9 @@ Source files are located on "Aspera on cloud", when :
|
|
|
1437
1581
|
* the server is Aspera on Cloud, and making a download / recv
|
|
1438
1582
|
* the agent is Aspera on Cloud, and making an upload / send
|
|
1439
1583
|
|
|
1440
|
-
### <a
|
|
1584
|
+
### <a id="multisession"></a>Support of multi-session
|
|
1441
1585
|
|
|
1442
|
-
Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.
|
|
1586
|
+
Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
|
|
1443
1587
|
|
|
1444
1588
|
* when agent=node :
|
|
1445
1589
|
|
|
@@ -1460,6 +1604,7 @@ shall be preferred.
|
|
|
1460
1604
|
|
|
1461
1605
|
Multi-session spawn is done by `ascli`.
|
|
1462
1606
|
|
|
1607
|
+
When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
|
|
1463
1608
|
|
|
1464
1609
|
### Examples
|
|
1465
1610
|
|
|
@@ -1487,19 +1632,162 @@ Multi-session spawn is done by `ascli`.
|
|
|
1487
1632
|
--ts=@json:'{"precalculate_job_size":true}'
|
|
1488
1633
|
```
|
|
1489
1634
|
|
|
1635
|
+
## <a id="scheduling"></a>Lock for exclusive execution
|
|
1636
|
+
|
|
1637
|
+
In some conditions, it may be desirable to ensure that `ascli` is not executed several times in parallel.
|
|
1638
|
+
|
|
1639
|
+
For instance when `ascli` is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
|
|
1640
|
+
|
|
1641
|
+
* Executing instances may pile-up and kill the system
|
|
1642
|
+
* The same file may be transferred by multiple instances at the same time.
|
|
1643
|
+
* `preview` may generate the same files in multiple instances.
|
|
1644
|
+
|
|
1645
|
+
Usually the OS native scheduler already provides some sort of protection against parallel execution:
|
|
1646
|
+
|
|
1647
|
+
* The Windows scheduler does this by default
|
|
1648
|
+
* Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
|
|
1649
|
+
|
|
1650
|
+
```
|
|
1651
|
+
/usr/bin/flock -w 0 /var/cron.lock ascli ...
|
|
1652
|
+
```
|
|
1653
|
+
|
|
1654
|
+
`ascli` natively supports a locking mechanism with option `lock_port`.
|
|
1655
|
+
(Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
|
|
1656
|
+
|
|
1657
|
+
Example:
|
|
1658
|
+
|
|
1659
|
+
Run this same command in two separate terminals within less than 30 seconds:
|
|
1660
|
+
|
|
1661
|
+
```
|
|
1662
|
+
ascli config echo @ruby:'sleep(30)' --lock-port=12345
|
|
1663
|
+
```
|
|
1664
|
+
|
|
1665
|
+
The first instance will sleep 30 seconds, the second one will immediately exit like this:
|
|
1666
|
+
|
|
1667
|
+
```
|
|
1668
|
+
WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
|
|
1669
|
+
```
|
|
1670
|
+
|
|
1671
|
+
## "Provençale"
|
|
1490
1672
|
|
|
1673
|
+
`ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
|
|
1674
|
+
, some are provided om shared libraries and must be activated. (e.g. using `trapd`)
|
|
1491
1675
|
|
|
1492
|
-
|
|
1676
|
+
The list of supported *PVCL* adapters can be retrieved with command:
|
|
1493
1677
|
|
|
1494
|
-
|
|
1678
|
+
```
|
|
1679
|
+
ascli conf ascp info
|
|
1680
|
+
+--------------------+-----------------------------------------------------------+
|
|
1681
|
+
| key | value |
|
|
1682
|
+
+--------------------+-----------------------------------------------------------+
|
|
1683
|
+
-----8<-----snip-----8<-----
|
|
1684
|
+
| product_name | IBM Aspera SDK |
|
|
1685
|
+
| product_version | 4.0.1.182389 |
|
|
1686
|
+
| process | pvcl |
|
|
1687
|
+
| shares | pvcl |
|
|
1688
|
+
| noded | pvcl |
|
|
1689
|
+
| faux | pvcl |
|
|
1690
|
+
| file | pvcl |
|
|
1691
|
+
| stdio | pvcl |
|
|
1692
|
+
| stdio-tar | pvcl |
|
|
1693
|
+
+--------------------+-----------------------------------------------------------+
|
|
1694
|
+
```
|
|
1695
|
+
|
|
1696
|
+
Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
|
|
1697
|
+
|
|
1698
|
+
Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
|
|
1495
1699
|
|
|
1496
|
-
|
|
1700
|
+
The simplified format is:
|
|
1701
|
+
|
|
1702
|
+
```
|
|
1703
|
+
<adapter>:///<sub file path>?<arg1>=<val1>&...
|
|
1704
|
+
```
|
|
1705
|
+
|
|
1706
|
+
One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
|
|
1707
|
+
|
|
1708
|
+
## <a id="faux_testing"></a>`faux:` for testing
|
|
1709
|
+
|
|
1710
|
+
This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not `ascli`.
|
|
1711
|
+
|
|
1712
|
+
This adapter can be used to simulate a file or a directory.
|
|
1713
|
+
|
|
1714
|
+
To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
|
|
1715
|
+
|
|
1716
|
+
```
|
|
1717
|
+
faux:///filename?filesize
|
|
1718
|
+
```
|
|
1719
|
+
|
|
1720
|
+
where:
|
|
1721
|
+
|
|
1722
|
+
* `filename` is the name that will be assigned to the file on the destination
|
|
1723
|
+
* `filesize` is the number of bytes that will be sent (in decimal).
|
|
1724
|
+
|
|
1725
|
+
Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
|
1726
|
+
|
|
1727
|
+
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
|
1728
|
+
|
|
1729
|
+
To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
|
|
1730
|
+
|
|
1731
|
+
```
|
|
1732
|
+
faux:///dirname?<arg1>=<val1>&...
|
|
1733
|
+
```
|
|
1734
|
+
|
|
1735
|
+
where:
|
|
1736
|
+
|
|
1737
|
+
* `dirname` is the folder name and can contain `/` to specify a subfolder.
|
|
1738
|
+
* supported arguments are:
|
|
1739
|
+
|
|
1740
|
+
<table>
|
|
1741
|
+
<tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
|
|
1742
|
+
<tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
|
|
1743
|
+
<tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
|
|
1744
|
+
<tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
|
|
1745
|
+
<tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
|
|
1746
|
+
<tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
|
|
1747
|
+
<tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
|
|
1748
|
+
</table>
|
|
1749
|
+
|
|
1750
|
+
The sequence parameter is applied as follows:
|
|
1751
|
+
|
|
1752
|
+
* If `seq` is `random` then each file size is:
|
|
1753
|
+
|
|
1754
|
+
* size +/- (inc * rand())
|
|
1755
|
+
* Where rand is a random number between 0 and 1
|
|
1756
|
+
* Note that file size must not be negative, inc will be set to size if it is greater than size
|
|
1757
|
+
* Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
|
|
1758
|
+
|
|
1759
|
+
* If `seq` is `sequential` then each file size is:
|
|
1760
|
+
|
|
1761
|
+
* `size + ((fileindex - 1) * inc)`
|
|
1762
|
+
* Where first file is index 1
|
|
1763
|
+
* So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
|
1764
|
+
* As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
|
1765
|
+
|
|
1766
|
+
Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
|
|
1767
|
+
|
|
1768
|
+
To discard data at the destination, the destination argument is set to `faux://` .
|
|
1769
|
+
|
|
1770
|
+
Examples:
|
|
1771
|
+
|
|
1772
|
+
* Upload 20 gibibytes of random data to file myfile to directory /Upload
|
|
1773
|
+
|
|
1774
|
+
```
|
|
1775
|
+
ascli server upload faux:///myfile\?20g --to-folder=/Upload
|
|
1776
|
+
```
|
|
1777
|
+
|
|
1778
|
+
* Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
|
|
1779
|
+
|
|
1780
|
+
```
|
|
1781
|
+
ascli server upload /tmp/sample --to-folder=faux://
|
|
1782
|
+
```
|
|
1497
1783
|
|
|
1498
|
-
|
|
1784
|
+
* Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
|
|
1499
1785
|
|
|
1500
|
-
|
|
1786
|
+
```
|
|
1787
|
+
ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
|
|
1788
|
+
```
|
|
1501
1789
|
|
|
1502
|
-
## <a
|
|
1790
|
+
## <a id="commands"></a>Sample Commands
|
|
1503
1791
|
|
|
1504
1792
|
A non complete list of commands used in unit tests:
|
|
1505
1793
|
|
|
@@ -1508,21 +1796,21 @@ ascli
|
|
|
1508
1796
|
ascli -h
|
|
1509
1797
|
ascli aoc -N remind --username=my_aoc_user_email
|
|
1510
1798
|
ascli aoc -N servers
|
|
1511
|
-
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}'
|
|
1512
|
-
ascli aoc admin ats access_key --id=akibmcloud --secret=somesecret node browse /
|
|
1513
|
-
ascli aoc admin ats access_key --id=akibmcloud delete
|
|
1799
|
+
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
|
1514
1800
|
ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
|
1515
1801
|
ascli aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
|
1802
|
+
ascli aoc admin ats access_key delete akibmcloud
|
|
1516
1803
|
ascli aoc admin ats access_key list --fields=name,id
|
|
1804
|
+
ascli aoc admin ats access_key node akibmcloud --secret=somesecret browse /
|
|
1517
1805
|
ascli aoc admin ats cluster clouds
|
|
1518
1806
|
ascli aoc admin ats cluster list
|
|
1519
1807
|
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
|
1520
|
-
ascli aoc admin ats cluster show
|
|
1521
|
-
ascli aoc admin res
|
|
1808
|
+
ascli aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
|
1809
|
+
ascli aoc admin res application list
|
|
1522
1810
|
ascli aoc admin res client list
|
|
1523
1811
|
ascli aoc admin res client_access_key list
|
|
1524
|
-
ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
|
|
1525
1812
|
ascli aoc admin res client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}'
|
|
1813
|
+
ascli aoc admin res client_registration_token delete my_clt_reg_id
|
|
1526
1814
|
ascli aoc admin res client_registration_token list
|
|
1527
1815
|
ascli aoc admin res contact list
|
|
1528
1816
|
ascli aoc admin res dropbox list
|
|
@@ -1532,34 +1820,35 @@ ascli aoc admin res kms_profile list
|
|
|
1532
1820
|
ascli aoc admin res node list
|
|
1533
1821
|
ascli aoc admin res operation list
|
|
1534
1822
|
ascli aoc admin res organization show
|
|
1535
|
-
ascli aoc admin res package list
|
|
1823
|
+
ascli aoc admin res package list --http-options=@json:'{"read_timeout":120.0}'
|
|
1536
1824
|
ascli aoc admin res saml_configuration list
|
|
1537
1825
|
ascli aoc admin res self show
|
|
1538
1826
|
ascli aoc admin res short_link list
|
|
1539
1827
|
ascli aoc admin res user list
|
|
1540
1828
|
ascli aoc admin res workspace_membership list
|
|
1541
1829
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
|
|
1542
|
-
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 access_key delete --id=testsub1
|
|
1543
1830
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v3 events
|
|
1544
1831
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 browse /
|
|
1545
1832
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 delete /folder1
|
|
1546
1833
|
ascli aoc admin resource node --name=AOC_NODE1_NAME --secret=AOC_NODE1_SECRET v4 mkdir /folder1
|
|
1834
|
+
ascli aoc admin resource node v3 name AOC_NODE1_NAME --secret=AOC_NODE1_SECRET access_key delete testsub1
|
|
1547
1835
|
ascli aoc admin resource workspace list
|
|
1548
1836
|
ascli aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
|
1549
|
-
ascli aoc automation workflow
|
|
1837
|
+
ascli aoc automation workflow "my_wf_id" action create --value=@json:'{"name":"toto"}' | tee action.info
|
|
1550
1838
|
ascli aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
|
|
1551
|
-
ascli aoc automation workflow delete
|
|
1839
|
+
ascli aoc automation workflow delete "my_wf_id"
|
|
1552
1840
|
ascli aoc automation workflow list
|
|
1553
1841
|
ascli aoc automation workflow list --select=@json:'{"name":"test_workflow"}' --fields=id --format=csv --display=data > test
|
|
1554
1842
|
ascli aoc automation workflow list --value=@json:'{"show_org_workflows":"true"}' --scope=admin:all
|
|
1555
1843
|
ascli aoc bearer_token --display=data --scope=user:all
|
|
1556
1844
|
ascli aoc faspex
|
|
1557
1845
|
ascli aoc files bearer /
|
|
1846
|
+
ascli aoc files bearer_token_node /
|
|
1558
1847
|
ascli aoc files browse /
|
|
1559
1848
|
ascli aoc files browse / -N --link=my_aoc_publink_folder
|
|
1560
1849
|
ascli aoc files delete /testsrc
|
|
1561
1850
|
ascli aoc files download --transfer=connect /200KB.1
|
|
1562
|
-
ascli aoc files file
|
|
1851
|
+
ascli aoc files file show my_file_id
|
|
1563
1852
|
ascli aoc files find / --value='\.partial$'
|
|
1564
1853
|
ascli aoc files http_node_download --to-folder=. /200KB.1
|
|
1565
1854
|
ascli aoc files mkdir /testsrc
|
|
@@ -1570,13 +1859,14 @@ ascli aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_li
|
|
|
1570
1859
|
ascli aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
|
|
1571
1860
|
ascli aoc files upload --to-folder=/testsrc testfile.bin
|
|
1572
1861
|
ascli aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
|
1862
|
+
ascli aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
|
1573
1863
|
ascli aoc files v3 info
|
|
1574
1864
|
ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
|
|
1575
1865
|
ascli aoc organization
|
|
1576
1866
|
ascli aoc packages list
|
|
1577
|
-
ascli aoc packages list --
|
|
1578
|
-
ascli aoc packages recv
|
|
1579
|
-
ascli aoc packages recv
|
|
1867
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"my_shbxid","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
|
1868
|
+
ascli aoc packages recv "my_package_id" --to-folder=.
|
|
1869
|
+
ascli aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
|
1580
1870
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
|
1581
1871
|
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["internal.user@example.com"],"note":"my note"}' testfile.bin
|
|
1582
1872
|
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
|
|
@@ -1584,21 +1874,23 @@ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' t
|
|
|
1584
1874
|
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
|
|
1585
1875
|
ascli aoc user info modify @json:'{"name":"dummy change"}'
|
|
1586
1876
|
ascli aoc user info show
|
|
1877
|
+
ascli aoc user shared_inboxes
|
|
1878
|
+
ascli aoc user workspaces
|
|
1587
1879
|
ascli aoc workspace
|
|
1588
|
-
ascli ats access_key --
|
|
1589
|
-
ascli ats access_key --id=akibmcloud --secret=somesecret cluster
|
|
1590
|
-
ascli ats access_key --id=akibmcloud --secret=somesecret node browse /
|
|
1591
|
-
ascli ats access_key --id=akibmcloud delete
|
|
1880
|
+
ascli ats access_key cluster akibmcloud --secret=somesecret
|
|
1592
1881
|
ascli ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
|
1593
1882
|
ascli ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
|
1883
|
+
ascli ats access_key delete ak_aws
|
|
1884
|
+
ascli ats access_key delete akibmcloud
|
|
1594
1885
|
ascli ats access_key list --fields=name,id
|
|
1886
|
+
ascli ats access_key node akibmcloud browse / --secret=somesecret
|
|
1595
1887
|
ascli ats api_key create
|
|
1596
1888
|
ascli ats api_key instances
|
|
1597
1889
|
ascli ats api_key list
|
|
1598
1890
|
ascli ats cluster clouds
|
|
1599
1891
|
ascli ats cluster list
|
|
1600
1892
|
ascli ats cluster show --cloud=aws --region=eu-west-1
|
|
1601
|
-
ascli ats cluster show
|
|
1893
|
+
ascli ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
|
|
1602
1894
|
ascli conf flush_tokens
|
|
1603
1895
|
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes
|
|
1604
1896
|
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath='' --username=my_aoc_user_email --test-mode=yes --use-generic-client=yes
|
|
@@ -1610,10 +1902,13 @@ ascli config ascp info
|
|
|
1610
1902
|
ascli config ascp install
|
|
1611
1903
|
ascli config ascp products list
|
|
1612
1904
|
ascli config ascp show
|
|
1905
|
+
ascli config ascp spec
|
|
1613
1906
|
ascli config check_update
|
|
1907
|
+
ascli config detect --url=https://my_aoc_org.ibmaspera.com;\
|
|
1908
|
+
ascli config detect --url=my_faspex_url
|
|
1614
1909
|
ascli config doc
|
|
1615
1910
|
ascli config doc transfer-parameters
|
|
1616
|
-
ascli config email_test
|
|
1911
|
+
ascli config email_test --notif-to=my_recipient_email
|
|
1617
1912
|
ascli config export
|
|
1618
1913
|
ascli config genkey mykey
|
|
1619
1914
|
ascli config plugins
|
|
@@ -1623,34 +1918,35 @@ ascli console transfer smart list
|
|
|
1623
1918
|
ascli console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
|
|
1624
1919
|
ascli cos -N --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
|
|
1625
1920
|
ascli cos -N --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
|
|
1626
|
-
ascli cos node access_key
|
|
1921
|
+
ascli cos node access_key show self
|
|
1627
1922
|
ascli cos node download testfile.bin --to-folder=.
|
|
1628
1923
|
ascli cos node info
|
|
1629
1924
|
ascli cos node upload testfile.bin
|
|
1630
1925
|
ascli faspex health
|
|
1631
1926
|
ascli faspex package list
|
|
1632
|
-
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data
|
|
1633
|
-
ascli faspex package list --fields=package_id --format=csv --display=data
|
|
1634
|
-
ascli faspex package recv --to-folder=.
|
|
1635
|
-
ascli faspex package recv --to-folder=. --
|
|
1636
|
-
ascli faspex package recv --to-folder=.
|
|
1927
|
+
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data --query=@json:'{"max":1}');\
|
|
1928
|
+
ascli faspex package list --fields=package_id --format=csv --display=data --query=@json:'{"max":1}');\
|
|
1929
|
+
ascli faspex package recv "my_package_id" --to-folder=.
|
|
1930
|
+
ascli faspex package recv "my_package_id" --to-folder=. --box=sent
|
|
1931
|
+
ascli faspex package recv --to-folder=. "my_package_id"
|
|
1637
1932
|
ascli faspex package recv --to-folder=. --link="my_faspex_publink_recv_from_fxuser"
|
|
1933
|
+
ascli faspex package recv ALL --to-folder=. --once-only=yes
|
|
1638
1934
|
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com","FASPEX_USERNAME"]}' testfile.bin
|
|
1639
1935
|
ascli faspex package send --link="my_faspex_publink_send_to_dropbox" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
|
1640
1936
|
ascli faspex package send --link="my_faspex_publink_send_to_fxuser" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
|
|
1641
1937
|
ascli faspex source name "Server Files" node br /
|
|
1642
1938
|
ascli faspex5 node list --value=@json:'{"type":"received","subtype":"mypackages"}'
|
|
1643
1939
|
ascli faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
|
1644
|
-
ascli faspex5 package receive
|
|
1940
|
+
ascli faspex5 package receive "my_package_id" --to-folder=.
|
|
1645
1941
|
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"${f5_user}"}]}' testfile.bin
|
|
1646
1942
|
ascli node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
|
|
1647
|
-
ascli node -N -Ptst_node_preview access_key delete
|
|
1648
|
-
ascli node async
|
|
1649
|
-
ascli node async
|
|
1650
|
-
ascli node async
|
|
1943
|
+
ascli node -N -Ptst_node_preview access_key delete aoc_1
|
|
1944
|
+
ascli node async bandwidth 1
|
|
1945
|
+
ascli node async counters 1
|
|
1946
|
+
ascli node async files 1
|
|
1651
1947
|
ascli node async list
|
|
1652
|
-
ascli node async show
|
|
1653
|
-
ascli node async show
|
|
1948
|
+
ascli node async show 1
|
|
1949
|
+
ascli node async show ALL
|
|
1654
1950
|
ascli node basic_token
|
|
1655
1951
|
ascli node browse / -r
|
|
1656
1952
|
ascli node delete folder_1/10MB.1
|
|
@@ -1659,8 +1955,8 @@ ascli node download --to-folder=. folder_1/testfile.bin
|
|
|
1659
1955
|
ascli node health
|
|
1660
1956
|
ascli node info
|
|
1661
1957
|
ascli node search / --value=@json:'{"sort":"mtime"}'
|
|
1662
|
-
ascli node service --id=service1 delete
|
|
1663
1958
|
ascli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
|
|
1959
|
+
ascli node service delete service1
|
|
1664
1960
|
ascli node service list
|
|
1665
1961
|
ascli node transfer list --value=@json:'{"active_only":true}'
|
|
1666
1962
|
ascli node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass"}'
|
|
@@ -1668,12 +1964,12 @@ ascli node upload --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000
|
|
|
1668
1964
|
ascli orchestrator info
|
|
1669
1965
|
ascli orchestrator plugins
|
|
1670
1966
|
ascli orchestrator processes
|
|
1671
|
-
ascli orchestrator workflow
|
|
1672
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID start --params=@json:'{"Param":"world !"}'
|
|
1673
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID start --params=@json:'{"Param":"world !"}' --result=ResultStep:Complete_status_message
|
|
1674
|
-
ascli orchestrator workflow --id=ORCH_WORKFLOW_ID status
|
|
1967
|
+
ascli orchestrator workflow inputs ORCH_WORKFLOW_ID
|
|
1675
1968
|
ascli orchestrator workflow list
|
|
1676
|
-
ascli orchestrator workflow
|
|
1969
|
+
ascli orchestrator workflow start ORCH_WORKFLOW_ID --params=@json:'{"Param":"world !"}'
|
|
1970
|
+
ascli orchestrator workflow start ORCH_WORKFLOW_ID --params=@json:'{"Param":"world !"}' --result=ResultStep:Complete_status_message
|
|
1971
|
+
ascli orchestrator workflow status ALL
|
|
1972
|
+
ascli orchestrator workflow status ORCH_WORKFLOW_ID
|
|
1677
1973
|
ascli preview check --skip-types=office
|
|
1678
1974
|
ascli preview folder 1 --skip-types=office --log-level=info --file-access=remote --ts=@json:'{"target_rate_kbps":1000000}'
|
|
1679
1975
|
ascli preview scan --skip-types=office --log-level=info
|
|
@@ -1709,7 +2005,7 @@ ascli server mkdir folder_1/target_hot
|
|
|
1709
2005
|
ascli server mv folder_1/200KB.2 folder_1/to.delete
|
|
1710
2006
|
ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
|
1711
2007
|
ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
|
1712
|
-
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername
|
|
2008
|
+
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
|
|
1713
2009
|
ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
|
1714
2010
|
ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
|
|
1715
2011
|
ascli shares admin share list
|
|
@@ -1729,12 +2025,12 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
|
|
|
1729
2025
|
...and more
|
|
1730
2026
|
```
|
|
1731
2027
|
|
|
1732
|
-
## <a
|
|
2028
|
+
## <a id="usage"></a>Usage
|
|
1733
2029
|
|
|
1734
2030
|
```
|
|
1735
|
-
|
|
2031
|
+
ascli -h
|
|
1736
2032
|
NAME
|
|
1737
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
|
2033
|
+
ascli -- a command line tool for Aspera Applications (v4.5.0)
|
|
1738
2034
|
|
|
1739
2035
|
SYNOPSIS
|
|
1740
2036
|
ascli COMMANDS [OPTIONS] [ARGS]
|
|
@@ -1744,10 +2040,11 @@ DESCRIPTION
|
|
|
1744
2040
|
Documentation and examples: https://rubygems.org/gems/aspera-cli
|
|
1745
2041
|
execute: ascli conf doc
|
|
1746
2042
|
or visit: http://www.rubydoc.info/gems/aspera-cli
|
|
2043
|
+
source repo: https://github.com/IBM/aspera-cli
|
|
1747
2044
|
|
|
1748
2045
|
ENVIRONMENT VARIABLES
|
|
1749
2046
|
ASCLI_HOME config folder, default: $HOME/.aspera/ascli
|
|
1750
|
-
|
|
2047
|
+
any option can be set as an environment variable, refer to the manual
|
|
1751
2048
|
|
|
1752
2049
|
COMMANDS
|
|
1753
2050
|
To list first level commands, execute: ascli
|
|
@@ -1755,7 +2052,8 @@ COMMANDS
|
|
|
1755
2052
|
|
|
1756
2053
|
OPTIONS
|
|
1757
2054
|
Options begin with a '-' (minus), and value is provided on command line.
|
|
1758
|
-
Special values are supported beginning with special prefix
|
|
2055
|
+
Special values are supported beginning with special prefix @pfx:, where pfx is one of:
|
|
2056
|
+
base64, json, zlib, ruby, csvt, lines, list, incps, val, file, path, env, stdin, preset
|
|
1759
2057
|
Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'
|
|
1760
2058
|
|
|
1761
2059
|
ARGS
|
|
@@ -1781,39 +2079,43 @@ OPTIONS: global
|
|
|
1781
2079
|
--logger=ENUM log method: stderr, stdout, syslog
|
|
1782
2080
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
|
1783
2081
|
--query=VALUE additional filter for API calls (extended value) (some commands)
|
|
2082
|
+
--http-options=VALUE options for http socket (extended value)
|
|
1784
2083
|
--insecure=ENUM do not validate HTTPS certificate: yes, no
|
|
1785
2084
|
--once-only=ENUM process only new items (some commands): yes, no
|
|
2085
|
+
--log-passwords=ENUM show passwords in logs: yes, no
|
|
1786
2086
|
|
|
1787
2087
|
COMMAND: config
|
|
1788
|
-
SUBCOMMANDS:
|
|
2088
|
+
SUBCOMMANDS: list overview id preset open documentation genkey gem_path plugins flush_tokens echo wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update initdemo vault
|
|
1789
2089
|
OPTIONS:
|
|
1790
2090
|
--value=VALUE extended value for create, update, list filter
|
|
1791
2091
|
--property=VALUE name of property to set
|
|
1792
2092
|
--id=VALUE resource identifier (modify,delete,show)
|
|
1793
2093
|
--config-file=VALUE read parameters from file in YAML format, current=/Users/FooBar/.aspera/ascli/config.yaml
|
|
1794
|
-
--override=ENUM override existing value: yes, no
|
|
1795
2094
|
-N, --no-default do not load default configuration for plugin
|
|
1796
|
-
--
|
|
1797
|
-
--
|
|
2095
|
+
--override=ENUM Wizard: override existing value: yes, no
|
|
2096
|
+
--use-generic-client=ENUM Wizard: AoC: use global or org specific jwt client id: yes, no
|
|
2097
|
+
--default=ENUM Wizard: set as default configuration for specified plugin (also: update): yes, no
|
|
2098
|
+
--test-mode=ENUM Wizard: skip private key check step: yes, no
|
|
2099
|
+
-P, --presetVALUE load the named option preset from current config file
|
|
2100
|
+
--pkeypath=VALUE Wizard: path to private key for JWT
|
|
1798
2101
|
--ascp-path=VALUE path to ascp
|
|
1799
2102
|
--use-product=VALUE use ascp from specified product
|
|
1800
2103
|
--smtp=VALUE smtp configuration (extended value: hash)
|
|
1801
2104
|
--fpac=VALUE proxy auto configuration URL
|
|
1802
|
-
-P, --presetVALUE load the named option preset from current config file
|
|
1803
|
-
--default=VALUE set as default configuration for specified plugin
|
|
1804
2105
|
--secret=VALUE default secret
|
|
1805
|
-
--secrets=VALUE secret
|
|
2106
|
+
--secrets=VALUE secret vault
|
|
1806
2107
|
--sdk-url=VALUE URL to get SDK
|
|
1807
|
-
--sdk-folder=VALUE SDK folder
|
|
1808
|
-
--
|
|
1809
|
-
--
|
|
2108
|
+
--sdk-folder=VALUE SDK folder path
|
|
2109
|
+
--notif-to=VALUE email recipient for notification of transfers
|
|
2110
|
+
--notif-template=VALUE email ERB template for notification of transfers
|
|
2111
|
+
--version-check-days=VALUE period in days to check new version (zero to disable)
|
|
1810
2112
|
--ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
|
|
1811
2113
|
--local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
|
|
1812
2114
|
--to-folder=VALUE destination folder for downloaded files
|
|
1813
2115
|
--sources=VALUE list of source files (see doc)
|
|
1814
|
-
--transfer-info=VALUE
|
|
2116
|
+
--transfer-info=VALUE parameters for transfer agent
|
|
1815
2117
|
--src-type=ENUM type of file list: list, pair
|
|
1816
|
-
--transfer=ENUM type of transfer: direct,
|
|
2118
|
+
--transfer=ENUM type of transfer agent: direct, node, connect, httpgw, trsdk
|
|
1817
2119
|
--progress=ENUM type of progress bar: none, native, multi
|
|
1818
2120
|
|
|
1819
2121
|
|
|
@@ -1834,7 +2136,7 @@ OPTIONS:
|
|
|
1834
2136
|
--validator=VALUE identifier of validator (optional for central)
|
|
1835
2137
|
--asperabrowserurl=VALUE URL for simple aspera web ui
|
|
1836
2138
|
--name=VALUE sync name
|
|
1837
|
-
--token=ENUM
|
|
2139
|
+
--token-type=ENUM Type of token used for transfers: aspera, basic, hybrid
|
|
1838
2140
|
|
|
1839
2141
|
|
|
1840
2142
|
COMMAND: orchestrator
|
|
@@ -1884,11 +2186,11 @@ OPTIONS:
|
|
|
1884
2186
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
|
1885
2187
|
--username=VALUE username to log in
|
|
1886
2188
|
--password=VALUE user's password
|
|
1887
|
-
--client-id=VALUE
|
|
1888
|
-
--client-secret=VALUE
|
|
1889
|
-
--redirect-uri=VALUE
|
|
1890
|
-
--auth=ENUM type of
|
|
1891
|
-
--private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
|
|
2189
|
+
--client-id=VALUE OAuth client identifier
|
|
2190
|
+
--client-secret=VALUE OAuth client secret
|
|
2191
|
+
--redirect-uri=VALUE OAuth redirect URI
|
|
2192
|
+
--auth=ENUM OAuth type of authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey, boot
|
|
2193
|
+
--private-key=VALUE Oauth RSA private key PEM value for JWT (prefix file path with @val:@file:)
|
|
1892
2194
|
|
|
1893
2195
|
|
|
1894
2196
|
COMMAND: cos
|
|
@@ -1913,7 +2215,7 @@ OPTIONS:
|
|
|
1913
2215
|
--source-name=VALUE create package from remote source (by name)
|
|
1914
2216
|
--storage=VALUE Faspex local storage definition
|
|
1915
2217
|
--recipient=VALUE use if recipient is a dropbox (with *)
|
|
1916
|
-
--box=ENUM package box: inbox,
|
|
2218
|
+
--box=ENUM package box: inbox, archive, sent
|
|
1917
2219
|
|
|
1918
2220
|
|
|
1919
2221
|
COMMAND: shares2
|
|
@@ -1987,7 +2289,6 @@ OPTIONS:
|
|
|
1987
2289
|
--new-user-option=VALUE new user creation option
|
|
1988
2290
|
--from-folder=VALUE share to share source folder
|
|
1989
2291
|
--scope=VALUE OAuth scope for AoC API calls
|
|
1990
|
-
--notify=VALUE notify users that file was received
|
|
1991
2292
|
--bulk=ENUM bulk operation: yes, no
|
|
1992
2293
|
--default-ports=ENUM use standard FASP ports or get from node api: yes, no
|
|
1993
2294
|
|
|
@@ -2017,34 +2318,18 @@ OPTIONS:
|
|
|
2017
2318
|
|
|
2018
2319
|
Note that actions and parameter values can be written in short form.
|
|
2019
2320
|
|
|
2020
|
-
# <a
|
|
2021
|
-
|
|
2022
|
-
`ascli` comes with several Aspera application plugins.
|
|
2023
|
-
|
|
2024
|
-
REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
|
|
2025
|
-
|
|
2026
|
-
Those are using options:
|
|
2027
|
-
|
|
2028
|
-
* url
|
|
2029
|
-
* username
|
|
2030
|
-
* password
|
|
2031
|
-
|
|
2032
|
-
Those can be provided using command line, parameter set, env var, see section above.
|
|
2033
|
-
|
|
2034
|
-
Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
|
|
2035
|
-
|
|
2036
|
-
# <a name="aoc"></a>Plugin: Aspera on Cloud
|
|
2321
|
+
# <a id="aoc"></a>Plugin: Aspera on Cloud
|
|
2037
2322
|
|
|
2038
2323
|
Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
|
|
2039
2324
|
|
|
2040
2325
|
It is recommended to use the wizard to set it up, but manual configuration is also possible.
|
|
2041
2326
|
|
|
2042
|
-
## <a
|
|
2327
|
+
## <a id="aocwizard"></a>Configuration: using Wizard
|
|
2043
2328
|
|
|
2044
2329
|
`ascli` provides a configuration wizard. Here is a sample invocation :
|
|
2045
2330
|
|
|
2046
2331
|
```
|
|
2047
|
-
|
|
2332
|
+
ascli config wizard
|
|
2048
2333
|
option: url> https://myorg.ibmaspera.com
|
|
2049
2334
|
Detected: Aspera on Cloud
|
|
2050
2335
|
Preparing preset: aoc_myorg
|
|
@@ -2060,7 +2345,7 @@ Setting config preset as default for aspera
|
|
|
2060
2345
|
saving config file
|
|
2061
2346
|
Done.
|
|
2062
2347
|
You can test with:
|
|
2063
|
-
|
|
2348
|
+
ascli aoc user info show
|
|
2064
2349
|
```
|
|
2065
2350
|
|
|
2066
2351
|
Optionally, it is possible to create a new organization-specific "integration".
|
|
@@ -2068,7 +2353,7 @@ For this, specify the option: `--use-generic-client=no`.
|
|
|
2068
2353
|
|
|
2069
2354
|
This will guide you through the steps to create.
|
|
2070
2355
|
|
|
2071
|
-
## <a
|
|
2356
|
+
## <a id="aocmanual"></a>Configuration: using manual setup
|
|
2072
2357
|
|
|
2073
2358
|
If you used the wizard (recommended): skip this section.
|
|
2074
2359
|
|
|
@@ -2080,7 +2365,7 @@ Several types of OAuth authentication are supported:
|
|
|
2080
2365
|
* Web based authentication : authentication is made by user using a browser
|
|
2081
2366
|
* URL Token : external users authentication with url tokens (public links)
|
|
2082
2367
|
|
|
2083
|
-
The authentication method is
|
|
2368
|
+
The authentication method is controlled by option `auth`.
|
|
2084
2369
|
|
|
2085
2370
|
For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [[option preset](#lprt) for Aspera on Cloud](#aocpreset).
|
|
2086
2371
|
|
|
@@ -2088,7 +2373,7 @@ For a more convenient, browser-less, experience follow the [JWT](#jwt) section (
|
|
|
2088
2373
|
|
|
2089
2374
|
In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.`ascli` saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
|
|
2090
2375
|
|
|
2091
|
-
### <a
|
|
2376
|
+
### <a id="clientreg"></a>Optional: API Client Registration
|
|
2092
2377
|
|
|
2093
2378
|
If you use the built-in client_id and client_secret, skip this and do not set them in next section.
|
|
2094
2379
|
|
|
@@ -2112,31 +2397,31 @@ Note: for web based authentication, `ascli` listens on a local port (e.g. specif
|
|
|
2112
2397
|
|
|
2113
2398
|
Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
|
|
2114
2399
|
|
|
2115
|
-
### <a
|
|
2400
|
+
### <a id="aocpreset"></a>[option preset](#lprt) for Aspera on Cloud
|
|
2116
2401
|
|
|
2117
2402
|
If you did not use the wizard, you can also manually create a [option preset](#lprt) for `ascli` in its configuration file.
|
|
2118
2403
|
|
|
2119
2404
|
Lets create an [option preset](#lprt) called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
|
2120
2405
|
|
|
2121
2406
|
```
|
|
2122
|
-
|
|
2407
|
+
ascli config preset ask my_aoc_org url client_id client_secret
|
|
2123
2408
|
option: url> https://myorg.ibmaspera.com/
|
|
2124
2409
|
option: client_id> BJLPObQiFw
|
|
2125
2410
|
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
|
|
2126
2411
|
updated: my_aoc_org
|
|
2127
2412
|
```
|
|
2128
2413
|
|
|
2129
|
-
(This can also be done in one line using the command `config
|
|
2414
|
+
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
|
2130
2415
|
|
|
2131
2416
|
Define this [option preset](#lprt) as default configuration for the `aspera` plugin:
|
|
2132
2417
|
|
|
2133
2418
|
```
|
|
2134
|
-
|
|
2419
|
+
ascli config preset set default aoc my_aoc_org
|
|
2135
2420
|
```
|
|
2136
2421
|
|
|
2137
2422
|
Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
|
|
2138
2423
|
|
|
2139
|
-
### <a
|
|
2424
|
+
### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
|
|
2140
2425
|
|
|
2141
2426
|
For a Browser-less, Private Key-based authentication, use the following steps.
|
|
2142
2427
|
|
|
@@ -2151,13 +2436,13 @@ This can be done using any of the following method:
|
|
|
2151
2436
|
* using the CLI:
|
|
2152
2437
|
|
|
2153
2438
|
```
|
|
2154
|
-
|
|
2439
|
+
ascli config genkey ~/.aspera/ascli/aocapikey
|
|
2155
2440
|
```
|
|
2156
2441
|
|
|
2157
2442
|
* `ssh-keygen`:
|
|
2158
2443
|
|
|
2159
2444
|
```
|
|
2160
|
-
|
|
2445
|
+
ssh-keygen -t rsa -f ~/.aspera/ascli/aocapikey -N ''
|
|
2161
2446
|
```
|
|
2162
2447
|
|
|
2163
2448
|
* `openssl`
|
|
@@ -2165,11 +2450,11 @@ $ ssh-keygen -t rsa -f ~/.aspera/ascli/aocapikey -N ''
|
|
|
2165
2450
|
(on some openssl implementation (mac) there is option: -nodes (no DES))
|
|
2166
2451
|
|
|
2167
2452
|
```
|
|
2168
|
-
|
|
2169
|
-
|
|
2170
|
-
|
|
2171
|
-
|
|
2172
|
-
|
|
2453
|
+
APIKEY=~/.aspera/ascli/aocapikey
|
|
2454
|
+
openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
|
|
2455
|
+
openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
|
|
2456
|
+
openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
|
|
2457
|
+
rm -f ${APIKEY}.protected
|
|
2173
2458
|
```
|
|
2174
2459
|
|
|
2175
2460
|
#### API Client JWT activation
|
|
@@ -2188,13 +2473,13 @@ If you are not using the built-in client_id and secret, JWT needs to be authoriz
|
|
|
2188
2473
|
* Using command line
|
|
2189
2474
|
|
|
2190
2475
|
```
|
|
2191
|
-
|
|
2476
|
+
ascli aoc admin res client list
|
|
2192
2477
|
:............:.........:
|
|
2193
2478
|
: id : name :
|
|
2194
2479
|
:............:.........:
|
|
2195
2480
|
: BJLPObQiFw : ascli :
|
|
2196
2481
|
:............:.........:
|
|
2197
|
-
|
|
2482
|
+
ascli aoc admin res client modify --id=BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
|
2198
2483
|
modified
|
|
2199
2484
|
```
|
|
2200
2485
|
|
|
@@ -2215,14 +2500,14 @@ open the previously generated public key located here: `$HOME/.aspera/ascli/aoca
|
|
|
2215
2500
|
* Using command line
|
|
2216
2501
|
|
|
2217
2502
|
```
|
|
2218
|
-
|
|
2503
|
+
ascli aoc admin res user list
|
|
2219
2504
|
:........:................:
|
|
2220
2505
|
: id : name :
|
|
2221
2506
|
:........:................:
|
|
2222
2507
|
: 109952 : Tech Support :
|
|
2223
2508
|
: 109951 : LAURENT MARTIN :
|
|
2224
2509
|
:........:................:
|
|
2225
|
-
|
|
2510
|
+
ascli aoc user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/ascli/aocapikey.pub"))}'
|
|
2226
2511
|
modified
|
|
2227
2512
|
```
|
|
2228
2513
|
|
|
@@ -2230,16 +2515,16 @@ Note: the `aspera user info show` command can be used to verify modifications.
|
|
|
2230
2515
|
|
|
2231
2516
|
### [option preset](#lprt) modification for JWT
|
|
2232
2517
|
|
|
2233
|
-
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the
|
|
2518
|
+
To activate default use of JWT authentication for `ascli` using the [option preset](#lprt), do the following:
|
|
2234
2519
|
|
|
2235
2520
|
* change auth method to JWT
|
|
2236
2521
|
* provide location of private key
|
|
2237
|
-
* provide username to login as (
|
|
2522
|
+
* provide username to login as (OAuth "subject")
|
|
2238
2523
|
|
|
2239
2524
|
Execute:
|
|
2240
2525
|
|
|
2241
2526
|
```
|
|
2242
|
-
|
|
2527
|
+
ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/ascli/aocapikey --username=laurent.martin.aspera@fr.ibm.com
|
|
2243
2528
|
```
|
|
2244
2529
|
|
|
2245
2530
|
Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
|
|
@@ -2247,73 +2532,157 @@ Note: the private key argument represents the actual PEM string. In order to rea
|
|
|
2247
2532
|
After this last step, commands do not require web login anymore.
|
|
2248
2533
|
|
|
2249
2534
|
|
|
2250
|
-
### <a
|
|
2535
|
+
### <a id="aocfirst"></a>First Use
|
|
2251
2536
|
|
|
2252
2537
|
Once client has been registered and [option preset](#lprt) created: `ascli` can be used:
|
|
2253
2538
|
|
|
2254
2539
|
```
|
|
2255
|
-
|
|
2540
|
+
ascli aoc files br /
|
|
2256
2541
|
Current Workspace: Default Workspace (default)
|
|
2257
2542
|
empty
|
|
2258
2543
|
```
|
|
2259
2544
|
|
|
2260
2545
|
|
|
2261
|
-
|
|
2546
|
+
## Administration
|
|
2262
2547
|
|
|
2263
2548
|
The `admin` command allows several administrative tasks (and require admin privilege).
|
|
2264
2549
|
|
|
2265
2550
|
It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
|
|
2266
2551
|
|
|
2267
|
-
Bulk
|
|
2552
|
+
### Bulk creation and deletion of resource
|
|
2553
|
+
|
|
2554
|
+
Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
|
|
2555
|
+
In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
|
|
2556
|
+
|
|
2557
|
+
### Listing resources
|
|
2558
|
+
|
|
2559
|
+
The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
|
|
2560
|
+
|
|
2561
|
+
The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
|
|
2562
|
+
|
|
2563
|
+
The following parameters are supported:
|
|
2564
|
+
|
|
2565
|
+
* `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
|
|
2566
|
+
* `sort`: name of fields to sort results, prefix with `-` for reverse order.
|
|
2567
|
+
* `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
|
|
2568
|
+
* `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
|
|
2569
|
+
* `page` : native api parameter, in general do not use (added by
|
|
2570
|
+
* `per_page` : native api parameter, number of items par api call, in general do not use
|
|
2571
|
+
* Other specific parameters depending on resource type.
|
|
2572
|
+
|
|
2573
|
+
Both `max` and `pmax` are processed internally in `ascli`, not included in actual API call and limit the number of successive pages requested to API. `ascli` will return all values using paging if not provided.
|
|
2574
|
+
|
|
2575
|
+
Other parameters are directly sent as parameters to the GET request on API.
|
|
2576
|
+
|
|
2577
|
+
`page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
|
|
2268
2578
|
|
|
2269
|
-
|
|
2579
|
+
`q` and `sort` are available on most resource types.
|
|
2580
|
+
|
|
2581
|
+
Other parameters depend on the type of entity (refer to AoC API).
|
|
2582
|
+
|
|
2583
|
+
Examples:
|
|
2584
|
+
|
|
2585
|
+
* List users with `laurent` in name:
|
|
2586
|
+
|
|
2587
|
+
```
|
|
2588
|
+
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
|
2589
|
+
```
|
|
2590
|
+
|
|
2591
|
+
* List users who logged-in before a date:
|
|
2270
2592
|
|
|
2271
2593
|
```
|
|
2272
|
-
--query=@json:'{"
|
|
2594
|
+
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
|
2273
2595
|
```
|
|
2274
2596
|
|
|
2275
|
-
|
|
2597
|
+
* List external users and sort in reverse alphabetical order using name:
|
|
2598
|
+
|
|
2599
|
+
```
|
|
2600
|
+
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
|
2601
|
+
```
|
|
2602
|
+
|
|
2603
|
+
Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
|
|
2604
|
+
|
|
2605
|
+
Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
|
|
2606
|
+
|
|
2607
|
+
### <a id="res_select"></a>Selecting a resource
|
|
2608
|
+
|
|
2609
|
+
Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
|
|
2610
|
+
|
|
2611
|
+
To execute an action on a specific resource, select it using one of those methods:
|
|
2612
|
+
|
|
2613
|
+
* *recommended:* give id directly on command line *after the action*: `aoc admin res node show 123`
|
|
2614
|
+
* give name on command line *after the action*: `aoc admin res node show name abc`
|
|
2615
|
+
* provide option `id` : `aoc admin res node show --id=123`
|
|
2616
|
+
* provide option `name` : `aoc admin res node show --name=abc`
|
|
2617
|
+
|
|
2618
|
+
### Access Key secrets
|
|
2619
|
+
|
|
2620
|
+
In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
|
|
2621
|
+
It is usually provided using the `secret` option.
|
|
2622
|
+
For example in a command like:
|
|
2623
|
+
|
|
2276
2624
|
```
|
|
2277
|
-
--
|
|
2278
|
-
--query=@json:'{"q":"laurent"}'
|
|
2625
|
+
ascli aoc admin res node --id=123 --secret="secret1" v3 info
|
|
2279
2626
|
```
|
|
2280
2627
|
|
|
2281
|
-
|
|
2628
|
+
It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
|
|
2282
2629
|
|
|
2283
|
-
|
|
2630
|
+
### Activity
|
|
2284
2631
|
|
|
2285
|
-
|
|
2286
|
-
secret is required, it is usually provided using the `secret` option. For example in a command like:
|
|
2632
|
+
The activity app can be queried with:
|
|
2287
2633
|
|
|
2288
2634
|
```
|
|
2289
|
-
|
|
2635
|
+
ascli aoc admin analytics transfers
|
|
2290
2636
|
```
|
|
2291
2637
|
|
|
2292
|
-
It
|
|
2638
|
+
It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
|
|
2293
2639
|
|
|
2294
|
-
|
|
2640
|
+
`mytemplate.erb`:
|
|
2295
2641
|
|
|
2296
2642
|
```
|
|
2297
|
-
|
|
2298
|
-
|
|
2299
|
-
|
|
2300
|
-
|
|
2643
|
+
From: <%=from_name%> <<%=from_email%>>
|
|
2644
|
+
To: <<%=ev['user_email']%>>
|
|
2645
|
+
Subject: <%=ev['files_completed']%> files received
|
|
2646
|
+
|
|
2647
|
+
Dear <%=ev[:user_email.to_s]%>,
|
|
2648
|
+
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
|
|
2649
|
+
<%=ev['content']%>
|
|
2650
|
+
|
|
2651
|
+
Thank you.
|
|
2301
2652
|
```
|
|
2653
|
+
The environment provided contains the following additional variable:
|
|
2654
|
+
|
|
2655
|
+
* ev : all details on the transfer event
|
|
2302
2656
|
|
|
2303
|
-
|
|
2657
|
+
Example:
|
|
2304
2658
|
|
|
2305
2659
|
```
|
|
2306
|
-
|
|
2660
|
+
ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
|
2661
|
+
--query=@json:'{"status":"completed","direction":"receive"}' \
|
|
2662
|
+
--notif-to=active --notif-template=@file:mytemplate.erb
|
|
2307
2663
|
```
|
|
2308
2664
|
|
|
2309
|
-
|
|
2665
|
+
Options:
|
|
2666
|
+
|
|
2667
|
+
* `once_only` keep track of last date it was called, so next call will get only new events
|
|
2668
|
+
* `query` filter (on API call)
|
|
2669
|
+
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
|
2670
|
+
|
|
2671
|
+
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
|
|
2310
2672
|
|
|
2311
|
-
|
|
2673
|
+
### Transfer: Using specific transfer ports
|
|
2312
2674
|
|
|
2313
|
-
|
|
2675
|
+
By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
|
|
2676
|
+
The option `default_ports` ([yes]/no) allows ascli to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
|
|
2677
|
+
|
|
2678
|
+
### Using ATS
|
|
2679
|
+
|
|
2680
|
+
Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
|
|
2681
|
+
|
|
2682
|
+
### Example: Bulk creation of users
|
|
2314
2683
|
|
|
2315
2684
|
```
|
|
2316
|
-
|
|
2685
|
+
ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
|
2317
2686
|
:.......:.........:
|
|
2318
2687
|
: id : status :
|
|
2319
2688
|
:.......:.........:
|
|
@@ -2322,20 +2691,20 @@ $ ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@exampl
|
|
|
2322
2691
|
:.......:.........:
|
|
2323
2692
|
```
|
|
2324
2693
|
|
|
2325
|
-
|
|
2694
|
+
### Example: Find with filter and delete
|
|
2326
2695
|
|
|
2327
2696
|
```
|
|
2328
|
-
|
|
2697
|
+
ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
|
2329
2698
|
:.......:........................:
|
|
2330
2699
|
: id : email :
|
|
2331
2700
|
:.......:........................:
|
|
2332
2701
|
: 98398 : dummyuser1@example.com :
|
|
2333
2702
|
: 98399 : dummyuser2@example.com :
|
|
2334
2703
|
:.......:........................:
|
|
2335
|
-
|
|
2336
|
-
|
|
2337
|
-
|
|
2338
|
-
|
|
2704
|
+
thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
|
|
2705
|
+
echo $thelist
|
|
2706
|
+
["113501","354061"]
|
|
2707
|
+
ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
2339
2708
|
:.......:.........:
|
|
2340
2709
|
: id : status :
|
|
2341
2710
|
:.......:.........:
|
|
@@ -2344,10 +2713,18 @@ $ ascli aoc admin res user --bulk=yes --id=@json:[$thelist] delete
|
|
|
2344
2713
|
:.......:.........:
|
|
2345
2714
|
```
|
|
2346
2715
|
|
|
2347
|
-
|
|
2716
|
+
### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
|
2717
|
+
|
|
2718
|
+
```
|
|
2719
|
+
ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
|
|
2720
|
+
```
|
|
2721
|
+
|
|
2722
|
+
To delete them use the same method as before
|
|
2723
|
+
|
|
2724
|
+
### Example: Display current user's workspaces
|
|
2348
2725
|
|
|
2349
2726
|
```
|
|
2350
|
-
|
|
2727
|
+
ascli aoc user workspaces
|
|
2351
2728
|
:......:............................:
|
|
2352
2729
|
: id : name :
|
|
2353
2730
|
:......:............................:
|
|
@@ -2357,42 +2734,45 @@ $ ascli aoc user workspaces
|
|
|
2357
2734
|
:......:............................:
|
|
2358
2735
|
```
|
|
2359
2736
|
|
|
2360
|
-
|
|
2737
|
+
### Example: Create a sub access key in a "node"
|
|
2361
2738
|
|
|
2362
2739
|
Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
|
|
2363
2740
|
|
|
2364
2741
|
```
|
|
2365
|
-
|
|
2742
|
+
ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
|
|
2366
2743
|
```
|
|
2367
2744
|
|
|
2368
|
-
|
|
2745
|
+
### Example: Display transfer events (ops/transfer)
|
|
2369
2746
|
|
|
2370
2747
|
```
|
|
2371
|
-
|
|
2748
|
+
ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
|
|
2372
2749
|
```
|
|
2373
2750
|
|
|
2374
|
-
|
|
2375
|
-
|
|
2376
|
-
|
|
2377
|
-
|
|
2751
|
+
Examples of query (TODO: cleanup):
|
|
2752
|
+
|
|
2753
|
+
```
|
|
2754
|
+
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
|
2755
|
+
|
|
2756
|
+
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
|
2757
|
+
|
|
2378
2758
|
# filter= 'id', 'short_summary', or 'summary'
|
|
2379
2759
|
# count=nnn
|
|
2380
2760
|
# tag=x.y.z%3Dvalue
|
|
2381
2761
|
# iteration_token=nnn
|
|
2382
2762
|
# after_time=2016-05-01T23:53:09Z
|
|
2383
2763
|
# active_only=true|false
|
|
2764
|
+
```
|
|
2384
2765
|
|
|
2385
|
-
|
|
2386
|
-
* Display node events (events)
|
|
2766
|
+
### Example: Display node events (events)
|
|
2387
2767
|
|
|
2388
2768
|
```
|
|
2389
|
-
|
|
2769
|
+
ascli aoc admin res node --secret=_secret_ v3 events
|
|
2390
2770
|
```
|
|
2391
2771
|
|
|
2392
|
-
|
|
2772
|
+
### Example: Display members of a workspace
|
|
2393
2773
|
|
|
2394
2774
|
```
|
|
2395
|
-
|
|
2775
|
+
ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
|
2396
2776
|
:.............:.........:..................................:
|
|
2397
2777
|
: member_type : manager : member.email :
|
|
2398
2778
|
:.............:.........:..................................:
|
|
@@ -2411,29 +2791,29 @@ other query parameters:
|
|
|
2411
2791
|
{"workspace_membership_through":true,"include_indirect":true}
|
|
2412
2792
|
```
|
|
2413
2793
|
|
|
2414
|
-
|
|
2794
|
+
### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
|
|
2415
2795
|
|
|
2416
|
-
a-
|
|
2796
|
+
a- Get id of first workspace
|
|
2417
2797
|
|
|
2418
2798
|
```
|
|
2419
2799
|
WS1='First Workspace'
|
|
2420
2800
|
WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
|
|
2421
2801
|
```
|
|
2422
2802
|
|
|
2423
|
-
b-
|
|
2803
|
+
b- Get id of second workspace
|
|
2424
2804
|
|
|
2425
2805
|
```
|
|
2426
2806
|
WS2='Second Workspace'
|
|
2427
2807
|
WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
|
|
2428
2808
|
```
|
|
2429
2809
|
|
|
2430
|
-
c-
|
|
2810
|
+
c- Extract membership information
|
|
2431
2811
|
|
|
2432
2812
|
```
|
|
2433
|
-
|
|
2813
|
+
ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
|
|
2434
2814
|
```
|
|
2435
2815
|
|
|
2436
|
-
d-
|
|
2816
|
+
d- Convert to creation data for second workspace:
|
|
2437
2817
|
|
|
2438
2818
|
```
|
|
2439
2819
|
grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
|
|
@@ -2445,16 +2825,16 @@ or, using jq:
|
|
|
2445
2825
|
jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
|
|
2446
2826
|
```
|
|
2447
2827
|
|
|
2448
|
-
e-
|
|
2828
|
+
e- Add members to second workspace
|
|
2449
2829
|
|
|
2450
2830
|
```
|
|
2451
|
-
|
|
2831
|
+
ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
|
|
2452
2832
|
```
|
|
2453
2833
|
|
|
2454
|
-
|
|
2834
|
+
### Example: Get users who did not log since a date
|
|
2455
2835
|
|
|
2456
2836
|
```
|
|
2457
|
-
|
|
2837
|
+
ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
|
2458
2838
|
:...............................:
|
|
2459
2839
|
: email :
|
|
2460
2840
|
:...............................:
|
|
@@ -2463,20 +2843,20 @@ $ ascli aoc admin res user list --fields=email --query=@json:'{"per_page":10000,
|
|
|
2463
2843
|
:...............................:
|
|
2464
2844
|
```
|
|
2465
2845
|
|
|
2466
|
-
|
|
2846
|
+
### Example: List "Limited" users
|
|
2467
2847
|
|
|
2468
2848
|
```
|
|
2469
|
-
|
|
2849
|
+
ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
|
2470
2850
|
```
|
|
2471
2851
|
|
|
2472
|
-
|
|
2852
|
+
### Example: Perform a multi Gbps transfer between two remote shared folders
|
|
2473
2853
|
|
|
2474
|
-
In this example, a user has access to a workspace where two shared folders are located on
|
|
2854
|
+
In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
|
|
2475
2855
|
|
|
2476
2856
|
First, setup the environment (skip if already done)
|
|
2477
2857
|
|
|
2478
2858
|
```
|
|
2479
|
-
|
|
2859
|
+
ascli conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
|
|
2480
2860
|
Detected: Aspera on Cloud
|
|
2481
2861
|
Preparing preset: aoc_sedemo
|
|
2482
2862
|
Using existing key:
|
|
@@ -2495,7 +2875,7 @@ Setting config preset as default for aspera
|
|
|
2495
2875
|
saving config file
|
|
2496
2876
|
Done.
|
|
2497
2877
|
You can test with:
|
|
2498
|
-
|
|
2878
|
+
ascli aoc user info show
|
|
2499
2879
|
```
|
|
2500
2880
|
|
|
2501
2881
|
This creates the option preset "aoc_<org name>" to allow seamless command line access and sets it as default for aspera on cloud.
|
|
@@ -2505,19 +2885,20 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
|
2505
2885
|
Then, transfer between those:
|
|
2506
2886
|
|
|
2507
2887
|
```
|
|
2508
|
-
|
|
2888
|
+
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
|
2509
2889
|
```
|
|
2510
2890
|
|
|
2511
|
-
|
|
2891
|
+
### Example: create registration key to register a node
|
|
2892
|
+
|
|
2512
2893
|
```
|
|
2513
|
-
|
|
2894
|
+
ascli aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
|
2514
2895
|
jfqslfdjlfdjfhdjklqfhdkl
|
|
2515
2896
|
```
|
|
2516
2897
|
|
|
2517
|
-
|
|
2898
|
+
### Example: delete all registration keys
|
|
2518
2899
|
|
|
2519
2900
|
```
|
|
2520
|
-
|
|
2901
|
+
ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res client delete --bulk=yes --id=@lines:@stdin:
|
|
2521
2902
|
+-----+---------+
|
|
2522
2903
|
| id | status |
|
|
2523
2904
|
+-----+---------+
|
|
@@ -2528,143 +2909,215 @@ $ ascli aoc admin res admin/client list --fields=id --format=csv|ascli aoc admin
|
|
|
2528
2909
|
+-----+---------+
|
|
2529
2910
|
```
|
|
2530
2911
|
|
|
2531
|
-
|
|
2912
|
+
### Example: Create a node
|
|
2532
2913
|
|
|
2533
|
-
|
|
2914
|
+
AoC nodes as actually composed with two related entities:
|
|
2915
|
+
|
|
2916
|
+
* An access key created on the Transfer Server (HSTS/ATS)
|
|
2917
|
+
* a `node` resource in the AoC application.
|
|
2918
|
+
|
|
2919
|
+
The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
|
|
2920
|
+
Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
|
|
2921
|
+
|
|
2922
|
+
So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
|
|
2923
|
+
|
|
2924
|
+
* create the access key on ATS
|
|
2534
2925
|
|
|
2535
2926
|
```
|
|
2536
|
-
|
|
2927
|
+
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
|
2537
2928
|
```
|
|
2538
2929
|
|
|
2539
|
-
|
|
2930
|
+
Take a note of the randomly generated `id` and `secret`.
|
|
2931
|
+
|
|
2932
|
+
* Retrieve the ATS node address
|
|
2540
2933
|
|
|
2541
2934
|
```
|
|
2542
|
-
|
|
2935
|
+
ascli aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv|cut -f2 -d,
|
|
2543
2936
|
```
|
|
2544
2937
|
|
|
2545
|
-
*
|
|
2938
|
+
* Create the node entity
|
|
2546
2939
|
|
|
2547
2940
|
```
|
|
2548
|
-
|
|
2941
|
+
ascli aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
|
2942
|
+
```
|
|
2943
|
+
|
|
2944
|
+
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
|
2945
|
+
|
|
2946
|
+
### Example: List packages in a given shared inbox
|
|
2947
|
+
|
|
2948
|
+
First retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
|
2949
|
+
(To find out available filters, consult the API definition, or use the web interface in developer mode).
|
|
2950
|
+
|
|
2951
|
+
Note that when no query is provided, the query used by default is: `{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}`. The workspace id is added if not already present in the query.
|
|
2952
|
+
|
|
2953
|
+
```
|
|
2954
|
+
shbxid=$(ascli aoc user shared_inboxes --select=@json:'{"dropbox.name":"My Shared Inbox"}' --format=csv --fields=dropbox_id --display=data)
|
|
2955
|
+
|
|
2956
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
|
2549
2957
|
```
|
|
2550
2958
|
|
|
2551
|
-
##
|
|
2959
|
+
## Packages
|
|
2960
|
+
|
|
2961
|
+
The webmail-like application.
|
|
2962
|
+
|
|
2963
|
+
### Send a Package
|
|
2552
2964
|
|
|
2553
2965
|
Send a package:
|
|
2554
2966
|
|
|
2555
2967
|
```
|
|
2556
|
-
|
|
2968
|
+
ascli aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
|
|
2557
2969
|
```
|
|
2558
2970
|
|
|
2559
2971
|
Notes:
|
|
2560
2972
|
|
|
2561
|
-
*
|
|
2562
|
-
*
|
|
2563
|
-
*
|
|
2564
|
-
*
|
|
2565
|
-
*
|
|
2973
|
+
* The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
|
|
2974
|
+
* List allowed shared inbox destinations with: `ascli aoc user shared_inboxes`
|
|
2975
|
+
* Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
|
|
2976
|
+
* Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
|
|
2977
|
+
* or just names: `"recipients":[{"The Dest"}]` . ascli will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
|
|
2978
|
+
* If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
|
|
2979
|
+
* if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
|
|
2566
2980
|
* if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
|
|
2567
2981
|
|
|
2568
2982
|
Examples:
|
|
2569
2983
|
|
|
2984
|
+
* Send a package with one file to two users, using their email
|
|
2985
|
+
|
|
2986
|
+
```
|
|
2987
|
+
ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
|
2988
|
+
```
|
|
2989
|
+
|
|
2990
|
+
* Send a package with one file to a shared inbox, using internal identifier, with specific transfer parameters
|
|
2991
|
+
|
|
2992
|
+
```
|
|
2993
|
+
ascli aoc package send --value=@json:'{"name":"my delivery","recipients":[{"type":"dropbox","id":"12345"}]}' --ts=@json:'{"target_rate_kbps":100000}' my_file.dat
|
|
2994
|
+
```
|
|
2995
|
+
|
|
2996
|
+
* Send a package with one file to a shared inbox (by name) with metadata
|
|
2997
|
+
|
|
2570
2998
|
```
|
|
2571
|
-
|
|
2572
|
-
$ ascli aoc package send --value=@json:'{"name":"my file in shared inbox","recipients":["The Shared Inbox"]}' my_file.dat --ts=@json:'{"target_rate_kbps":100000}'
|
|
2573
|
-
$ ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
|
2999
|
+
ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox Name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
|
2574
3000
|
```
|
|
2575
3001
|
|
|
2576
|
-
|
|
3002
|
+
### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
|
2577
3003
|
|
|
2578
3004
|
It is possible to automatically download new packages, like using Aspera Cargo:
|
|
2579
3005
|
|
|
2580
3006
|
```
|
|
2581
|
-
|
|
3007
|
+
ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
2582
3008
|
```
|
|
2583
3009
|
|
|
2584
3010
|
* `--id=ALL` (case sensitive) will download all packages
|
|
2585
|
-
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
|
2586
|
-
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid
|
|
3011
|
+
* `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
|
|
3012
|
+
* `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
|
|
2587
3013
|
|
|
2588
|
-
Typically, one would
|
|
3014
|
+
Typically, one would execute this command on a regular basis, using the method of your choice:
|
|
2589
3015
|
|
|
2590
3016
|
* Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
|
|
2591
3017
|
* Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
|
|
2592
3018
|
* etc...
|
|
2593
3019
|
|
|
2594
|
-
##
|
|
2595
|
-
|
|
2596
|
-
Download of files is straightforward with a specific syntax for the `aspera files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
|
2597
|
-
|
|
2598
|
-
* if only one source is provided, it is downloaded
|
|
2599
|
-
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
|
2600
|
-
|
|
2601
|
-
## Find Files
|
|
3020
|
+
## Files
|
|
2602
3021
|
|
|
2603
|
-
|
|
3022
|
+
Folder sharing app.
|
|
2604
3023
|
|
|
2605
|
-
|
|
2606
|
-
|
|
2607
|
-
* empty (default) : all files, equivalent to: `exec:true`
|
|
2608
|
-
* not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
|
|
3024
|
+
### Download Files
|
|
2609
3025
|
|
|
2610
|
-
|
|
3026
|
+
Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
|
|
2611
3027
|
|
|
2612
|
-
*
|
|
3028
|
+
* if only one source is provided, it is downloaded
|
|
3029
|
+
* if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
|
|
2613
3030
|
|
|
2614
|
-
|
|
3031
|
+
### Shared folders
|
|
2615
3032
|
|
|
2616
|
-
*
|
|
3033
|
+
* list shared folders in node
|
|
2617
3034
|
|
|
2618
3035
|
```
|
|
2619
|
-
|
|
3036
|
+
ascli aoc admin res node --id=8669 shared_folders
|
|
2620
3037
|
```
|
|
2621
3038
|
|
|
2622
|
-
*
|
|
3039
|
+
* list shared folders in workspace
|
|
2623
3040
|
|
|
2624
3041
|
```
|
|
2625
|
-
|
|
3042
|
+
ascli aoc admin res workspace --id=10818 shared_folders
|
|
2626
3043
|
```
|
|
2627
3044
|
|
|
2628
|
-
*
|
|
3045
|
+
* list members of shared folder
|
|
2629
3046
|
|
|
2630
3047
|
```
|
|
2631
|
-
|
|
3048
|
+
ascli aoc admin res node --id=8669 v4 perm 82 show
|
|
2632
3049
|
```
|
|
2633
3050
|
|
|
2634
|
-
|
|
3051
|
+
### Cross Organization transfers
|
|
3052
|
+
|
|
3053
|
+
It is possible to transfer files directly between organizations without having to first download locally and then upload...
|
|
3054
|
+
|
|
3055
|
+
Although optional, the creation of [option preset](#lprt) is recommended to avoid placing all parameters in the command line.
|
|
3056
|
+
|
|
3057
|
+
Procedure to send a file from org1 to org2:
|
|
3058
|
+
|
|
3059
|
+
* Get access to Organization 1 and create a [option preset](#lprt): e.g. `org1`, for instance, use the [Wizard](#aocwizard)
|
|
3060
|
+
* Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
|
|
3061
|
+
* Get access to Organization 2 and create a [option preset](#lprt): e.g. `org2`
|
|
3062
|
+
* Check that access works and locate the destination folder `mydestfolder`
|
|
3063
|
+
* execute the following:
|
|
2635
3064
|
|
|
2636
3065
|
```
|
|
2637
|
-
|
|
3066
|
+
ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
|
|
2638
3067
|
```
|
|
2639
3068
|
|
|
2640
|
-
|
|
3069
|
+
Explanation:
|
|
2641
3070
|
|
|
2642
|
-
|
|
3071
|
+
* `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
|
|
3072
|
+
* `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
|
|
3073
|
+
* `--format=json` format the output in JSON (instead of default text table)
|
|
3074
|
+
* `--display=data` display only the result, and remove other information, such as workspace name
|
|
3075
|
+
* `|` the standard output of the first command is fed into the second one
|
|
3076
|
+
* `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
|
|
3077
|
+
* `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
|
|
3078
|
+
* `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
|
|
3079
|
+
* `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
|
|
3080
|
+
|
|
3081
|
+
### Find Files
|
|
3082
|
+
|
|
3083
|
+
The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
|
|
3084
|
+
|
|
3085
|
+
The expression can be of 3 formats:
|
|
3086
|
+
|
|
3087
|
+
* empty (default) : all files, equivalent to value: `exec:true`
|
|
3088
|
+
* not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
|
|
3089
|
+
|
|
3090
|
+
For instance, to find files with a special extension, use `--value='\.myext$'`
|
|
3091
|
+
|
|
3092
|
+
* starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
|
|
3093
|
+
|
|
3094
|
+
Examples of expressions: (using like this: `--value=exec:'<expression>'`)
|
|
3095
|
+
|
|
3096
|
+
* Find files more recent than 100 days
|
|
2643
3097
|
|
|
2644
3098
|
```
|
|
2645
|
-
|
|
3099
|
+
f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
|
2646
3100
|
```
|
|
2647
3101
|
|
|
2648
|
-
|
|
3102
|
+
* Find files older than 1 year on a given node and store in file list
|
|
2649
3103
|
|
|
2650
3104
|
```
|
|
2651
|
-
|
|
2652
|
-
--query=@json:'{"status":"completed","direction":"receive"}' \
|
|
2653
|
-
--notify=@json:'{"to":"<''%=transfer[:user_email.to_s]%>","subject":"<''%=transfer[:files_completed.to_s]%> files received","body":"Dear <''%=transfer[:user_email.to_s]%>\nWe received <''%=transfer[:files_completed.to_s]%> files for a total of <''%=transfer[:transferred_bytes.to_s]%> bytes, starting with file:\n<''%=transfer[:content.to_s]%>\n\nThank you."}'
|
|
3105
|
+
ascli aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
|
2654
3106
|
```
|
|
2655
3107
|
|
|
2656
|
-
*
|
|
2657
|
-
* `query` filter (on API call)
|
|
2658
|
-
* `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
|
|
2659
|
-
|
|
2660
|
-
Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
|
|
3108
|
+
* Delete the files, one by one
|
|
2661
3109
|
|
|
2662
|
-
|
|
3110
|
+
```
|
|
3111
|
+
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
|
|
3112
|
+
```
|
|
2663
3113
|
|
|
2664
|
-
|
|
3114
|
+
* Delete the files in bulk
|
|
2665
3115
|
|
|
3116
|
+
```
|
|
3117
|
+
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
|
|
3118
|
+
```
|
|
2666
3119
|
|
|
2667
|
-
# Plugin: Aspera Transfer Service
|
|
3120
|
+
# <a id="ats"></a>Plugin: Aspera Transfer Service
|
|
2668
3121
|
|
|
2669
3122
|
ATS is usable either :
|
|
2670
3123
|
|
|
@@ -2674,10 +3127,13 @@ ATS is usable either :
|
|
|
2674
3127
|
|
|
2675
3128
|
## IBM Cloud ATS : creation of api key
|
|
2676
3129
|
|
|
3130
|
+
This section is about using ATS with an IBM cloud subscription.
|
|
3131
|
+
If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
|
|
3132
|
+
|
|
2677
3133
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
|
2678
3134
|
|
|
2679
3135
|
```
|
|
2680
|
-
|
|
3136
|
+
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
|
2681
3137
|
OK
|
|
2682
3138
|
API key mykeyname was created
|
|
2683
3139
|
|
|
@@ -2696,48 +3152,47 @@ References:
|
|
|
2696
3152
|
* [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
|
2697
3153
|
* [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
|
2698
3154
|
|
|
2699
|
-
|
|
2700
3155
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
|
2701
3156
|
|
|
2702
3157
|
```
|
|
2703
|
-
|
|
2704
|
-
|
|
2705
|
-
|
|
3158
|
+
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
|
|
3159
|
+
ascli config preset set default ats my_ibm_ats
|
|
3160
|
+
ascli ats api_key instances
|
|
2706
3161
|
+--------------------------------------+
|
|
2707
3162
|
| instance |
|
|
2708
3163
|
+--------------------------------------+
|
|
2709
3164
|
| aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
|
|
2710
3165
|
+--------------------------------------+
|
|
2711
|
-
|
|
2712
|
-
|
|
3166
|
+
ascli config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
|
|
3167
|
+
ascli ats api_key create
|
|
2713
3168
|
+--------+----------------------------------------------+
|
|
2714
3169
|
| key | value |
|
|
2715
3170
|
+--------+----------------------------------------------+
|
|
2716
3171
|
| id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
|
|
2717
3172
|
| secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
|
|
2718
3173
|
+--------+----------------------------------------------+
|
|
2719
|
-
|
|
3174
|
+
ascli config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
|
2720
3175
|
```
|
|
2721
3176
|
|
|
2722
3177
|
## Examples
|
|
2723
3178
|
|
|
2724
|
-
Example: create access key on softlayer:
|
|
3179
|
+
Example: create access key on IBM Cloud (softlayer):
|
|
2725
3180
|
|
|
2726
3181
|
```
|
|
2727
|
-
|
|
3182
|
+
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
|
2728
3183
|
```
|
|
2729
3184
|
|
|
2730
3185
|
Example: create access key on AWS:
|
|
2731
3186
|
|
|
2732
3187
|
```
|
|
2733
|
-
|
|
3188
|
+
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
|
|
2734
3189
|
|
|
2735
3190
|
```
|
|
2736
3191
|
|
|
2737
3192
|
Example: create access key on Azure SAS:
|
|
2738
3193
|
|
|
2739
3194
|
```
|
|
2740
|
-
|
|
3195
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
|
2741
3196
|
|
|
2742
3197
|
```
|
|
2743
3198
|
|
|
@@ -2746,7 +3201,7 @@ $ ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id
|
|
|
2746
3201
|
Example: create access key on Azure:
|
|
2747
3202
|
|
|
2748
3203
|
```
|
|
2749
|
-
|
|
3204
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
|
2750
3205
|
|
|
2751
3206
|
```
|
|
2752
3207
|
|
|
@@ -2756,85 +3211,114 @@ delete all my access keys:
|
|
|
2756
3211
|
for k in $(ascli ats access_key list --field=id --format=csv);do ascli ats access_key id $k delete;done
|
|
2757
3212
|
```
|
|
2758
3213
|
|
|
3214
|
+
The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
|
|
3215
|
+
|
|
2759
3216
|
# Plugin: IBM Aspera High Speed Transfer Server (transfer)
|
|
2760
3217
|
|
|
2761
|
-
This plugin
|
|
3218
|
+
This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
|
|
3219
|
+
It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
|
|
3220
|
+
Modern mode is to use the node API and transfer tokens.
|
|
2762
3221
|
|
|
2763
3222
|
## Authentication
|
|
2764
3223
|
|
|
2765
3224
|
Both password and SSH keys auth are supported.
|
|
2766
3225
|
|
|
2767
|
-
|
|
3226
|
+
If username is not provided, the default transfer user `xfer` is used.
|
|
3227
|
+
|
|
3228
|
+
If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
|
|
3229
|
+
|
|
3230
|
+
```
|
|
3231
|
+
ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
|
|
3232
|
+
```
|
|
3233
|
+
|
|
3234
|
+
Multiple SSH key paths can be provided.
|
|
3235
|
+
The value of the parameter `ssh_keys` can be a single value or an array.
|
|
3236
|
+
Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
|
|
2768
3237
|
|
|
2769
3238
|
Examples:
|
|
2770
3239
|
|
|
2771
3240
|
```
|
|
2772
|
-
|
|
2773
|
-
|
|
2774
|
-
|
|
3241
|
+
ascli server --ssh-keys=~/.ssh/id_rsa
|
|
3242
|
+
ascli server --ssh-keys=@list:,~/.ssh/id_rsa
|
|
3243
|
+
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
|
2775
3244
|
```
|
|
2776
3245
|
|
|
2777
|
-
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
|
3246
|
+
The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
|
|
3247
|
+
By default the ssh library expect that an ssh-agent is running.
|
|
2778
3248
|
|
|
2779
|
-
|
|
3249
|
+
On Linux, if you get an error message such as:
|
|
2780
3250
|
|
|
2781
3251
|
```
|
|
2782
|
-
[Linux]
|
|
2783
3252
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
|
|
2784
3253
|
```
|
|
2785
3254
|
|
|
2786
|
-
or
|
|
3255
|
+
or on Windows:
|
|
2787
3256
|
|
|
2788
3257
|
```
|
|
2789
|
-
[Windows]
|
|
2790
3258
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
|
2791
3259
|
```
|
|
2792
3260
|
|
|
2793
|
-
This means that you
|
|
3261
|
+
This means that you don't have such an ssh agent running, then:
|
|
2794
3262
|
|
|
2795
3263
|
* check env var: `SSH_AGENT_SOCK`
|
|
2796
|
-
* check if the key is protected with a passphrase
|
|
3264
|
+
* check if the ssh key is protected with a passphrase
|
|
2797
3265
|
* [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
|
|
2798
|
-
* To
|
|
3266
|
+
* To disable use of `ssh-agent`, use the option `ssh_option` like this:
|
|
2799
3267
|
|
|
2800
3268
|
```
|
|
2801
|
-
|
|
3269
|
+
ascli server --ssh-options=@ruby:'{use_agent: false}' ...
|
|
2802
3270
|
```
|
|
2803
3271
|
|
|
2804
|
-
This can also be set as default using a preset
|
|
3272
|
+
This can also be set as default using a global preset.
|
|
2805
3273
|
|
|
2806
3274
|
## Example
|
|
2807
3275
|
|
|
2808
|
-
One can test the
|
|
3276
|
+
One can test the `server` application using the well known demo server:
|
|
2809
3277
|
|
|
2810
3278
|
```
|
|
2811
|
-
|
|
2812
|
-
|
|
2813
|
-
|
|
2814
|
-
$ ascli server download /aspera-test-dir-large/200MB
|
|
3279
|
+
ascli config initdemo
|
|
3280
|
+
ascli server browse /aspera-test-dir-large
|
|
3281
|
+
ascli server download /aspera-test-dir-large/200MB
|
|
2815
3282
|
```
|
|
2816
3283
|
|
|
2817
|
-
|
|
2818
|
-
|
|
3284
|
+
`initdemo` creates a [option preset](#lprt) `demoserver` and set it as default for plugin `server`.
|
|
2819
3285
|
|
|
2820
3286
|
# Plugin: IBM Aspera High Speed Transfer Server (node)
|
|
2821
3287
|
|
|
2822
3288
|
This plugin gives access to capabilities provided by HSTS node API.
|
|
2823
3289
|
|
|
2824
|
-
##
|
|
3290
|
+
## File Operations
|
|
2825
3291
|
|
|
2826
3292
|
It is possible to:
|
|
2827
3293
|
* browse
|
|
2828
3294
|
* transfer (upload / download)
|
|
2829
3295
|
* ...
|
|
2830
3296
|
|
|
3297
|
+
For transfers, it is possible to control how transfer is authorized using option: `token_type`:
|
|
3298
|
+
|
|
3299
|
+
* `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
|
|
3300
|
+
* `basic` : transfer spec is created like this:
|
|
3301
|
+
|
|
3302
|
+
```
|
|
3303
|
+
{
|
|
3304
|
+
"remote_host": address of node url,
|
|
3305
|
+
"remote_user": "xfer",
|
|
3306
|
+
"ssh_port": 33001,
|
|
3307
|
+
"token": "Basic <base 64 encoded user/pass>",
|
|
3308
|
+
"direction": send/recv
|
|
3309
|
+
}
|
|
3310
|
+
```
|
|
3311
|
+
|
|
3312
|
+
* `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
|
3313
|
+
|
|
2831
3314
|
## Central
|
|
2832
3315
|
|
|
2833
|
-
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and
|
|
3316
|
+
The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
|
|
2834
3317
|
|
|
2835
3318
|
Filtering can be applied:
|
|
3319
|
+
|
|
2836
3320
|
```
|
|
2837
|
-
|
|
3321
|
+
ascli node central file list
|
|
2838
3322
|
```
|
|
2839
3323
|
|
|
2840
3324
|
by providing the `validator` option, offline transfer validation can be done.
|
|
@@ -2846,7 +3330,7 @@ It is possible to start a FASPStream session using the node API:
|
|
|
2846
3330
|
Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
|
|
2847
3331
|
|
|
2848
3332
|
```
|
|
2849
|
-
|
|
3333
|
+
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
|
|
2850
3334
|
```
|
|
2851
3335
|
|
|
2852
3336
|
## Watchfolder
|
|
@@ -2858,11 +3342,10 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
|
2858
3342
|
* Start watchd and watchfolderd services running as a system user having access to files
|
|
2859
3343
|
* configure a watchfolder to define automated transfers
|
|
2860
3344
|
|
|
2861
|
-
|
|
2862
3345
|
```
|
|
2863
|
-
|
|
2864
|
-
|
|
2865
|
-
|
|
3346
|
+
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
|
3347
|
+
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
|
3348
|
+
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
|
2866
3349
|
```
|
|
2867
3350
|
|
|
2868
3351
|
## Out of Transfer File Validation
|
|
@@ -2870,13 +3353,13 @@ $ ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1"
|
|
|
2870
3353
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
|
2871
3354
|
|
|
2872
3355
|
```
|
|
2873
|
-
|
|
3356
|
+
ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
|
2874
3357
|
:..............:..............:............:......................................:
|
|
2875
3358
|
: session_uuid : file_id : status : path :
|
|
2876
3359
|
:..............:..............:............:......................................:
|
|
2877
3360
|
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
|
|
2878
3361
|
:..............:..............:............:......................................:
|
|
2879
|
-
|
|
3362
|
+
ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
|
2880
3363
|
updated
|
|
2881
3364
|
```
|
|
2882
3365
|
|
|
@@ -2892,7 +3375,7 @@ Create another configuration for the Azure ATS instance: in section "node", name
|
|
|
2892
3375
|
Then execute the following command:
|
|
2893
3376
|
|
|
2894
3377
|
```
|
|
2895
|
-
|
|
3378
|
+
ascli node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
|
|
2896
3379
|
```
|
|
2897
3380
|
|
|
2898
3381
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance
|
|
@@ -2901,18 +3384,22 @@ to download files.
|
|
|
2901
3384
|
## Create access key
|
|
2902
3385
|
|
|
2903
3386
|
```
|
|
2904
|
-
|
|
3387
|
+
ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
|
|
2905
3388
|
```
|
|
2906
3389
|
|
|
2907
3390
|
# Plugin: IBM Aspera Faspex5
|
|
2908
3391
|
|
|
2909
3392
|
3 authentication methods are supported:
|
|
2910
3393
|
|
|
2911
|
-
* boot
|
|
2912
|
-
* web
|
|
2913
3394
|
* jwt
|
|
3395
|
+
* web
|
|
3396
|
+
* boot
|
|
3397
|
+
|
|
3398
|
+
For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
|
|
3399
|
+
|
|
3400
|
+
For web method, create an API client in Faspex, and use: --auth=web
|
|
2914
3401
|
|
|
2915
|
-
For boot method:
|
|
3402
|
+
For boot method: (will be removed in future)
|
|
2916
3403
|
|
|
2917
3404
|
* open a browser
|
|
2918
3405
|
* start developer mode
|
|
@@ -2922,45 +3409,77 @@ For boot method:
|
|
|
2922
3409
|
Use it as password and use `--auth=boot`.
|
|
2923
3410
|
|
|
2924
3411
|
```
|
|
2925
|
-
|
|
3412
|
+
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
|
|
2926
3413
|
```
|
|
2927
3414
|
|
|
2928
|
-
For web method, create an API client in Faspex, and use: --auth=web
|
|
2929
|
-
|
|
2930
|
-
For JWT, create an API client in Faspex with jwt supporot, and use: --auth=jwt
|
|
2931
|
-
as of beta£3 this does not allow regular users.
|
|
2932
|
-
|
|
2933
3415
|
Ready to use Faspex5 with CLI.
|
|
2934
3416
|
|
|
2935
|
-
Once the graphical registration form exist, ther bootstrap method can be removed.
|
|
2936
|
-
|
|
2937
3417
|
# Plugin: IBM Aspera Faspex (4.x)
|
|
2938
3418
|
|
|
2939
3419
|
Notes:
|
|
2940
3420
|
|
|
2941
|
-
*
|
|
2942
|
-
*
|
|
3421
|
+
* The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
|
3422
|
+
* For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
|
3423
|
+
|
|
3424
|
+
## Listing Packages
|
|
3425
|
+
|
|
3426
|
+
Command: `faspex package list`
|
|
3427
|
+
|
|
3428
|
+
### Option `box`
|
|
3429
|
+
|
|
3430
|
+
By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
|
|
3431
|
+
|
|
3432
|
+
### Option `recipient`
|
|
3433
|
+
|
|
3434
|
+
A user can receive a package because the recipient is:
|
|
3435
|
+
|
|
3436
|
+
* the user himself (default)
|
|
3437
|
+
* the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
|
|
3438
|
+
|
|
3439
|
+
### Option `query`
|
|
3440
|
+
|
|
3441
|
+
As inboxes may be large, it is possible to use the following query parameters:
|
|
3442
|
+
|
|
3443
|
+
* `count` : (native) number items in one API call (default=0, equivalent to 10)
|
|
3444
|
+
* `page` : (native) id of page in call (default=0)
|
|
3445
|
+
* `startIndex` : (native) index of item to start, default=0, oldest index=0
|
|
3446
|
+
* `max` : maximum number of items
|
|
3447
|
+
* `pmax` : maximum number of pages
|
|
3448
|
+
|
|
3449
|
+
(SQL query is `LIMIT <startIndex>, <count>`)
|
|
3450
|
+
|
|
3451
|
+
The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
|
|
3452
|
+
|
|
3453
|
+
If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
|
|
3454
|
+
|
|
3455
|
+
### Example
|
|
3456
|
+
|
|
3457
|
+
```
|
|
3458
|
+
ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
|
3459
|
+
```
|
|
3460
|
+
|
|
3461
|
+
List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
|
|
2943
3462
|
|
|
2944
3463
|
## Receiving a Package
|
|
2945
3464
|
|
|
2946
|
-
The command is `package recv`, possible
|
|
3465
|
+
The command is `package recv`, possible methods are:
|
|
2947
3466
|
|
|
2948
3467
|
* provide a package id with option `id`
|
|
2949
3468
|
* provide a public link with option `link`
|
|
2950
3469
|
* provide a `faspe:` URI with option `link`
|
|
2951
3470
|
|
|
2952
3471
|
```
|
|
2953
|
-
|
|
2954
|
-
|
|
3472
|
+
ascli faspex package recv --id=12345
|
|
3473
|
+
ascli faspex package recv --link=faspe://...
|
|
2955
3474
|
```
|
|
2956
3475
|
|
|
2957
3476
|
If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
|
|
2958
3477
|
|
|
2959
3478
|
```
|
|
2960
|
-
|
|
3479
|
+
ascli faspex package list --recipient='*thedropboxname'
|
|
2961
3480
|
```
|
|
2962
3481
|
|
|
2963
|
-
|
|
3482
|
+
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
|
2964
3483
|
|
|
2965
3484
|
## Sending a Package
|
|
2966
3485
|
|
|
@@ -2969,7 +3488,7 @@ The command is `faspex package send`. Package information (title, note, metadata
|
|
|
2969
3488
|
Example:
|
|
2970
3489
|
|
|
2971
3490
|
```
|
|
2972
|
-
|
|
3491
|
+
ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
|
2973
3492
|
```
|
|
2974
3493
|
|
|
2975
3494
|
If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
|
|
@@ -2979,17 +3498,29 @@ Additional optional parameters in `delivery_info`:
|
|
|
2979
3498
|
* Package Note: : `"note":"note this and that"`
|
|
2980
3499
|
* Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
|
|
2981
3500
|
|
|
2982
|
-
##
|
|
3501
|
+
## Email notification on transfer
|
|
3502
|
+
|
|
3503
|
+
Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
|
|
3504
|
+
|
|
3505
|
+
Example:
|
|
3506
|
+
|
|
3507
|
+
```
|
|
3508
|
+
ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
|
3509
|
+
```
|
|
3510
|
+
|
|
3511
|
+
In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
|
|
3512
|
+
|
|
3513
|
+
## Operation on dropboxes
|
|
2983
3514
|
|
|
2984
3515
|
Example:
|
|
2985
3516
|
|
|
2986
3517
|
```
|
|
2987
|
-
|
|
2988
|
-
|
|
2989
|
-
|
|
3518
|
+
ascli faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
|
3519
|
+
ascli faspex v4 dropbox list
|
|
3520
|
+
ascli faspex v4 dropbox delete --id=36
|
|
2990
3521
|
```
|
|
2991
3522
|
|
|
2992
|
-
##
|
|
3523
|
+
## Remote sources
|
|
2993
3524
|
|
|
2994
3525
|
Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
|
|
2995
3526
|
the node API is used, for this it is required to add a section ":storage" that links
|
|
@@ -3024,7 +3555,7 @@ It is possible to tell `ascli` to download newly received packages, much like th
|
|
|
3024
3555
|
cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
|
|
3025
3556
|
|
|
3026
3557
|
```
|
|
3027
|
-
|
|
3558
|
+
ascli faspex packages recv --id=ALL --once-only=yes --lock-port=12345
|
|
3028
3559
|
```
|
|
3029
3560
|
|
|
3030
3561
|
# Plugin: IBM Aspera Shares
|
|
@@ -3034,42 +3565,51 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
|
3034
3565
|
In Shares2, users, groups listing are paged, to display sequential pages:
|
|
3035
3566
|
|
|
3036
3567
|
```
|
|
3037
|
-
|
|
3568
|
+
for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}';done
|
|
3038
3569
|
```
|
|
3039
3570
|
|
|
3040
3571
|
# Plugin: IBM Cloud Object Storage
|
|
3041
3572
|
|
|
3042
3573
|
The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
|
|
3043
|
-
It uses the same transfer service as Aspera on Cloud.
|
|
3044
|
-
|
|
3574
|
+
It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
|
|
3575
|
+
Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
|
|
3045
3576
|
|
|
3046
|
-
|
|
3577
|
+
There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
|
|
3578
|
+
|
|
3579
|
+
## Using endpoint, apikey and Resource Instance ID (CRN)
|
|
3580
|
+
|
|
3581
|
+
If you have those parameters already, then following options shall be provided:
|
|
3047
3582
|
|
|
3048
3583
|
* `bucket` bucket name
|
|
3049
3584
|
* `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
|
|
3050
3585
|
* `apikey` API Key
|
|
3051
3586
|
* `crn` resource instance id
|
|
3052
3587
|
|
|
3053
|
-
|
|
3588
|
+
For example, let us create a default configuration:
|
|
3054
3589
|
|
|
3055
|
-
|
|
3056
|
-
|
|
3057
|
-
|
|
3590
|
+
```
|
|
3591
|
+
ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
|
|
3592
|
+
ascli conf id default set cos mycos
|
|
3593
|
+
```
|
|
3594
|
+
|
|
3595
|
+
Then, jump to the transfer example.
|
|
3058
3596
|
|
|
3059
|
-
|
|
3597
|
+
## Using service credential file
|
|
3598
|
+
|
|
3599
|
+
If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
|
|
3060
3600
|
|
|
3061
3601
|
Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy
|
|
3062
3602
|
|
|
3063
3603
|
Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
|
|
3064
3604
|
|
|
3065
|
-
or using the CLI:
|
|
3605
|
+
or using the IBM Cloud CLI:
|
|
3066
3606
|
|
|
3067
3607
|
```
|
|
3068
|
-
|
|
3069
|
-
|
|
3608
|
+
ibmcloud resource service-keys
|
|
3609
|
+
ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
|
3070
3610
|
```
|
|
3071
3611
|
|
|
3072
|
-
(if you
|
|
3612
|
+
(if you don't have `jq` installed, extract the structure as follows)
|
|
3073
3613
|
|
|
3074
3614
|
It consists in the following structure:
|
|
3075
3615
|
|
|
@@ -3093,33 +3633,38 @@ The field `resource_instance_id` is for option `crn`
|
|
|
3093
3633
|
|
|
3094
3634
|
The field `apikey` is for option `apikey`
|
|
3095
3635
|
|
|
3096
|
-
|
|
3636
|
+
(If needed: endpoints for regions can be found by querying the `endpoints` URL.)
|
|
3097
3637
|
|
|
3098
|
-
|
|
3638
|
+
The required options for this method are:
|
|
3099
3639
|
|
|
3100
|
-
|
|
3101
|
-
|
|
3102
|
-
|
|
3103
|
-
```
|
|
3640
|
+
* `bucket` bucket name
|
|
3641
|
+
* `region` bucket region, e.g. eu-de
|
|
3642
|
+
* `service_credentials` see below
|
|
3104
3643
|
|
|
3105
|
-
|
|
3644
|
+
For example, let us create a default configuration:
|
|
3106
3645
|
|
|
3107
3646
|
```
|
|
3108
|
-
|
|
3109
|
-
|
|
3647
|
+
ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
|
|
3648
|
+
ascli conf id default set cos mycos
|
|
3110
3649
|
```
|
|
3111
3650
|
|
|
3112
|
-
|
|
3651
|
+
## Operations, transfers
|
|
3652
|
+
|
|
3653
|
+
Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
|
|
3654
|
+
|
|
3655
|
+
A subset of `node` plugin operations are supported, basically node API:
|
|
3113
3656
|
|
|
3114
3657
|
```
|
|
3115
|
-
|
|
3116
|
-
|
|
3658
|
+
ascli cos node info
|
|
3659
|
+
ascli cos node upload 'faux:///sample1G?1g'
|
|
3117
3660
|
```
|
|
3118
3661
|
|
|
3662
|
+
Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
|
|
3663
|
+
|
|
3119
3664
|
# Plugin: IBM Aspera Sync
|
|
3120
3665
|
|
|
3121
|
-
A basic plugin to start an "async" using `ascli`.
|
|
3122
|
-
to start from ma configuration file, using `ascli` standard options.
|
|
3666
|
+
A basic plugin to start an "async" using `ascli`.
|
|
3667
|
+
The main advantage is the possibility to start from ma configuration file, using `ascli` standard options.
|
|
3123
3668
|
|
|
3124
3669
|
# Plugin: Preview
|
|
3125
3670
|
|
|
@@ -3140,8 +3685,10 @@ Specify the previews folder as shown in:
|
|
|
3140
3685
|
By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
|
|
3141
3686
|
|
|
3142
3687
|
```
|
|
3143
|
-
|
|
3144
|
-
|
|
3688
|
+
PATH=/opt/aspera/bin:$PATH
|
|
3689
|
+
|
|
3690
|
+
asconfigurator -x "server;preview_dir,previews"
|
|
3691
|
+
asnodeadmin --reload
|
|
3145
3692
|
```
|
|
3146
3693
|
|
|
3147
3694
|
Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
|
|
@@ -3154,16 +3701,18 @@ This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
|
|
|
3154
3701
|
To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
|
|
3155
3702
|
|
|
3156
3703
|
```
|
|
3157
|
-
|
|
3704
|
+
asuserdata -a | grep max_request_file_create_size_kb
|
|
3705
|
+
|
|
3158
3706
|
max_request_file_create_size_kb: "1024"
|
|
3159
|
-
|
|
3707
|
+
|
|
3708
|
+
asconfigurator -x "server; max_request_file_create_size_kb,16384"
|
|
3160
3709
|
```
|
|
3161
3710
|
|
|
3162
3711
|
If you use a value different than 16777216, then specify it using option `max_size`.
|
|
3163
3712
|
|
|
3164
3713
|
Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
|
|
3165
3714
|
|
|
3166
|
-
## <a
|
|
3715
|
+
## <a id="prev_ext"></a>External tools: Linux
|
|
3167
3716
|
|
|
3168
3717
|
The tool requires the following external tools available in the `PATH`:
|
|
3169
3718
|
|
|
@@ -3180,7 +3729,7 @@ Other OSes should work as well, but are note tested.
|
|
|
3180
3729
|
To check if all tools are found properly, execute:
|
|
3181
3730
|
|
|
3182
3731
|
```
|
|
3183
|
-
|
|
3732
|
+
ascli preview check
|
|
3184
3733
|
```
|
|
3185
3734
|
|
|
3186
3735
|
### mimemagic
|
|
@@ -3188,22 +3737,22 @@ $ ascli preview check
|
|
|
3188
3737
|
To benefit from extra mime type detection install gem mimemagic:
|
|
3189
3738
|
|
|
3190
3739
|
```
|
|
3191
|
-
|
|
3740
|
+
gem install mimemagic
|
|
3192
3741
|
```
|
|
3193
3742
|
|
|
3194
3743
|
or to install an earlier version if any problem:
|
|
3195
3744
|
|
|
3196
3745
|
```
|
|
3197
|
-
|
|
3746
|
+
gem install mimemagic -v '~> 0.3.0'
|
|
3198
3747
|
```
|
|
3199
3748
|
|
|
3200
3749
|
To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
|
|
3201
3750
|
|
|
3202
3751
|
If not used, Mime type used for conversion is the one provided by the node API.
|
|
3203
3752
|
|
|
3204
|
-
If used, it the `preview` command will first
|
|
3753
|
+
If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
|
|
3205
3754
|
|
|
3206
|
-
### Image:
|
|
3755
|
+
### Image: ImageMagick and optipng
|
|
3207
3756
|
|
|
3208
3757
|
```
|
|
3209
3758
|
yum install -y ImageMagick optipng
|
|
@@ -3211,31 +3760,33 @@ yum install -y ImageMagick optipng
|
|
|
3211
3760
|
|
|
3212
3761
|
### Video: FFmpeg
|
|
3213
3762
|
|
|
3763
|
+
The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
|
|
3764
|
+
|
|
3214
3765
|
```
|
|
3215
|
-
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt &&
|
|
3766
|
+
curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
|
|
3216
3767
|
```
|
|
3217
3768
|
|
|
3218
3769
|
### Office: Unoconv and Libreoffice
|
|
3219
3770
|
|
|
3220
|
-
If you
|
|
3771
|
+
If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
|
|
3221
3772
|
|
|
3222
3773
|
The generation of preview in based on the use of `unoconv` and `libreoffice`
|
|
3223
3774
|
|
|
3224
|
-
*
|
|
3775
|
+
* CentOS 8
|
|
3225
3776
|
|
|
3226
3777
|
```
|
|
3227
|
-
|
|
3778
|
+
dnf install unoconv
|
|
3228
3779
|
```
|
|
3229
3780
|
|
|
3230
3781
|
* Amazon Linux
|
|
3231
3782
|
|
|
3232
3783
|
```
|
|
3233
|
-
|
|
3234
|
-
|
|
3235
|
-
|
|
3236
|
-
|
|
3237
|
-
|
|
3238
|
-
|
|
3784
|
+
amazon-linux-extras enable libreoffice
|
|
3785
|
+
yum clean metadata
|
|
3786
|
+
yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
|
|
3787
|
+
wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
|
3788
|
+
mv unoconv /usr/bin
|
|
3789
|
+
chmod a+x /usr/bin/unoconv
|
|
3239
3790
|
```
|
|
3240
3791
|
|
|
3241
3792
|
## Configuration
|
|
@@ -3247,9 +3798,11 @@ Like any `ascli` commands, parameters can be passed on command line or using a c
|
|
|
3247
3798
|
Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
|
|
3248
3799
|
|
|
3249
3800
|
```
|
|
3250
|
-
|
|
3251
|
-
|
|
3252
|
-
|
|
3801
|
+
su -s /bin/bash - xfer
|
|
3802
|
+
|
|
3803
|
+
ascli config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
|
3804
|
+
|
|
3805
|
+
ascli config preset set default preview previewconf
|
|
3253
3806
|
```
|
|
3254
3807
|
|
|
3255
3808
|
Here we assume that Office file generation is disabled, else remove this option.
|
|
@@ -3258,7 +3811,7 @@ Here we assume that Office file generation is disabled, else remove this option.
|
|
|
3258
3811
|
One can check if the access key is well configured using:
|
|
3259
3812
|
|
|
3260
3813
|
```
|
|
3261
|
-
|
|
3814
|
+
ascli -Ppreviewconf node browse /
|
|
3262
3815
|
```
|
|
3263
3816
|
|
|
3264
3817
|
This shall list the contents of the storage root of the access key.
|
|
@@ -3274,8 +3827,9 @@ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order
|
|
|
3274
3827
|
Lets do a one shot test, using the configuration previously created:
|
|
3275
3828
|
|
|
3276
3829
|
```
|
|
3277
|
-
|
|
3278
|
-
|
|
3830
|
+
su -s /bin/bash - xfer
|
|
3831
|
+
|
|
3832
|
+
ascli preview scan --overwrite=always
|
|
3279
3833
|
```
|
|
3280
3834
|
|
|
3281
3835
|
When the preview generator is first executed it will create a file: `.aspera_access_key`
|
|
@@ -3289,13 +3843,13 @@ Adapt the scripts to your own preference.
|
|
|
3289
3843
|
|
|
3290
3844
|
We assume here that a configuration preset was created as shown previously.
|
|
3291
3845
|
|
|
3292
|
-
Lets first setup a script that will be used in the
|
|
3846
|
+
Lets first setup a script that will be used in the scheduler and sets up the environment.
|
|
3293
3847
|
|
|
3294
3848
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
|
3295
3849
|
|
|
3296
3850
|
```
|
|
3297
|
-
#!/bin/bash
|
|
3298
|
-
# set a timeout protection, just in case
|
|
3851
|
+
#!/bin/bash
|
|
3852
|
+
# set a timeout protection, just in case
|
|
3299
3853
|
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
|
3300
3854
|
. /etc/profile.d/rvm.sh
|
|
3301
3855
|
rvm use 2.6 --quiet
|
|
@@ -3305,13 +3859,13 @@ exec timeout ${tmout} ascli "${@}"
|
|
|
3305
3859
|
Here the cronjob is created for user `xfer`.
|
|
3306
3860
|
|
|
3307
3861
|
```
|
|
3308
|
-
|
|
3862
|
+
crontab<<EOF
|
|
3309
3863
|
0 * * * * /home/xfer/cron_ascli preview scan --logger=syslog --display=error
|
|
3310
3864
|
2-59 * * * * /home/xfer/cron_ascli preview trev --logger=syslog --display=error
|
|
3311
3865
|
EOF
|
|
3312
3866
|
```
|
|
3313
3867
|
|
|
3314
|
-
Note that the
|
|
3868
|
+
Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
|
|
3315
3869
|
|
|
3316
3870
|
## Candidate detection for creation or update (or deletion)
|
|
3317
3871
|
|
|
@@ -3336,12 +3890,12 @@ Deletion of preview for deleted source files: not implemented yet (TODO).
|
|
|
3336
3890
|
If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
|
|
3337
3891
|
|
|
3338
3892
|
```
|
|
3339
|
-
|
|
3893
|
+
ascli preview scan --skip-folders=@json:'["/not_here"]'
|
|
3340
3894
|
```
|
|
3341
3895
|
|
|
3342
3896
|
The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
|
|
3343
3897
|
|
|
3344
|
-
When scanning the option `value` has the same
|
|
3898
|
+
When scanning the option `value` has the same behavior as for the `node find` command.
|
|
3345
3899
|
|
|
3346
3900
|
For instance to filter out files beginning with `._` do:
|
|
3347
3901
|
|
|
@@ -3360,7 +3914,7 @@ Use option `skip_format` to skip generation of a format.
|
|
|
3360
3914
|
|
|
3361
3915
|
## Supported input Files types
|
|
3362
3916
|
|
|
3363
|
-
The preview generator supports
|
|
3917
|
+
The preview generator supports rendering of those file categories:
|
|
3364
3918
|
|
|
3365
3919
|
* image
|
|
3366
3920
|
* pdf
|
|
@@ -3380,7 +3934,7 @@ The tool can also locally detect the mime type using gem `mimemagic`.
|
|
|
3380
3934
|
|
|
3381
3935
|
## Access to original files and preview creation
|
|
3382
3936
|
|
|
3383
|
-
Standard open source tools are used to create
|
|
3937
|
+
Standard open source tools are used to create thumbnails and video previews.
|
|
3384
3938
|
Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
|
|
3385
3939
|
The tool provides 2 ways to read and write files with the option: `file_access`
|
|
3386
3940
|
|
|
@@ -3397,50 +3951,89 @@ Aspera CLI can send email, for that setup SMTP configuration. This is done with
|
|
|
3397
3951
|
The `smtp` option is a hash table (extended value) with the following fields:
|
|
3398
3952
|
<table>
|
|
3399
3953
|
<tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
|
|
3400
|
-
<tr><td
|
|
3401
|
-
<tr><td
|
|
3402
|
-
<tr><td
|
|
3403
|
-
<tr><td
|
|
3404
|
-
<tr><td
|
|
3405
|
-
<tr><td
|
|
3406
|
-
<tr><td
|
|
3407
|
-
<tr><td
|
|
3954
|
+
<tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
|
|
3955
|
+
<tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
|
|
3956
|
+
<tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
|
|
3957
|
+
<tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
|
|
3958
|
+
<tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
|
|
3959
|
+
<tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
|
|
3960
|
+
<tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
|
|
3961
|
+
<tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
|
|
3408
3962
|
</table>
|
|
3409
3963
|
|
|
3410
3964
|
## Example of configuration:
|
|
3411
3965
|
|
|
3412
3966
|
```
|
|
3413
|
-
|
|
3414
|
-
|
|
3415
|
-
|
|
3967
|
+
ascli config preset set smtp_google server smtp.google.com
|
|
3968
|
+
ascli config preset set smtp_google username john@gmail.com
|
|
3969
|
+
ascli config preset set smtp_google password P@ssw0rd
|
|
3416
3970
|
```
|
|
3417
3971
|
|
|
3418
3972
|
or
|
|
3419
3973
|
|
|
3420
3974
|
```
|
|
3421
|
-
|
|
3975
|
+
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
|
|
3422
3976
|
```
|
|
3423
3977
|
|
|
3424
3978
|
or
|
|
3425
3979
|
|
|
3426
3980
|
```
|
|
3427
|
-
|
|
3981
|
+
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
|
|
3428
3982
|
```
|
|
3429
3983
|
|
|
3430
|
-
Set this
|
|
3984
|
+
Set this configuration as global default, for instance:
|
|
3431
3985
|
|
|
3432
3986
|
```
|
|
3433
|
-
|
|
3434
|
-
|
|
3987
|
+
ascli config preset set cli_default smtp @val:@preset:smtp_google
|
|
3988
|
+
ascli config preset set default config cli_default
|
|
3435
3989
|
```
|
|
3436
3990
|
|
|
3991
|
+
## Email templates
|
|
3992
|
+
|
|
3993
|
+
Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
|
|
3994
|
+
|
|
3995
|
+
The template is the full SMTP message, including headers.
|
|
3996
|
+
|
|
3997
|
+
The following variables are defined by default:
|
|
3998
|
+
|
|
3999
|
+
* from_name
|
|
4000
|
+
* from_email
|
|
4001
|
+
* to
|
|
4002
|
+
|
|
4003
|
+
Other variables are defined depending on context.
|
|
4004
|
+
|
|
3437
4005
|
## Test
|
|
3438
4006
|
|
|
3439
4007
|
Check settings with `smtp_settings` command. Send test email with `email_test`.
|
|
3440
4008
|
|
|
3441
4009
|
```
|
|
3442
|
-
|
|
3443
|
-
|
|
4010
|
+
ascli config --smtp=@preset:smtp_google smtp
|
|
4011
|
+
ascli config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
|
|
4012
|
+
```
|
|
4013
|
+
|
|
4014
|
+
## Notifications for transfer status
|
|
4015
|
+
|
|
4016
|
+
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
|
|
4017
|
+
|
|
4018
|
+
To activate, use option `notif_to`.
|
|
4019
|
+
|
|
4020
|
+
A default e-mail template is used, but it can be overridden with option `notif_template`.
|
|
4021
|
+
|
|
4022
|
+
The environment provided contains the following additional variables:
|
|
4023
|
+
|
|
4024
|
+
* subject
|
|
4025
|
+
* body
|
|
4026
|
+
* global_transfer_status
|
|
4027
|
+
* ts
|
|
4028
|
+
|
|
4029
|
+
Example of template:
|
|
4030
|
+
|
|
4031
|
+
```
|
|
4032
|
+
From: <%=from_name%> <<%=from_email%>>
|
|
4033
|
+
To: <<%=to%>>
|
|
4034
|
+
Subject: <%=subject%>
|
|
4035
|
+
|
|
4036
|
+
Transfer is: <%=global_transfer_status%>
|
|
3444
4037
|
```
|
|
3445
4038
|
|
|
3446
4039
|
# Tool: `asession`
|
|
@@ -3448,7 +4041,7 @@ $ ascli config --smtp=@preset:smtp_google email sample.dest@example.com
|
|
|
3448
4041
|
This gem comes with a second executable tool providing a simplified standardized interface
|
|
3449
4042
|
to start a FASP session: `asession`.
|
|
3450
4043
|
|
|
3451
|
-
It aims at simplifying the startup of a FASP session from a programmatic stand point as
|
|
4044
|
+
It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
|
|
3452
4045
|
|
|
3453
4046
|
* common to Aspera Node API (HTTP POST /ops/transfer)
|
|
3454
4047
|
* common to Aspera Connect API (browser javascript startTransfer)
|
|
@@ -3460,7 +4053,7 @@ This makes it easy to integrate with any language provided that one can spawn a
|
|
|
3460
4053
|
|
|
3461
4054
|
The tool expect one single argument: a [_transfer-spec_](#transferspec).
|
|
3462
4055
|
|
|
3463
|
-
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON
|
|
4056
|
+
If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
|
|
3464
4057
|
|
|
3465
4058
|
Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
|
|
3466
4059
|
|
|
@@ -3471,7 +4064,7 @@ Note that there are special "extended" [_transfer-spec_](#transferspec) paramete
|
|
|
3471
4064
|
* `EX_loglevel` to change log level of the tool
|
|
3472
4065
|
* `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
|
|
3473
4066
|
|
|
3474
|
-
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the
|
|
4067
|
+
Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
|
|
3475
4068
|
|
|
3476
4069
|
## Comparison of interfaces
|
|
3477
4070
|
|
|
@@ -3498,7 +4091,7 @@ echo "${MY_TSPEC}"|asession
|
|
|
3498
4091
|
This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
|
|
3499
4092
|
|
|
3500
4093
|
```
|
|
3501
|
-
|
|
4094
|
+
asession
|
|
3502
4095
|
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
|
3503
4096
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
|
3504
4097
|
{"type":"DONE"}
|
|
@@ -3513,7 +4106,7 @@ Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/asp
|
|
|
3513
4106
|
## Help
|
|
3514
4107
|
|
|
3515
4108
|
```
|
|
3516
|
-
|
|
4109
|
+
asession -h
|
|
3517
4110
|
USAGE
|
|
3518
4111
|
asession
|
|
3519
4112
|
asession -h|--help
|
|
@@ -3571,8 +4164,8 @@ Interesting ascp features are found in its arguments: (see ascp manual):
|
|
|
3571
4164
|
Note that:
|
|
3572
4165
|
|
|
3573
4166
|
* `ascli` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
|
|
3574
|
-
* not all native ascp arguments are available as standard transfer_spec parameters
|
|
3575
|
-
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the
|
|
4167
|
+
* most, but not all native ascp arguments are available as standard transfer_spec parameters
|
|
4168
|
+
* native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
|
|
3576
4169
|
|
|
3577
4170
|
### server side and configuration
|
|
3578
4171
|
|
|
@@ -3587,18 +4180,18 @@ Once `ascli` parameters are defined, run the command using the OS native schedul
|
|
|
3587
4180
|
## Example
|
|
3588
4181
|
|
|
3589
4182
|
```
|
|
3590
|
-
|
|
4183
|
+
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
|
|
3591
4184
|
|
|
3592
4185
|
```
|
|
3593
4186
|
|
|
3594
|
-
The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they
|
|
4187
|
+
The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
|
|
3595
4188
|
|
|
3596
|
-
#
|
|
4189
|
+
# Health check and Nagios
|
|
3597
4190
|
|
|
3598
|
-
|
|
4191
|
+
Most plugin provide a `health` command that will check the health status of the application. Example:
|
|
3599
4192
|
|
|
3600
4193
|
```
|
|
3601
|
-
|
|
4194
|
+
ascli console health
|
|
3602
4195
|
+--------+-------------+------------+
|
|
3603
4196
|
| status | component | message |
|
|
3604
4197
|
+--------+-------------+------------+
|
|
@@ -3615,13 +4208,13 @@ Typically, the health check uses the REST API of the application with the follow
|
|
|
3615
4208
|
`ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
|
|
3616
4209
|
|
|
3617
4210
|
```
|
|
3618
|
-
|
|
4211
|
+
ascli server health transfer --to-folder=/Upload --format=nagios --progress=none
|
|
3619
4212
|
OK - [transfer:ok]
|
|
3620
|
-
|
|
4213
|
+
ascli server health asctlstatus --cmd_prefix='sudo ' --format=nagios
|
|
3621
4214
|
OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
|
|
3622
4215
|
```
|
|
3623
4216
|
|
|
3624
|
-
# Module: `Aspera`
|
|
4217
|
+
# Ruby Module: `Aspera`
|
|
3625
4218
|
|
|
3626
4219
|
Main components:
|
|
3627
4220
|
|
|
@@ -3632,8 +4225,8 @@ Main components:
|
|
|
3632
4225
|
A working example can be found in the gem, example:
|
|
3633
4226
|
|
|
3634
4227
|
```
|
|
3635
|
-
|
|
3636
|
-
|
|
4228
|
+
ascli config gem_path
|
|
4229
|
+
cat $(ascli config gem_path)/../examples/transfer.rb
|
|
3637
4230
|
```
|
|
3638
4231
|
|
|
3639
4232
|
This sample code shows some example of use of the API as well as
|
|
@@ -3668,106 +4261,143 @@ So, it evolved into `ascli`:
|
|
|
3668
4261
|
|
|
3669
4262
|
# Changes (Release notes)
|
|
3670
4263
|
|
|
4264
|
+
* 4.5.0
|
|
4265
|
+
|
|
4266
|
+
* new: support transfer agent: [Transfer SDK](#agt_trsdk)
|
|
4267
|
+
* new: support [http socket options](#http_options)
|
|
4268
|
+
* new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
|
|
4269
|
+
* new: `config vault` supports encrypted passwords, also macos keychain
|
|
4270
|
+
* new: `config preset` command for consistency with id
|
|
4271
|
+
* new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
|
|
4272
|
+
* change: when using wss, use [ruby's CA certs](#certificates)
|
|
4273
|
+
* change: unexpected parameter makes exit code not zero
|
|
4274
|
+
* change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
|
|
4275
|
+
* change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
|
|
4276
|
+
* change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
|
|
4277
|
+
* fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
|
|
4278
|
+
|
|
4279
|
+
* 4.4.0
|
|
4280
|
+
|
|
4281
|
+
* new: `aoc packages list` add possibility to add filter with option `query`
|
|
4282
|
+
* new: `aoc admin res xxx list` now get all items by default #50
|
|
4283
|
+
* new: `preset` option can specify name or hash value
|
|
4284
|
+
* new: `node` plugin accepts bearer token and access key as credential
|
|
4285
|
+
* new: `node` option `token_type` allows using basic token in addition to aspera type.
|
|
4286
|
+
* change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
|
|
4287
|
+
* change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
|
|
4288
|
+
|
|
4289
|
+
* 4.3.0
|
|
4290
|
+
|
|
4291
|
+
* new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
|
|
4292
|
+
* new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
|
|
4293
|
+
|
|
4294
|
+
* 4.2.2
|
|
4295
|
+
|
|
4296
|
+
* new: `faspex package list` retrieves the whole list, not just first page
|
|
4297
|
+
* new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
|
|
4298
|
+
* new: the error "Remote host is not who we expected" displays a special remediation message
|
|
4299
|
+
* new: `conf ascp spec` displays supported transfer spec
|
|
4300
|
+
* new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
|
|
4301
|
+
* fix: space character in `faspe:` url are precent encoded if needed
|
|
4302
|
+
* fix: `preview scan`: if file_id is unknown, ignore and continue scan
|
|
4303
|
+
* change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then `ascli` exits with code 1 (instead of zero=success)
|
|
4304
|
+
* change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
|
|
4305
|
+
|
|
3671
4306
|
* 4.2.1
|
|
3672
4307
|
|
|
3673
|
-
|
|
3674
|
-
|
|
4308
|
+
* new: command `faspex package recv` supports link of type: `faspe:`
|
|
4309
|
+
* new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
|
|
3675
4310
|
|
|
3676
4311
|
* 4.2.0
|
|
3677
4312
|
|
|
3678
|
-
|
|
3679
|
-
|
|
3680
|
-
|
|
3681
|
-
|
|
3682
|
-
|
|
3683
|
-
|
|
3684
|
-
|
|
3685
|
-
|
|
3686
|
-
|
|
3687
|
-
|
|
4313
|
+
* new: command `aoc remind` to receive organization membership by email
|
|
4314
|
+
* new: in `preview` option `value` to filter out on file name
|
|
4315
|
+
* new: `initdemo` to initialize for demo server
|
|
4316
|
+
* new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
|
|
4317
|
+
* fix: on Windows `conf ascp use` expects ascp.exe
|
|
4318
|
+
* fix: (break) multi_session_threshold is Integer, not String
|
|
4319
|
+
* fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
|
|
4320
|
+
* fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
|
|
4321
|
+
* change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
|
|
4322
|
+
* change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
|
|
3688
4323
|
|
|
3689
4324
|
* 4.1.0
|
|
3690
4325
|
|
|
3691
|
-
|
|
3692
|
-
|
|
3693
|
-
|
|
3694
|
-
|
|
3695
|
-
|
|
3696
|
-
|
|
3697
|
-
|
|
3698
|
-
|
|
3699
|
-
|
|
3700
|
-
|
|
3701
|
-
|
|
3702
|
-
|
|
3703
|
-
* new: initial support for Faspex 5 Beta 1
|
|
4326
|
+
* fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
|
|
4327
|
+
* new: update documentation with regard to offline and docker installation
|
|
4328
|
+
* new: renamed command `nagios_check` to `health`
|
|
4329
|
+
* new: agent `http_gw` now supports upload
|
|
4330
|
+
* new: added option `sdk_url` to install SDK from local file for offline install
|
|
4331
|
+
* new: check new gem version periodically
|
|
4332
|
+
* new: the --fields= option, support -_fieldname_ to remove a field from default fields
|
|
4333
|
+
* new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
|
|
4334
|
+
* new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
|
|
4335
|
+
* new: AoC a password can be provided for a public link
|
|
4336
|
+
* new: `conf doc` take an optional parameter to go to a section
|
|
4337
|
+
* new: initial support for Faspex 5 Beta 1
|
|
3704
4338
|
|
|
3705
4339
|
* 4.0.0
|
|
3706
4340
|
|
|
3707
|
-
|
|
3708
|
-
|
|
3709
|
-
|
|
3710
|
-
|
|
3711
|
-
|
|
3712
|
-
|
|
3713
|
-
|
|
3714
|
-
|
|
4341
|
+
* now available as open source at [https://github.com/IBM/aspera-cli](https://github.com/IBM/aspera-cli) with general cleanup
|
|
4342
|
+
* changed default tool name from `mlia` to `ascli`
|
|
4343
|
+
* changed `aspera` command to `aoc`
|
|
4344
|
+
* changed gem name from `asperalm` to `aspera-cli`
|
|
4345
|
+
* changed module name from `Asperalm` to `Aspera`
|
|
4346
|
+
* removed command `folder` in `preview`, merged to `scan`
|
|
4347
|
+
* persistency files go to sub folder instead of main folder
|
|
4348
|
+
* added possibility to install SDK: `config ascp install`
|
|
3715
4349
|
|
|
3716
4350
|
* 0.11.8
|
|
3717
4351
|
|
|
3718
|
-
|
|
4352
|
+
* Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
|
|
3719
4353
|
|
|
3720
4354
|
* 0.11.7
|
|
3721
4355
|
|
|
3722
|
-
|
|
3723
|
-
|
|
3724
|
-
|
|
3725
|
-
|
|
3726
|
-
|
|
3727
|
-
|
|
4356
|
+
* rework on rest call error handling
|
|
4357
|
+
* use option `display` with value `data` to remove out of extraneous information
|
|
4358
|
+
* fixed option `lock_port` not working
|
|
4359
|
+
* generate special icon if preview failed
|
|
4360
|
+
* possibility to choose transfer progress bar type with option `progress`
|
|
4361
|
+
* AoC package creation now output package id
|
|
3728
4362
|
|
|
3729
4363
|
* 0.11.6
|
|
3730
4364
|
|
|
3731
|
-
|
|
3732
|
-
|
|
3733
|
-
|
|
3734
|
-
|
|
3735
|
-
|
|
3736
|
-
|
|
3737
|
-
|
|
4365
|
+
* orchestrator : added more choice in auth type
|
|
4366
|
+
* preview: cleanup in generator (removed and renamed parameters)
|
|
4367
|
+
* preview: better documentation
|
|
4368
|
+
* preview: animated thumbnails for video (option: `video_png_conv=animated`)
|
|
4369
|
+
* preview: new event trigger: `trevents` (`events` seems broken)
|
|
4370
|
+
* preview: unique tmp folder to avoid clash of multiple instances
|
|
4371
|
+
* repo: added template for secrets used for testing
|
|
3738
4372
|
|
|
3739
4373
|
* 0.11.5
|
|
3740
4374
|
|
|
3741
|
-
|
|
3742
|
-
|
|
3743
|
-
|
|
3744
|
-
|
|
3745
|
-
|
|
3746
|
-
|
|
4375
|
+
* added option `default_ports` for AoC (see manual)
|
|
4376
|
+
* allow bulk delete in `aspera files` with option `bulk=yes`
|
|
4377
|
+
* fix getting connect versions
|
|
4378
|
+
* added section for Aix
|
|
4379
|
+
* support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
|
|
4380
|
+
* added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
|
|
3747
4381
|
|
|
3748
4382
|
* 0.11.4
|
|
3749
4383
|
|
|
3750
|
-
|
|
4384
|
+
* possibility to give shared inbox name when sending a package (else use id and type)
|
|
3751
4385
|
|
|
3752
4386
|
* 0.11.3
|
|
3753
4387
|
|
|
3754
|
-
|
|
4388
|
+
* minor fixes on multi-session: avoid exception on progress bar
|
|
3755
4389
|
|
|
3756
4390
|
* 0.11.2
|
|
3757
4391
|
|
|
3758
|
-
|
|
4392
|
+
* fixes on multi-session: progress bat and transfer spec param for "direct"
|
|
3759
4393
|
|
|
3760
4394
|
* 0.11.1
|
|
3761
4395
|
|
|
3762
|
-
|
|
4396
|
+
* enhanced short_link creation commands (see examples)
|
|
3763
4397
|
|
|
3764
4398
|
* 0.11
|
|
3765
4399
|
|
|
3766
|
-
|
|
3767
|
-
|
|
3768
|
-
```
|
|
3769
|
-
... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist"}'
|
|
3770
|
-
```
|
|
4400
|
+
* add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
|
|
3771
4401
|
|
|
3772
4402
|
* 0.10.18
|
|
3773
4403
|
|
|
@@ -3776,7 +4406,7 @@ So, it evolved into `ascli`:
|
|
|
3776
4406
|
* 0.10.17
|
|
3777
4407
|
|
|
3778
4408
|
* fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
|
|
3779
|
-
* new modifier: `@list:<
|
|
4409
|
+
* new modifier: `@list:<separator>val1<separator>...`
|
|
3780
4410
|
|
|
3781
4411
|
* 0.10.16
|
|
3782
4412
|
|
|
@@ -3798,7 +4428,7 @@ So, it evolved into `ascli`:
|
|
|
3798
4428
|
* 0.10.12
|
|
3799
4429
|
|
|
3800
4430
|
* added support for AoC node registration keys
|
|
3801
|
-
* replaced option : `local_resume` with `transfer_info` for agent `direct`
|
|
4431
|
+
* replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
|
|
3802
4432
|
* Transfer agent is no more a Singleton instance, but only one is used in CLI
|
|
3803
4433
|
* `@incps` : new extended value modifier
|
|
3804
4434
|
* ATS: no more provides access keys secrets: now user must provide it
|
|
@@ -3830,7 +4460,7 @@ So, it evolved into `ascli`:
|
|
|
3830
4460
|
|
|
3831
4461
|
* 0.10.6
|
|
3832
4462
|
|
|
3833
|
-
* FaspManager: transfer spec `authentication` no more needed for local
|
|
4463
|
+
* FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
|
|
3834
4464
|
* gem version requirements made more open
|
|
3835
4465
|
|
|
3836
4466
|
* 0.10.5
|
|
@@ -3848,9 +4478,9 @@ So, it evolved into `ascli`:
|
|
|
3848
4478
|
|
|
3849
4479
|
* 0.10.2
|
|
3850
4480
|
|
|
3851
|
-
|
|
3852
|
-
|
|
3853
|
-
|
|
4481
|
+
* updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
|
|
4482
|
+
* added doc for "cargo" like actions
|
|
4483
|
+
* added doc for multi-session
|
|
3854
4484
|
|
|
3855
4485
|
* 0.10.1
|
|
3856
4486
|
|
|
@@ -3878,22 +4508,22 @@ So, it evolved into `ascli`:
|
|
|
3878
4508
|
|
|
3879
4509
|
* 0.9.33
|
|
3880
4510
|
|
|
3881
|
-
|
|
3882
|
-
|
|
3883
|
-
|
|
3884
|
-
|
|
4511
|
+
* new command to display basic token of node
|
|
4512
|
+
* new command to display bearer token of node in AoC
|
|
4513
|
+
* the --fields= option, support +_fieldname_ to add a field to default fields
|
|
4514
|
+
* many small changes
|
|
3885
4515
|
|
|
3886
4516
|
* 0.9.32
|
|
3887
4517
|
|
|
3888
|
-
|
|
3889
|
-
|
|
3890
|
-
|
|
4518
|
+
* all Faspex public links are now supported
|
|
4519
|
+
* removed faspex operation recv_publink
|
|
4520
|
+
* replaced with option `link` (consistent with AoC)
|
|
3891
4521
|
|
|
3892
4522
|
* 0.9.31
|
|
3893
4523
|
|
|
3894
|
-
|
|
3895
|
-
|
|
3896
|
-
|
|
4524
|
+
* added more support for public link: receive and send package, to user or dropbox and files view.
|
|
4525
|
+
* delete expired file lists
|
|
4526
|
+
* changed text table gem from text-table to terminal-table because it supports multiline values
|
|
3897
4527
|
|
|
3898
4528
|
* 0.9.27
|
|
3899
4529
|
|
|
@@ -3913,149 +4543,149 @@ So, it evolved into `ascli`:
|
|
|
3913
4543
|
|
|
3914
4544
|
* 0.9.24
|
|
3915
4545
|
|
|
3916
|
-
|
|
3917
|
-
|
|
4546
|
+
* fix bug where AoC node to node transfer did not work
|
|
4547
|
+
* fix bug on error if ED25519 private key is defined in .ssh
|
|
3918
4548
|
|
|
3919
4549
|
* 0.9.23
|
|
3920
4550
|
|
|
3921
|
-
|
|
3922
|
-
|
|
4551
|
+
* defined REST error handlers, more error conditions detected
|
|
4552
|
+
* commands to select specific ascp location
|
|
3923
4553
|
|
|
3924
4554
|
* 0.9.21
|
|
3925
4555
|
|
|
3926
|
-
|
|
3927
|
-
|
|
4556
|
+
* supports simplified wizard using global client
|
|
4557
|
+
* only ascp binary is required, other SDK (keys) files are now generated
|
|
3928
4558
|
|
|
3929
4559
|
* 0.9.20
|
|
3930
4560
|
|
|
3931
|
-
|
|
3932
|
-
|
|
3933
|
-
|
|
4561
|
+
* improved wizard (prepare for AoC global client id)
|
|
4562
|
+
* preview generator: addedoption : --skip-format=<png,mp4>
|
|
4563
|
+
* removed outdated pictures from this doc
|
|
3934
4564
|
|
|
3935
4565
|
* 0.9.19
|
|
3936
4566
|
|
|
3937
|
-
|
|
4567
|
+
* added command aspera bearer --scope=xx
|
|
3938
4568
|
|
|
3939
4569
|
* 0.9.18
|
|
3940
4570
|
|
|
3941
|
-
|
|
4571
|
+
* enhanced aspera admin events to support query
|
|
3942
4572
|
|
|
3943
4573
|
* 0.9.16
|
|
3944
4574
|
|
|
3945
|
-
|
|
3946
|
-
|
|
4575
|
+
* AoC transfers are now reported in activity app
|
|
4576
|
+
* new interface for Rest class authentication (keep backward compatibility)
|
|
3947
4577
|
|
|
3948
4578
|
* 0.9.15
|
|
3949
4579
|
|
|
3950
|
-
|
|
3951
|
-
|
|
4580
|
+
* new feature: "find" command in aspera files
|
|
4581
|
+
* sample code for transfer API
|
|
3952
4582
|
|
|
3953
4583
|
* 0.9.12
|
|
3954
4584
|
|
|
3955
|
-
|
|
3956
|
-
|
|
4585
|
+
* add nagios commands
|
|
4586
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
|
3957
4587
|
|
|
3958
4588
|
* 0.9.11
|
|
3959
4589
|
|
|
3960
|
-
|
|
3961
|
-
|
|
4590
|
+
* Breaking change: @stdin is now @stdin:
|
|
4591
|
+
* support of ATS for IBM Cloud, removed old version based on aspera id
|
|
3962
4592
|
|
|
3963
4593
|
|
|
3964
4594
|
* 0.9.10
|
|
3965
4595
|
|
|
3966
|
-
|
|
3967
|
-
|
|
3968
|
-
|
|
3969
|
-
|
|
4596
|
+
* Breaking change: parameter transfer-node becomes more generic: transfer-info
|
|
4597
|
+
* Display SaaS storage usage with command: aspera admin res node --id=nn info
|
|
4598
|
+
* cleaner way of specifying source file list for transfers
|
|
4599
|
+
* Breaking change: replaced download_mode option with http_download action
|
|
3970
4600
|
|
|
3971
4601
|
* 0.9.9
|
|
3972
4602
|
|
|
3973
|
-
|
|
3974
|
-
|
|
4603
|
+
* Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
|
|
4604
|
+
* Now supports "cargo" for Aspera on Cloud (automatic package download)
|
|
3975
4605
|
|
|
3976
4606
|
* 0.9.8
|
|
3977
4607
|
|
|
3978
|
-
|
|
3979
|
-
|
|
4608
|
+
* Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
|
|
4609
|
+
* AoC: share to share transfer with command "transfer"
|
|
3980
4610
|
|
|
3981
4611
|
* 0.9.7
|
|
3982
4612
|
|
|
3983
|
-
|
|
3984
|
-
|
|
3985
|
-
|
|
3986
|
-
|
|
3987
|
-
|
|
4613
|
+
* homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
|
|
4614
|
+
* preview persistency goes to unique file by default
|
|
4615
|
+
* catch mxf extension in preview as video
|
|
4616
|
+
* Faspex: possibility to download all packages by specifying id=ALL
|
|
4617
|
+
* Faspex: to come: cargo-like function to download only new packages with id=NEW
|
|
3988
4618
|
|
|
3989
4619
|
* 0.9.6
|
|
3990
4620
|
|
|
3991
|
-
|
|
3992
|
-
|
|
4621
|
+
* Breaking change: `@param:`is now `@preset:` and is generic
|
|
4622
|
+
* AoC: added command to display current workspace information
|
|
3993
4623
|
|
|
3994
4624
|
* 0.9.5
|
|
3995
4625
|
|
|
3996
|
-
|
|
3997
|
-
|
|
4626
|
+
* new parameter: new_user_option used to choose between public_link and invite of external users.
|
|
4627
|
+
* fixed bug in wizard, and wizard uses now product detection
|
|
3998
4628
|
|
|
3999
4629
|
* 0.9.4
|
|
4000
4630
|
|
|
4001
|
-
|
|
4002
|
-
|
|
4003
|
-
|
|
4631
|
+
* Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
|
|
4632
|
+
* AoC Package send supports external users
|
|
4633
|
+
* new command to export AoC config to Aspera CLI config
|
|
4004
4634
|
|
|
4005
4635
|
* 0.9.3
|
|
4006
4636
|
|
|
4007
|
-
|
|
4008
|
-
|
|
4009
|
-
|
|
4010
|
-
|
|
4011
|
-
|
|
4012
|
-
|
|
4637
|
+
* REST error message show host and code
|
|
4638
|
+
* option for quiet display
|
|
4639
|
+
* modified transfer interface and allow token re-generation on error
|
|
4640
|
+
* async add admin command
|
|
4641
|
+
* async add db parameters
|
|
4642
|
+
* Breaking change: new option "sources" to specify files to transfer
|
|
4013
4643
|
|
|
4014
4644
|
* 0.9.2
|
|
4015
4645
|
|
|
4016
|
-
|
|
4646
|
+
* Breaking change: changed AoC package creation to match API, see AoC section
|
|
4017
4647
|
|
|
4018
4648
|
* 0.9.1
|
|
4019
4649
|
|
|
4020
|
-
|
|
4650
|
+
* Breaking change: changed faspex package creation to match API, see Faspex section
|
|
4021
4651
|
|
|
4022
4652
|
* 0.9
|
|
4023
4653
|
|
|
4024
|
-
|
|
4025
|
-
|
|
4654
|
+
* Renamed the CLI from aslmcli to `ascli`
|
|
4655
|
+
* Automatic rename and conversion of former config folder from aslmcli to `ascli`
|
|
4026
4656
|
|
|
4027
4657
|
* 0.7.6
|
|
4028
4658
|
|
|
4029
|
-
|
|
4659
|
+
* add "sync" plugin
|
|
4030
4660
|
|
|
4031
4661
|
* 0.7
|
|
4032
4662
|
|
|
4033
|
-
|
|
4034
|
-
|
|
4035
|
-
|
|
4036
|
-
|
|
4037
|
-
|
|
4663
|
+
* Breaking change: AoC package recv take option if for package instead of argument.
|
|
4664
|
+
* Breaking change: Rest class and Oauth class changed init parameters
|
|
4665
|
+
* AoC: receive package from public link
|
|
4666
|
+
* select by col value on output
|
|
4667
|
+
* added rename (AoC, node)
|
|
4038
4668
|
|
|
4039
4669
|
* 0.6.19
|
|
4040
4670
|
|
|
4041
4671
|
Breaking change:
|
|
4042
4672
|
|
|
4043
|
-
|
|
4044
|
-
|
|
4045
|
-
|
|
4046
|
-
|
|
4047
|
-
|
|
4048
|
-
|
|
4049
|
-
|
|
4050
|
-
|
|
4673
|
+
* ats server list provisioned → ats cluster list
|
|
4674
|
+
* ats server list clouds → ats cluster clouds
|
|
4675
|
+
* ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
|
|
4676
|
+
* ats server id xxx → ats cluster show --id=xxx
|
|
4677
|
+
* ats subscriptions → ats credential subscriptions
|
|
4678
|
+
* ats api_key repository list → ats credential cache list
|
|
4679
|
+
* ats api_key list → ats credential list
|
|
4680
|
+
* ats access_key id xxx → ats access_key --id=xxx
|
|
4051
4681
|
|
|
4052
4682
|
* 0.6.18
|
|
4053
4683
|
|
|
4054
|
-
|
|
4684
|
+
* some commands take now --id option instead of id command.
|
|
4055
4685
|
|
|
4056
4686
|
* 0.6.15
|
|
4057
4687
|
|
|
4058
|
-
|
|
4688
|
+
* Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in [option preset](#lprt) "default".
|
|
4059
4689
|
|
|
4060
4690
|
# BUGS, FEATURES, CONTRIBUTION
|
|
4061
4691
|
|
|
@@ -4063,16 +4693,16 @@ For issues or feature requests use the Github repository and issues.
|
|
|
4063
4693
|
|
|
4064
4694
|
You can also contribute to this open source project.
|
|
4065
4695
|
|
|
4066
|
-
One can also create one's own
|
|
4696
|
+
One can also [create one's own plugin](#createownplugin).
|
|
4067
4697
|
|
|
4068
4698
|
## Only one value for any option
|
|
4069
4699
|
|
|
4070
4700
|
Some commands and sub commands may ask for the same option name.
|
|
4071
|
-
Currently, since option definition is position
|
|
4701
|
+
Currently, since option definition is position independent (last one wins), it is not possible
|
|
4072
4702
|
to give an option to a command and the same option with different value to a sub command.
|
|
4073
4703
|
|
|
4074
|
-
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, the later will override the earlier one, and both entity will use the same id.
|
|
4075
|
-
As a
|
|
4704
|
+
For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
|
|
4705
|
+
As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
|
|
4076
4706
|
|
|
4077
4707
|
This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
|
|
4078
4708
|
|
|
@@ -4092,20 +4722,24 @@ You may either install the suggested Gems, or remove your ed25519 key from your
|
|
|
4092
4722
|
|
|
4093
4723
|
## Error "Remote host is not who we expected"
|
|
4094
4724
|
|
|
4095
|
-
`ascp`
|
|
4725
|
+
Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
|
|
4726
|
+
|
|
4727
|
+
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
|
4096
4728
|
|
|
4097
4729
|
```
|
|
4098
4730
|
--ts=@json:'{"sshfp":null}'
|
|
4099
4731
|
```
|
|
4100
4732
|
|
|
4101
|
-
|
|
4733
|
+
Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
|
|
4734
|
+
|
|
4735
|
+
References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
|
|
4102
4736
|
|
|
4103
|
-
##
|
|
4737
|
+
## Miscellaneous
|
|
4104
4738
|
|
|
4105
4739
|
* remove rest and oauth classes and use ruby standard gems:
|
|
4106
4740
|
|
|
4107
|
-
|
|
4108
|
-
|
|
4741
|
+
* oauth
|
|
4742
|
+
* https://github.com/rest-client/rest-client
|
|
4109
4743
|
|
|
4110
4744
|
* use Thor or any standard Ruby CLI manager
|
|
4111
4745
|
|