aspera-cli 4.0.0 → 4.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 7172a58b5d9de3455d25667a9eb6d3475d66b2a334be6e2cdee987a041d1d89f
4
- data.tar.gz: 7408405f727f10b137a15c453cd3d89770256fc8392ab311b1318b873bffd18d
3
+ metadata.gz: 902a8a3e2d46ccbb5cd86a29019008d42bca1ec21b4085b3dc9b689a48f5047e
4
+ data.tar.gz: 4e43c371b4867c4c0c3fd7466153c3e98e300befd45d9670a9a7e5b06739dcff
5
5
  SHA512:
6
- metadata.gz: 80fff0696b7a65c05b823483e8f07f34d37a3b0c95c6658ed23e4837b4b223143b46146aae53604a5f80e1695e12f598d53bbb900d3d34a5c31094e441904e82
7
- data.tar.gz: 76c299e2daed96ff33af518f1d1df97eba35bab38c9c8eec86f2bf5b623455dec6bc6bbddba3fbd390b3ebb7d34a68ed7054a74999e215cbbfa88840af8d311a
6
+ metadata.gz: 1f24130c6088bf6ba0abdd9d2b66557b867a8507975d014301e87ad12524b9625070346de6fcbf87ae4d3f32e2b0f5666b34657d7ab88fdcd331fd59632e936d
7
+ data.tar.gz: 79c2ada8efbd5cec946cefb0b7e7363b4323c619783ed65862c0963d07ac1e76a4bea51c0d426f7eaf4b336731f49c07e8b6e74f6830ae0e872954521ac1af8e
data/README.md CHANGED
@@ -1,25 +1,76 @@
1
1
  [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
2
- # `ascli` : a Command Line for IBM Aspera products
2
+ # `ascli` : Command Line Interface for IBM Aspera products
3
3
 
4
- Version : 4.0.0
4
+ Version : 4.1.0
5
5
 
6
6
  _Laurent/2016-2021_
7
7
 
8
- This gem provides a command line interface to Aspera Applications.
8
+ This gem provides `ascli`: a command line interface to Aspera Applications.
9
9
 
10
- Location (once released):
11
- [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
10
+ `ascli` is a also great tool to learn Aspera APIs.
12
11
 
13
- Disclaimers:
12
+ Ruby Gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli)
14
13
 
15
- * This has not yet been officially released so things may change
14
+ Ruby Doc: [https://www.rubydoc.info/gems/aspera-cli](https://www.rubydoc.info/gems/aspera-cli)
16
15
 
17
- `ascli` is also a great tool to learn Aspera APIs.
16
+ Ruby version must be >= > 2.4
17
+
18
+ # <a name="when_to_use"></a>When to use and when not to use
19
+
20
+ `ascli` is designed to be used as a command line tool to:
21
+
22
+ * execute commands on Aspera products
23
+ * transfer to/from Aspera products
24
+
25
+ So it is designed for:
26
+
27
+ * Interactive operations on a text terminal (typically, VT100 compatible)
28
+ * Batch operations in (shell) scripts (e.g. cron job)
29
+
30
+ `ascli` can be seen as a command line tool integrating:
31
+
32
+ * a configuration file (config.yaml) and advanced command line options
33
+ * cURL (for REST calls)
34
+ * Aspera transfer (ascp)
35
+
36
+ One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea.
37
+ For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
38
+
39
+ * Product APIs (REST) : e.g. AoC, Faspex, node
40
+ * Transfer SDK : with gRPC interface and laguage stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
41
+
42
+ Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
43
+
44
+ For scripting and ad'hoc command line operations, `ascli` is perfect.
45
+
46
+ # Notations
18
47
 
19
48
  In examples, command line operations (starting with `$`) are shown using a standard shell: `bash` or `zsh`.
49
+ Prompt `# ` refers to user `root`, prompt `xfer$ ` refer to user `xfer`.
20
50
 
21
51
  Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
22
52
 
53
+ # <a name="parsing"></a>Shell and Command line parsing
54
+
55
+ `ascli` is typically executed in a shell, either interactively or in a script. `ascli` receives its arguments from this shell.
56
+
57
+ On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before `ascli` (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to `ascli`. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
58
+
59
+ On Windows, `cmd` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
60
+
61
+ * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
62
+ * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
63
+
64
+ In case of doubt of argument values after parsing test like this:
65
+
66
+ ```
67
+ $ ascli conf echo "Hello World" arg2 3
68
+ "Hello World"
69
+ ERROR: Argument: unprocessed values: ["arg2", "3"]
70
+ ```
71
+
72
+ `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.
73
+
23
74
  # Quick Start
24
75
 
25
76
  This section guides you from installation, first use and advanced use.
@@ -30,7 +81,7 @@ Once the gem is installed, `ascli` shall be accessible:
30
81
 
31
82
  ```
32
83
  $ ascli --version
33
- 4.0.0
84
+ 4.1.0
34
85
  ```
35
86
 
36
87
  ## First use
@@ -95,77 +146,154 @@ Then, follow the section relative to the product you want to interact with ( Asp
95
146
 
96
147
  # <a name="installation"></a>Installation
97
148
 
98
- In order to use the tool or the gem, it is necessary to install those components:
149
+ It is possible to install *either* as a docker container or step by step on the native host:
99
150
 
100
- * [Ruby](#ruby)
151
+ * [Ruby](#ruby) version >= > 2.4
101
152
  * [aspera-cli](#the_gem)
102
- * [FASP](#fasp_prot)
153
+ * [Aspera SDK (ascp)](#fasp_prot)
103
154
 
104
155
  The following sections provide information on the installation.
105
156
 
157
+ An internet connection is required for the installation. If you dont have internet for the installation, refer to section [Installation without internet access](#offline_install).
158
+
159
+ ## Docker container
160
+
161
+ This method installs a docker image that contains: Ruby, ascli and the FASP sdk.
162
+
163
+ Ensure that you have Docker installed.
164
+
165
+ ```
166
+ $ docker --version
167
+ ```
168
+
169
+ Download the wrapping script:
170
+
171
+ ```
172
+ $ curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
173
+ $ chmod a+x ascli
174
+ ```
175
+
176
+ Install the container image:
177
+
178
+ ```
179
+ $ ./ascli install
180
+ ```
181
+
182
+ Start using it !
183
+
184
+ Note that the tool is run in the container.
185
+
186
+ The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/ascli`.
187
+
188
+ To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
189
+
106
190
  ## <a name="ruby"></a>Ruby
107
191
 
108
- A ruby interpreter is required to run the tool or to use the gem and tool.
192
+ Use this method to install on the native host.
109
193
 
110
- Ruby minimum version: > 2.4
194
+ A ruby interpreter is required to run the tool or to use the gem and tool.
111
195
 
112
- Ruby 3 is also supported.
196
+ Ruby minimum version: > 2.4. Ruby version 3 is also supported.
113
197
 
114
- Any type of Ruby installation can be used (installer, rpm, rvm, ...).
198
+ *Any type of Ruby installation can be used* : rpm, yum, dnf, rvm, brew, windows installer, ... .
115
199
 
116
200
  Refer to the following sections for a proposed method for specific operating systems.
117
201
 
118
- ### macOS
202
+ The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, Macos, Windows with cygwin, etc...).
203
+ If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
204
+ If you have a simpler better way to install Ruby version >= > 2.4 : use it !
205
+
206
+ ### Generic: RVM: single user installation (not root)
119
207
 
120
- MacOS 10.13+ (High Sierra) comes with a recent Ruby, so you can use it directly, you will need to install aspera-cli using `sudo` :
208
+ Use this method which provides more flexibility.
209
+
210
+ Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
211
+
212
+ Install the 2 keys
121
213
 
122
214
  ```
123
- $ sudo gem install aspera-cli
215
+ $ gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
124
216
  ```
125
217
 
126
- Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
218
+ Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
127
219
 
128
220
  ```
129
- $ brew install ruby
221
+ $ \curl -sSL https://get.rvm.io | bash -s stable
222
+ ```
223
+
224
+ If you keep the same terminal (ont needed if re-login):
225
+
226
+ ```
227
+ $ source ~/.rvm/scripts/rvm
130
228
  ```
131
229
 
132
- It is also possib le to use `rvm`
230
+ It is advised to get one of the pre-compiled ruby version, you can list with:
133
231
 
134
- ### Windows
232
+ ```
233
+ $ rvm list --remote
234
+ ```
135
235
 
136
- Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/).
236
+ Install the chosen pre-compiled Ruby version:
137
237
 
138
- Go to "Downloads".
238
+ ```
239
+ $ rvm install 2.7.2 --binary
240
+ ```
139
241
 
140
- Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
242
+ Ruby is now installed for the user, go on to Gem installation.
141
243
 
142
- At the end of the installer uncheck the box to skip the installation of "MSys2".
244
+ ### Generic: RVM: global installation (as root)
143
245
 
144
- ### Linux
246
+ Follow the same method as single user install, but execute as "root".
145
247
 
146
- Install Latest Ruby 2 using "rvm" [https://rvm.io/](https://rvm.io/) .
147
- It installs by default in /usr/local/rvm , but you can install in another location:
248
+ As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
249
+ One can install in another location with :
148
250
 
149
251
  ```
150
- curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
252
+ # curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
151
253
  ```
152
254
 
153
- Once installed, you can install latest ruby:
255
+ As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
256
+ If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok`.
257
+ To activate ruby (and ascli) later, source it:
154
258
 
155
259
  ```
156
- # rvm install ruby
260
+ # source /etc/profile.d/rvm.sh.ok
261
+ # rvm version
157
262
  ```
158
263
 
159
- If you dont want all users to have ruby by default,
160
- rename the file: `/etc/profile.d/rvm.sh` with another extension, and source it to get rvm.
264
+ ### Windows: Installer
265
+
266
+ Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/) :
267
+
268
+ * Go to "Downloads".
269
+ * Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
270
+ * At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.
271
+
272
+ ### macOS: pre-installed or `brew`
161
273
 
162
- Alternatively, only if you know what you do, on RPM based systems (CentOs, Redhat), install the ruby provided by yum which may be 2.0.
274
+ MacOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using `sudo` :
275
+
276
+ ```
277
+ $ sudo gem install aspera-cli
278
+ ```
279
+
280
+ Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
281
+
282
+ ```
283
+ $ brew install ruby
284
+ ```
285
+
286
+ ### Linux: package
287
+
288
+ If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
289
+
290
+ Example:
163
291
 
164
292
  ```
165
293
  # yum install -y ruby rubygems ruby-json
166
294
  ```
167
295
 
168
- One can cleanup your whole yum-installed ruby environment like this to uninstall:
296
+ One can cleanup the whole yum-installed ruby environment like this to uninstall:
169
297
 
170
298
  ```
171
299
  gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
@@ -190,6 +318,50 @@ For instance to build from source, and install in `/opt/ruby` :
190
318
  # make install
191
319
  ```
192
320
 
321
+ ### <a name="offline_install"></a>Installation without internet access
322
+
323
+ Note that currently no pre-packaged version exist yet.
324
+ A method to build one provided here:
325
+
326
+ On a server with the same OS version and with internet access follow the "Generic single user installation" method.
327
+
328
+ Then create an archive:
329
+
330
+ ```
331
+ $ cd
332
+ $ tar zcvf rvm-ascli.tgz .rvm
333
+ ```
334
+
335
+ Get the Aspera SDK. Execute:
336
+
337
+ ```
338
+ $ ascli conf --show-config|grep sdk_url
339
+ ```
340
+
341
+ Then download the SDK archive from that URL.
342
+
343
+ Another method for the SDK is to install the SDK (`ascli conf ascp install`) on the first system, and archive `$HOME/.aspera`.
344
+
345
+ Transfer those 2 archives to the target system without internet access.
346
+
347
+ On the target system:
348
+
349
+ * Extract the RVM archive either in a global location, or in a user's home folder : `path_to_rvm_root`
350
+ * in the user's `.profile` add this line: (replace `path_to_rvm_root` with the actual location)
351
+
352
+ ```
353
+ source path_to_rvm_root/scripts/rvm
354
+ rvm use 2.7.2
355
+ ```
356
+
357
+ For the SDK, either install from archive:
358
+
359
+ ```
360
+ $ ascli conf ascp install --sdk-url=file:///SDK.zip
361
+ ```
362
+
363
+ or restore the `$HOME/.aspera` folder for the user.
364
+
193
365
  ## <a name="the_gem"></a>`aspera-cli` gem
194
366
 
195
367
  Once you have Ruby and rights to install gems: Install the gem and its dependencies:
@@ -204,21 +376,40 @@ To upgrade to the latest version:
204
376
  # gem update aspera-cli
205
377
  ```
206
378
 
379
+ `ascli` checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
380
+
381
+ To check manually:
382
+
383
+ ```
384
+ # ascli conf check_update
385
+ ```
386
+
387
+
388
+
207
389
  ## <a name="fasp_prot"></a>FASP Protocol
208
390
 
209
- Most file transfers will be done using the FASP protocol. Only two additional files are required to perform
210
- an Aspera Transfer:
391
+ Most file transfers will be done using the FASP protocol, using `ascp`.
392
+ Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
211
393
 
212
394
  * ascp
213
395
  * aspera-license (in same folder, or ../etc)
214
396
 
215
- This can be installed directly with
397
+ This can be installed either be installing an Aspera transfer sofware, or using an embedded command:
216
398
 
217
399
  ```
218
400
  $ ascli conf ascp install
219
401
  ```
220
402
 
221
- Those can be found in one of IBM Aspera transfer server or client with its license file (some are free):
403
+ If a local SDK installation is prefered instead of fetching from internet: one can specify the location of the SDK file:
404
+
405
+ ```
406
+ $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
407
+ $ ascli conf ascp install --sdk-url=file:///SDK.zip
408
+ ```
409
+
410
+ The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
411
+
412
+ If the embedded method is not used, the following packages are also suitable:
222
413
 
223
414
  * IBM Aspera Connect Client (Free)
224
415
  * IBM Aspera Desktop Client (Free)
@@ -227,7 +418,7 @@ Those can be found in one of IBM Aspera transfer server or client with its licen
227
418
  * IBM Aspera High Speed Transfer EndPoint (Licensed)
228
419
 
229
420
  For instance, Aspera Connect Client can be installed
230
- by visiting the page: [http://downloads.asperasoft.com/connect2/](http://downloads.asperasoft.com/connect2/).
421
+ by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
231
422
 
232
423
  `ascli` will detect most of Aspera transfer products in standard locations and use the first one found.
233
424
  Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
@@ -235,6 +426,33 @@ Refer to section [FASP](#client) for details on how to select a client or set pa
235
426
  Several methods are provided on how to start a transfer. Use of a local client is one of them, but
236
427
  other methods are available. Refer to section: [Transfer Agents](#agents)
237
428
 
429
+ ## <a name="offline_install"></a>Offline Installation (without internet)
430
+
431
+ The procedure consists in:
432
+
433
+ * Follow the non-root installation procedure with RVM, including gem
434
+ * archive (zip, tar) the main RVM folder (includes ascli):
435
+
436
+ ```
437
+ $ cd ~
438
+ $ tar zcvf rvm_ascli.tgz .rvm
439
+ ```
440
+
441
+ * retrieve the SDK:
442
+
443
+ ```
444
+ $ curl -Lso SDK.zip https://ibm.biz/aspera_sdk
445
+ ```
446
+
447
+ * on the system without internet access:
448
+
449
+ ```
450
+ $ cd ~
451
+ $ tar zxvf rvm_ascli.tgz
452
+ $ source ~/.rvm/scripts/rvm
453
+ $ ascli conf ascp install --sdk-url=file:///SDK.zip
454
+ ```
455
+
238
456
  # <a name="cli"></a>Command Line Interface: `ascli`
239
457
 
240
458
  The `aspera-cli` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
@@ -415,6 +633,7 @@ By default, a table output will display one line per entry, and columns for each
415
633
  * a,b,c : the list of attributes specified by the comma separated list
416
634
  * Array extended value: for instance, @json:'["a","b","c"]' same as above
417
635
  * +a,b,c : add selected properties to the default selection.
636
+ * -a,b,c : remove selected properties from the default selection.
418
637
 
419
638
  ## <a name="extended"></a>Extended Value Syntax
420
639
 
@@ -428,25 +647,25 @@ The extended value syntax is:
428
647
 
429
648
  The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
430
649
 
431
- The following "readers" are supported:
650
+ The following "readers" are supported (returns value in []):
432
651
 
433
- * @val:VALUE , prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
434
- * @file:PATH , read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
435
- * @path:PATH , performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
436
- * @env:ENVVAR , read from a named env var, e.g.--password=@env:MYPASSVAR
437
- * @stdin: , read from stdin (no value on right)
438
- * @preset:NAME , get whole option preset value by name
652
+ * @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
653
+ * @file:PATH : [String] read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
654
+ * @path:PATH : [String] performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
655
+ * @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
656
+ * @stdin: : [String] read from stdin (no value on right)
657
+ * @preset:NAME : [Hash] get whole option preset value by name
439
658
 
440
659
  In addition it is possible to decode a value, using one or multiple decoders :
441
660
 
442
- * @base64: decode a base64 encoded string
443
- * @json: decode JSON values (convenient to provide complex structures)
444
- * @zlib: uncompress data
445
- * @ruby: execute ruby code
446
- * @csvt: decode a titled CSV value
447
- * @lines: split a string in multiple lines and return an array
448
- * @list: split a string in multiple items taking first character as separator and return an array
449
- * @incps: include values of presets specified by key include_presets in hash
661
+ * @base64: [String] decode a base64 encoded string
662
+ * @json: [any] decode JSON values (convenient to provide complex structures)
663
+ * @zlib: [String] uncompress data
664
+ * @ruby: [any] execute ruby code
665
+ * @csvt: [Array] decode a titled CSV value
666
+ * @lines: [Array] split a string in multiple lines and return an array
667
+ * @list: [Array] split a string in multiple items taking first character as separator and return an array
668
+ * @incps: [Hash] include values of presets specified by key `incps` in input hash
450
669
 
451
670
  To display the result of an extended value, use the `config echo` command.
452
671
 
@@ -478,7 +697,7 @@ $ ascli config echo @csvt:@file:test.csv
478
697
  :......:.....................:
479
698
  ```
480
699
 
481
- Example: create a hash and include values from preset named "config" of config file
700
+ Example: create a hash and include values from preset named "config" of config file in this hash
482
701
 
483
702
  ```
484
703
  $ ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
@@ -967,9 +1186,11 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
967
1186
  Example:
968
1187
 
969
1188
  ```
970
- $ ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://eudemo.asperademo.com:9443/aspera/http-gwy/v1"}'
1189
+ $ ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
971
1190
  ```
972
1191
 
1192
+ Note that the gateway only supports transfers authorized with a token.
1193
+
973
1194
  ## <a name="transferspec"></a>Transfer Specification
974
1195
 
975
1196
  Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
@@ -1020,7 +1241,7 @@ arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)
1020
1241
  </p>
1021
1242
  <p>
1022
1243
  UNDER CONSTRUCTION<br/>
1023
- <a href="https://developer.ibm.com/api/view/aspera-prod:ibm-aspera:title-IBM_Aspera#id90944">Documentation&rarr;Node API&rarr;/opt/transfers</a><br/>
1244
+ <a href="https://developer.ibm.com/apis/catalog/?search=aspera">Aspera API Documentation</a>&rarr;Node API&rarr;/opt/transfers<br/>
1024
1245
  </p>
1025
1246
 
1026
1247
  <table>
@@ -1102,8 +1323,8 @@ The option `to_folder` provides an equivalent and convenient way to change this
1102
1323
 
1103
1324
  ### List of files for transfers
1104
1325
 
1105
- When uploading, downloading or sending files, the user must specify
1106
- the list of files to transfer. Most of the time, the list of files to transfer will be simply specified on the command line:
1326
+ When uploading, downloading or sending files, the user must specify the list of files to transfer.
1327
+ Most of the time, the list of files to transfer will be simply specified on the command line:
1107
1328
 
1108
1329
  ```
1109
1330
  $ ascli server upload ~/mysample.file secondfile
@@ -1117,7 +1338,8 @@ $ ascli server upload --sources=@args ~/mysample.file secondfile
1117
1338
 
1118
1339
  More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1119
1340
 
1120
- Note that this is different from the "ascp" command line. The paradigm used by `ascli` is: all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
1341
+ Note that this is different from the "ascp" command line. The paradigm used by `ascli` is:
1342
+ all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
1121
1343
 
1122
1344
  For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. Accepted values are:
1123
1345
 
@@ -1247,6 +1469,8 @@ ascli aoc admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
1247
1469
  ascli aoc admin res apps_new list
1248
1470
  ascli aoc admin res client list
1249
1471
  ascli aoc admin res client_access_key list
1472
+ ascli aoc admin res client_registration_token --id=my_clt_reg_id delete
1473
+ ascli aoc admin res client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}'
1250
1474
  ascli aoc admin res client_registration_token list
1251
1475
  ascli aoc admin res contact list
1252
1476
  ascli aoc admin res dropbox list
@@ -1284,7 +1508,7 @@ ascli aoc files browse /
1284
1508
  ascli aoc files browse / -N --link=my_aoc_publink_folder
1285
1509
  ascli aoc files delete /testsrc
1286
1510
  ascli aoc files download --transfer=connect /200KB.1
1287
- ascli aoc files file 18891
1511
+ ascli aoc files file --id=my_file_id show
1288
1512
  ascli aoc files find / --value='\.partial$'
1289
1513
  ascli aoc files http_node_download --to-folder=. /200KB.1
1290
1514
  ascli aoc files mkdir /testsrc
@@ -1305,8 +1529,9 @@ ascli aoc packages recv --id=ALL --to-folder=. --once-only=yes --lock-port=12345
1305
1529
  ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["external.user@example.com"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
1306
1530
  ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["internal.user@example.com"],"note":"my note"}' testfile.bin
1307
1531
  ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
1308
- ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user
1532
+ ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
1309
1533
  ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
1534
+ ascli aoc servers
1310
1535
  ascli aoc user info modify @json:'{"name":"dummy change"}'
1311
1536
  ascli aoc user info show
1312
1537
  ascli aoc workspace
@@ -1335,6 +1560,9 @@ ascli config ascp info
1335
1560
  ascli config ascp install
1336
1561
  ascli config ascp products list
1337
1562
  ascli config ascp show
1563
+ ascli config check_update
1564
+ ascli config doc
1565
+ ascli config doc transfer-parameters
1338
1566
  ascli config email_test aspera.user1@gmail.com
1339
1567
  ascli config export
1340
1568
  ascli config genkey mykey
@@ -1349,21 +1577,22 @@ ascli cos node access_key --id=self show
1349
1577
  ascli cos node download testfile.bin --to-folder=.
1350
1578
  ascli cos node info
1351
1579
  ascli cos node upload testfile.bin
1352
- ascli faspex nagios_check
1580
+ ascli faspex health
1353
1581
  ascli faspex package list
1354
1582
  ascli faspex package list --box=sent --fields=package_id --format=csv --display=data|tail -n 1);\
1355
- ascli faspex package recv --box=sent --to-folder=. --id="my_package_id"
1583
+ ascli faspex package list --fields=package_id --format=csv --display=data|tail -n 1);\
1584
+ ascli faspex package recv --to-folder=. --box=sent --id="my_package_id"
1356
1585
  ascli faspex package recv --to-folder=. --id="my_package_id"
1357
1586
  ascli faspex package recv --to-folder=. --id=ALL --once-only=yes
1358
1587
  ascli faspex package recv --to-folder=. --link="my_faspex_publink_recv_from_fxuser"
1359
- ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com"]}' testfile.bin
1588
+ ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["internal.user@example.com","FASPEX_USERNAME"]}' testfile.bin
1360
1589
  ascli faspex package send --link="my_faspex_publink_send_to_dropbox" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
1361
1590
  ascli faspex package send --link="my_faspex_publink_send_to_fxuser" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
1362
1591
  ascli faspex source name "Server Files" node br /
1363
1592
  ascli faspex5 node list --value=@json:'{"type":"received","subtype":"mypackages"}'
1364
- ascli faspex5 package list --value=@json:'{"state":["released"]}'
1593
+ ascli faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
1365
1594
  ascli faspex5 package receive --id="my_package_id" --to-folder=.
1366
- ascli faspex5 package send --value=@json:'{"title":"test title","recipients":["admin"]}' testfile.bin
1595
+ ascli faspex5 package send --value=@json:'{"title":"test title","recipients":["${f5_user}"]}' testfile.bin
1367
1596
  ascli node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
1368
1597
  ascli node -N -Ptst_node_preview access_key delete --id=aoc_1
1369
1598
  ascli node async --id=1 bandwidth
@@ -1377,8 +1606,8 @@ ascli node browse / -r
1377
1606
  ascli node delete folder_1/10MB.1
1378
1607
  ascli node delete folder_1/testfile.bin
1379
1608
  ascli node download --to-folder=. folder_1/testfile.bin
1609
+ ascli node health
1380
1610
  ascli node info
1381
- ascli node nagios_check
1382
1611
  ascli node search / --value=@json:'{"sort":"mtime"}'
1383
1612
  ascli node service --id=service1 delete
1384
1613
  ascli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
@@ -1408,7 +1637,8 @@ ascli preview test --case=test png "TSTFILE_MXF" --video-png-conv=fixed --log-le
1408
1637
  ascli preview test --case=test png "TSTFILE_PDF" --log-level=debug
1409
1638
  ascli preview trevents --once-only=yes --skip-types=office --log-level=info
1410
1639
  ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user ctl all:status
1411
- ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user nagios app_services --format=nagios
1640
+ ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health app_services --format=nagios
1641
+ ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health asctlstatus --format=nagios --cmd-prefix='sudo '
1412
1642
  ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user nodeadmin -- -l
1413
1643
  ascli server -N -Ptst_server_bykey -Plocal_user br /
1414
1644
  ascli server browse /
@@ -1421,21 +1651,24 @@ ascli server df
1421
1651
  ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=.
1422
1652
  ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=folder_1 --transfer=node
1423
1653
  ascli server du /
1654
+ ascli server health transfer --to-folder=folder_1 --format=nagios
1424
1655
  ascli server info
1425
1656
  ascli server md5sum NEW_SERVER_FOLDER/testfile.bin
1426
1657
  ascli server mkdir NEW_SERVER_FOLDER --logger=stdout
1427
1658
  ascli server mkdir folder_1/target_hot
1428
1659
  ascli server mv folder_1/200KB.2 folder_1/to.delete
1429
- ascli server nagios transfer --to-folder=folder_1 --format=nagios
1430
1660
  ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
1431
1661
  ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
1432
1662
  ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername
1433
1663
  ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
1434
1664
  ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER
1665
+ ascli shares admin share list
1435
1666
  ascli shares repository browse /
1436
1667
  ascli shares repository delete /SHARES_UPLOAD/testfile.bin
1437
1668
  ascli shares repository download --to-folder=. /SHARES_UPLOAD/testfile.bin
1669
+ ascli shares repository download --to-folder=. /SHARES_UPLOAD/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://HTTP_GW_FQDN/aspera/http-gwy/v1"}'
1438
1670
  ascli shares repository upload --to-folder=/SHARES_UPLOAD testfile.bin
1671
+ ascli shares repository upload --to-folder=/SHARES_UPLOAD testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://HTTP_GW_FQDN/aspera/http-gwy/v1"}'
1439
1672
  ascli shares2 appinfo
1440
1673
  ascli shares2 organization list
1441
1674
  ascli shares2 project list --organization=Sport
@@ -1451,7 +1684,7 @@ ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"r
1451
1684
  ```
1452
1685
  $ ascli -h
1453
1686
  NAME
1454
- ascli -- a command line tool for Aspera Applications (v4.0.0)
1687
+ ascli -- a command line tool for Aspera Applications (v4.1.0)
1455
1688
 
1456
1689
  SYNOPSIS
1457
1690
  ascli COMMANDS [OPTIONS] [ARGS]
@@ -1462,6 +1695,10 @@ DESCRIPTION
1462
1695
  execute: ascli conf doc
1463
1696
  or visit: http://www.rubydoc.info/gems/aspera-cli
1464
1697
 
1698
+ ENVIRONMENT VARIABLES
1699
+ ASCLI_HOME config folder, default: $HOME/.aspera/ascli
1700
+ #any option can be set as an environment variable, refer to the manual
1701
+
1465
1702
  COMMANDS
1466
1703
  To list first level commands, execute: ascli
1467
1704
  Note that commands can be written shortened (provided it is unique).
@@ -1490,7 +1727,7 @@ OPTIONS: global
1490
1727
  -v, --version display version
1491
1728
  -w, --warnings check for language warnings
1492
1729
  --ui=ENUM method to start browser: text, graphical
1493
- --log-level=ENUM Log level: debug, info, error, warn, fatal, unknown
1730
+ --log-level=ENUM Log level: debug, info, warn, error, fatal, unknown
1494
1731
  --logger=ENUM log method: stderr, stdout, syslog
1495
1732
  --lock-port=VALUE prevent dual execution of a command, e.g. in cron
1496
1733
  --query=VALUE additional filter for API calls (extended value) (some commands)
@@ -1498,7 +1735,7 @@ OPTIONS: global
1498
1735
  --once-only=ENUM process only new items (some commands): yes, no
1499
1736
 
1500
1737
  COMMAND: config
1501
- SUBCOMMANDS: gem_path genkey plugins flush_tokens list overview open echo id documentation wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file
1738
+ SUBCOMMANDS: gem_path genkey plugins flush_tokens list overview open echo id documentation wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update
1502
1739
  OPTIONS:
1503
1740
  --value=VALUE extended value for create, update, list filter
1504
1741
  --property=VALUE name of property to set
@@ -1516,8 +1753,11 @@ OPTIONS:
1516
1753
  --default=VALUE set as default configuration for specified plugin
1517
1754
  --secret=VALUE access key secret for node
1518
1755
  --secrets=VALUE access key secret for node
1756
+ --sdk-url=VALUE URL to get SDK
1757
+ --sdk-folder=VALUE SDK folder location
1519
1758
  --test-mode=ENUM skip user validation in wizard mode: yes, no
1520
- --ts=VALUE override transfer spec values (Hash, use @json: prefix), current={}
1759
+ --version-check-days=VALUE period to check neew version in days (zero to disable)
1760
+ --ts=VALUE override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
1521
1761
  --local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
1522
1762
  --to-folder=VALUE destination folder for downloaded files
1523
1763
  --sources=VALUE list of source files (see doc)
@@ -1536,7 +1776,7 @@ OPTIONS:
1536
1776
 
1537
1777
 
1538
1778
  COMMAND: node
1539
- SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details nagios_check events space info license mkdir mklink mkfile rename delete search
1779
+ SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details health events space info license mkdir mklink mkfile rename delete search
1540
1780
  OPTIONS:
1541
1781
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1542
1782
  --username=VALUE username to log in
@@ -1589,11 +1829,16 @@ OPTIONS:
1589
1829
 
1590
1830
 
1591
1831
  COMMAND: faspex5
1592
- SUBCOMMANDS: node package
1832
+ SUBCOMMANDS: node package auth_client
1593
1833
  OPTIONS:
1594
1834
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1595
1835
  --username=VALUE username to log in
1596
1836
  --password=VALUE user's password
1837
+ --client-id=VALUE API client identifier in application
1838
+ --client-secret=VALUE API client secret in application
1839
+ --redirect-uri=VALUE API client redirect URI
1840
+ --auth=ENUM type of Oauth authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey, boot
1841
+ --private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
1597
1842
 
1598
1843
 
1599
1844
  COMMAND: cos
@@ -1608,7 +1853,7 @@ OPTIONS:
1608
1853
 
1609
1854
 
1610
1855
  COMMAND: faspex
1611
- SUBCOMMANDS: nagios_check package source me dropbox v4 address_book login_methods
1856
+ SUBCOMMANDS: health package source me dropbox v4 address_book login_methods
1612
1857
  OPTIONS:
1613
1858
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1614
1859
  --username=VALUE username to log in
@@ -1646,6 +1891,7 @@ OPTIONS:
1646
1891
  --case=VALUE basename of output for for test
1647
1892
  --scan-path=VALUE subpath in folder id to start scan in (default=/)
1648
1893
  --scan-id=VALUE forder id in storage to start scan in, default is access key main folder id
1894
+ --mimemagic=ENUM use Mime type detection of gem mimemagic: yes, no
1649
1895
  --overwrite=ENUM when to overwrite result file: always, never, mtime
1650
1896
  --file-access=ENUM how to read and write files in repository: local, remote
1651
1897
  --max-size=VALUE maximum size (in bytes) of preview file
@@ -1677,17 +1923,6 @@ OPTIONS:
1677
1923
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1678
1924
  --username=VALUE username to log in
1679
1925
  --password=VALUE user's password
1680
-
1681
- COMMAND: ats
1682
- SUBCOMMANDS: cluster access_key api_key aws_trust_policy
1683
- OPTIONS:
1684
- --ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
1685
- --instance=VALUE ATS instance in ibm cloud
1686
- --ats-key=VALUE ATS key identifier (ats_xxx)
1687
- --ats-secret=VALUE ATS key secret
1688
- --params=VALUE Parameters access key creation (@json:)
1689
- --cloud=VALUE Cloud provider
1690
- --region=VALUE Cloud region
1691
1926
  --auth=ENUM type of Oauth authentication: body_userpass, header_userpass, web, jwt, url_token, ibm_apikey
1692
1927
  --operation=ENUM client operation for transfers: push, pull
1693
1928
  --client-id=VALUE API client identifier in application
@@ -1695,8 +1930,8 @@ OPTIONS:
1695
1930
  --redirect-uri=VALUE API client redirect URI
1696
1931
  --private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
1697
1932
  --workspace=VALUE name of workspace
1698
- --eid=VALUE identifier
1699
1933
  --name=VALUE resource name
1934
+ --path=VALUE file or folder path
1700
1935
  --link=VALUE public link to shared resource
1701
1936
  --new-user-option=VALUE new user creation option
1702
1937
  --from-folder=VALUE share to share source folder
@@ -1707,7 +1942,7 @@ OPTIONS:
1707
1942
 
1708
1943
 
1709
1944
  COMMAND: server
1710
- SUBCOMMANDS: nagios nodeadmin userdata configurator ctl download upload browse delete rename ls rm mv du info mkdir cp df md5sum
1945
+ SUBCOMMANDS: health nodeadmin userdata configurator ctl download upload browse delete rename ls rm mv du info mkdir cp df md5sum
1711
1946
  OPTIONS:
1712
1947
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1713
1948
  --username=VALUE username to log in
@@ -1718,7 +1953,7 @@ OPTIONS:
1718
1953
 
1719
1954
 
1720
1955
  COMMAND: console
1721
- SUBCOMMANDS: transfer nagios_check
1956
+ SUBCOMMANDS: transfer health
1722
1957
  OPTIONS:
1723
1958
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1724
1959
  --username=VALUE username to log in
@@ -1845,7 +2080,7 @@ updated: my_aoc_org
1845
2080
  Define this [option preset](#lprt) as default configuration for the `aspera` plugin:
1846
2081
 
1847
2082
  ```
1848
- $ ascli config id default set aspera my_aoc_org
2083
+ $ ascli config id default set aoc my_aoc_org
1849
2084
  ```
1850
2085
 
1851
2086
  Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
@@ -2219,7 +2454,7 @@ Then, create two shared folders located in two regions, in your files home, in a
2219
2454
  Then, transfer between those:
2220
2455
 
2221
2456
  ```
2222
- $ ascli -Paoc_show aspera files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2457
+ $ ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2223
2458
  ```
2224
2459
 
2225
2460
  * create registration key to register a node
@@ -2620,23 +2855,67 @@ $ ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"my
2620
2855
 
2621
2856
  # Plugin: IBM Aspera Faspex
2622
2857
 
2623
- Note that the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2858
+ Notes:
2624
2859
 
2625
- ## Sending a Package
2860
+ * the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
2861
+ * for full details on Faspex API, refer to: [Reference on Developer Site](https://www.ibm.com/products/aspera/developer)
2862
+
2863
+ ## Faspex 5 Beta1
2864
+
2865
+ As the web UI does not yet allow adding API client yet, the way to use CLI is:
2866
+
2867
+ * open a browser
2868
+ * start developer mode
2869
+ * login to faspex 5
2870
+ * find the first API call with `Authorization` token, and copy it (kind of base64 long string)
2871
+
2872
+ Use it as password and use `--auth=boot`.
2626
2873
 
2627
- Provide delivery info in JSON, example:
2874
+ ```
2875
+ $ ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
2876
+ ```
2877
+
2878
+ An JWT client can then be created with a private key:
2628
2879
 
2629
2880
  ```
2630
- --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}'
2881
+ $ jsonk=$(openssl rsa -in ~/.aspera/ascli/aspera_on_cloud_key -pubout 2> /dev/null | sed -e :a -e N -e '$!ba' -e 's/\n/\\n/g')
2882
+ $ ascli faspex5 -Pf5boot auth_client create --value=@json:'{"name":"hello","client_type":"public","redirect_uris":["https://localhost:12345"],"allow_jwt_grant":true,"public_key":"'$jsonk'"}'
2631
2883
  ```
2632
2884
 
2633
- a note can be added: `"note":"Please ..."`
2885
+ or deleted by name:
2634
2886
 
2635
- metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2887
+ ```
2888
+ $ id=$(ascli faspex5 auth_client list --select=@json:'{"name":"hello"}' --fields=client_id --format=csv)
2889
+ $ ascli faspex5 auth_client delete --id=$id
2890
+ ```
2891
+
2892
+ Once the API client is created with a client_id and secret (result of create command), create a configuration:
2636
2893
 
2894
+ ```
2895
+ $ ascli conf id f5 update --url=https://localhost/aspera/faspex --auth=jwt --client-id=abcd --client-secret=def --username=pierre@example.com --private-key=@val:@file:~/.aspera/ascli/aspera_on_cloud_key
2896
+ $ ascli conf id default set faspex5 f5
2897
+ ```
2637
2898
 
2638
- Note for full details, refer to:
2639
- [Reference on Developer Site](https://developer.asperasoft.com/web/faspex/sending)
2899
+ Ready to use Faspex5 with CLI.
2900
+
2901
+ Once the graphical registration form exist, ther bootstrap method can be removed.
2902
+
2903
+ ## Sending a Package
2904
+
2905
+ The command is `faspex package send`. Package information (title, note, metadata, options) is provided in option `delivery_info`. (Refer to Faspex API).
2906
+
2907
+ Example:
2908
+
2909
+ ```
2910
+ $ ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
2911
+ ```
2912
+
2913
+ If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
2914
+
2915
+ Additional optional parameters in `delivery_info`:
2916
+
2917
+ * Package Note: : `"note":"note this and that"`
2918
+ * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
2640
2919
 
2641
2920
  ## operation on dropboxes
2642
2921
 
@@ -2699,6 +2978,8 @@ $ for p in 1 2 3;do ascli shares2 admin users list --value=@json:'{"page":'$p'}'
2699
2978
  # Plugin: IBM Cloud Object Storage
2700
2979
 
2701
2980
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2981
+ It uses the same transfer service as Aspera on Cloud.
2982
+ see [https://status.aspera.io](https://status.aspera.io)
2702
2983
 
2703
2984
  Required options are either:
2704
2985
 
@@ -2755,7 +3036,14 @@ Endpoints for regions can be found by querying the `endpoints` URL.
2755
3036
  For convenience, let us create a default configuration, for example:
2756
3037
 
2757
3038
  ```
2758
- $ ascli conf id mycos update --service-credentials=@val:@json:@file:$HOME/service_creds.json --region=us-south --bucket=laurent
3039
+ $ ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
3040
+ $ ascli conf id default set cos mycos
3041
+ ```
3042
+
3043
+ or using direct parameters:
3044
+
3045
+ ```
3046
+ $ ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
2759
3047
  $ ascli conf id default set cos mycos
2760
3048
  ```
2761
3049
 
@@ -2821,6 +3109,7 @@ The tool requires the following external tools available in the `PATH`:
2821
3109
  * OptiPNG : `optipng`
2822
3110
  * FFmpeg : `ffmpeg` `ffprobe`
2823
3111
  * Libreoffice : `libreoffice`
3112
+ * ruby gem `mimemagic`
2824
3113
 
2825
3114
  Here shown on Redhat/CentOS.
2826
3115
 
@@ -2832,6 +3121,26 @@ To check if all tools are found properly, execute:
2832
3121
  $ ascli preview check
2833
3122
  ```
2834
3123
 
3124
+ ### mimemagic
3125
+
3126
+ To benefit from extra mime type detection install gem mimemagic:
3127
+
3128
+ ```
3129
+ # gem install mimemagic
3130
+ ```
3131
+
3132
+ or to install an earlier version if any problem:
3133
+
3134
+ ```
3135
+ # gem install mimemagic -v '~> 0.3.0'
3136
+ ```
3137
+
3138
+ To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
3139
+
3140
+ If not used, Mime type used for conversion is the one provided by the node API.
3141
+
3142
+ If used, it the `preview` command will first analyse the file content using mimemagic, and if no match, will try by extension.
3143
+
2835
3144
  ### Image: Imagemagick and optipng
2836
3145
 
2837
3146
  ```
@@ -2856,90 +3165,110 @@ The generation of preview in based on the use of `unoconv` and `libreoffice`
2856
3165
  # dnf install unoconv
2857
3166
  ```
2858
3167
 
3168
+ * Amazon Linux
3169
+
3170
+ ```
3171
+ # amazon-linux-extras enable libreoffice
3172
+ # yum clean metadata
3173
+ # yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
3174
+ # wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
3175
+ # mv unoconv /usr/bin
3176
+ # chmod a+x /usr/bin/unoconv
3177
+ ```
2859
3178
 
2860
3179
  ## Configuration
2861
3180
 
2862
- Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). Note that if you use the `ascli` run as `xfer` user, like here, the configuration file must be created as the same user. Example using a [option preset](#lprt) named `my_preset_name` (choose any name relevant to you, e.g. the AoC node name, and replace in the following lines):
3181
+ The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)
3182
+
3183
+ Like any `ascli` commands, parameters can be passed on command line or using a configuration [option preset](#lprt). The configuration file must be created with the same user used to run so that it is properly used on runtime.
3184
+
3185
+ Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
2863
3186
 
2864
3187
  ```
2865
3188
  # su -s /bin/bash - xfer
2866
- $ ascli config id my_preset_name update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
2867
- $ ascli config id default set preview my_preset_name
3189
+ $ ascli config id previewconf update --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
3190
+ $ ascli config id default set preview previewconf
2868
3191
  ```
2869
3192
 
2870
- Here we assume that Office file generation is disabled, else remove the option. For the `lock_port` option refer to a previous section in thsi manual.
3193
+ Here we assume that Office file generation is disabled, else remove the option. `lock_port` prevents concurrent execution of generation when using a scheduler.
2871
3194
 
2872
3195
  Once can check if the access key is well configured using:
2873
3196
 
2874
3197
  ```
2875
- $ ascli -Pmy_preset_name node browse /
3198
+ $ ascli -Ppreviewconf node browse /
2876
3199
  ```
2877
3200
 
2878
3201
  This shall list the contents of the storage root of the access key.
2879
3202
 
2880
3203
  ## Execution
2881
3204
 
2882
- The tool intentionally supports only a "one shot" mode in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
2883
- It needs to be run regularly to create or update preview files. For that use your best
3205
+ The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
3206
+ It needs to be run on a regular basis to create or update preview files. For that use your best
2884
3207
  reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.
2885
3208
 
2886
- Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consiustent have generate the appropriate access rights, the generation process
2887
- should be run as user `xfer`.
3209
+ Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
2888
3210
 
2889
3211
  Lets do a one shot test, using the configuration previously created:
2890
3212
 
2891
3213
  ```
2892
3214
  # su -s /bin/bash - xfer
2893
- $ ascli preview scan --overwrite=always
3215
+ xfer$ ascli preview scan --overwrite=always
2894
3216
  ```
2895
3217
 
2896
3218
  When the preview generator is first executed it will create a file: `.aspera_access_key`
2897
- which contains the access key used.
3219
+ in the previews folder which contains the access key used.
2898
3220
  On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
2899
3221
 
2900
3222
  ## Configuration for Execution in scheduler
2901
3223
 
2902
- Here is an example of configuration for use with cron on Linux. Adapt the scripts to your own preference.
3224
+ Here is an example of configuration for use with cron on Linux.
3225
+ Adapt the scripts to your own preference.
2903
3226
 
2904
3227
  We assume here that a configuration preset was created as shown previously.
2905
3228
 
2906
- Here the cronjob is created for `root`, and changes the user to `xfer`, also overriding the shell which should be `aspshell`. (adapt the command below, as it would override existing crontab). It is also up to you to use directly the `xfer` user's crontab. This is an example only.
3229
+ Lets first setup a script that will be used in the sceduler and sets up the environment.
3230
+
3231
+ Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
3232
+
3233
+ ```
3234
+ #!/bin/bash
3235
+ # set a timeout protection, just in case
3236
+ case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
3237
+ . /etc/profile.d/rvm.sh
3238
+ rvm use 2.6 --quiet
3239
+ exec timeout ${tmout} ascli "${@}"
3240
+ ```
3241
+
3242
+ Here the cronjob is created for user `xfer`.
2907
3243
 
2908
3244
  ```
2909
- # crontab<<EOF
2910
- 2-59 * * * * su -s /bin/bash - xfer -c 'nice +10 timeout 10m ascli preview event --log-level=info --logger=syslog --iteration-file=/tmp/preview_restart.txt'
2911
- 0 * * * * su -s /bin/bash - xfer -c 'nice +10 timeout 30m ascli preview scan --log-level=info --logger=syslog'
3245
+ xfer$ crontab<<EOF
3246
+ 0 * * * * /home/xfer/cron_ascli preview scan --logger=syslog --display=error
3247
+ 2-59 * * * * /home/xfer/cron_ascli preview trev --logger=syslog --display=error
2912
3248
  EOF
2913
3249
  ```
2914
3250
 
2915
- Nopte that the options here may be located in the config preset, but it was left on the command line to keep stdout for command line execution of preview.
3251
+ Note that the loging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
2916
3252
 
2917
3253
  ## Candidate detection for creation or update (or deletion)
2918
3254
 
2919
- The tool will find candidates for preview generation using three commands:
3255
+ The tool generates preview files using those commands:
2920
3256
 
2921
3257
  * `trevents` : only recently uploaded files will be tested (transfer events)
2922
3258
  * `events` : only recently uploaded files will be tested (file events: not working)
2923
- * `scan` : deeply scan all files under the access key&apos;s "storage root"
2924
- * `folder` : same as `scan`, but only on the specified folder&apos;s "file identifier"
2925
- * `file` : for an individual file generation
2926
-
2927
- Note that for the `event`, the option `iteration_file` should be specified so that
2928
- successive calls only process new events. This file will hold an identifier
2929
- telling from where to get new events.
2930
-
2931
- It is also possible to test a local file, using the `test` command.
3259
+ * `scan` : recursively scan all files under the access key&apos;s "storage root"
3260
+ * `test` : test using a local file
2932
3261
 
2933
3262
  Once candidate are selected, once candidates are selected,
2934
3263
  a preview is always generated if it does not exist already,
2935
3264
  else if a preview already exist, it will be generated
2936
- using one of three overwrite method:
3265
+ using one of three values for the `overwrite` option:
2937
3266
 
2938
3267
  * `always` : preview is always generated, even if it already exists and is newer than original
2939
3268
  * `never` : preview is generated only if it does not exist already
2940
3269
  * `mtime` : preview is generated only if the original file is newer than the existing
2941
3270
 
2942
- Deletion of preview for deleted source files: not implemented yet.
3271
+ Deletion of preview for deleted source files: not implemented yet (TODO).
2943
3272
 
2944
3273
  If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
2945
3274
 
@@ -3193,6 +3522,34 @@ $ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=1234
3193
3522
 
3194
3523
  The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3195
3524
 
3525
+ # Aspera Health check and Nagios
3526
+
3527
+ Each plugin provide a `health` command that will check the health status of the application. Example:
3528
+
3529
+ ```
3530
+ $ ascli console health
3531
+ +--------+-------------+------------+
3532
+ | status | component | message |
3533
+ +--------+-------------+------------+
3534
+ | ok | console api | accessible |
3535
+ +--------+-------------+------------+
3536
+ ```
3537
+
3538
+ Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
3539
+
3540
+ * issuing a transfer to the server
3541
+ * checking web app status with `asctl all:status`
3542
+ * checking daemons process status
3543
+
3544
+ `ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
3545
+
3546
+ ```
3547
+ $ ascli server health transfer --to-folder=/Upload --format=nagios --progress=none
3548
+ OK - [transfer:ok]
3549
+ $ ascli server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3550
+ OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
3551
+ ```
3552
+
3196
3553
  # Module: `Aspera`
3197
3554
 
3198
3555
  Main components:
@@ -3212,6 +3569,16 @@ This sample code shows some example of use of the API as well as
3212
3569
  REST API.
3213
3570
  Note: although nice, it's probably a good idea to use RestClient for REST.
3214
3571
 
3572
+ Example of use of the API of Aspera on Cloud:
3573
+
3574
+ ```
3575
+ require 'aspera/aoc'
3576
+
3577
+ aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/ascli/aspera_on_cloud_key')),username: 'laurent.martin.aspera@fr.ibm.com',subpath: 'api/v1')
3578
+
3579
+ aoc.read('self')
3580
+ ```
3581
+
3215
3582
  # History
3216
3583
 
3217
3584
  When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
@@ -3228,11 +3595,25 @@ So, it evolved into `ascli`:
3228
3595
  * supports transfers with multiple [Transfer Agents](#agents), that&apos;s why transfer parameters moved from ascp command line to [_transfer-spec_](#transferspec) (more reliable , more standard)
3229
3596
  * `ruby` is consistent with other Aspera products
3230
3597
 
3598
+ # Changes (Release notes)
3231
3599
 
3600
+ * 4.x
3232
3601
 
3233
- # Release Notes
3602
+ * fix: remove keys from transfer spec and command line when not needed
3603
+ * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
3604
+ * new: update documentation with regard to offline and docker installation
3605
+ * new: renamed command `nagios_check` to `health`
3606
+ * new: agent `http_gw` now supports upload
3607
+ * new: added option `sdk_url` to install SDK from local file for offline install
3608
+ * new: check new gem version periodically
3609
+ * new: the --fields= option, support -_fieldname_ to remove a field from default fields
3610
+ * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
3611
+ * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
3612
+ * new: AoC a password can be provided for a public link
3613
+ * new: `conf doc` take an optional parameter to go to a section
3614
+ * new: initial support for Faspex 5 Beta 1
3234
3615
 
3235
- * 4.0.0.pre2
3616
+ * 4.0.0
3236
3617
 
3237
3618
  * now available as open source at [https://github.com/IBM/aspera-cli](https://github.com/IBM/aspera-cli) with general cleanup
3238
3619
  * changed default tool name from `mlia` to `ascli`
@@ -3360,7 +3741,7 @@ So, it evolved into `ascli`:
3360
3741
 
3361
3742
  * 0.10.6
3362
3743
 
3363
- * FaspManager: transfer spec `authentication` no more needed for local tranfer to use aspera public keys. public keys will be used if there is a token and no key or password is provided.
3744
+ * FaspManager: transfer spec `authentication` no more needed for local tranfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
3364
3745
  * gem version requirements made more open
3365
3746
 
3366
3747
  * 0.10.5